Method and program for identifying correction candidate

Abstract

To provide a method for identifying a text candidate.SOLUTION: A method which identifies a candidate for correction, in which a computer identifies a candidate for a text that is recommended to be corrected from one of a plurality of classes including a plurality of texts to another one of the plurality of classes, causes the computer to perform processing of: identifying, for each of the plurality of classes, a representative text from among the plurality of texts; generating a clustering model using some of the plurality of texts, including the representative text, as correct answer data for supervised machine learning; clustering the plurality of texts using a clustering model; and identifying, for each of the plurality of classes, on the basis of a feature amount of the plurality of texts, a predetermined text as a candidate, which is outside a predetermined class obtained by clustering and has a possibility of being included in any of another classes.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to a correction candidate identification method and a correction candidate identification program. [Background technology]

[0002] A technique for clustering data is known (see, for example, Patent Document 1). In addition, a technique for automatically classifying text data by category and a system for classifying text information are also known (see, for example, Patent Documents 2 and 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2011-509472 [Patent Document 2] JP 2008-071283 A [Patent Document 3] JP 2004-126815 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, text data (hereinafter simply referred to as text) may be converted into vector-format features using a predetermined conversion method such as Word2Vec, and text may be clustered based on these features. In such cases, texts with similar feature distances are often included in the same cluster. Features with similar distances between them represent texts with similar meanings to a certain extent, so that texts with similar meanings can be grouped together into a cluster, which can be used for analysis in which text is treated as a numerical value.

[0005] On the other hand, when clustering texts that correspond to industry terms, the texts may be included in a cluster that differs from the user's intention. For example, driver's licenses for vehicles include licenses for driving regular cars, and licenses for driving heavy machinery such as cranes. In this case, it may be more in line with the user's intention if the text representing the heavy machinery driver's license is included in a cluster that includes architecture-related texts such as architect, rather than in a cluster that includes texts representing driver's licenses for regular cars.

[0006] In this way, if text is included in a cluster different from the user's intention, the user is required to correct the text to a cluster that matches the user's intention. However, when a huge amount of text is clustered and various clusters are generated, it is difficult to find text that is included in a cluster different from the user's intention. This may result in a lot of time being spent on correcting the text to the cluster that the user intended.

[0007] Therefore, in one aspect, an object of the present invention is to provide a correction candidate specifying method and a correction candidate specifying program for specifying correction candidates for text. [Means for solving the problem]

[0008] In one embodiment, a correction candidate identification method is a correction candidate identification method in which a computer identifies candidates for text that are recommended to be corrected from one of a plurality of classes containing a plurality of texts to another of the plurality of classes, in which the computer executes the following processes: for each of the plurality of classes, a representative text that represents the plurality of texts from among the plurality of texts is identified; a clustering model is generated using a portion of the plurality of texts including the representative text as correct answer data for supervised machine learning; clustering is performed on the plurality of texts based on the clustering model; and for each of the plurality of classes, a predetermined text that falls outside a predetermined class determined by the clustering and has the potential to be included from one of the classes into one of the other classes is identified as the candidate based on features of the plurality of texts. Effect of the Invention

[0009] Identify possible text corrections. [Brief description of the drawings]

[0010] [Figure 1] 2 is an example of a hardware configuration of a correction candidate identification device. [Diagram 2] 2 is an example of a functional configuration of a correction candidate identification device. [Diagram 3] 1 is an example of text input to a correction candidate identification device. [Figure 4] 11 is a flowchart illustrating a process executed by a correction candidate identification device. [Diagram 5] 11A and 11B are diagrams illustrating an example of a process for extracting features of words and word groups from each text. [Figure 6] 1A and 1B are diagrams illustrating an example of a process for extracting features of words and word groups from a specific text. [Figure 7] 1A and 1B are diagrams illustrating an example of clustering of a word group and a plurality of clusters generated by the clustering. [Figure 8] FIG. 11 is a diagram illustrating an example of a process for identifying representative text. [Figure 9] This is an example of training data in which either a cluster number or a non-cluster number is set for each word group. [Figure 10] 1 is an example of a word group likelihood matrix. [Figure 11] 13 is an example of a correction recommendation screen. [Figure 12] 13 is a flowchart showing an example of a representative text specification process. [Figure 13] 11A and 11B are diagrams illustrating an example of a process for extracting text features from feature amounts of a word group. [Figure 14] 11A and 11B are diagrams illustrating an example of a process for extracting a central vector of text and a process for identifying representative text. [Figure 15] FIG. 13 is a diagram for explaining an example of the action and effect of the present invention. [Figure 16] 1A is a flowchart (part 1) illustrating a part of the process executed by the correction candidate identification device, and FIG. 1B is a flowchart (part 2) illustrating a part of the process executed by the correction candidate identification device. [Figure 17] FIG. 11 is a diagram illustrating an example of a process for generating sub-clusters. [Figure 18] 1 is an example of a likelihood matrix of sub-clusters. [Figure 19] 13 is a flowchart showing another example of the representative text identification process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] (First embodiment) First, a correction candidate identification device 100 that executes a correction candidate identification method will be described with reference to Fig. 1. More specifically, a hardware configuration of the correction candidate identification device 100 will be described. The correction candidate identification device 100 corresponds to a control device provided in a terminal device such as a PC (Personal Computer) or a tablet terminal.

[0013] 1, the correction candidate identification device 100 includes a CPU (Central Processing Unit) 100A as a processor, and a RAM (Random Access Memory) 100B and a ROM (Read Only Memory) 100C as memories. The correction candidate identification device 100 includes a network I / F (Interface) 100D and a HDD (Hard Disk Drive) 100E. Instead of the HDD (Hard Disk Drive) 100E, a SSD (Solid State Drive) may be adopted.

[0014] The correction candidate identification device 100 may include at least one of an input I / F 100F, an output I / F 100G, an input / output I / F 100H, and a drive device 100I, as necessary. The CPU 100A to the drive device 100I are connected to each other by an internal bus 100J. In other words, the correction candidate identification device 100 can be realized by a computer.

[0015] An input device 710 is connected to the input I / F 100F. Examples of the input device 710 include a keyboard, a mouse, and a touch panel. A display device 720 is connected to the output I / F 100G. Examples of the display device 720 include a liquid crystal display. A semiconductor memory 730 is connected to the input / output I / F 100H. Examples of the semiconductor memory 730 include a USB (Universal Serial Bus) memory and a flash memory. The input / output I / F 100H reads the correction candidate identification program stored in the semiconductor memory 730. The input I / F 100F and the input / output I / F 100H each include a USB port, for example. The output I / F 100G includes a display port, for example.

[0016] A portable recording medium 740 is inserted into the drive device 100I. The portable recording medium 740 may be a removable disk such as a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc). The drive device 100I reads the correction candidate identification program recorded on the portable recording medium 740. The network I / F 100D includes, for example, a LAN port and a communication circuit. The communication circuit includes either or both of a wired communication circuit and a wireless communication circuit. The network I / F 100D may be connected to a communication network such as a LAN (Local Area Network) or the Internet.

[0017] A correction candidate identification program stored in at least one of ROM 100C, HDD 100E, and semiconductor memory 730 is temporarily stored in RAM 100B by CPU 100A. A correction candidate identification program recorded on portable recording medium 740 is temporarily stored in RAM 100B by CPU 100A. By executing the stored correction candidate identification program, CPU 100A realizes various functions described later and executes a correction candidate identification method including various processes described later. The correction candidate identification program may be one according to a flowchart described later.

[0018] The functional configuration of the correction candidate identification device 100 will be described with reference to Fig. 2 and Fig. 3. Note that Fig. 2 shows the main functional parts of the correction candidate identification device 100.

[0019] As shown in Fig. 2, the correction candidate identification device 100 includes a storage unit 110, a processing unit 120, an input unit 130, and an output unit 140. The storage unit 110 can be realized by either one or both of the RAM 100B and the HDD 100E described above. The processing unit 120 can be realized by the CPU 100A described above. The input unit 130 can be realized by the input I / F 100F described above. The output unit 140 can be realized by the output I / F 100G described above.

[0020] The storage unit 110, the processing unit 120, the input unit 130, and the output unit 140 are connected to one another. The storage unit 110 includes an input text storage unit 111 and a word feature dictionary 112. The processing unit 120 includes a first extraction unit 121, a second extraction unit 122, a clustering unit 123, and a specification unit 124. The processing unit 120 also includes a generation unit 125, a learning unit 126, and a recommendation unit 127.

[0021] The input text storage unit 111 stores text input from the input device 710. Specifically, as shown in FIG. 3, the input text storage unit 111 stores a plurality of texts, such as the text "architect" and the text "crane operation". The plurality of texts represent, for example, the qualifications held in job seeker information and the qualifications required in job information. Since the qualifications held and the qualifications required are often freely written, the same content as text may be written with different keywords. In addition, in the case of texts containing industry jargon, different words may have the same meaning in the target field, and the same word may have different meanings in the target field. The input text storage unit 111 stores such various texts.

[0022] The word feature dictionary 112 is a dictionary that stores word features, which are features of various words, in the form of numerical vectors of several hundred dimensions (e.g., 300 dimensions). A word feature is associated with each word. The word feature dictionary 112 can convert a word into a word feature expressed in the form of a numerical vector. The word feature makes it possible to understand the meaning of a word. The word feature dictionary 112 can be realized by a known natural language processing method (specifically, a word feature extraction method) such as Word2Vec or GloVe (Global Vectors for Word Representation).

[0023] The first extraction unit 121 acquires text stored in the input text storage unit 111, and extracts words from the acquired text. More specifically, the first extraction unit 121 performs morphological analysis on each acquired text, and divides each text into a plurality of words. The first extraction unit 121 also performs a cleaning process on the plurality of words divided from the text. The cleaning process is a process of removing noise-like characters that have no meaning as words, such as symbols and grades (e.g., grade 1 and grade 2). The first extraction unit 121 divides the plurality of texts into a plurality of words each having a meaning.

[0024] The second extraction unit 122 extracts word features in the form of numerical vectors corresponding to each of the multiple words from the word feature dictionary 112 based on the word feature dictionary 112 and the multiple words segmented from the text. After extracting the word features, the second extraction unit 122 associates the extracted word features with the word group as features of the word group including the multiple words. In this way, the word group including the multiple words segmented from the text is represented by features in the form of vectors.

[0025] The clustering unit 123 clusters the word groups based on the feature amounts of the word groups extracted by the second extraction unit 122 to generate clusters. A cluster is an example of a class. The clustering unit 123 generates clusters that aggregate word groups that are similar to each other due to the close distance of their feature amounts. As a result, word groups whose feature amounts are within a predetermined range of distance are aggregated into the same cluster.

[0026] The identification unit 124 identifies a word that represents a group of words from among the group of words belonging to the same cluster as a representative text. Although details will be described later, the identification unit 124 averages the feature amounts of the above-mentioned group of words for each dimension, and extracts the feature amount of the text including the group of words for each text. When the identification unit 124 extracts the feature amount of each text, it averages the feature amount of each text and extracts the center of the cluster as a central vector. The average may be a simple average or a weighted average. Since the feature amount of the text is expressed in vector form, the central vector can be expressed by the average feature amount. When the identification unit 124 extracts the central vector of the text, it extracts a plurality of texts that belong within a predetermined first range from the central vector, and extracts a word close to the central vector from the extracted text as a representative text. That is, the identification unit 124 extracts words that belong within a predetermined second range from the central vector from the text, and extracts the extracted words as a representative text.

[0027] The generating unit 125 generates training teacher data as training data for the learning unit 126 to perform machine learning. The generating unit 125 also generates cluster discrimination data as training data for the learning unit 126 to classify. Although details will be described later, the generating unit 125 generates training data in which a cluster number is set as correct answer data for supervised machine learning for a part of word groups including a representative text among word groups belonging to the same cluster as training teacher data. The generating unit 125 also generates training data in which a non-cluster number is set as unassessed data for unsupervised machine learning for the remaining word groups not including a representative text among word groups belonging to the same cluster as cluster discrimination data. The non-cluster number may be any number other than the cluster number. In this embodiment, the generating unit 125 sets the non-cluster number "-1" as unassessed data for word groups not including a representative text. In this way, the generating unit 125 generates training teacher data for correct answer data and cluster discrimination data for unassessed data as training data.

[0028] The learning unit 126 executes semi-supervised machine learning using the feature amounts of the word groups as explanatory variables, and the training teacher data and the cluster discrimination data as objective variables. More specifically, the learning unit 126 executes supervised machine learning using the feature amounts of the part of the word groups described above as explanatory variables, and the training teacher data as objective variables. In this way, the learning unit 126 generates a clustering model as a trained model. Next, the learning unit 126 executes clustering on the feature amounts of the remaining word groups described above based on the clustering model, assigns cluster numbers in descending order of accuracy, and updates the clustering model. That is, the learning unit 126 may execute a method called self-training. As another method, a method may be used in which supervised machine learning is executed on the feature amounts of all word groups by pseudo-assigning cluster numbers (labels) to cluster discrimination data having feature amounts of word groups close to the feature amounts of the word groups corresponding to the training teacher data, thereby generating a clustering model. This method is sometimes called a label propagation method or a label diffusion method.

[0029] In this way, the learning unit 126 executes semi-supervised machine learning. As a result, the learning unit 126 generates a word group probability matrix including a probability (hereinafter, referred to as a cluster belonging probability) that indicates the possibility of the word group belonging to each cluster. In this way, the learning unit 126 learns using the training teacher data as correct answer data to generate a clustering model. Then, the learning unit 126 generates a word group probability matrix in which both the training teacher data and the cluster discrimination data are collectively classified by this clustering model. The learning unit 126 can reveal candidates of texts that are recommended to be corrected from the original cluster to another cluster based on the cluster belonging probability included in the probability matrix.

[0030] The recommendation unit 127 displays a specified number of texts including the word group on the display device 720 in descending order of cluster belonging probability based on the word group probability matrix generated by the learning unit 126. This makes it easier for the user to find texts that are candidates for correction. As a result, even if the text is included in a cluster not intended by the user, the user can quickly perform the correction work of including the text back in the cluster intended by the user.

[0031] Next, the operation of the correction candidate identification device 100 will be described with reference to FIGS.

[0032] 4, the first extraction unit 121 waits until a clustering instruction is detected (step S1: NO). For example, when a user performs a predetermined operation to issue a clustering instruction on the input device 710, the first extraction unit 121 detects the clustering instruction (step S1: YES). As a result, the first extraction unit 121 extracts words (step S2).

[0033] Specifically, the first extraction unit 121 obtains text (see FIG. 3) from the input text storage unit 111, and extracts words from the text as shown in FIG. 5. For example, for the text "Driver's License (AT Only)" 10, the first extraction unit 121 extracts a word group 11 including the word "Driver's License", the symbol "(", the word "AT", the word "limited", and the symbol ")" from the text "Driver's License (AT Only)" 10 as shown in FIG. 6. The first extraction unit 121 executes a morphological analysis on the text "Driver's License (AT Only)" 10, thereby being able to extract a word group 11 including multiple words and multiple symbols from the text "Driver's License (AT Only)" 10.

[0034] Furthermore, when the word group 11 is extracted, the first extraction unit 121 executes a cleaning process on the word group 11, as shown in Fig. 6. For example, the first extraction unit 121 removes the symbols "(" and ")" by executing the cleaning process. This allows the first extraction unit 121 to extract the word group 12 after the symbols have been removed.

[0035] When the process of step S2 is completed, the second extraction unit 122 extracts features of the word groups (step S3). Specifically, as shown in Fig. 5, the second extraction unit 122 extracts features of each of the word groups extracted from each text. For example, as shown in Fig. 6, the second extraction unit 122 extracts features 13 of the word group 12 from the word group 12 using the word feature dictionary 112. After extracting the features 13, the second extraction unit 122 associates them with the word group 12 as shown in Fig. 5.

[0036] When the process of step S3 is completed, the second extraction unit 122 judges whether or not all the feature quantities have been extracted (step S4). If all the feature quantities have not been extracted (step S4: NO), the second extraction unit 122 executes the process of step S3. If all the feature quantities have been extracted (step S4: YES), the clustering unit 123 executes clustering for the word groups based on the feature quantities of the word groups (step S5). That is, the clustering unit 123 executes unsupervised machine learning for the word groups based on the feature quantities of the word groups. As a result, for example, as shown in FIG. 7, the clustering unit 123 generates clusters CL1, CL2, and CL3 that aggregate multiple word groups that are similar to each other due to the close distance of the feature quantities. Note that the clustering executed by the clustering unit 123 is different from the clustering based on the clustering model executed by the learning unit 126 described later. That is, the clustering executed by the clustering unit 123 is clustering independent of the clustering model executed by the learning unit 126.

[0037] When the process of step S5 is completed, the clustering unit 123 assigns cluster numbers to the clusters (step S6). Although the cluster numbers are not shown in Fig. 7, for example, the clustering unit 123 assigns the cluster number "1" to the cluster CL1. Similarly, the clustering unit 123 assigns the cluster number "2" to the cluster CL2, and the cluster number "3" to the cluster CL3.

[0038] When the process of step S6 is completed, the identification unit 124 executes a representative text identification process (step S7). The representative text identification process is a process for identifying a word that represents a word group belonging to the same cluster as a representative text. The identification unit 124 identifies the representative text by executing the representative text identification process. The details of the representative text identification process will be described later.

[0039] When the process of step S7 is completed, the identification unit 124 judges whether or not the representative text has been identified for all clusters (step S8). If the representative text has not been identified for all clusters (step S8: NO), the identification unit 124 executes the process of step S7. As a result, as shown in FIG. 8, the identification unit 124 identifies the representative text for each of the clusters CL1, CL2, and CL3. For example, the identification unit 124 identifies the representative text "architect" and the representative text "engineer" for the cluster CL1. Similarly, the identification unit 124 identifies the representative text "nurse" and the representative text "assistant nurse" for the cluster CL2. Furthermore, the identification unit 124 identifies the representative text "regular license", the representative text "driver's license", and the representative text "driver's license" for the cluster CL3.

[0040] When the identification unit 124 identifies representative texts for all clusters (step S8: YES), the generation unit 125 judges whether or not the same word as the representative text of a cluster is present in the word group of the cluster (step S9). Specifically, as shown in Fig. 9, the generation unit 125 judges for each cluster whether or not there is a word that overlaps with the representative text among the multiple words contained in the word group.

[0041] For example, for cluster CL1, there is a word group that includes the word "architect" that overlaps with the representative text "architect". Such a word group is marked with a positive symbol "O" indicating that there is a word that overlaps with the representative text. On the other hand, for example, a word group that includes the word "electricity" and the word "construction" does not overlap with the representative text "architect". Such a word group is marked with a negative symbol "X" indicating that there is no word that overlaps with the representative text.

[0042] If the word group of the cluster contains the same word as the representative text of the cluster (step S9: YES), the generation unit 125 sets a cluster number (step S10). For example, as shown in FIG. 9, for a word group containing a positive symbol "o", the generation unit 125 sets the cluster number "1" assigned to the cluster CL1 to which the word group belongs. On the other hand, if the word group of the cluster does not contain the same word as the representative text of the cluster (step S9: NO), the generation unit 125 sets a non-cluster number (step S11). For example, as shown in FIG. 9, for a word group containing a negative symbol "x", the generation unit 125 sets the non-cluster number "-1".

[0043] When the process of either step S10 or S11 is completed, the generation unit 125 judges whether or not cluster numbers or non-cluster numbers have been set for all word groups (step S12). If cluster numbers or non-cluster numbers have not been set for all word groups (step S12: NO), the generation unit 125 executes the process of step S9, and then executes the process of either step S10 or S11. As a result, as shown in Fig. 9, the generation unit 125 sets cluster numbers or non-cluster numbers for all word groups. In this way, the generation unit 125 generates training teacher data in which cluster numbers are set and cluster discrimination data in which non-cluster numbers are set.

[0044] On the other hand, if cluster numbers or non-cluster numbers are set for all word groups (step S12: YES), the learning unit 126 generates a clustering model (step S13). Specifically, the learning unit 126 executes supervised machine learning using the feature quantities of the word groups associated with the above-mentioned partial word group as explanatory variables and the learning teacher data corresponding to the partial word group as objective variables. In this way, the learning unit 126 generates a clustering model.

[0045] When the process of step S13 is completed, the learning unit 126 executes clustering and generates a likelihood matrix of the word groups (steps S14 and S15). Specifically, the learning unit 126 executes clustering as unsupervised machine learning on the feature amounts of the remaining word groups described above. That is, the learning unit 126 executes clustering based on a clustering model on the feature amounts of the word groups associated with the remaining word groups described above.

[0046] The learning unit 126 performs such clustering to classify the word groups collectively, thereby generating a probability matrix of the word groups including various cluster belonging probabilities, as shown in Fig. 10. As described above, the clustering model is generated using the word groups including words that overlap with the representative text as the correct answer data alone. Therefore, as shown in Fig. 10, even if the word groups including words that overlap with the representative text are classified into any cluster based on the clustering model, they will be classified into a cluster that does not require correction with a high probability.

[0047] For example, the word "architect" in the word group belonging to cluster CL1 overlaps with the representative text "architect". Therefore, the learning unit 126 calculates a cluster belonging accuracy of "99%" (abbreviated as cluster #1 accuracy; the same applies below) for cluster CL1 to which cluster number "1" is assigned. In other words, it is determined that the need for cluster correction work for text corresponding to the word group containing the word "architect" is extremely low.

[0048] On the other hand, the words "crane" and "driving" in the word group belonging to cluster CL3 do not overlap with any of the representative texts "regular license", "driver's license", and "driver's license". Therefore, the learning unit 126 calculates a cluster belonging accuracy of "45%" for belonging to cluster CL1, but also calculates a cluster belonging accuracy of "55%" for belonging to cluster CL3. In other words, the cluster belonging cannot be determined with high accuracy for the text corresponding to the word group including the words "crane" and "driving". In other words, it is determined that there is a high need for cluster correction work.

[0049] When the process of step S15 is completed, the recommendation unit 127 identifies a recommended correction candidate (step S16). Specifically, the recommendation unit 127 identifies the remaining cluster membership accuracy excluding the cluster membership accuracy in which the cluster number assigned to the cluster matches the cluster number of the cluster membership accuracy from the word group accuracy matrix. In this embodiment, as shown in FIG. 10, the recommendation unit 127 identifies the remaining cluster membership accuracy excluding the cluster membership accuracy with diagonal lines (hatching) from the word group accuracy matrix. When the recommendation unit 127 identifies the cluster membership accuracy, it identifies a predetermined text corresponding to the word group for which the identified cluster membership accuracy was calculated as a recommended correction candidate.

[0050] When the process of step S16 is completed, the recommendation unit 127 displays the recommended correction candidates (step S17) and ends the process. Specifically, the recommendation unit 127 sorts the recommended correction candidates in descending order of cluster belonging accuracy. Then, the recommendation unit 127 displays a correction recommendation screen including the designated number of recommended correction candidates corresponding to the word group for which the top cluster belonging accuracy has been calculated and the cluster belonging accuracy on the display device 720. As a result, the correction recommendation screen appears on the display device 720 as shown in FIG. 11. By checking the correction recommendation screen, the user can easily find the text of the correction candidate.

[0051] When displaying the correction recommendation screen, the recommendation unit 127 may include the cluster number of the source and the cluster number of the recommended destination, etc. This allows the user to instantly grasp the cluster number to which the text of the correction candidate currently belongs and the cluster number of the destination, and allows the user to move on to the correction work in a short time.

[0052] Next, the above-mentioned representative text specification process will be described with reference to FIGS.

[0053] 12, when the representative text identification process starts, the identification unit 124 extracts a central vector (step S21). Specifically, the identification unit 124 specifies one of the clusters, and extracts the feature amounts of the text corresponding to each word group by averaging the feature amounts of the word groups belonging to the specified cluster.

[0054] 13, when the second extraction unit 122 extracts feature quantity 13 of the word group 12 (see FIG. 6), the identification unit 124 averages the feature quantities in vector format assigned to each of the words "driver's license", "AT", and "restricted" included in the word group 12 to extract feature quantity 14 of the text "driver's license (AT restricted)" corresponding to the word group 12. The identification unit 124 similarly extracts text feature quantities for each cluster for word groups other than the word group 12.

[0055] After extracting the feature amount of each text for each cluster, the identification unit 124 averages the feature amount of each text for each cluster to extract a central vector for each cluster. For example, as shown in the upper part of Fig. 14, when cluster CL3 with cluster number "3" (expressed as cluster #3) is specified, the identification unit 124 averages the feature amount of each text to extract central vector V3. Although not shown, the identification unit 124 similarly extracts central vectors for clusters CL1 and CL2.

[0056] When the process of step S21 is completed, the identification unit 124 extracts text near the central vector (step S22). For example, as shown in the upper part of Fig. 14, the identification unit 124 extracts a plurality of texts belonging to a predetermined first range R1 from the central vector V3 as text near the central vector V3. Although not shown, the identification unit 124 similarly extracts text near the central vector for clusters CL1 and CL2.

[0057] When the process of step S22 is completed, the identification unit 124 extracts words near the central vector from the extracted text (step S23). For example, as shown in the lower part of Fig. 14, the identification unit 124 extracts a plurality of words that belong to a predetermined second range R2 from the central vector V3 as words near the central vector V3. Although not shown, the identification unit 124 similarly extracts words near the central vectors for the clusters CL1 and CL2.

[0058] When the process of step S23 is completed, the identification unit 124 judges whether the first condition is satisfied (step S24). The identification unit 124 judges that the first condition is satisfied if the number of extracted words is equal to or greater than the threshold number, or if all of the extracted words have been added to a word list, which will be described later (step S24: YES). On the other hand, the identification unit 124 judges that the first condition is not satisfied if the number of extracted words is not equal to or greater than the threshold number and if all of the extracted words have not been added to the word list (step S24: NO).

[0059] If the first condition is not satisfied, the identification unit 124 adds the extracted word to the word list (step S25) and executes the process of step S24. The identification unit 124 repeats the processes of steps S24 and S25 until the first condition is satisfied. This adds one or more words to the word list. If the first condition is satisfied, the identification unit 124 determines whether the second condition is satisfied (step S26).

[0060] The identifying unit 124 determines that the second condition is satisfied when all the text has been processed or the upper limit of the number of words set in advance in the word list has been reached (step S26: YES). On the other hand, the identifying unit 124 determines that the second condition is not satisfied when all the text has not been processed and the upper limit of the number of words has not been reached (step S26: NO). When the second condition is not satisfied, the identifying unit 124 repeats the processes of steps S23 to S26 until the second condition is satisfied. When the second condition is satisfied, the identifying unit 124 outputs one or more words added to the word list as representative text (step S27) and ends the representative text identification process. By identifying the representative text in this way, the identifying unit 124 can accurately identify the representative text that represents the word group. As a result, the correction candidate identifying device 100 can accurately identify the text of the correction candidate.

[0061] The effects of the present invention will be described with reference to FIG.

[0062] First, as shown in the upper part of Fig. 15, when the correction candidate identification device 100 performs known clustering (see Fig. 4) on the input text (see Fig. 3) before performing the representative text identification process, the correction candidate identification device 100 generates multiple clusters CL1, CL2, and CL3. In this case, since the representative text identification process, etc. is not performed, for example, the correction candidate identification device 100 may generate cluster CL1 including the text "electrical work". Similarly, the correction candidate identification device 100 may generate cluster CL3 including the text "crane operation".

[0063] Some users may wish to classify the text "Crane operation" included in cluster CL3 into cluster CL1. Similarly, some users may wish to classify the text "Electrical work" included in cluster CL1 into cluster CL3. However, when the amount of text becomes huge, the number of clusters may also increase. As a result, the user may not be able to quickly grasp the text that has been classified into a cluster contrary to the user's intention, and may be delayed in finding the cluster to be corrected.

[0064] However, according to this embodiment, as shown in the lower part of FIG. 15, the correction candidate identification device 100 performs a known clustering and then performs a representative text identification process, re-clustering, etc. As a result, for example, in cluster CL1, the correction candidate identification device 100 determines that the texts "architect" and "engineer" are similar to each other because the feature distances are close. On the other hand, the correction candidate identification device 100 determines that the text "electrical work" is not similar to these texts because the feature distances are far from these texts. Similarly, for example, in cluster CL3, the correction candidate identification device 100 determines that the texts "driver's license" and "regular license" are similar to each other because the feature distances are close. On the other hand, the correction candidate identification device 100 determines that the text "crane operation" is not similar to these texts because the feature distances are far from these texts.

[0065] In this way, the correction candidate identification device 100 executes the representative text identification process and the like to perform clustering again, and thus the text "electrical work" and the text "crane operation" become apparent as correction candidates. By the correction candidate identification device 100 presenting the text thus made apparent as a correction candidate to the user, the user can quickly grasp the text that has been classified into a cluster contrary to the user's intention. As a result, the user can quickly find the cluster to be corrected.

[0066] Second embodiment Next, a second embodiment of the present invention will be described with reference to Figures 16 to 19. Note that, in Figures 16(a) and (b) and Figure 19, the same processes as those in the flowcharts shown in Figures 4 and 12 are denoted by the same reference numerals, and the description thereof will be omitted.

[0067] In the first embodiment, a single-level (i.e., one-level) cluster has been described as an example. On the other hand, as described in the second embodiment, the inside of the above-mentioned cluster may be further clustered to generate sub-clusters of multiple levels (i.e., two or more levels), and the sub-clusters may be targets for correction. This may enable the user to quickly find the sub-cluster to be corrected.

[0068] First, as shown in Fig. 16(a), when the clustering unit 123 completes the process of step S5, it performs sub-clustering (step S31), and then performs the process of step S6. Specifically, as shown in the upper part of Fig. 17, the clustering unit 123 generates clusters CL1, CL2, and CL3 by performing clustering on the input text (see Fig. 3).

[0069] After generating clusters CL1, CL2, and CL3, the clustering unit 123 performs sub-clustering as a second clustering for each of the clusters CL1, CL2, and CL3. As a result, as shown in the lower part of Fig. 17, the clustering unit 123 generates sub-clusters SC1A and SC1B for the cluster CL1. Similarly, the clustering unit 123 generates sub-clusters SC2A and SC2B for the cluster CL2, and generates sub-clusters SC3A and SC3B for the cluster CL3.

[0070] Thereafter, the clustering unit 123 executes the process of step S6 to assign, for example, a sub-cluster number "1A" to the sub-cluster SC1A. The clustering unit 123 similarly assigns sub-cluster numbers to the sub-clusters SC1B, SC2A, SC2B, SC3A, and SC3B.

[0071] Next, as shown in Fig. 16(b), when the learning unit 126 completes the process of step S15, it generates an accuracy matrix of the sub-cluster (step S41), and then executes the process of step S16. Specifically, when the learning unit 126 generates the accuracy matrix of the word group (see Fig. 10), it generates an accuracy matrix of the sub-cluster including various cluster belonging accuracy together with the representative text of the sub-cluster, as shown in Fig. 18. For example, the learning unit 126 averages the cluster belonging accuracy included in the accuracy matrix of the word group for each sub-cluster to generate the accuracy matrix of the sub-cluster. The representative text of the sub-cluster will be described later.

[0072] This allows the recommendation unit 127 to identify a recommended correction candidate based on the certainty matrix of the sub-cluster. In this embodiment, as shown in Fig. 18, the recommendation unit 127 identifies the remaining cluster membership certainties excluding the cluster membership certainties indicated by diagonal lines (hatching) from the certainty matrix of the sub-cluster. When the recommendation unit 127 identifies the cluster membership certainties, it identifies the sub-cluster for which the identified cluster membership certainties have been calculated as a recommended correction candidate.

[0073] Next, the representative text identification process according to this embodiment will be described with reference to FIG.

[0074] In the process of step S26 described in the first embodiment, if it is determined that the second condition is satisfied, the identification unit 124 acquires the representative text of the sub-cluster (step S51). For example, the identification unit 124 acquires the representative text "Electricity" from the sub-cluster SC1B (see FIG. 17) by the same process as steps S21 to S23. Similarly, the identification unit 124 acquires the representative text "Cook" and the representative text "Crane" from the sub-clusters SC2B and SC3B.

[0075] When the process of step S51 is completed, the identification unit 124 judges whether the third condition is satisfied (step S52). If the number of extracted words is equal to or greater than the threshold number or all of the extracted words have been added to the word list, the identification unit 124 judges that the third condition is satisfied (step S52: YES). On the other hand, if the number of extracted words is not equal to or greater than the threshold number and all of the extracted words have not been added to the word list, the identification unit 124 judges that the third condition is not satisfied (step S52: NO).

[0076] If the third condition is not satisfied, the identification unit 124 adds the extracted word to the word list (step S53) and executes the process of step S52. The identification unit 124 repeats the processes of steps S52 and S53 until the third condition is satisfied. This causes one or more words to be added to the word list. If the third condition is satisfied, the identification unit 124 determines whether the fourth condition is satisfied (step S54).

[0077] The identification unit 124 determines that the fourth condition is satisfied when all sub-clusters have been processed or the upper limit of the number of words previously set in the word list has been reached (step S54: YES). On the other hand, the identification unit 124 determines that the fourth condition is not satisfied when all text has not been processed and the upper limit of the number of words has not been reached (step S54: NO). If the fourth condition is not satisfied, the identification unit 124 repeats the processes of steps S52 to S54 until the fourth condition is satisfied. If the fourth condition is satisfied, the identification unit 124 executes the process of step S27 and outputs one or more words added to the word list as representative text. This ends the representative text identification process.

[0078] Although the preferred embodiment of the present invention has been described above in detail, the present invention is not limited to the specific embodiment of the present invention, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. For example, in the above embodiment, the qualifications held and the necessary qualifications are described as an example of multiple texts, but the items of fixed property tax may be multiple texts and used in the assessment decision of a local government.

[0079] In addition, the following supplementary notes are provided in relation to the above description. (Supplementary Note 1) A correction candidate identification method in which a computer identifies candidates for text that are recommended to be corrected from one of a plurality of classes containing a plurality of texts to another of the plurality of classes, the correction candidate identification method comprising the steps of: identifying, for each of the plurality of classes, a representative text that represents the plurality of texts from among the plurality of texts; generating a clustering model using a portion of the plurality of texts including the representative text as correct answer data for supervised machine learning; clustering the plurality of texts based on the clustering model; and identifying, for each of the plurality of classes, a predetermined text that falls outside a predetermined class determined by the clustering and that has the potential to be moved from one of the classes to be included in one of the other classes, based on features of the plurality of texts, as the candidate. (Supplementary Note 2) The method for identifying correction candidates according to Supplementary Note 1, characterized in that the process of identifying the predetermined text includes displaying a designated number of the predetermined texts on a display device in order of the likelihood. (Supplementary Note 3) The method for identifying correction candidates described in Supplementary Note 1 or 2, characterized in that the process of identifying the representative text includes averaging features of the plurality of texts, extracting a center of the plurality of classes for each of the plurality of classes, extracting text that falls within a first range from the center, and extracting words that fall within a second range from the center from the extracted text, thereby identifying the representative text. (Appendix 4) The method for identifying correction candidates according to appendix 1 or 2, characterized in that the process of clustering performs the clustering on both the part of text and the remaining text of the plurality of texts that does not include the representative text, based on the clustering model. (Supplementary Note 5) The method for identifying correction candidates according to Supplementary Note 1 or 2, further comprising a process of generating the plurality of classes by performing a clustering process on the plurality of texts, which is independent of and different from the clustering model, before identifying the representative text. (Supplementary Note 6) The method of identifying correction candidates as described in Supplementary Note 5, further comprising a process of generating subclasses within the plurality of classes by further performing the separate clustering on in-class text contained in each of the plurality of classes. (Appendix 7) The method of identifying correction candidates according to appendix 1 or 2, wherein the plurality of texts include one or both of a first text having different words but the same meaning and a second text having the same words but the different meaning. (Appendix 8) A correction candidate identification program that causes a computer to identify candidates for text that are recommended to be corrected from one of a plurality of classes containing a plurality of texts to another of the plurality of classes, the correction candidate identification program causing the computer to execute the following processes: for each of the plurality of classes, identify a representative text that represents the plurality of texts from among the plurality of texts, generate a clustering model using a portion of the plurality of texts including the representative text as correct answer data for supervised machine learning, perform clustering on the plurality of texts based on the clustering model, and identify, for each of the plurality of classes, a predetermined text that falls outside a predetermined class determined by the clustering and that has the potential to be included from one of the classes to one of the other classes, based on features of the plurality of texts. [Explanation of symbols]

[0080] 100 Correction candidate identification device 111 Input text storage unit 112 Word Feature Dictionary 121 1st extraction part 122 Second extraction part 123 Clustering Department 124 Specific part 125 Generation part 126 Learning Department 127 Recommendation section CL1, CL2, CL3 Cluster

Claims

1. 1. A method for identifying a correction candidate, in which a computer identifies a text candidate that is recommended to be corrected from one of a plurality of classes including a plurality of texts to another of the plurality of classes, the method comprising: identifying a representative text from among the plurality of texts for each of the plurality of classes; A part of the texts including the representative text is used as correct answer data for supervised machine learning to generate a clustering model; clustering the plurality of texts based on the clustering model; identifying, for each of the plurality of classes, a predetermined text that is outside the predetermined class determined by the clustering and has a possibility of being included in one of the classes from another of the classes, as the candidate, based on the feature amounts of the plurality of texts; The method of identifying correction candidates, the processing of which is carried out by the computer.

2. The process of identifying the predetermined text includes displaying a specified number of the predetermined texts on a display device in order of the likelihood.

2. The method of claim 1, wherein the correction candidates are identified based on the first and second inputs.

3. The process of identifying the representative text includes averaging feature amounts of the plurality of texts, extracting a center of the plurality of classes for each of the plurality of classes, extracting text that falls within a first range from the center, and extracting words that fall within a second range from the center from the extracted text, thereby identifying the representative text.

3. A method according to claim 1 or 2, characterized in that:

4. the process of performing clustering includes performing the clustering on both the part of text and the remaining text of the plurality of texts not including the representative text based on the clustering model; 3. A method according to claim 1 or 2, characterized in that:

5. generating the plurality of classes by performing another clustering process, which is independent of the clustering model and different from the clustering process, on the plurality of texts before identifying the representative text; 3. A method according to claim 1 or 2, characterized in that:

6. A correction candidate identification program that causes a computer to identify a text candidate that is recommended to be corrected from one of a plurality of classes including a plurality of texts to another of the plurality of classes, the program comprising: identifying a representative text from among the plurality of texts for each of the plurality of classes; A part of the texts including the representative text is used as correct answer data for supervised machine learning to generate a clustering model; clustering the plurality of texts based on the clustering model; identifying, for each of the plurality of classes, a predetermined text that is outside the predetermined class determined by the clustering and has a possibility of being included in one of the classes from another of the classes, as the candidate, based on the feature amounts of the plurality of texts; A correction candidate identification program that causes the computer to execute a process.

Images