Classification system, method and program

Abstract

In the present invention, an input means 181 receives the input of: test data, which is document data to be classified; a hierarchical structure in which the lowermost layer node indicates the class of a classification destination; and a classification score for a known class, which is a class in which classified document data is present, as the classification score indicating the probability with which the test data is classified into each class of the hierarchical structure. An unknown class score calculation means 182 calculates a classification score for an unknown class, which is a class in which no classified documents are present, on the basis of the uniformity of the classification scores for each known class in the hierarchical structure. A matching score calculation means 183 calculates a matching score indicating the similarity of the test data with a label of a known class or the label of the unknown class. A final classification score calculation means 184 calculates a final classification score indicating the probability with which the test data is classified into a class, the final classification score increasing as the classification score of the known class or the classification score of the unknown class, and the matching score, increase.

Description

[Technical field]

[0001] The present invention relates to a classification system, a classification method, and a classification program for classifying document data. [Background technology]

[0002] There are various methods for classifying a large amount of data by labeling the data. For example, data classification methods include the flat approach, which ignores the hierarchical structure, and the top-down approach, which classifies the data by successively dividing the hierarchical structure from the top node. Note that these methods are based on the assumption that labeled data exists for all classes.

[0003] On the other hand, as time goes by, new classes to be classified may arise. For such new classes, there is generally no initially labeled data. Hereinafter, such classes for which no labeled data exists may be referred to as unknown classes. Also, classes for which labeled data exists may be referred to as known classes. It is required to be able to appropriately classify data for such unknown classes as well.

[0004] With the above-mentioned commonly known methods, it is difficult to appropriately classify data into new classes. In response to such a problem, Patent Document 1 describes an image classification system for identifying unknown images. The image classification system described in Patent Document 1 calculates the probability that an input image corresponds to a known image whose label is known, calculates the likelihood that the input image will be classified as an unknown image whose label is unknown, and the likelihood that the input image will be classified as a known image, and corrects each likelihood by the calculated probability. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 171416 Summary of the Invention [Problem to be solved by the invention]

[0006] When applying predictive labels indicating class contents to data using trained AI (Artificial Intelligence) (hereinafter referred to as inference), as mentioned above, data that should be classified into a class that does not exist may appear. For example, suppose that "tapioca milk tea" is set as the class to which products should be classified, based on the popularity of tapioca milk tea.

[0007] However, in the initial state, there is no data labeled with the new class “tapioca milk tea,” so it is difficult for a typical hierarchical classifier to classify the data to be classified into an unknown class.

[0008] The system described in Patent Document 1 improves the accuracy of the estimated label by calculating the likelihood of an unknown image to be lower as the probability of a known image increases. On the other hand, in a situation where document data is classified into an unknown class, the probability that document data will be classified into an unknown class is low to begin with, so if processing is performed to further reduce this low probability, the probability that document data will be classified into an unknown class will be further reduced. Therefore, it is difficult to classify document data into an unknown class.

[0009] Therefore, an object of the present invention is to provide a classification system, a classification method, and a classification program that can appropriately classify data to be classified, even into unknown classes for which no classified document data exists. [Means for solving the problem]

[0010] The classification system according to the present invention is characterized by comprising: test data, which is document data to be classified; a hierarchical structure in which the lowest-level node indicates the class to be classified into; an input means for accepting input of classification scores of known classes, which are classes in which classified document data exists, as classification scores indicating the probability that the test data will be classified into each class in the hierarchical structure; unknown class score calculation means for calculating a classification score of an unknown class, which is a class in which classified document data does not exist, based on the uniformity of the classification scores of each known class in the hierarchical structure; a matching score calculation means for calculating a matching score indicating the similarity between the test data and a label of a known class or a label of an unknown class; and a final classification score calculation means for calculating the classification score of the known class or the classification score of the unknown class, and a final classification score indicating the probability that the test data will be classified into a class, the larger the matching score, the larger the classification score.

[0011] The classification method according to the present invention comprises the steps of: The computer The method receives input of test data, which is document data to be classified, a hierarchical structure in which the lowest-level node indicates a classification target class, and classification scores of known classes, which are classes in which classified document data exists, as classification scores indicating the probability that the test data will be classified into each class in the hierarchical structure; The computer In the hierarchical structure, a classification score of an unknown class, which is a class for which no classified document data exists, is calculated based on the uniformity of the classification scores of each known class; The computer Calculate a matching score indicating the degree of similarity between the test data and the labels of the known classes or the labels of the unknown classes; The computer The method is characterized in that a final classification score indicating the probability that the test data will be classified into a class is calculated, the classification score of the known class or the classification score of the unknown class, and the probability that the test data will be classified into a class are calculated so that the larger the matching score is, the larger the final classification score is.

[0012] The classification program according to the present invention is characterized in that it causes a computer to execute an input process for accepting input of test data, which is document data to be classified, a hierarchical structure in which the lowest-level node indicates the class to be classified into, and classification scores of known classes, which are classes in which classified document data exists, as classification scores indicating the probability that the test data will be classified into each class in the hierarchical structure; an unknown class score calculation process for calculating a classification score of an unknown class, which is a class in which classified document data does not exist, based on the uniformity of the classification scores of each known class in the hierarchical structure; a matching score calculation process for calculating a matching score indicating the similarity between the test data and a label of a known class or a label of an unknown class; and a final classification score calculation process for calculating the classification score of the known class or the classification score of the unknown class, and a final classification score indicating the probability that the test data will be classified into a class, the larger the matching score, the larger the classification score. Effect of the Invention

[0013] According to the present invention, data to be classified can be appropriately classified even into unknown classes for which no classified document data exists. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a classification system according to the present invention. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of a hierarchical structure. [Diagram 3] FIG. 11 is an explanatory diagram showing an example of data. [Figure 4] FIG. 1 is an explanatory diagram illustrating an example of the operation of a general hierarchical classifier. [Diagram 5] FIG. 13 is an explanatory diagram showing an example of a matching device. [Figure 6] FIG. 11 is an explanatory diagram showing an example of a process for calculating a classification score of an unknown class. [Figure 7] FIG. 11 is an explanatory diagram showing an example of a process for calculating a matching score. [Figure 8] FIG. 11 is an explanatory diagram showing an example of a process for calculating a final classification score. [Figure 9] FIG. 11 is an explanatory diagram showing an example of a process for calculating a final classification score. [Figure 10] FIG. 11 is an explanatory diagram showing an example of a process for calculating a final classification score. [Figure 11] FIG. 11 is an explanatory diagram showing an example of a process for calculating a final classification score. [Figure 12] 13 is a flowchart showing an operation example of a learning process. [Figure 13] FIG. 11 is an explanatory diagram showing an example of a classification process. [Figure 14] FIG. 1 is a block diagram showing an overview of a classification system according to the present invention. [Figure 15] FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of an embodiment of a classification system according to the present invention.

[0016] First, a situation in which the classification system of this embodiment is used will be described. In this embodiment, classification of document data is represented in a hierarchical structure, and the lowest-level nodes (leaf nodes) represent the classes to which the data is classified. Furthermore, upper nodes in the hierarchical structure correspond to parent nodes, and lower nodes connected to parent nodes correspond to child nodes. Figure 2 is an explanatory diagram showing an example of a hierarchical structure.

[0017] In the example shown in Fig. 2, the topmost node is a class "drinks", and when the class "drinks" is taken as a parent node, its child nodes are the classes "tea" and "coffee". Also, the bottommost node 101 represents a class.

[0018] In the following explanation, the linguistic meaning of the class assigned to the data to be classified will be referred to as a class label. Furthermore, among the class labels, labels assigned manually will be distinguished as correct labels, and labels assigned by AI or the like will be distinguished as predicted labels. Note that classifying data into classes and assigning class labels to data are synonymous. Furthermore, data with correct labels will be referred to as training data. Training data can be said to be data that has already been classified into classes.

[0019] Moreover, data without a correct answer label is referred to as test data. The test data is data to be classified in this embodiment. Note that there is no distinction between the learning data and the test data, and both data are collectively referred to as all data or simply as data.

[0020] Fig. 3 is an explanatory diagram showing an example of data used in this embodiment. Fig. 3 shows training data 104 in which a correct answer label 103 is added to document data 102, and test data 105 in which a correct answer label is not added to document data 102.

[0021] In this embodiment, we assume a situation where data of a class for which no learning data exists appears at the time of inference (in other words, is included in the test data). For example, this is a situation where "tapioca milk tea" is manually set in the product classification hierarchy because tapioca milk tea is popular, but no data exists yet.

[0022] As described above, a class for which no training data exists is referred to as an unknown class, and a class for which training data exists is referred to as a known class. A typical hierarchical classifier cannot classify test data into an unknown class. This is because a typical hierarchical classifier classifies test data into one of the known classes, and is not intended to classify data into an unknown class in the first place.

[0023] Fig. 4 is an explanatory diagram showing an example of the operation of a general hierarchical classifier. The example shown in Fig. 4 shows a situation in which one piece of test data 106 (sometimes referred to as a test sample) is classified into classes of a hierarchical classification 107. In the hierarchical classification 107, "Game Machine A" and "Game Machine B ver.4" exist as known classes, and "Game Machine B ver.5" has been newly added as an unknown class. That is, each known class is classified with the learning data 108, and the learning data is not classified into the unknown class.

[0024] 4 is a product name and description of a game software for game console B ver.5. When classifying a test sample, a typical hierarchical classifier 109 attempts to classify the test sample into a known class in which the learning data 108 exists. Therefore, even a test sample including "game console B ver.5" will be classified into the known class "game console B ver.4."

[0025] In this embodiment, an object is to appropriately classify test data (test samples) into an unknown class that does not include such learning data.

[0026] As illustrated in FIG. 1, the classification system 1 of this embodiment includes a memory unit 10, a learning data input unit 20, a hierarchical classifier learning unit 30, a matching device learning unit 40, a test data input unit 50, a classification score calculation unit 60, an unknown class score calculation unit 70, a matching score calculation unit 80, a final classification score calculation unit 90, and an output unit 100.

[0027] The storage unit 10 stores various information used for processing by the classification system 1 of this embodiment. The storage unit 10 may store, for example, a hierarchical classifier trained by a hierarchical classifier training unit 30 described below, or a matcher trained by a matcher training unit 40. The storage unit 10 may also store a hierarchical classifier or a matcher generated by another system (not shown). The storage unit 10 is realized by, for example, a magnetic disk or the like.

[0028] The learning data input unit 20 accepts input of learning data used by the hierarchical classifier learning unit 30 and the matching device learning unit 40. The specific contents of the learning data will be described later.

[0029] The hierarchical classifier learning unit 30 learns a hierarchical classifier that classifies test data according to a hierarchical structure based on the input learning data. The hierarchical classifier of this embodiment is implemented as a classifier that outputs the probability that the test data is classified into each class (hereinafter, referred to as a classification score). More specifically, the hierarchical classifier of this embodiment is implemented as a classifier that outputs a classification score with which the test data is classified into each known class.

[0030] The hierarchical classifier learning unit 30 may learn the hierarchical classifier by any method. For example, the hierarchical classifier learning unit 30 may learn the hierarchical classifier by a support vector machine (SVM) that performs binary classification by supervised learning. The hierarchical classifier learning unit 30 stores the hierarchical classifier generated by learning in the storage unit 10.

[0031] It should be noted that, when a hierarchical classifier generated in advance is used in the classification process described below, the classification system 1 does not need to include the hierarchical classifier learning unit 30.

[0032] The matcher learning unit 40 receives as input a class label indicating the linguistic meaning of a class and a document sample, and learns a matcher that outputs a matching score according to the degree of similarity between the class label and the document sample.

[0033] The content of the similarity calculated by the matcher is arbitrary, and may be, for example, semantic similarity, character string similarity, or a combination of these. Examples of semantic similarity include the cosine similarity between the embedding vector of the class label and the embedding vector of the document sample, and the Euclidean distance. Examples of character string similarity include the longest common substring length and edit distance between the class label and the document sample.

[0034] The matcher learning unit 40 learns a matcher that calculates a matching score as an overall similarity between a document sample and a class label from one or more of these similarities. The matcher learning unit 40 may train a matcher that calculates a weighted linear sum of at least one or both of the semantic similarity between a document sample and a class label and the similarity of the included character strings as a matching score, for example. Furthermore, the matcher learning unit 40 may train a matcher that calculates a matching score using a sigmoid function that takes the calculated weighted linear sum as an argument.

[0035] Fig. 5 is an explanatory diagram showing an example of a matcher. The matcher shown in Fig. 5 is expressed by a function sim(i,j) that calculates a weighted linear sum of N types of similarity simN from a document sample 110 and a class label 111, and applies the calculated value to a sigmoid function to calculate a matching score.

[0036] The method by which the matcher learning unit 40 learns the matcher is arbitrary. For example, the matcher learning unit 40 may learn a matcher (function) as illustrated in FIG. 5 by using an arbitrary error function (loss function) and learning data. In this case, for example, binary cross entropy indicating whether or not a document sample i is classified into class j may be used. The matcher learning unit 40 stores the matcher generated by learning in the storage unit 10.

[0037] In addition, when a matching device generated in advance is used in the classification process described below, the classification system 1 does not need to include the matching device learning unit 40.

[0038] In addition, since the hierarchical classifier and the matcher are learned by the learning data input unit 20, the hierarchical classifier learning unit 30, and the matching device learning unit 40, the device 2 including the learning data input unit 20, the hierarchical classifier learning unit 30, and the matching device learning unit 40 can be called a learning device.

[0039] The test data input unit 50 receives input of document data to be classified (i.e., test data) and a hierarchical structure including classes to be classified. The test data input unit 50 may read a hierarchical classifier from the storage unit 10 as the input of the hierarchical structure. The test data input here may be data that is thought to be classified into a known class, or may be data that is thought to be classified into an unknown class.

[0040] The classification score calculation unit 60 calculates the probability that the input test data is classified into each known class (i.e., the classification score of the known class). In this embodiment, the classification score calculation unit 60 applies the input test data to a hierarchical classifier to calculate the classification score of each known class.

[0041] In this manner, the test data, the hierarchical structure, and the classification scores are inputted by the test data input unit 50 and the classification score calculation unit 60.

[0042] The unknown class score calculation unit 70 calculates the classification score of the unknown class. As described above, a typical hierarchical classifier does not output the probability of classifying test data into a class in which classified document data does not exist (i.e., an unknown class).

[0043] Therefore, in this embodiment, the unknown class score calculation unit 70 performs a process of allocating the classification scores of known classes to unknown classes. Specifically, the unknown class score calculation unit 70 calculates the classification score of the unknown class based on the uniformity of the classification scores of each known class in the input hierarchical structure. At this time, the unknown class score calculation unit 70 calculates the classification score of the unknown class so that the more uniform the classification scores of each known class are, the higher the classification score becomes.

[0044] This is because the more the hierarchical classifier is confused about classification, the less likely it is that the classification scores will be concentrated in one class, and the classification scores should be uniform for each class. In such a situation, it is assumed that the input test data is not test data that should be classified into a known class (i.e., it is test data that should be classified into an unknown class), so it is considered appropriate to increase the classification score of the unknown class.

[0045] When calculating the classification score of an unknown class, it is necessary to ensure that the relationship between the classification scores in the entire hierarchical structure is appropriate. Therefore, the unknown class score calculation unit 70 allocates the classification scores of the known classes existing under the parent node of the unknown class for which the classification score is to be calculated to the classification score of the unknown class so that the sums of the classification scores of the known classes and the unknown classes existing under the parent node of the unknown class are equal.

[0046] In addition, when there are multiple unknown classes, it is difficult to determine which unknown class to classify into, so the unknown class score calculation section 70 may equally allocate the classification scores of the known classes to the classification scores of the unknown classes.

[0047] Fig. 6 is an explanatory diagram showing an example of a process for calculating a classification score of an unknown class. Fig. 6 shows an example in which a part of a hierarchical structure is extracted. In the example shown in Fig. 6, the hierarchical classifier calculates the classification score of parent node 112 to be 0.8, and the classification scores of three child nodes 113 of known classes subordinate thereto to be 0.4, 0.2, and 0.2, respectively. Furthermore, no classification score has been calculated for two child nodes 114 of an unknown class subordinate to parent node 112.

[0048] The unknown class score calculation unit 70 allocates a uniform classification score to unknown classes of the same parent node. In the example shown in Fig. 6, the unknown class score calculation unit 70 calculates the classification score of each of the two child nodes 114 as 0.2, indicating that the classification scores of the known classes have been allocated to each of them.

[0049] A specific example of how the unknown class score calculation unit 70 calculates the classification score of an unknown class will be described below. In the following specific example, the KL divergence is used to judge the uniformity of the classification score. Here, the parent node is represented as s, and the i-th child node of the unknown class of node s is represented as N(s). i In addition, c(s) is the entire set of child nodes of a known class of node s. In this case, the child node N(s) i The classification score p(N(s) i |x,s) can be calculated by the following formula 1. Note that node s corresponds to the parent node 112 in FIG. 6, and child node N(s) i corresponds to one of the child nodes 114 in FIG. 6, and c(s) corresponds to the child node 113.

[0050]

number

[0051] In Equation 1, KL stands for KL divergence, and p uniform represents a uniform distribution. T is a parameter representing the gradualness of the distribution change to the classification score of the unknown class, and ths represents the value of the KL divergence when the classification score of the unknown class is equal to the classification score of the known class. Both T and ths may be determined in advance by a user or the like, or may be determined from training data.

[0052] In formula 1, the more uniform the classification probability of the known classes is, the smaller the value of the KL divergence becomes, and therefore the value of the exp argument becomes smaller. Therefore, as the value of the exp term approaches 0, the denominator on the right hand side approaches 1, and the classification score of the unknown class becomes larger. The unknown class score calculation unit 70 calculates the classification score of the unknown class using the above-mentioned formula 1. may be calculated.

[0053] The matching score calculation unit 80 calculates the degree of similarity (hereinafter, referred to as a matching score) between the input test data and the class label of a known class or the class label of an unknown class. Specifically, the matching score calculation unit 80 calculates a matching score by applying the class label and test data of each class to the above-mentioned matcher. The matching score calculated in this way can be said to be an index showing how similar the class label and the document sample are for each class in terms of strings or semantics.

[0054] The matching score calculation unit 80 calculates values ​​used to calculate similarity in the matcher from the input test data and class labels, and applies the calculated values ​​to the matcher to calculate a matching score that indicates the overall similarity.

[0055] Fig. 7 is an explanatory diagram showing an example of a process for calculating a matching score. The hierarchical structure 115 shown in Fig. 7 is a hierarchical structure showing classification of food and drink, with black circles indicating known classes and white circles indicating unknown classes. Here, it is assumed that the input test data 116 includes the sentence "XX company low-calorie tapioca karate."

[0056] The matching score calculation unit 80 calculates the matching score as the degree of similarity between the test data and the class label. In this case, the matching score is calculated to be higher for the class "tapioca" classified as a food and the class "tapioca milk tea" classified as a drink than for the other classes.

[0057] The final classification score calculation unit 90 calculates the probability that the test data will be classified into a class (hereinafter, referred to as the final classification score) based on the calculated classification score and matching score. Specifically, the final classification score calculation unit 90 calculates the final classification score so that the larger the classification score of the known class or the classification score of the unknown class and the matching score are, the larger the final classification score becomes. Note that the method of calculating the final classification score is arbitrary.

[0058] The final classification score calculation unit 90 may, for example, multiply the classification score by the matching score to calculate the final classification score. The final classification score calculation unit 90 may also determine in advance a function f that receives the classification score as input and outputs a value determined based on the magnitude of the classification score. The final classification score calculation unit 90 may then multiply a value calculated by applying the classification score of each class to the function f by the matching score to calculate the final classification score. Examples of the function f include an identity map and a step function.

[0059] Figures 8 to 11 are explanatory diagrams showing an example of a process for calculating a final classification score. For example, as a result of the processes shown in Figures 6 and 7, the classification scores and matching scores are calculated as values ​​shown in Figure 8. The final classification score calculation unit 90 calculates the final classification score based on the calculated classification scores and matching scores.

[0060] As illustrated in Fig. 9, the final classification score calculation unit 90 calculates the final classification score based on the calculated classification score and matching score. The final classification score calculation unit 90 may calculate a value obtained by converting the classification score using a function f: [0,1] → [0,1] illustrated in Fig. 9, and multiply the calculated value by the matching score to calculate the final classification score. From this, the function f can be said to be a function that determines how the classification score is used to calculate the final classification score.

[0061] Fig. 10 is an explanatory diagram showing an example in which the identity mapping I is used as the function f. The example shown in Fig. 10 is equivalent to the process of multiplying the classification score and the matching score to calculate the final classification score.

[0062] 11 is an explanatory diagram showing an example in which a step function θ is used as the function f. When x is a document sample and y is a class label, the step function θ is expressed, for example, by the following formula 2. For example, in the example shown in FIG. 11, the final classification score of the class "tapioca milk tea" is calculated to be 0.9, and the final classification score of the class "tapioca" is calculated to be 0.

[0063]

number

[0064] In Equation 2, h is a function for calculating the classification score, and is a function used when the unknown class score calculation unit 70 calculates the classification score. Furthermore, ths2 is a threshold value of the step function, and is determined in advance by a user or the like. The example shown in FIG. 11 is an example in which the threshold value ths2 is determined to be 0. Classes with classification scores smaller than the threshold value ths2 are replaced with a classification score of 0. In this way, by using the step function, it is possible to narrow down candidate classes for classification to those with high classification scores.

[0065] For example, the unknown class score calculation unit 70 can calculate the classification score of an unknown class, but the classification scores of child nodes under the same parent node are calculated equally. Therefore, it is difficult to determine which child node the test data should be classified into based on this classification score alone. On the other hand, the matching score calculation unit 80 calculates the semantic or character string similarity between the test data and the class label. Therefore, the final classification score calculation unit 90 calculates the final classification score by taking into account both this classification score and the matching score. This makes it possible to calculate different classification scores (final classification scores) for child nodes of unknown classes under the same parent node.

[0066] The output unit 100 outputs the classification result of the test data. Specifically, the output unit 100 determines the class with the highest final classification score as the class into which the test data is classified, and outputs information about the class. For example, the output unit 100 may output the class label of the class into which the test data is classified, or may display the part corresponding to the class in the classified hierarchical structure in a manner that makes it distinguishable from other classes. The output unit 100 may also output the classification score calculated by the above-mentioned process.

[0067] The test data is classified by the test data input unit 50, the classification score calculation unit 60, the unknown class score calculation unit 70, the matching score calculation unit 80, the final classification score calculation unit 90, and the output unit 100. Therefore, the device 3 including the test data input unit 50, the classification score calculation unit 60, the unknown class score calculation unit 70, the matching score calculation unit 80, the final classification score calculation unit 90, and the output unit 100 can be called a classification device.

[0068] The training data input unit 20, the hierarchical classifier training unit 30, the matching device training unit 40, the test data input unit 50, the classification score calculation unit 60, the unknown class score calculation unit 70, the matching score calculation unit 80, the final classification score calculation unit 90, and the output unit 100 are realized by a computer processor (e.g., a CPU (Central Processing Unit)) that operates according to a program.

[0069] For example, a program may be stored in storage unit 10, and a processor may read the program and operate, in accordance with the program, as learning data input unit 20, hierarchical classifier learning unit 30, matcher learning unit 40, test data input unit 50, classification score calculation unit 60, unknown class score calculation unit 70, matching score calculation unit 80, final classification score calculation unit 90, and output unit 100. Furthermore, the functions of classification system 1 may be provided in the form of SaaS (Software as a Service).

[0070] In addition, the training data input unit 20, the hierarchical classifier training unit 30, the matching device training unit 40, the test data input unit 50, the classification score calculation unit 60, the unknown class score calculation unit 70, the matching score calculation unit 80, the final classification score calculation unit 90, and the output unit 100 may each be realized by dedicated hardware.

[0071] In addition, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and programs.

[0072] Furthermore, when some or all of the components of the classification system 1 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, cloud computing system, etc., in a form in which each is connected via a communication network.

[0073] Next, the operation of the classification system 1 of this embodiment will be described. FIG. 12 is a flowchart showing an example of the operation of the learning process in the classification system 1. The learning data input unit 20 accepts input of learning data (step S11). The hierarchical classifier learning unit 30 learns a hierarchical classifier that classifies test data based on the input learning data (step S12). In addition, the matching device learning unit 40 learns a matcher that outputs a matching score according to the similarity between a class label and a document sample (step S13). The learned hierarchical classifier and matching device are stored in the storage unit 10.

[0074] 13 is an explanatory diagram showing an example of classification processing in the classification system 1. The test data input unit 50 accepts input of test data and a hierarchical structure (step S21). The classification score calculation unit 60 applies the input test data to the hierarchical classifier to calculate the classification score of each class (step S22). In addition, the unknown class score calculation unit 70 calculates the classification score of the unknown class based on the uniformity of the classification scores of the known classes (step S23).

[0075] The matching score calculation unit 80 calculates a matching score between the test data and each class label (step S24). The final classification score calculation unit 90 calculates a final classification score based on the classification score and matching score of each class (step S25). Then, the output unit 100 outputs the class with the highest final classification score as the class into which the test data is to be classified (step S26).

[0076] As described above, in this embodiment, the test data input unit 50 and the classification score calculation unit 60 accept input of test data, a hierarchical structure, and classification scores classified into each class in the hierarchical structure. The unknown class score calculation unit 70 calculates the classification score of the unknown class based on the uniformity of the classification scores of each known class in the hierarchical structure. In addition, the matching score calculation unit 80 calculates a matching score between the test data and the class label of the known class or the class label of the unknown class. Then, the final classification score calculation unit 90 calculates the classification score and the final classification score so that the final classification score increases as the matching score increases. Therefore, data to be classified can be appropriately classified into unknown classes for which no classified document data exists.

[0077] For example, in a typical hierarchical classification, test data can be classified into known classes with high accuracy, and correspondence to intermediate nodes (rough classification) can also be performed with high accuracy. On the other hand, it is difficult to classify test data into unknown classes for which there is no training data. In addition, a method that matches class labels with document samples uses information indicating the similarity between the class labels and documents, so it is possible to classify test data into unknown classes for which there is no training data. However, since classification is based only on the similarity with the label, the classification accuracy is generally low.

[0078] In response to the above-mentioned problems, the classification system 1 of this embodiment performs classification by combining hierarchical classification and matching, which is expected to compensate for the weaknesses of each method and improve the classification accuracy of data in unknown classes while maintaining high classification accuracy of data in known classes.

[0079] For example, it is not easy to distinguish between "tapioca" and "tapioca milk tea" by matching alone. On the other hand, hierarchical classification allows us to classify drinks and foods, so for example, data on "tapioca latte" can be distinguished as a drink by hierarchical classification, and the possibility that it is "tapioca" can be reduced.

[0080] Next, an example of the classification system 1 of this embodiment will be described. As an example of applying the classification system 1 of this embodiment, a situation can be considered in which labeled product document data (e.g., documents including product names and product descriptions) are available for most of the classes among all the classes, but labeled product document data is not available for some classes.

[0081] In this situation, if new, unlabeled product document data is obtained in addition to the above data, it is necessary to classify the unlabeled product document data into one of the classes in the set hierarchical structure and assign a single label to it.

[0082] To achieve this, first, the training data input unit 20 accepts input of a hierarchical structure and labeled product document data, which is training data. Using the input training data, the hierarchical classifier training unit 30 trains the classifier, and the matcher training unit 40 trains the matcher. Then, the classification system 1 classifies unlabeled product document data based on a classification process as shown in FIG.

[0083] By performing such processing, it is possible to label unlabeled product document data, and in particular, it becomes possible to correctly classify test data that should be classified into unknown classes that could not be correctly classified using only a general hierarchical classifier.

[0084] Next, an overview of the present invention will be described below. Fig. 14 is a block diagram showing an overview of a classification system according to the present invention. A classification system 180 (e.g., classification system 1) according to the present invention includes test data, which is document data to be classified, a hierarchical structure in which the lowest-level node indicates a class to be classified into, and an input means 181 (e.g., test data input unit 50 and classification score calculation unit 60) for accepting input of classification scores of known classes, which are classes in which classified document data exists, as classification scores indicating the probability that the test data will be classified into each class in the hierarchical structure; unknown class score calculation means 182 (e.g., unknown class score calculation unit 70) for calculating a classification score of an unknown class, which is a class in which classified document data does not exist, based on the uniformity of the classification scores of each known class in the hierarchical structure; matching score calculation means 183 (e.g., matching score calculation unit 80) for calculating a matching score indicating the similarity between the test data and a label of a known class or a label of an unknown class; and final classification score calculation means 184 (e.g., final classification score calculation unit 90) for calculating the classification score of the known class or the classification score of the unknown class, and a final classification score indicating the probability that the test data will be classified into a class, such that the larger the matching score, the larger the classification score.

[0085] With such a configuration, data to be classified can be appropriately classified even into unknown classes for which no classified document data exists.

[0086] Specifically, the unknown class score calculation means 182 may calculate the classification score of the unknown class (for example, using the above-mentioned formula 1) so that the more uniform the classification scores of the known classes are, the higher the classification score becomes.

[0087] In addition, the matching score calculation means 183 may calculate the matching score by applying the class label and test data of each class to a matcher (for example, the matcher illustrated in FIG. 5) that receives a class label indicating the linguistic meaning of the class and a document sample as input and outputs a matching score according to the similarity between the class label and the document sample.

[0088] Specifically, the matching score calculation means 183 may calculate the matching score by applying the class labels and test data of each class to a matcher that calculates the matching score using a sigmoid function that takes as an argument a weighted linear sum of at least one or both of semantic similarity (e.g., cosine similarity, Euclidean distance, etc.) and similarity of included strings (e.g., longest common substring length, edit distance, etc.).

[0089] The final classification score calculation means 184 may calculate the final classification score by multiplying the matching score and a value calculated by applying the classification score of each class to a function (e.g., an identity mapping I or a step function θ) that takes the classification score as input and outputs a value determined based on the magnitude of the classification score.

[0090] In addition, the unknown class score calculation means 182 may allocate the classification scores of the known classes existing under the parent node of the unknown class for which the classification score is to be calculated to the classification score of the unknown class so that the sum of the classification scores of the known classes and the unknown classes existing under the parent node of the unknown class for which the classification score is to be calculated is equal.

[0091] Specifically, the unknown class score calculation means 182 may equally allocate the classification scores of the known classes to the classification scores of the unknown classes.

[0092] 15 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. A computer 1000 includes a processor 1001, a main memory device 1002, an auxiliary memory device 1003, and an interface 1004.

[0093] The above-mentioned classification system 180 is implemented in a computer 1000. The operations of the above-mentioned processing units are stored in the form of a program (classification program) in the auxiliary storage device 1003. The processor 1001 reads the program from the auxiliary storage device 1003, loads it in the main storage device 1002, and executes the above-mentioned processing in accordance with the program.

[0094] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disc Read-only memory), a DVD-ROM (Read-only memory), and a semiconductor memory connected via the interface 1004. In addition, when this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may load the program into the main storage device 1002 and execute the above-mentioned process.

[0095] The program may be for realizing part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with another program already stored in the auxiliary storage device 1003.

[0096] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0097] (Supplementary Note 1) An input means for receiving input of test data which is document data to be classified, a hierarchical structure in which a node at the bottom level indicates a class to be classified into, and a classification score indicating a probability that the test data will be classified into each class in the hierarchical structure, the classification score of a known class which is a class in which classified document data exists; an unknown class score calculation means for calculating a classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; a matching score calculation means for calculating a matching score indicating a degree of similarity between the test data and a label of the known class or a label of the unknown class; a final classification score calculation means for calculating a final classification score indicating a probability that the test data is classified into the class, the final classification score being larger as the classification score of the known class or the classification score of the unknown class and the matching score are larger. A classification system characterized by:

[0098] (Appendix 2) The unknown class score calculation means calculates the classification score of the unknown class so that the classification scores of each known class are as uniform as possible. Classification system as described in Appendix 1.

[0099] (Appendix 3) The matching score calculation means calculates a matching score by applying the class label and test data of each class to a matcher that receives a class label indicating the linguistic meaning of the class and a document sample as input and outputs a matching score according to the similarity between the class label and the document sample. A classification system as described in Appendix 1 or Appendix 2.

[0100] (Appendix 4) The matching score calculation means calculates a matching score by applying the class label and test data of each class to a matcher that calculates a matching score using a sigmoid function that takes as an argument a weighted linear sum of at least one of the semantic similarity and the similarity of the included character strings, or both of these similarities. A classification system according to any one of Appendix 1 to Appendix 3.

[0101] (Appendix 5) The final classification score calculation means calculates the final classification score by multiplying a value calculated by applying the classification score of each class to a function that receives the classification score as an input and outputs a value determined based on the magnitude of the classification score, and the value calculated by the matching score. A classification system according to any one of Appendix 1 to Appendix 4.

[0102] (Appendix 6) The unknown class score calculation means allocates the classification scores of the known classes present under the parent node of the unknown class for which the classification score is to be calculated to the classification score of the unknown class so that the sum of the classification scores of the known classes and the unknown classes present under the parent node of the unknown class for which the classification score is to be calculated is equal to the sum of the classification scores of the known classes and the unknown classes present under the parent node of the unknown class. A classification system according to any one of Appendix 1 to Appendix 5.

[0103] (Appendix 7) The unknown class score calculation means equally distributes the classification scores of the known classes to the classification scores of the unknown classes. Classification system described in Appendix 6.

[0104] (Appendix 8) The method receives input of test data, which is document data to be classified, a hierarchical structure in which the lowest-level node indicates a class to be classified into, and classification scores indicating the probability that the test data will be classified into each class in the hierarchical structure, the classification scores of known classes, which are classes in which classified document data exists; calculating the classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; Calculating a matching score indicating a similarity between the test data and the label of the known class or the label of the unknown class; Calculate the classification score of the known class or the classification score of the unknown class, and a final classification score indicating the probability that the test data is classified into the class, so that the final classification score is larger as the matching score is larger. A classification method characterized by:

[0105] (Appendix 9) Calculate the classification score of the unknown class so that the classification scores of each known class are as uniform as possible. Classification method described in Appendix 8.

[0106] (Appendix 10) To the computer, an input process for receiving an input of test data, which is document data to be classified, a hierarchical structure in which a node at the bottom level indicates a class to which the test data is to be classified, and a classification score indicating a probability that the test data will be classified into each class in the hierarchical structure, the classification score of a known class, which is a class in which classified document data exists; an unknown class score calculation process for calculating a classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; A matching score calculation process for calculating a matching score indicating a similarity between the test data and the label of the known class or the label of the unknown class; and A final classification score calculation process for calculating a final classification score indicating a probability that the test data is classified into the class, the final classification score being larger as the classification score of the known class or the classification score of the unknown class and the matching score are larger. A program storage medium storing a classification program for executing the above.

[0107] (Appendix 11) To the computer, In the unknown class score calculation process, the classification score of the unknown class is calculated so that the more uniform the classification scores of each known class are, the higher the score will be. A program storage medium according to claim 10, storing a classification program for

[0108] (Appendix 12) To the computer, an input process for receiving an input of test data, which is document data to be classified, a hierarchical structure in which a node at the bottom level indicates a class to which the test data is to be classified, and a classification score indicating a probability that the test data will be classified into each class in the hierarchical structure, the classification score of a known class, which is a class in which classified document data exists; an unknown class score calculation process for calculating a classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; A matching score calculation process for calculating a matching score indicating a similarity between the test data and the label of the known class or the label of the unknown class; and A final classification score calculation process for calculating a final classification score indicating a probability that the test data is classified into the class, the final classification score being larger as the classification score of the known class or the classification score of the unknown class and the matching score are larger. A classification program to execute the above.

[0109] (Appendix 13) To the computer, In the unknown class score calculation process, the classification score of the unknown class is calculated so that the more uniform the classification scores of each known class are, the higher the score will be. 13. The classification program according to claim 12.

[0110] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0111] 1 Classification System 10 Storage section 20 Learning data input section 30 Hierarchical Classifier Training Unit 40 Matching device learning unit 50 Test data input section 60 Classification score calculation unit 70 Unknown class score calculation unit 80 Matching score calculation unit 90 Final classification score calculation section 100 Output section

Claims

1. an input means for receiving an input of test data which is document data to be classified, a hierarchical structure in which a node at the bottom level indicates a class to be classified into, and a classification score indicating a probability that the test data will be classified into each class in the hierarchical structure, the classification score being a known class which is a class in which classified document data exists; an unknown class score calculation means for calculating a classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; a matching score calculation means for calculating a matching score indicating a degree of similarity between the test data and a label of the known class or a label of the unknown class; a final classification score calculation means for calculating a final classification score indicating a probability that the test data is classified into the class, the final classification score being larger as the classification score of the known class or the classification score of the unknown class and the matching score are larger. A classification system characterized by:

2. The unknown class score calculation means calculates the classification score of the unknown class so that the classification score of each known class is higher as the classification score becomes more uniform. The classification system of claim 1.

3. The matching score calculation means calculates a matching score by applying the class label and test data of each class to a matcher that receives a class label indicating the linguistic meaning of the class and a document sample as input and outputs a matching score according to the similarity between the class label and the document sample. A classification system according to claim 1 or claim 2.

4. The matching score calculation means calculates a matching score by applying the class label and test data of each class to a matcher that calculates a matching score using a sigmoid function having as an argument a weighted linear sum of at least one of the semantic similarity and the similarity of the included character strings, or both of these similarities. A classification system according to any one of claims 1 to 3.

5. The final classification score calculation means calculates a final classification score by multiplying a value calculated by applying the classification score of each class to a function that receives the classification score as an input and outputs a value determined based on the magnitude of the classification score, and the value calculated by the matching score. A classification system according to any one of claims 1 to 4.

6. The unknown class score calculation means allocates the classification scores of the known classes present under the parent node of the unknown class for which the classification score is to be calculated to the classification score of the unknown class such that the sum of the classification scores of the known classes and the unknown class present under the parent node of the unknown class for which the classification score is to be calculated is equal to the sum of the classification scores of the known classes and the unknown class present under the parent node of the unknown class. A classification system according to any one of claims 1 to 5.

7. The unknown class score calculation means equally distributes the classification scores of the known classes to the classification scores of the unknown classes. The classification system of claim 6.

8. A computer receives input of test data which is document data to be classified, a hierarchical structure in which the lowest node indicates a class to be classified into, and classification scores which indicate the probability that the test data will be classified into each class in the hierarchical structure, the classification scores being of known classes in which classified document data exists; the computer calculates the classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; The computer calculates a matching score indicating a similarity between the test data and the label of the known class or the label of the unknown class; The computer calculates a classification score for the known class or a classification score for the unknown class, and a final classification score indicating a probability that the test data is classified into the class, the final classification score being larger as the matching score increases. A classification method characterized by:

9. The computer calculates the classification score of the unknown class so that the classification scores of each known class are as uniform as possible. The classification method according to claim 8.

10. On the computer, an input process for receiving an input of test data, which is document data to be classified, a hierarchical structure in which a node at the bottom level indicates a class to which the test data is to be classified, and a classification score indicating a probability that the test data will be classified into each class in the hierarchical structure, the classification score of a known class, which is a class in which classified document data exists; an unknown class score calculation process for calculating a classification score of an unknown class, which is a class for which no classified document data exists, based on uniformity of the classification scores of the known classes in the hierarchical structure; A matching score calculation process for calculating a matching score indicating a similarity between the test data and the label of the known class or the label of the unknown class; and A final classification score calculation process for calculating a final classification score indicating a probability that the test data is classified into the class, the final classification score being larger as the classification score of the known class or the classification score of the unknown class and the matching score are larger. A classification program to execute the above.

Images