Data classification system, data classification method, program, method for evaluating machine learning model

Abstract

To perform processing which predicts one correct label from a plurality of labels and processing which predicts a plurality of correct labels from the plurality of labels without increase in a load of an annotation operation.SOLUTION: An aspect of the present invention is a learning device which includes: a conversion unit which converts first learning data with which a first machine learning model which outputs a plurality of correct labels when data of interest is input thereinto is caused to learn, into second learning data including an integrated label obtained by integrating a plurality of labels included in the first learning data on the basis of a flag of the plurality of labels; and a learning unit which causes a second machine learning model which outputs one correct label when data of interest is input thereinto to learn, with the second learning data converted by the conversion unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to Data classification system, data classification method a method for evaluating programs and machine learning models.

Background Art

[0002] Conventionally, a technique for classifying target data using a machine learning model has been known. As this classification type of machine learning model, there are a single label model that classifies target data into one label among a plurality of labels, and a multi-label model that classifies target data into a plurality of labels simultaneously.

[0003] As techniques related to the single label model and the multi-label model, for example, Patent Documents 1 and 2 are known. Patent Document 1 describes determining a threshold value and / or a scale factor to improve the classification of the multi-label model. Patent Document 2 describes, for creating teacher data for a multi-label model and a single label model, for input data, for each label, obtaining any one of a positive evaluation indicating that the content of the input data matches the label, a negative evaluation indicating that the content of the input data does not match the label, and a disregard evaluation indicating exclusion from the learning target label, creating teacher data, and when the learning device learns a neural network for learning, adjusting the weight coefficient of the intermediate layer so that the recognition score of the positive evaluation or negative evaluation label and the correct answer score of the positive evaluation or negative evaluation approach, and making the recognition score of the disregard evaluation label not affect the adjustment of the weight coefficient of the intermediate layer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] FIG. 14 and FIG. 15 are diagrams for explaining the single-label model. The single-label model performs learning and prediction such that the sum of outputs for each label among a plurality of labels is 1. Therefore, as shown in FIG. 14(A), when the correct label among labels 1 to 4 is label 3, label 3 can be accurately output as the correct label.

[0006] However, as shown in FIG. 14(B), the single-label model cannot be applied to a multi-labeling problem in which a plurality of labels (for example, label 2 and label 3) apply to the target data. That is, in the single-label model, although the correct labels are labels 2 and 3, label 3 with the highest score is output as the correct label. Also, in the single-label model, in order to make the sum of each label 1, the scores of label 2 and label 3 become small.

[0007] As shown in FIG. 15, in the single-label model, the score of each correct label becomes smaller as the number of correct labels increases. For this reason, it becomes difficult to extract the correct label by comparing the score of each label output by the single-label model with the threshold value. Furthermore, since it is necessary to calculate the threshold value based on the difference between the score of the correct label output from the single-label model and the scores of other labels, there is a problem that it cannot be learned.

[0008] FIG. 16 is a diagram for explaining a multi-label model. The multi-label model performs learning and prediction so that the score for each label ranges from 0 to 1. Therefore, even when there is one correct label of label 3 as shown in FIG. 16(A) or when there are two correct labels of labels 2 and 3 as shown in FIG. 16(B), a high score can be output. Also, in the multi-label model, even if one threshold is set without considering the number of correct labels, the correct labels can be output with high accuracy. However, for example, as a prediction model for classifying target data such as documents, it is common to use a single-label model. However, the single-label model has a problem that it is difficult to extract a plurality of correct labels.

[0009] When it is unknown whether the number of correct labels corresponding to the target data is one or plural, it is conceivable to verify both the prediction result of the single-label model and the prediction result of the multi-label model. However, since it is necessary to separately perform the annotation work for the multi-label model and the annotation work for the single-label model, there is a problem that the work burden increases. Also, since the labels for the multi-label model and the labels for the single-label model are different, there is a problem that the prediction accuracy of the multi-label model and the prediction accuracy of the single-label model cannot be compared.

[0010] The present invention has been made in view of the above problems, and it is possible to perform processing for predicting one correct label from a plurality of labels and processing for predicting a plurality of correct labels from a plurality of labels without increasing the burden of the annotation work Data classification system, data classification method , and aims to provide a program and a method for evaluating a machine learning model.

Means for Solving the Problems

[0011] (1) One aspect of the present invention is A multi-label storage unit that stores information for attaching a plurality of correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; first learning data for learning a first machine learning model that outputs a plurality of correct labels when input with target data Of these, one new integrated label is generated from a plurality of labels whose flag values are the first value, the flag of the integrated label is set to the first value, and the flags of the plurality of labels that are the source of the integrated label are set to the second value, thereby the A conversion unit that converts into second learning data including integrated labels, and a learning unit that uses the second learning data converted by the conversion unit to train a second machine learning model that outputs one correct label when inputting target data. A prediction unit that inputs target data to the second machine learning model and obtains a prediction result including a correct label output from the second machine learning model; and an inverse conversion unit that, when the correct label obtained by the prediction unit is the integrated label, converts the flags of the plurality of labels that are the source of the integrated label to the first value and deletes the integrated label from the prediction result; Comprising Data classification system It is.

[0012] (2) One aspect of the present invention is A step in which a storage unit stores information for attaching a plurality of correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; a step in which a learning device First learning data for training a first machine learning model that outputs a plurality of correct labels when inputting target data Of these, one new integrated label is generated from a plurality of labels whose flag values are the first value, the flag of the integrated label is set to the first value, and the flags of the plurality of labels that are the source of the integrated label are set to the second value, thereby the Converting into second learning data including integrated labels, The learning device Using the converted second learning data to train a second machine learning model that outputs one correct label when inputting target data, A step in which a prediction device inputs target data to the second machine learning model and obtains a prediction result including a correct label output from the second machine learning model; and a step in which the prediction device, when the obtained correct label is the integrated label, converts the flags of the plurality of labels that are the source of the integrated label to the first value and deletes the integrated label from the prediction result; A data classification method including.

[0013] (3) One aspect of the present invention is to cause a computer to A step of storing information for attaching a plurality of correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; First learning data for training a first machine learning model that outputs a plurality of correct labels when inputting target data Of these, one new integrated label is generated from a plurality of labels whose flag values are the first value, the flag of the integrated label is set to the first value, and the flags of the plurality of labels that are the source of the integrated label are set to the second value, thereby the Converting into second learning data including integrated labels, and using the converted second learning data to train a second machine learning model that outputs one correct label when inputting target data, Inputting the target data into the second machine learning model, and obtaining a prediction result including the correct label output from the second machine learning model; when the obtained correct label is the integrated label, converting the flags of the plurality of labels that are the source of the integrated label to the first value, and deleting the integrated label from the prediction result; A program for executing.

[0017] (4) One aspect of the present invention is The storage unit stores information for assigning a flag having a plurality of correct labels and a first value to one piece of learning data obtained from the annotation terminal; the learning device First learning data for training a first machine learning model that outputs a plurality of correct labels when inputting target data generates one new integrated label from the plurality of labels among which the value of the flag is the first value, sets the flag of the integrated label to the first value, and sets the flags of the plurality of labels that are the source of the integrated label to the second value, thereby Converting into second learning data including integrated labels, the learning device Using the converted second learning data to train a second machine learning model that outputs one correct label when inputting target data, the prediction device Inputting target data into a first machine learning model trained using first learning data with a plurality of correct labels assigned to the data, and obtaining a prediction result of the first machine learning model. the prediction device Input the target data into a second machine learning model trained using second learning data including an integrated label obtained by integrating a plurality of labels included in the first learning data based on flags of the plurality of labels, and when the correct label is the integrated label, flags of the plurality of labels that are the source of the integrated label are to the first value converted, the integrated label is deleted from the prediction result, and a prediction result of the second machine learning model is obtained; the prediction device evaluating the second machine learning model by comparing the prediction result of the first machine learning model with the prediction result of the second machine learning model; A method for evaluating a machine learning model, comprising:

Effect of the Invention

[0018] According to one aspect of the present invention, it is possible to perform a process of predicting one correct label from a plurality of labels and a process of predicting a plurality of correct labels from a plurality of labels without increasing the burden of annotation work.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0020] Hereinafter, a learning device and method, a prediction device and method, a program, and an evaluation method for a machine learning model to which the present invention is applied will be described with reference to the drawings.

[0021] [Overview of Embodiment] The data classification system 1 to which the present invention is applied causes, for example, a single-label model to learn to which label the data to be classified belongs, and classifies to which label the information belongs using the learned single-label model. The data to be classified in the embodiment is, for example, text data, document data, voice data, or image data. Hereinafter, the data to be classified will be referred to as "target data". In the embodiment, a label is information indicating the correct answer given to the target data. The single-label model in the embodiment is a machine learning model that predicts that the target data will be classified into one label when the target data is input. The multi-label model in the embodiment is a machine learning model that predicts that the target data will be classified into a plurality of labels when the target data is input.

[0022] In particular, the data classification system 1 of the embodiment generates an integrated label by integrating a plurality of labels in the learning data for the multi-label model annotated for learning the multi-label model, and converts it into learning data for a single-label model including the integrated label as one correct label. The learning data for this single-label model is data in which a single integrated label is pseudo-labeled instead of a plurality of correct labels in the multi-label model. The data classification system 1 learns the single-label model using the converted learning data for the single-label model. The data classification system 1 inputs the target data to the learned single-label model, and when the correct label output from the single-label model is the integrated label, converts the integrated label into a plurality of labels.

[0023] As a result, once the data classification system 1 of the embodiment creates learning data for the multi-label model through a single annotation operation, it can perform learning of the single-label model by converting the learning data. Further, according to the data classification system 1, by converting the correct label output from the learned single-label model, it is possible to output a prediction result equivalent to that of the multi-label model. As a result, the data classification system 1 can perform prediction processing by the single-label model and prediction processing by the multi-label model without increasing the burden of the annotation operation. Hereinafter, the data classification system 1 of the embodiment will be described in detail.

[0024] [Configuration of Data Classification System 1] FIG. 1 is a block diagram showing an example of the configuration of the data classification system in the embodiment. The data classification system 1 includes, for example, a request terminal 10, an annotation terminal 20, a multi-label storage unit 30, a learning device 40, a single-label storage unit 50, a model parameter storage unit 60, a prediction device 70, and a prediction result storage unit 80. The request terminal 10, the annotation terminal 20, the multi-label storage unit 30, the learning device 40, the single-label storage unit 50, the model parameter storage unit 60, the prediction device 70, and the prediction result storage unit 80 are connected to the network NW. Each device connected to the network NW is provided with a communication interface such as a NIC (Network Interface Card) or a wireless communication module. The network NW includes, for example, a general-purpose Internet, a WAN (Wide Area Network), a LAN (Local Area Network), Wifi (registered trademark), a cellular network, and the like.

[0025] The request terminal 10 is a mobile phone such as a smartphone, a tablet terminal, a personal computer, or the like. The request terminal 10 shares, for example, target data and information for requesting the correct label of the target data with the prediction device 70 according to the user's operation. The request terminal 10 receives information regarding the prediction result corresponding to the request from the prediction device 70 and presents the prediction result to the user.

[0026] The annotation terminal 20 is, for example, a personal computer or the like, and is a terminal for performing annotation work. The annotation work is an operation of attaching correct labels to data. The annotation terminal 20 associates data with correct labels according to the operator's operation and stores them in the multi-label storage unit 30. The annotation in the embodiment executes multi-labeling that allows multiple correct labels to be assigned to one piece of data. Note that, in the embodiment, the data to which labels are assigned may be text data, document data, image data, audio data, or data in which these data are combined, and is not particularly limited.

[0027] The multi-label storage unit 30 includes a storage device such as an HDD and control software. The multi-label storage unit 30 performs processes such as registration of data, registration of labels, and registration of the correspondence between data and labels according to a request from the annotation terminal 20. Thereby, the multi-label storage unit 30 stores learning data of a multi-label model in which multiple correct labels are assigned to one piece of data.

[0028] The learning device 40 is a computer that executes learning processing by executing a learning program by, for example, a CPU or the like. The learning device 40 converts the learning data of the multi-label model into the learning data of the single-label model, and learns the single-label model using the converted learning data of the single-label model. The learning device 40 stores the converted single-label model in the single-label storage unit 50. The learning device 40 learns the multi-label model using the learning data of the multi-label model stored in the multi-label storage unit 30.

[0029] The single-label storage unit 50 includes a storage device such as an HDD, and control software and the like. The single-label storage unit 50 performs processes such as registration of learning data of the multi-label model stored in the multi-label storage unit 30, registration of integrated labels, change of flags in the multi-label model, and registration of the correspondence between data and integrated labels according to requests from the learning device 40. Thereby, the single-label storage unit 50 stores learning data of a single-label model to which integration as one correct label is given for one data.

[0030] The model parameter storage unit 60 includes a storage device such as an HDD, and control software and the like. The model parameter storage unit 60 stores the model parameters of the multi-label model and the model parameters of the single-label model as the results learned by the learning device 40.

[0031] The prediction device 70 is a computer that executes prediction processing, for example, by executing a prediction program by a CPU or the like. The prediction device 70 inputs the target data supplied from the request terminal 10 into the multi-label model, and obtains a prediction result based on the information output from the multi-label model. The prediction result of the multi-label model may include a plurality of correct labels. The prediction device 70 inputs the target data supplied from the request terminal 10 into the single-label model, and obtains a prediction result based on the information output from the single-label model. When the correct label included in the single-label model is an integrated label, the prediction device 70 converts the integrated model into a plurality of correct labels, and generates a prediction result including the converted plurality of correct labels. The prediction result of the prediction device 70 is stored, for example, in the prediction result storage unit 80.

[0032] The prediction result storage unit 80 includes a storage device such as an HDD, and control software and the like. The prediction result stored in the prediction result storage unit 80 is provided to the request terminal 10, for example. The prediction result stored in the prediction result storage unit 80 may be used for comparison between the prediction result of the multi-label model and the prediction result of the single-label model.

[0033] FIG. 2 is a block diagram showing an example of the learning device 40 in the embodiment, (A) is a block diagram regarding the multi-label model, and (B) is a block diagram regarding the single-label model. FIG. 3 is a block diagram showing an example of the prediction device 70 in the embodiment, (A) is a block diagram regarding the multi-label model, and (B) is a block diagram regarding the single-label model.

[0034] As shown in FIG. 2(A), the learning device 40 includes, for example, a learning processing unit 42 and a multi-label model 44. The multi-label model 44 is a machine learning model, for example, a convolutional neural network (CNN). The learning processing unit 42 acquires the learning data of the multi-label model from the multi-label storage unit 30. FIG. 4 is a diagram showing an example of the learning data of the multi-label model in the embodiment. The learning processing unit 42 inputs the learning data of the multi-label model to the multi-label model 44 and acquires the prediction result output from the multi-label model 44. The learning processing unit 42 recursively updates the model parameters of the multi-label model 44 so that the correct label output from the multi-label model 44 matches the correct label in the learning data. That is, the learning processing unit 42 updates the processing parameters regarding the multi-label model 44 in the model parameter storage unit 60 and reflects the updated processing parameters in the multi-label model 44. The processing parameters are, for example, at least one of the number of layers, the number of nodes in each layer, the connection method of the nodes between each layer, the activation function, the error function, and the gradient descent algorithm, the pooling area, the kernel, the weight coefficient, and the weight matrix in a convolutional neural network. Thereby, the learning processing unit 42 performs, for example, deep learning to acquire the processing parameters. Deep learning is machine learning using a multi-layer structure, particularly a neural network of three or more layers.

[0035] As shown in FIG. 2(B), for example, the learning device 40 includes a conversion unit 46, a learning processing unit 42, and a single-label model 48. The conversion unit 46 generates an integrated label by integrating a plurality of correct labels in the learning data (first learning data) of a multi-label model to which a plurality of correct labels are assigned to the data, and converts it into learning data (second learning data) of a single-label model to which the integrated label is assigned as one correct label for the data. The conversion unit 46 stores the converted learning data of the single-label model in the single-label storage unit 50.

[0036] FIG. 5 is a diagram showing an example of the learning data (second learning data) of the converted single-label model in the embodiment. For example, as shown in FIG. 4, it is assumed that for one piece of data "Hello,...", the flags of the labels "The call cannot be connected" and "I want the person in charge to come out" are both "1". In this case, as shown in FIG. 5, the conversion unit 46 generates an integrated label "The call cannot be connected_I want the person in charge to come out" by integrating the two labels, sets the flag of the integrated model, and changes the flags of the two labels "The call cannot be connected" and "I want the person in charge to come out" that are the source of the integrated label to "0". Similarly, when the flags of all the labels for the data "Repaired yesterday..." are "1" for the labels "The call cannot be connected", "The network cannot be connected", and "Claim", the conversion unit 46 generates an integrated label "The call cannot be connected_The network cannot be connected_Claim", sets the flag of the integrated model, and changes the flag of the label that is the source of the integrated model to "0". Further, when the flags of all the labels for the data "Recording or..." are "0", the conversion unit 46 generates an integrated label "NoMatch" and sets the flag of the integrated model. Note that the label name of the integrated label is an example, and any other label name may be used as long as it can correspond to the plurality of labels that are the source of the integrated label. Also, the integrated label name "NoMatch" is an example, and any other label name may be used as long as it integrates the fact that the flags of all the labels for the data are "0".

[0037] The learning processing unit 42 acquires the learning data of the single-label model stored in the single-label storage unit 50. The learning processing unit 42 inputs the learning data of the single-label model into the single-label model 48 and acquires the prediction result output from the single-label model 48. The learning processing unit 42 recursively updates the model parameters of the single-label model 48 so that the correct label output from the single-label model 48 matches the correct label in the learning data. That is, the learning processing unit 42 updates the processing parameters regarding the single-label model 48 in the model parameter storage unit 60 and reflects the updated processing parameters in the single-label model 48. Note that the processing parameters of the single-label model 48 may be the same as those of the multi-label model 44.

[0038] As shown in FIG. 3(A), for example, the prediction device 70 includes a prediction unit 72 and a prediction result output unit 76. The prediction unit 72 acquires the model parameters of the multi-label model from the model parameter storage unit 60. The prediction unit 72 inputs the target data, predicts the correct label for the target data, and stores it in the prediction result storage unit 80. The prediction result output unit 76 answers the correct label for the target data stored in the prediction result storage unit 80 as the prediction result.

[0039] As shown in FIG. 3(B), for example, the prediction device 70 includes a prediction unit 72, an inverse conversion unit 74, and a prediction result output unit 76. The prediction unit 72 acquires the model parameters of the single-label model from the model parameter storage unit 60. The prediction unit 72 inputs the target data, predicts the correct label for the target data, and outputs it to the inverse conversion unit 74. The inverse conversion unit 74 refers to the single-label storage unit 50. When the output correct label is an integrated label, the integrated label is converted into a plurality of correct labels and stored in the prediction result storage unit 80. The prediction result output unit 76 answers the correct label for the target data stored in the prediction result storage unit 80 as the prediction result.

[0040] [Learning Processing] FIG. 6 is a flowchart showing an example of the learning process of the embodiment. The data classification system 1 stores the training data of the multi-label model by performing the annotation work (step S100) (step S102). The data classification system 1 acquires the training data of the multi-label model at the timing when the timing of the learning process arrives, for example (step S104), and determines the type of the model to be learned (step S106).

[0041] When the type of the model to be learned by the data classification system 1 is a single-label model, the data classification system 1 converts the training data of the multi-label model into the training data of the single-label model (step S108), and performs the learning process of the single-label model using the converted training data of the single-label model (step S110). Thereby, the data classification system 1 can store the model parameters of the single-label model (step S112).

[0042] When the type of the model to be learned by the data classification system 1 is a multi-label model, the data classification system 1 performs the learning process of the multi-label model using the acquired training data of the multi-label model (step S114). Thereby, the data classification system 1 can store the model parameters of the multi-label model (step S116).

[0043] FIG. 7 is a flowchart showing an example of the conversion process (step S108) of the learning data of the embodiment. First, the learning device 40 selects one row in the learning data of the multi-label model (step S200). The learning data of the multi-label model in the embodiment is assumed to have data and label flags associated with each other in one row as shown in FIG. 4. Next, the learning device 40 determines whether or not there are two or more labels with the flag set (step S202). When there are two or more labels with the flag set (step S202: YES), the learning device 40 generates an integrated label by integrating the two or more labels (step S204). Next, the learning device 40 adds the generated integrated label to the column of the learning data (step S206), sets the flag of the integrated label to "1", and sets the flag of the original label that is the source of the integrated label to "0" (step S208).

[0044] When there are not two or more labels with the flag set (step S202: NO), the learning device 40 determines whether or not all the flags are "0" (step S210). When not all the flags are "0" (step S210: NO), the learning device 40 proceeds to step S216. When all the flags are "0" (step S210: YES), the learning device 40 adds a NoMatch label to the column of the learning data (step S212), sets the flag of the NoMatch label to "1", and sets the flags of the other labels of the NoMatch label to "0" (step S214).

[0045] In step S214, the learning device 40 determines whether or not the process has been performed for all the rows of the learning data of the multi-label model. When the process has not been performed for all the rows (step S214: NO), the learning device 40 returns the process to step S200. When the process has been performed for all the rows (step S214: YES), the process of this flowchart ends.

[0046] [Prediction process] FIG. 8 is a flowchart showing an example of the prediction process of the embodiment. The prediction device 70 acquires target data from the request terminal 10 (step S300), and determines the type of the model to be predicted (step S302). Note that the type of the model to be predicted may be set in advance based on, for example, a user operation.

[0047] When the type of the model to be predicted by the prediction device 70 is a single-label model, the prediction device 70 inputs the target data into the single-label model and acquires the prediction result output from the single-label model (step S304). Next, the prediction device 70 inverse-transforms the prediction result output from the single-label model (step S306). Next, the prediction device 70 stores the inverse-transformed prediction result in the prediction result storage unit 80 (step S308). Next, the prediction device 70 answers the stored prediction result (step S314).

[0048] When the type of the model to be predicted by the prediction device 70 is a multi-label model, the prediction device 70 inputs the target data into the multi-label model and acquires the prediction result output from the multi-label model (step S310). Next, the prediction device 70 stores the prediction result output from the multi-label model in the prediction result storage unit 80 (step S312). Next, the prediction device 70 answers the stored prediction result (step S314).

[0049] FIG. 9 is a flowchart showing an example of the inverse transformation process (step S306) of the prediction result of the embodiment. First, the prediction device 70 selects one row in the prediction result of the single-label model (step S400). For example, when a plurality of target data are included in the request, or when the data included in the request is divided into a plurality of target data according to a predetermined rule, the prediction result of the single-label model is data in which the target data and the score for each label are associated with each row. Next, the prediction device 70 acquires, as the prediction result, the label with the highest score among the labels included in the prediction result (step S402).

[0050] Next, the prediction device 70 determines whether the label with the highest score is the NoMatch label (step S406). If the label with the highest score is the NoMatch label (step S406: YES), the prediction device 70 sets the flags of all the labels included in the prediction result to "0" (step S408), and proceeds to step S416. If the label with the highest score is not the NoMatch label (step S406: NO), the prediction device 70 proceeds to step S410.

[0051] In step S410, the prediction device 70 determines whether the label with the highest score is the integrated label. If the label with the highest score is the integrated label (step S410: YES), the prediction device 70 sets the flag of the original label of the integrated label to "1" and sets the flags of the other labels of the original label to "0" (step S412). If the label with the highest score is not the integrated label (step S410: NO), the prediction device 70 sets the flag of the label with the highest score to "1" and sets the flags of the other labels of the label with the highest score to "0" (step S414).

[0052] In step S416, the prediction device 70 determines whether processing has been performed for all rows of the prediction result. If processing has not been performed for all rows (step S416: NO), the prediction device 70 returns the processing to step S400. If processing has been performed for all rows (step S416: YES), the prediction device 70 discards the integrated label column included in the prediction result (step S418), and ends the processing of this flowchart.

[0053] FIG. 10 is a diagram for explaining an example of the conversion process and the inverse conversion process of the embodiment. (A) is an example of learning data of a multi-label model, (B) is an example of the learning data subjected to the conversion process, (C) is an example of a prediction result, and (D) is an example of the prediction result subjected to the inverse conversion process.

[0054] Assume that the annotation terminal 20 generates learning data for a multi-label model as shown in Fig. 10(A) through annotation work. Before training the single-label model, the training device 40 extracts a plurality of labels 1 to 3 whose flags are "1" from the learning data of the multi-label model, and generates a new label x by integrating the extracted labels 1 to 3. When the flags of a plurality of labels other than labels 1 to 3 in the learning data of the multi-label model are "1", the training device 40 may integrate the plurality of labels to generate a new label x+1. As shown in Fig. 10(B), the training device 40 creates learning data for the single-label model in which the flags of labels 1 to 3 that are the source of the newly generated integrated label x are set to "0" and the flag of label x is set to "1".

[0055] The prediction device 70 uses the single-label model to predict the most likely single label among a plurality of labels. As shown in Fig. 10(C), the prediction result is output such that the sum of the scores (likelihood, probability) of each label is 1. The prediction device 70 determines that the score of label x is the highest among the scores of all labels. Since label x is an integrated label, as shown in Fig. 10(D), the prediction device 70 converts it to a prediction result in which the flags of labels 1 to 3 that are the source of label x are set to "1", and deletes the integrated label.

[0056] Fig. 11 is a diagram for explaining an example of the conversion process and the inverse conversion process of the embodiment, (A) is an example of the learning data of the multi-label model, (B) is an example of the learning data after the conversion process, (C) is an example of the prediction result, and (D) is an example of the prediction result after the inverse conversion process.

[0057] Assume that the annotation terminal 20 generates learning data for a multi-label model as shown in Fig. 11(A) through annotation work. Before training the single-label model, since all the flags in the learning data of the multi-label model are "0", the learning device 40 generates a new NoMatch label by integrating all the labels. As shown in Fig. 11(B), the learning device 40 creates learning data for the single-label model with the flag of the generated NoMatch label set to "1".

[0058] The prediction device 70 uses the single-label model to predict the most likely single label among multiple labels. The prediction result is output such that the sum of the scores (likelihood, probability) of each label is 1, as shown in Fig. 11(C). The prediction device 70 determines that the score of the NoMatch label is the highest among the scores of all labels. Since the NoMatch label is the integrated label, as shown in Fig. 11(D), the prediction device 70 converts the prediction result with the flag of the label that was the source of the NoMatch label set to "0" and deletes the NoMatch label.

[0059] [Model Evaluation Process] The data classification system 1 may evaluate the single-label model trained using the learning data of the single-label model. Fig. 12 is a flowchart showing an example of the model evaluation process of the embodiment. The prediction device 70 acquires evaluation data (step S300#). The evaluation data is data including target data and correct labels. Next, the prediction device 70 determines the type of the model (step S302).

[0060] When the type of the model is a single-label model, the prediction device 70 inputs the evaluation data into the single-label model and obtains the prediction result output from the single-label model (step S304). Next, the prediction device 70 inverse-transforms the prediction result output from the single-label model (step S306). Next, the prediction device 70 stores the inverse-transformed prediction result in the prediction result storage unit 80 (step S308). When the type of the model is a multi-label model, the prediction device 70 inputs the evaluation data into the multi-label model and obtains the prediction result output from the multi-label model (step S310). Next, the prediction device 70 stores the prediction result output from the multi-label model in the prediction result storage unit 80 (step S312).

[0061] Next, the prediction device 70 compares the prediction result of the single-label model with the correct label in the evaluation data, and compares the prediction result of the multi-label model with the correct label in the evaluation data (step S320). Thereby, the prediction device 70 can evaluate the single-label model based on the prediction accuracy of the multi-label model and the prediction accuracy of the single-label model (step S322). For example, the prediction device 70 can evaluate whether the single-label model has a prediction accuracy close to that of the multi-label model.

[0062] FIG. 13 is a diagram showing the relationship between the number of labels included in the learning data of the multi-label model, the score predicted by the single-label model, and the score predicted by the multi-label model. In FIG. 13, the score corresponding to the label is shown as a line graph. According to FIG. 13, it can be seen that even when the same evaluation data is input to both the multi-label model and the single-label model, there is no significant difference in the scores predicted by the multi-label model and the single-label model. Therefore, it can be seen that even if the learning data of the multi-label model is converted to train the single-label model as in the embodiment, prediction can be performed with the same accuracy as the multi-label model.

[0063] As described above, according to the learning device 40 of the embodiment, a conversion unit 46 that converts first learning data for training a first machine learning model (multi-label model) that outputs a plurality of correct labels when inputting target data into second learning data including an integrated label in which a plurality of labels included in the first learning data are integrated based on flags of the plurality of labels, and a learning unit (learning processing unit 42, model parameter storage unit 60, and single-label model 48) that trains a second machine learning model (single-label model) that outputs one correct label when inputting target data using the second learning data converted by the conversion unit 46, can be provided to realize the learning device. Further, according to the learning device 40, a learning method corresponding to the learning device and a program that executes processing corresponding to the learning method can be realized. According to the learning device 40 of the embodiment, learning of the single-label model can be performed using the learning data of the multi-label model. As a result, according to the learning device 40 of the embodiment, if the annotation work of the learning data of the multi-label model is performed, the learning process of the single-label model that predicts one correct label from a plurality of labels and the learning process of the multi-label model that predicts a plurality of correct labels from a plurality of labels can be performed without increasing the burden of the annotation work.

[0064] According to the prediction device 70 of the embodiment, a prediction unit 72 that inputs target data to a second machine learning model (single label model) learned using second training data obtained by converting first training data for training a first machine learning model (multi-label model) that outputs a plurality of correct labels when the target data is input, and that acquires a prediction result including the correct label output from the second machine learning model; and an inverse conversion unit 74 that, when the correct label acquired by the prediction unit 72 is an integrated label, converts the flags of the plurality of labels that are the source of the integrated label and deletes the integrated label from the prediction result, can be provided. Further, according to the prediction device 70, a prediction method corresponding to the prediction device and a program that executes a process corresponding to the prediction method can be realized. According to the prediction device 70 of the embodiment, learning of a single label model can be performed using training data obtained by converting the training data of the multi-label model, and the prediction result of the single label model can be inversely converted to output the prediction result of the multi-label model. As a result, according to the prediction device 70 of the embodiment, if the annotation work of the training data of the multi-label model is performed, the prediction process of the single label model that predicts one correct label from a plurality of labels and the prediction process of the multi-label model that predicts a plurality of correct labels from a plurality of labels can be performed without increasing the burden of the annotation work.

[0065] According to the data classification system 1 of the embodiment, a first machine learning model (multi-label model) trained using first learning data in which a plurality of correct labels are assigned to data is input with target data, and a prediction result of the first machine learning model is obtained; inputting the target data into a second machine learning model (single-label model) trained using second learning data including an integrated label obtained by integrating a plurality of labels included in the first learning data based on the flags of the plurality of labels, and when the correct label is the integrated label, converting the flags of the plurality of labels that are the source of the integrated label, deleting the integrated label from the prediction result, and obtaining the prediction result of the second machine learning model; and evaluating the second machine learning model by comparing the prediction result of the first machine learning model with the prediction result of the second machine learning model. According to this data classification system 1, when learning data for a multi-label model is generated by a single annotation operation and learning of the multi-label model and the single-label model is performed, the prediction result of the single-label model can be evaluated.

[0066] Although each embodiment and modification example have been described, these are merely examples and are not limited thereto. For example, any one of the embodiments or modification examples, or a part of each embodiment or a part of each modification example, may be combined with one or more other embodiments or one or more other modification examples to implement an aspect of the present invention.

[0067] Note that a program for executing each process of the user terminal device 100, the search server device 200, and the batch server device 300 in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, whereby the various processes described above related to the user terminal device 100, the search server device 200, and the batch server device 300 may be performed.

[0068] Note that the "computer system" mentioned here may include hardware such as an OS and peripheral devices. Also, the "computer system" shall include a homepage providing environment (or display environment) if the WWW system is being used. Further, the "computer-readable recording medium" refers to a writable non-volatile memory such as a flexible disk, magneto-optical disk, ROM, flash memory, a portable medium such as a CD-ROM, and a storage device such as a hard disk built into a computer system.

[0069] Furthermore, the "computer-readable recording medium" also includes a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and that holds the program for a certain period of time. Also, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication wire) such as a telephone line. Also, the above program may be for realizing a part of the aforementioned functions. Further, it may be a so-called difference file (difference program) that can realize the aforementioned functions in combination with a program already recorded in a computer system.

[0070] As described above, the embodiments of the present invention have been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and designs within the scope not departing from the gist of the present invention are also included.

[0071]

Explanation of Reference Numerals

[0072] ​10 Request terminal 20 Annotation terminal 30 Multi-label memory unit 40 Learning device 42 Learning processing unit 44 Multi-label model 46 Conversion unit 48 Single-label model 50 Single-label memory unit 60 Model parameter memory unit 70 Prediction device 72 Prediction unit 74 Inverse conversion unit 76 Prediction result output unit 80 Prediction result memory unit

Claims

[

1. ] A multi-label storage unit that stores information for attaching a plurality of correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal. Among the first learning data for training a first machine learning model that outputs a plurality of correct labels when target data is input, one new integrated label is generated from a plurality of labels whose flag values are the first value, the flag of the integrated label is set to the first value, and the flags of the plurality of labels that are the source of the integrated label are set to a second value, thereby converting to second learning data including the integrated label. A conversion unit. A learning unit that trains a second machine learning model that outputs one correct label when target data is input, using the second learning data converted by the conversion unit. A prediction unit that inputs target data to the second machine learning model and obtains a prediction result including the correct label output from the second machine learning model. When the correct label obtained by the prediction unit is the integrated label, the flags of the plurality of labels that are the source of the integrated label are converted to the first value, and the integrated label is deleted from the prediction result. An inverse conversion unit. A data classification system comprising: [

2. ] A step in which a storage unit stores information for attaching a plurality of correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal. A step in which a learning device generates one new integrated label from a plurality of labels whose flag values are the first value among the first learning data for training a first machine learning model that outputs a plurality of correct labels when target data is input, sets the flag of the integrated label to the first value, and sets the flags of the plurality of labels that are the source of the integrated label to a second value, thereby converting to second learning data including the integrated label. A step in which the learning device trains a second machine learning model that outputs one correct label when target data is input, using the converted second learning data. A step in which a prediction device inputs target data to the second machine learning model and obtains a prediction result including the correct label output from the second machine learning model. When the acquired correct label is the integrated label, the prediction device converts the flags of a plurality of labels that are the source of the integrated label into the first value and deletes the integrated label from the prediction result; A data classification method including this.

3. Causing a computer to: store information for attaching a flag having a plurality of correct labels and a first value to one piece of learning data acquired from an annotation terminal; generate a new integrated label from a plurality of labels in which the value of the flag is the first value among the first learning data for training a first machine learning model that outputs a plurality of correct labels when inputting target data, set the flag of the integrated label to the first value, and set the flags of the plurality of labels that are the source of the integrated label to a second value, thereby converting the first learning data including the integrated label into second learning data; training a second machine learning model that outputs one correct label when inputting target data using the converted second learning data; inputting target data to the second machine learning model and obtaining a prediction result including the correct label output from the second machine learning model; When the acquired correct label is the integrated label, converting the flags of a plurality of labels that are the source of the integrated label into the first value and deleting the integrated label from the prediction result; A program for causing the above to be executed.

4. The storage unit stores information for attaching a flag having a plurality of correct labels and a first value to one piece of learning data acquired from an annotation terminal; The learning device generates a new integrated label from a plurality of labels in which the value of the flag is the first value among the first learning data for training a first machine learning model that outputs a plurality of correct labels when inputting target data, sets the flag of the integrated label to the first value, and sets the flags of the plurality of labels that are the source of the integrated label to a second value, thereby converting the first learning data including the integrated label into second learning data; The learning device trains a second machine learning model that outputs one correct label when inputting target data using the converted second learning data; The prediction device inputs target data into a first machine learning model trained using first training data in which a plurality of correct labels are assigned to the data, and obtains a prediction result of the first machine learning model; The prediction device inputs target data into a second machine learning model trained using second training data including an integrated label obtained by integrating a plurality of labels included in the first training data based on flags of the plurality of labels, and when the correct label is the integrated label, converts the flags of the plurality of labels that are the source of the integrated label into the first value, deletes the integrated label from the prediction result, and obtains a prediction result of the second machine learning model; The prediction device evaluates the second machine learning model by comparing the prediction result of the first machine learning model and the prediction result of the second machine learning model; A method for evaluating a machine learning model, including the above steps.

Images