Information processing method, control program, and information processing system

Abstract

To determine whether a label definition for a teacher label needs to be modified.SOLUTION: An information processing method performs processing including a step (a) of inputting data into a learned classification model and classifying the learned classification model, a step (b) of generating determination information that determines whether a label definition needs to be modified based on a classification result classified in the step (a), and a step (c) of outputting the determination information of the step (b).SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing method and a control program. 、 and information processing systems. [Background technology]

[0002] In recent years, machine learning such as deep learning has been widely used in various fields. In one machine learning technique, data to be classified is input and the data is classified using a classification model to determine the type of data. This classification model is trained using a training dataset that contains data and its teacher labels in advance. Because the classification model is trained using the training dataset, if the teacher labels assigned to the data in the training dataset are inappropriate, the classification accuracy of the classification model obtained by training will be low.

[0003] Since annotation work for generating a training dataset is generally performed by an operator, there are many cases where the teacher labels are inappropriate. In the technology disclosed in Non-Patent Document 1, a classification model is trained using a training dataset (a set of data and teacher labels), and the trained classification model is used to classify the data of the training dataset as input data. Then, based on the obtained output label (classified class) and the teacher labels previously assigned to this data, the joint distribution of the true label and the teacher labels is estimated, thereby determining whether the teacher labels have been assigned incorrectly.

[0004] Similarly, in the technology disclosed in Patent Document 1, a classification model is trained using a training dataset, and the trained model is used to classify the data in the training dataset. Then, based on the obtained output labels and likelihoods and the teacher labels assigned to the data, it is determined whether the teacher labels have been assigned incorrectly. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-111279 [Non-patent literature]

[0006] [Non-Patent Document 1] Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang, "Confident Learning:Estimating Uncertainty in Dataset Labels", ICML2020, Internet (URL: https: / / arxiv.org / pdf / 1911.00068.pdf) Summary of the Invention [Problem to be solved by the invention]

[0007] In annotation work, in addition to simple mistakes in assigning teacher labels, there are also cases where teacher labels cannot be assigned appropriately due to inappropriate label definitions. For example, unexpected data may be discovered during annotation work, and the previous label definition may not have an appropriate class, making it necessary to change the label definition by adding a new class, for example. In this case, there is a problem that the consistency of definitions may become unclear when multiple workers are involved, and consistency cannot be ensured as the workload increases. The technologies disclosed in Non-Patent Document 1 and Patent Document 1 detect teacher label errors but cannot determine whether such label definitions are appropriate.

[0008] The present invention has been made to solve such problems, and has an object to provide an information processing method and an information processing system that can determine whether or not a label definition needs to be modified. [Means for solving the problem]

[0009] The above-mentioned problems of the present invention are solved by the following means.

[0010] (1) Step (a) of inputting data into a trained classification model and classifying the data using the trained classification model; a step (b) of generating determination information that determines whether or not the label definition needs to be corrected based on the classification result obtained in the step (a); Step (c) of outputting the determination information of step (b); An information processing method for performing processing including the steps of:

[0011] (2) In the information processing method described in (1) above, if it is determined in step (b) that the label definition needs to be modified, the determination information further includes at least one of a proposal for adding a class to be classified, merging a class, and subdividing a class.

[0012] (3) The information processing method according to (1) or (2), wherein in the step (a), a multi-label classification model that outputs an independent likelihood for each class is used as the trained model.

[0013] (4) In the step (a), label noise data is input, and in the step (b), the label noise data is judged to generate the judgment information. Here, in the information processing method described in (3) above, when a dataset is classified using a multi-class classification model trained using a dataset in which data is assigned a teacher label, if data classified into a class different from the teacher label is called first classification data and data other than the first classification data is called second classification data, then the label noise data is the first classification data.

[0014] (5) further comprising a step (d) of setting a decision threshold based on the likelihood for each class into which the plurality of data are classified in the step (a); The information processing method according to (3) or (4) above, wherein in the step (b), the determination is made using the determination threshold set in the step (d).

[0015] (6) An information processing method according to (5) above, wherein in step (b), if there are multiple classes in the classification results of one target data that have a likelihood greater than or equal to the judgment threshold, the judgment information suggests class integration or class subdivision.

[0016] (7) An information processing method according to (5) or (6) above, wherein in step (b), if the likelihood of all classes in the classification results of one target data is less than the judgment threshold, the judgment information suggests adding a class.

[0017] (8) An information processing method according to any one of (5) to (7) above, wherein in step (b), if the likelihood of only one class in the classification result of one piece of data is equal to or greater than the judgment threshold, the judgment information is that the label definition should not be corrected, and it is proposed to change the teacher label to the class having a likelihood equal to or greater than the judgment threshold.

[0018] (9) An information processing method according to any one of (5) to (8) above, wherein in step (d), a plurality of data including the first classification data and the second classification data are classified in step (a), and the average value of the obtained likelihoods is used as the judgment threshold.

[0019] (10) An information processing method described in any one of (4) to (9) above, wherein the trained classification model is a classification model trained using multiple data sets based on the second classification data.

[0020] (11) A control program for causing a computer to execute the information processing method according to any one of (1) to (10) above.

[0021] (12) An information processing device comprising a control unit that executes the program described in (11) above.

[0022] (13) an acquisition unit for acquiring data; a control unit that generates determination information for label definition from the acquired data; an output unit that outputs determination information of the label definition; The control unit includes a label classification unit that inputs the data into a trained classification model and classifies the data using the trained classification model; and An information processing system that functions as a label definition determination unit that performs classification using the trained classification model and generates determination information that determines whether or not a label definition needs to be corrected based on the classification result.

[0023] (14) In the information processing system described in (13) above, when the label definition determination unit determines that the label definition needs to be modified, the determination information further includes at least one suggestion of adding a class to be classified, merging classes, and subdividing classes.

[0024] (15) The information processing system according to (13) or (14), wherein the label classification unit uses a multi-label classification model that outputs an independent likelihood for each class as the trained model.

[0025] (16) The acquisition unit acquires label noise data, the control unit generates the determination information from the label noise data. Here, when classifying a dataset using a multi-class classification model trained using a dataset in which a teacher label is assigned to the data, if data classified into the same class as the teacher label is defined as second classified data and data classified into a class different from the teacher label is defined as first classified data, the label noise data is the first classified data, as described in (15) above. [Effects of the Invention]

[0026] The information processing method according to the present invention executes a process including the steps of: inputting data into a trained classification model and classifying the data using the trained classification model; generating determination information for determining whether or not a label definition needs to be corrected based on the classification results obtained in the step (a); and outputting the determination information obtained in the step (b). In this way, it is possible to appropriately determine whether or not a label definition needs to be corrected. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a block diagram showing a schematic configuration of an information processing system according to an embodiment of the present invention. [Figure 2A] FIG. 2 is a functional block diagram showing the flow of data in each function of the control unit. [Figure 2B] FIG. 2 is a functional block diagram showing the flow of data in each function of the control unit. [Figure 3] 10 is a flowchart showing the processing of the first pre-stage. [Figure 4] 10 is a table showing an example of data to which inappropriate teacher labels have been assigned. [Figure 5] 10 is a flowchart showing the process of learning a multi-label classification model and setting a decision threshold as the process of pre-stage 2. [Figure 6] 6 is a subroutine flowchart showing the determination threshold setting process in step S22 of FIG. 5. [Figure 7] 10 is a flowchart showing a process for determining whether or not a label definition needs to be corrected in this embodiment. [Figure 8] 8 is a subroutine flowchart showing the determination information generation process in step S33 of FIG. 7. [Figure 9] 10 is a table illustrating an example of likelihood for each label and the relationship with determination information. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and duplicate explanations will be omitted. Furthermore, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0029] 1 is a block diagram showing a schematic configuration of an information processing system 10 according to this embodiment. The information processing system 10 includes a control unit 11, a first storage unit 21, a second storage unit 22, and a communication unit 23. These are interconnected via signal lines such as a bus for exchanging signals.

[0030] The control unit 11 functions as an information processing device and also functions as a machine learning unit. The control unit 11 includes multiple CPUs, multiple GPUs (Graphics Processing Units), RAM, ROM, etc., and controls each device, performs machine learning, and determines whether or not the definition of the teacher label needs to be modified according to a program. The information processing system 10 may be an on-premise server or a cloud server using a commercial cloud service. Furthermore, some of the functions of the information processing system 10 (for example, only the functions of the machine learning unit) may be implemented by the cloud server.

[0031] The control unit 11 also functions as a first learning unit 111, a class classification unit 112, a noise determination unit 113, a second learning unit 121, a label classification unit 122, and a label definition determination unit 123. These functions will be described later.

[0032] The first storage unit 21 and the second storage unit 22 are configured from semiconductor memory or magnetic memory such as a hard disk that stores various programs and data in advance. The first storage unit 21 and the second storage unit 22 may each use a memory area logically divided in a single piece of hardware, or may be configured as separate units. In the latter case, one or both may be located on a cloud server.

[0033] Classification models (multi-class classification models and multi-label classification models) that are learned, generated, and updated by machine learning on a cloud server or on the information processing system 10 itself are stored in the first storage unit 21. The multi-class classification models and multi-label classification models may have a neural network structure such as CNN or RNN.

[0034] A multi-class classification model assigns only one label from multiple label definitions as output for a single input object. This classification is also called single-label classification in contrast to multi-label classification below. In the case of multi-class classification, the sum of the likelihoods of the outputs is 1 (100%).

[0035] A multi-label classification model can simultaneously assign two or more labels as output to a single input object. In the case of multi-label classification, the sum of the likelihoods of the outputs does not necessarily equal 1. Multi-class classification models and multi-label classification models can use images or documents as input objects.

[0036] The second storage unit 22 stores a training data set. Furthermore, the training data set is divided into first and second categories by a process described later by the noise determination unit 113. The first category is label noise data, which is data to which an inappropriate teacher label is assigned. The second category is label clean data, which is data to which an appropriate teacher label is assigned.

[0037] The communication unit 23 is an interface that transmits and receives data via a network. For example, communication is performed using standards such as Ethernet, Bluetooth (registered trademark), and IEEE802.11 (Wi-Fi). The communication unit 23 functions as an acquisition unit and an output unit by working in cooperation with the control unit 11. The acquisition unit functions to acquire a learning dataset from a terminal device such as a PC connected to the network. The output unit functions to transmit determination information indicating whether the label definitions made by the control unit 11 are appropriate to the terminal device, and to present the user with suggested information regarding corrections to the label definitions.

[0038] 2A and 2B are functional block diagrams showing the flow of data in each function of the control unit 11 (information processing device). FIG. 2B is performed following FIG. 2A. As shown in FIG. 2A, a training data set is stored in the second storage unit 22. The training data set is composed of a large number of training data to which teacher labels are respectively assigned. The teacher labels are assigned by a user via a terminal device. Alternatively, the teacher labels may be assigned in a pre-labeling process using a rule-based algorithm or a machine learning model. Label definition (class classification) for these teacher labels is performed in advance by the user.

[0039] (First learning unit 111, classification unit 112, noise determination unit 113) 1 and 2A, the first learning unit 111 trains a multi-class classification model that is before or during training and that is stored in the first storage unit 21, using a training dataset in the second storage unit 22. The trained multi-class classification model, whose parameters have been updated by training, is stored in the first storage unit 21, and the multi-class classification model before training is updated.

[0040] The classification unit 112 performs inference processing using a trained multi-class classification model stored in the first storage unit 21. By inputting training data (target data), the classification unit 112 performs multi-class classification of the target data and outputs only one output label from among multiple classes and its likelihood.

[0041] The noise determination unit 113 determines whether the teacher label assigned to the target data is appropriate or not, depending on the output of the classification unit 112. Specifically, when the classification unit 112 determines that the output label (different class) is different from the teacher label assigned to the target data, the noise determination unit 113 determines that the teacher label is inappropriate and determines that the target data is first classified data (label noise data). As a modified example, the noise determination unit 113 may set a first determination threshold (hereinafter also referred to as determination threshold a) for likelihood by processing described later (FIG. 3), and determine that the teacher label for target data output with a likelihood lower than the determination threshold a is inappropriate and that the target data is first classified data. In the latter modified example, even if the assigned teacher label and the output label are the same, the target data is determined to be first classified data (label noise data). In both the former and latter (modified examples), target data other than first classified data is determined to be second classified data (label clean data).

[0042] (Second learning unit 121, label classification unit 122, label definition determination unit 123) 1 and 2B, the components of the second learning unit 121, label classification unit 122, and label definition determination unit 123, which are the main components of this embodiment, will be described. As will be described below, in this embodiment, these components train a multi-label classification model using the second classified data (label-clean data), and the trained multi-label classification model performs inference processing on the first classified data (label noise), and determines whether or not the label definition needs to be modified based on the likelihood for each obtained label.

[0043] 1 and 2B, the second learning unit 121 learns a multi-label classification model that has not yet been learned or is currently being learned and that is stored in the first storage unit 21, using a plurality of data sets that have been classified into the second category (label-clean data) by the noise determination unit 113, among the learning data sets in the second storage unit 22. The trained multi-label classification model whose parameters have been updated by learning is stored in the first storage unit 21, and the multi-label classification model before learning is updated.

[0044] The label classification unit 122 performs inference processing using a trained multi-label classification model stored in the first storage unit 21. By inputting data determined to be first classified data (label noise) from among the training data stored in the second storage unit 22, the label classification unit 122 can perform multi-label classification of the target data and select multiple classes from the multiple classes as output labels. The label classification unit 122 also outputs the likelihood of each output label.

[0045] The label definition determination unit 123 generates determination information that determines whether or not the label definition needs to be modified, based on the classification result of the label classification unit 122. If the label definition determination unit 123 determines that the label definition needs to be modified, the unit 123 further includes in the determination information at least one proposal from among adding a class to be classified, integrating classes, and subdividing classes. In addition, the unit 123 determines a proposal to change the teacher label.

[0046] Furthermore, the label definition determination unit 123 sets a second determination threshold (hereinafter also referred to as determination threshold b) for likelihood by processing described later (FIG. 6), and proposes a change in the label definition accordingly. Specifically, the label definition determination unit 123 makes at least one of the following determinations, "Determination 1" to "Determination 3," depending on the classification result (classified class and likelihood of each class) of one target data (label noise data). (Decision 1) If there are multiple classes with likelihoods equal to or greater than the decision threshold b, the decision information suggests class integration or class subdivision. For example, this is the case when the label definitions are not exclusive, and the concepts of one class are contained within the concepts of another class, or when the category hierarchy is inappropriate. (Decision 2) If the likelihood of all classes is less than the decision threshold b, we propose adding a class as decision information. For example, this is the case when data other than that initially assumed exists and there is no appropriate class based on the label definitions up to now. (Decision 3) If the likelihood of only one class is greater than or equal to the decision threshold b, a change in the teacher label is proposed. In this case, it is suspected that the teacher label for this data set has been misassigned, but it is determined that there is no problem with the label definition and that correction of the label definition is unnecessary. In addition, as a response to the misassigned teacher label for this data set, it is proposed to change the class assigned in the original training dataset to the class whose likelihood is greater than or equal to the decision threshold b.

[0047] (Label definition determination process) Next, an information processing method for determining label definitions executed by the information processing system of this embodiment will be described with reference to FIGS.

[0048] (Pre-processing) First, the processing of the previous stage will be explained with reference to Figures 3 to 6. Figures 3 and 5 (Figure 6) are flowcharts showing the processing of previous stages 1 and 2, respectively, and Figure 4 is a table showing an example of data to which inappropriate teacher labels have been assigned. The processing of this previous stage 1 corresponds to the data flow shown in Figure 2A, and the processing of previous stage 2 corresponds to the flow in the upper half of Figure 2B.

[0049] (Preliminary stage 1) In the process of pre-stage 1 in FIG. 3 described below, data in the learning dataset is divided into first and second classifications (label-noise data, label-clean data).

[0050] (Step S11) The first learning unit 111 trains the multi-class classification model stored in the first storage unit 21 using the learning dataset stored in the second storage unit 22. The number of epochs for repeatedly using the learning dataset can be set as appropriate.

[0051] (Steps S12 and S13) The classification unit 112 performs multi-class classification processing on each piece of data included in the training dataset using a trained multi-class classification model. As a result, for each piece of target data input, one class and its likelihood are obtained as output. This is performed for all training data (hereinafter simply referred to as data). The classification results are temporarily stored in the storage unit 22 or the like.

[0052] (Step S14) The noise determination unit 113 uses the stored classification results to set determination thresholds a1 to an (collectively referred to as determination threshold a) for each of the n classes 1 to n. The determination threshold a can be the average value of the likelihood of each class.

[0053] (Step S15) The noise determination unit 113 determines whether the teacher label of each piece of data is inappropriate based on the classification results obtained up to step S13. It determines that the data is inappropriate if either (Condition 1) or (Condition 2) below is satisfied. Note that as a condition for determining inappropriateness, setting of the determination threshold value a may be omitted, and the determination may be made based only on Condition 1. (Condition 1) The output label and the teacher label originally assigned in the initial training dataset are different. (Condition 2) If the likelihood is less than the decision threshold a (even if the labels match).

[0054] (Step S16) The noise determination unit 113 extracts data to which an inappropriate teacher label has been assigned as first classified data. FIG. 4 is a table showing the results of classifying a training dataset using a multi-class classification model when the label definitions of classes 1 to 3 are "mandarin orange," "apple," and "citrus fruits," respectively. The numbers in the table indicate the number of output data items classified into classes. For example, if data with a teacher label of "mandarin orange" is assigned an output label of "apple" or "citrus fruits" through inference by the multi-class classification model, the data is extracted as first classified data (label noise data) that satisfies condition 1 shown in the dashed frame in FIG. 4.

[0055] (Step S17) The noise determination unit 113 extracts data other than the first classification data as second classification data. Specifically, this is the case when the output label and the teacher label match and the likelihood is equal to or greater than the determination threshold a. This completes the process of pre-stage 1 in Figure 3.

[0056] (Preliminary stage 2) 5 and 6, as the processing of the pre-stage 2, the information processing system 10 performs the processing of learning the multi-label classification model and setting the decision threshold b.

[0057] (Step S21) Here, the second learning unit 121 uses clean-label data to learn the multi-label classification model stored in the first storage unit 21. Specifically, learning is performed using multiple sets of the second classification data (clean-label data) and its teacher labels (hereinafter also referred to as second learning datasets) extracted in step S17 from the learning dataset stored in the second storage unit 22. The number of epochs for repeatedly using this second learning dataset can be set as appropriate.

[0058] (Step S22) Here, determination thresholds b1 to bn (collectively referred to as determination threshold b) are set for the classes 1 to n, respectively. Fig. 6 is a subroutine flowchart showing the determination threshold setting process in step S22 of Fig. 5.

[0059] (Step S221) The label classification unit 122 performs multi-label classification processing on all data included in the training dataset using the multi-label classification model trained in step S21. As a result, multiple classes can be obtained as output labels for a single target data input. The likelihood for the output labels is also obtained.

[0060] (Step S222) Based on the output of step S221, the label definition determination unit 123 sets the average likelihood of all data of each of classes 1 to n as determination thresholds b1 to bn (hereinafter, these will be collectively referred to as determination threshold b). This completes the processing up to previous stage 2, and the next process of determining whether or not the label definition needs to be modified is then carried out.

[0061] (Determining whether label definition needs to be modified) FIG. 7 is a flowchart showing the process of determining whether or not a label definition needs to be corrected in this embodiment.

[0062] (Step S31) The control unit 11 inputs data to the multi-label classification model. In this embodiment, data (label noise data) classified into the first classification data in the process of Fig. 3 is subject to determination of whether or not the label definition needs to be corrected.

[0063] (Step S32) The label classification unit 122 performs multi-label classification processing on the target data input in step S31, i.e., label noise data, using a multi-label classification model. One or more classes and their likelihoods are obtained as outputs.

[0064] (Step S33) The label definition determination unit 123 generates determination information that determines whether or not the label definition needs to be corrected, based on the classification result obtained in step S32. Fig. 8 is a subroutine flowchart showing the determination information generation process in step S33 of Fig. 7.

[0065] (Step S331) The label definition determination unit 123 determines whether or not the label definition needs to be corrected. Specifically, if there is only one class whose likelihood is equal to or greater than the determination threshold b in the classification result of one labeled noise data that is the target of the processing in steps S31 and S32, the label definition determination unit 123 determines that there is no need to correct the label definition (NO). This is a situation in which an incorrect teacher label is suspected, and corresponds to the above-mentioned determination 3. On the other hand, in cases other than when there is only one class whose likelihood is equal to or greater than the determination threshold b, the control unit 11 determines that there is a need to correct the label definition (YES). If the determination is YES, the control unit 11 proceeds to step S335, and if the determination is NO, the control unit 11 proceeds to step S333.

[0066] (Step S332) The label definition determination unit 123 counts the number of classes whose likelihood is equal to or greater than the determination threshold b in the classification result of the target label noise data. If there are multiple classes, the label definition determination unit 123 proceeds to step S333, and if there are zero classes, the label definition determination unit 123 proceeds to step S334.

[0067] FIG. 9 is a table illustrating the relationship between the likelihood for each label and the judgment information. In the figure, the label definitions for classes 1 to 3 are "mandarin orange," "apple," and "citrus fruits," respectively, and the judgment thresholds b1 to 3 are all the same 50%. In the example of FIG. 9, data 1 corresponds to step S333 because it is equal to or greater than the judgment threshold b (b1, b3) in two classes 1 and 3, "mandarin orange" and "citrus fruits." Data 2 corresponds to step S334 because the likelihoods of all classes 1 to 3 are less than the judgment threshold b (b1 to b3). Data 3 corresponds to step S335 because it is equal to or greater than the judgment threshold b in only one class.

[0068] (Step S333) The label definition determination unit 123 provides the determination information (1) that the label definition needs to be corrected, and (2) that it proposes class integration or class subdivision.

[0069] (Step S334) The label definition determining unit 123 (1) determines that the label definition needs to be modified, and (2) proposes adding a class.

[0070] (Step S335) Here, the label definition determination unit 123 (1) determines that the label definition does not need to be corrected, and (2) proposes changing the truth label.

[0071] As described above, the information processing method according to this embodiment includes a step (a) of inputting data into a trained classification model and classifying the data using the trained classification model, a step (b) of generating determination information for determining whether or not the label definition needs to be modified based on the classification result obtained in the step (a), and a step (c) of outputting the determination information obtained in the step (b). This makes it possible to appropriately determine whether or not the label definition needs to be modified.

[0072] Furthermore, in step (b), if it is determined that the label definition needs to be modified, the determination information further includes at least one suggestion from among adding a class to be classified, integrating classes, and subdividing classes. This allows the user to easily understand how to modify the label definition.

[0073] The configurations of the information processing device and information processing system described above are the main configurations described in order to explain the features of the above-mentioned embodiments, but are not limited to the above configurations and may be modified in various ways within the scope of the claims. Furthermore, configurations that are included in general information processing devices or information processing systems are not excluded.

[0074] In addition, some steps in the flowchart relating to the information processing method described above may be omitted, other steps may be added, the order of some of the steps may be changed, some steps may be executed simultaneously, or one step may be divided into multiple steps and executed.

[0075] The means and methods for performing various processes in the information processing system 10 described above can be realized by either dedicated hardware circuits or a programmed computer. The programs may be provided by computer-readable recording media such as USB memory or DVD (Digital Versatile Disc)-ROM, or online via a network such as the Internet. In this case, the programs recorded on the computer-readable recording media are typically transferred to and stored in a storage unit such as a hard disk. The programs may be provided as standalone application software or may be incorporated into the software of a device as a function of the device. [Explanation of symbols]

[0076] 10 Information Processing System (Machine Learning Device) 11 Control unit (information processing device) 111 First Learning Section 112 Classification Unit 113 Noise determination unit 121 Second Learning Section 122 Label Classification Unit 123 Label definition judgment part 21 1st memory section 22 2nd memory section 23 Communications Department

Claims

1. Step (a) of inputting data into a trained classification model and classifying the data using the trained classification model; a step (b) of generating determination information that determines whether or not the label definition needs to be corrected based on the classification result obtained in the step (a); Step (c) of outputting the determination information of step (b); Including, In the step (b), when it is determined that the label definition needs to be corrected, the determination information further includes at least one proposal of adding a class to be classified, integrating classes, and subdividing classes; In the step (a), a multi-label classification model that outputs an independent likelihood for each class is used as the trained classification model; In the step (a), label noise data is input, and in the step (b), the label noise data is judged to generate the judgment information; Here, when a dataset is classified using a multi-class classification model trained using a dataset in which a teacher label is assigned to the data, if data classified into a class different from the teacher label is defined as first classified data and data other than the first classified data is defined as second classified data, the label noise data is the first classified data. An information processing method in which processing is performed by a computer.

2. The method further includes a step (d) of setting a decision threshold based on the likelihood of each class into which the plurality of data are classified in the step (a), The information processing method according to claim 1 , wherein the determination in said step (b) is performed using a determination threshold set in said step (d).

3. 3. The information processing method according to claim 2, wherein, in step (b), if there are multiple classes in the classification results of one target data that have a likelihood greater than or equal to the judgment threshold, the judgment information suggests class integration or class subdivision.

4. 4. The information processing method according to claim 2, wherein in step (b), if the likelihoods of all classes in the classification results of one target data are less than the decision threshold, adding a class is suggested as the decision information.

5. 5. The information processing method according to claim 2, wherein, in step (b), if the likelihood of only one class in the classification result of one piece of data is equal to or greater than the judgment threshold, the judgment information is that modification of the label definition is not permitted, and a suggestion is made to change the teacher label to the class having a likelihood equal to or greater than the judgment threshold.

6. 6. The information processing method according to claim 2, wherein in step (d), a plurality of data including the first classification data and the second classification data are classified in step (a), and an average value of the likelihoods obtained is used as the judgment threshold.

7. The information processing method according to claim 1 , wherein the trained classification model is a classification model trained using a plurality of data sets based on the second classification data.

8. A step (a) of inputting data into a trained classification model and classifying the data using the trained classification model; a step (b) of generating determination information that determines whether or not the label definition needs to be corrected based on the classification result obtained in the step (a); Step (c) of outputting the determination information of step (b); Including, In the step (b), when it is determined that the label definition needs to be corrected, the determination information further includes at least one proposal of adding a class to be classified, integrating classes, and subdividing classes; In the step (a), a multi-label classification model that outputs an independent likelihood for each class is used as the trained classification model; The method further includes a step (d) of setting a decision threshold based on the likelihood of each class into which the plurality of data are classified in the step (a), In the step (b), the determination is made using the determination threshold value set in the step (d). An information processing method in which processing is performed by a computer.

9. A control program for causing a computer to execute the information processing method according to any one of claims 1 to 8.

10. an acquisition unit for acquiring data; a control unit that generates determination information for label definition from the acquired data; an output unit that outputs determination information of the label definition; The control unit includes a label classification unit that inputs the data into a trained classification model and classifies the data using the trained classification model; and a label definition determination unit that performs classification using the trained classification model and generates determination information that determines whether or not a label definition needs to be corrected based on the classification result; When the label definition determination unit determines that the label definition needs to be corrected, the determination information further includes at least one proposal of adding a class to be classified, integrating classes, and subdividing classes; the label classification unit uses a multi-label classification model that outputs an independent likelihood for each class as the trained classification model, The acquisition unit acquires label noise data, the control unit generates the determination information from the label noise data. Here, when a dataset is classified using a multi-class classification model trained using a dataset in which teacher labels are assigned to the data, if data classified into the same class as the teacher label is defined as second classified data and data classified into a class different from the teacher label is defined as first classified data, the label noise data is the first classified data, in this information processing system.

11. An acquisition unit that acquires data; a control unit that generates determination information for label definition from the acquired data; an output unit that outputs determination information of the label definition; The control unit includes a label classification unit that inputs the data into a trained classification model and classifies the data using the trained classification model; and a label definition determination unit that performs classification using the trained classification model and generates determination information that determines whether or not a label definition needs to be corrected based on the classification result; When the label definition determination unit determines that the label definition needs to be corrected, the determination information further includes at least one proposal of adding a class to be classified, integrating classes, and subdividing classes; the label classification unit uses a multi-label classification model that outputs an independent likelihood for each class as the trained classification model, The label definition determination unit sets a determination threshold based on a likelihood for each class into which a plurality of data are classified, and the label definition determination unit determines whether or not a label definition needs to be corrected using the set determination threshold.

Images