Learning Data Management System

Abstract

To make it possible to selectively detect and correct only an error of ground truth label due to label noise after excluding errors of ground truth label due to concept drift included in existing learning data.SOLUTION: A learning data management system comprises: a test data management unit that manages test data to which a label is to be attached; a classification model selection unit that selects a classification model for selecting a label; a classification unit that calculates a degree of certainty indicating the necessity for correcting a label selected by the classification model; and a correction candidate calculation unit that recommends a correction candidate for the label when the degree of certainty exceeds a predetermined value.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a system for managing learning data used when attaching labels to defects occurring in industrial products and the like.

Background Art

[0002] The task of reading a document described in natural language and classifying the inquiry content described in the document frequently appears in the operations of call centers and customer support. When a large number of documents need to be classified every day, it takes a long time to perform this classification task manually. One way to streamline such work is to build a document classification model through supervised learning, which is a type of machine learning, and automatically solve the classification task.

[0003] Specifically, for example, it is performed as follows. First, learning data consisting of a large number of pairs of target variables and explanatory variables is created. In the classification task, this target variable is also called the correct label. In the above-mentioned document classification work, examples of the correct labels of the learning data include categories indicating the content described in documents such as "requests" and "warranty applications". In addition, examples of the explanatory variables include the text described in documents such as "want the operation method to be made easier to understand" and "it was damaged when opened, so please exchange it", and supplementary information such as the document description time and the describer. Next, in order to make the learning data in a form easy to process mechanically, after encoding the correct label as a scalar value and encoding the explanatory variable as a multi-dimensional feature vector, it is input into a machine learning algorithm such as a support vector machine or a random forest, and learning processing is performed to optimize the parameters of the classification model that outputs the label predicted from the explanatory variable.

[0004] For a classification model obtained through supervised learning to be highly accurate and capable of correct classification, the accuracy of the training data is crucial. For example, suppose there is training data where the correct label for the explanatory variable "I want the operation method to be easy to understand" is incorrectly set to "request," but the correct label is "warranty application." A classification model trained using such erroneous training data will have low classification accuracy; for example, when the explanatory variable "I want the operation method to be easy to understand" is taken as input, it will output the incorrect predicted label "warranty application."

[0005] There are two main reasons why such erroneous training data is created.

[0006] The first cause is label noise. Label noise refers to the phenomenon where incorrect labels are assigned due to a lack of knowledge or misunderstanding on the part of the label assigner who reads the explanatory variables of the training data and assigns the correct labels.

[0007] The second cause is concept drift. Concept drift refers to the phenomenon where, due to a change in the rules (concepts) for assigning correct labels to explanatory variables, a label that was correct at the time the training data was created may be considered incorrect after a certain period of time. For example, suppose that initially, the correct label for an explanatory variable with the value "It was damaged when opened, so please replace it" was "Request," and training data was created accordingly. Later, suppose a new label, "Warranty Request," is added, and a new rule is made known to label assigners that explanatory variables like "It was damaged when opened, so please replace it" should be labeled with "Warranty Request." Then, training data where the explanatory variable "It was damaged when opened, so please replace it" was labeled "Request" before the new rule was established would be considered correct training data according to the rule at the time, but would be considered incorrect according to the new rule.

[0008] Regarding the problem of label noise, Non-Patent Document 1 discloses a method for correcting errors by first estimating pairs of labels that labelers are likely to confuse, and then replacing the correct label with the predicted label for data where the classification model predicted a label different from the correct label with high confidence. Furthermore, Patent Document 1 discloses a method for managing the confidence level of labels in image data and recommending label revision for data with low confidence levels. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2019-101560 [Non-patent literature]

[0010] [Non-Patent Document 1] C. Northcutt,et.al,“Confident Learning:Estimating Uncertainty in Dataset Labels,”J.Artif.Int.Res.,vol.70,pp.1373-1411,May 2021. [Overview of the project] [Problems that the invention aims to solve]

[0011] However, the method disclosed in Non-Patent Document 1 has the problem of correcting the correct labels assigned based on new rules after conceptual drift has occurred to incorrect labels assigned based on old rules, because it uses a classification model that has learned old rules from past training data to determine the presence or absence of label noise. Furthermore, the method disclosed in Patent Document 1 has the problem of not being able to detect label noise in existing training data or training data created by third parties, because it uses information that can only be measured at the time of label assignment, such as the time required for label assignment and whether or not the worker fell asleep during the assignment, as the confidence level of the correct labels.

[0012] This invention has been made in view of the above problems, and aims to provide a training data management system that excludes the influence of conceptual drift contained in existing training data, selectively detects only label noise, and enables label correction. [Means for solving the problem]

[0013] To solve the above problems, the learning data management system according to the present invention comprises: a test data management unit that manages test data to which labels are to be assigned; a classification model selection unit that selects a classification model for selecting labels; a classification unit that calculates a confidence level indicating the need to correct the labels selected by the classification model; and a correction candidate calculation unit that recommends correction candidates for the labels when the confidence level exceeds a predetermined value. [Effects of the Invention]

[0014] According to the present invention, it becomes possible to exclude the effects of conceptual drift contained in existing training data, selectively detect only label noise, and enable label correction. Further features related to the present invention will become apparent from the description herein and the accompanying drawings. Problems, configurations, and effects other than those described above will be revealed by the following description of the embodiments. [Brief explanation of the drawing]

[0015] [Figure 1] This figure shows an example of the hardware configuration of a learning data management server according to one embodiment of the present invention. [Figure 2] A diagram showing an example configuration of a learning data management system according to one embodiment of the present invention. [Figure 3] A figure showing an example of the configuration of learning data information according to one embodiment of the present invention. [Figure 4] A diagram showing an example of the configuration of classification model information according to one embodiment of the present invention. [Figure 5] A figure showing an example of the configuration of test data information according to one embodiment of the present invention. [Figure 6] A diagram showing an example of the configuration of classification result information according to one embodiment of the present invention. [Figure 7] A diagram showing a configuration example of event information according to an embodiment of the present invention. [Figure 8] A diagram showing a configuration example of related event information according to an embodiment of the present invention. [Figure 9] A diagram showing a configuration example of correction candidate information according to an embodiment of the present invention. [Figure 10] A flowchart showing the processing executed by the learning data management server according to an embodiment of the present invention. [Figure 11] A diagram showing a configuration example of a correction recommendation screen according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

[0017] (Hardware Configuration) FIG. 1 is a configuration diagram showing an example of the hardware configuration of the learning data management server 100.

[0018] As shown in FIG. 1, the learning data management server 100 includes a storage device 401, a program 402, an arithmetic device 403, and a communication device 405, and each part is interconnected via a bus.

[0019] The storage device 401 is composed of non-volatile memory elements such as an SSD (Solid State Drive) and a hard disk drive. The storage device 401 stores the program 402 that defines the operation of the arithmetic device 403, and various information 111, 113, 121, 124, 131, 133, 142 used or generated by the arithmetic device 403. The memory 404 is composed of a volatile memory element such as a RAM (Random Access Memory).

[0020] The arithmetic unit 403 consists of a processor such as a CPU (Central Processing Unit). The arithmetic unit 403 reads the program 402 stored in the storage device 401 into memory 404 and executes it, realizing the processing by each functional unit shown in Figure 2. The communication device 405 communicates with external devices such as the display unit 200 via the network 406. Hereafter, the learning data management server 100 will be simply referred to as server 100.

[0021] (System Configuration) Figure 2 shows an example configuration of a learning data management system according to one embodiment of the present invention. The learning data management system shown in Figure 2 includes a learning data management server 100 and a display unit 200. The learning data management server 100 is communicated with the display unit 200 used by the user of the learning data management system.

[0022] The learning data management server 100 includes learning data information 111, classification model information 113, test data information 121, classification result information 124, event information 131, related event information 133, and correction candidate information 142 as information stored in its memory unit. It also includes a learning unit 112, a classification model selection unit 122, a classification unit 123, an event selection unit 132, and a correction candidate calculation unit 141 as functional units that perform calculations using this information.

[0023] The server 100 performs the following processing as a whole. First, the learning unit 112 reads the learning data information 111, learns a classification model based on a predetermined learning method, and saves the resulting classification model to a file. Then, it stores the path to the saved file and the learning period as classification model information 113. At this time, any known method such as random forest or support vector machine may be used as the learning method. The learning period may be calculated as the range of dates on which the learning data used to learn the classification model was assigned correct labels, or as a list of those dates.

[0024] Meanwhile, the event selection unit 132 reads the test data to be judged from the test data information 121, extracts only the events related to the correct label assigned to the test data, and stores the ID and occurrence date of the event in the storage unit as related event information 133. At this time, the event may be any event that may be related to the correspondence between explanatory variables and correct labels based on the knowledge of experts in the field, such as updates to the manual describing the label assignment rules, or, if the explanatory variable is text describing product information, changes in the vendor that manufactures the product's parts or the launch of a new product.

[0025] Furthermore, if no ground truth label related to the event is specified in event information 131, the event may be calculated as unrelated to the ground truth label if multiple training and test data sets with similar explanatory variables around the time of the event are assigned the same ground truth label. In this case, the similarity of the explanatory variables can be calculated using known methods such as cosine similarity or K-means clustering. Moreover, if the event information 131 does not contain information about the time of the event, the time of the event may be estimated by detecting changes in the explanatory variables.

[0026] The classification model selection unit 122 then reads the related event information 133, selects the most recent event occurrence date included in the related event information 133, and selects a classification model from the classification model information 113 that includes only dates newer than the event occurrence date as the learning period.

[0027] The classification unit 123 classifies the test data to be judged using the classification model selected by the classification model selection unit 122, and stores the predicted label and confidence level obtained as the classification result in the storage unit as classification result information 124. At this time, the predicted label is the label output by the classification model. The confidence level is an index indicating the likelihood of the predicted label, and can be calculated using known methods such as the method disclosed in Non-Patent Literature 1 or, in the case of binary classification, the difference in classification probabilities for the two labels. The higher the confidence level, the greater the need to correct the original correct label to the predicted label output by the classification model.

[0028] The correction candidate calculation unit 141 reads the classification result information 124 and stores the explanatory variable, the correct label, the predicted label of the correction candidate, and the name of the person who assigned the correct label as correction candidate information 142 in the storage unit for test data where the confidence level exceeds a predetermined value, and instructs the display unit 200 to display the correction candidate information 142. At this time, the correction may be requested from a different assigner than the aforementioned assigner.

[0029] (Example data structure) Next, using Figures 3 to 9, we will explain examples of the data structure of each piece of information stored in the storage device of the server 100. First, Figure 3 shows an example of the data structure of the training data information 111. The training data information 111 shown in Figure 3 has fields 111a to 111d. Field 111a stores the training data ID, which is identification information for identifying the training data. Field 111b stores the explanatory variables of the training data. Field 111c stores the correct label, which is the target variable assigned to the training data. Field 111d stores the label assignment date on which the correct label was assigned to the training data.

[0030] Figure 4 shows an example of the data structure of classification model information 113. The classification model information 113 shown in Figure 4 has fields 113a to 113c. Field 113a stores the classification model ID, which is identification information for identifying the classification model. Field 113b stores the file path where the classification model is stored. Field 113c stores the training period, which indicates the period of the data used to train the classification model. The training period may be a range of dates on which correct labels were assigned to the training data used to train the classification model, or it may be a list of dates.

[0031] Figure 5 shows an example of the data structure of test data information 121. The test data information 121 shown in Figure 5 has fields 121a to 121e. Field 121a stores the test data ID, which is identification information for identifying the test data. Field 121b stores the explanatory variables of the test data. Field 121c stores the correct answer label, which is the target variable assigned to the test data. Field 121d stores the label assignment date on which the correct answer label was assigned to the test data. Field 121e stores the label assigner, which is identification information for identifying the person who assigned the correct answer label to the test data.

[0032] Figure 6 shows an example of the data structure of classification result information 124. The classification result information 124 shown in Figure 6 has fields 124a to 124c. Field 124a stores the test data ID, which is identification information for identifying the test data. Field 124b stores the predicted label, which is the value of the target variable predicted by the classification model for the test data. Field 124c stores the confidence value calculated from the output of the classification model for the test data.

[0033] Figure 7 shows an example of the data structure of event information 131. The event information 131 shown in Figure 7 has fields 131a to 131d. Field 131a stores the event ID, which is identification information for identifying the event. Field 131b stores the type of the event. Field 131c stores the date the event occurred. Field 131d stores the associated label, which indicates the target variable related to the event.

[0034] Figure 8 shows an example of the data structure of related event information 133. The related event information 133 shown in Figure 8 has fields 133a to 133d. Field 133a stores the test data ID, which is identification information for identifying test data. Field 133b stores the correct label for the test data. Field 133c stores the related event ID, which is identification information for identifying the event related to the correct label for the test data. Field 133d stores the occurrence date of the related event.

[0035] Figure 9 shows an example of the data structure of correction candidate information 142. The correction candidate information 142 shown in Figure 9 has fields 142a to 142e. Field 142a stores the test data ID, which is identification information for identifying the test data of the correction candidate. Field 142b stores the explanatory variables of the test data. Field 142c stores the correct label assigned to the test data. Field 142d stores the correction candidate label that will be corrected to the correct label of the test data. Field 142e stores identification information for identifying the label assigner who assigned the correct label to the test data.

[0036] (flowchart) Figure 10 is a flowchart illustrating an example of the operation of server 100.

[0037] First, the event selection unit 132 of the server 100 reads the test data to be judged from the test data information 121 (step S101). Next, the event selection unit 132 extracts only the events related to the correct labels assigned to the test data and stores the ID and occurrence date of the event as related event information 133 (step S102). At this time, any event that may be related to the correspondence between explanatory variables and correct labels based on the knowledge of experts in the field may be specified, such as an update of the manual describing the label assignment rules, or, if the explanatory variable is text describing product information, a change in the vendor that manufactures the product's parts or the launch of a new product. Furthermore, if no correct label related to the event is specified in the event information 131, the event may be calculated as unrelated to the correct label if the same correct label is assigned to multiple training data and test data with similar explanatory variables before and after the occurrence date of the event. At this time, the similarity of the explanatory variables can be calculated using known methods such as cosine similarity or K-means clustering. Furthermore, if the event information 131 does not contain information about the time of occurrence of the event, the time of occurrence may be estimated by detecting changes in the explanatory variables.

[0038] Next, the classification model selection unit 122 reads the related event information 133, extracts the date of the most recent event included in the related event information 133, and selects a classification model from the classification model information 113 that includes only dates newer than that event date as the learning period (step S103).

[0039] Next, the classification unit 123 classifies the test data to be judged using the classification model selected by the classification model selection unit 122, and stores the predicted label and confidence level obtained as the classification result as classification result information 124 (step S104). This predicted label is the label output by the classification model. The confidence level is an indicator of the likelihood of the predicted label and can be calculated using known methods such as the method disclosed in Non-Patent Literature 1 or, in the case of binary classification, the difference in classification probabilities for the two labels.

[0040] Next, the correction candidate calculation unit 141 reads the classification result information 124 and determines whether the already assigned correct label matches the predicted label. If they match, the process ends (step S105). If the correct label and the predicted label do not match, it determines whether the confidence level exceeds a predetermined value. If it does not, the process ends (step S106). If the confidence level exceeds a predetermined value, the explanatory variable, correct label, predicted label of the correction candidate, and name of the person who assigned the correct label are stored for the test data as correction candidate information 142, and the display unit 200 is instructed to display the correction candidate information 142, i.e., a correction is recommended (step S107). At this time, the correction may be recommended to a label assigner different from the person who assigned the label as listed in the label assigner name.

[0041] (Screen example) Figure 11 shows an example of the correction recommendation screen 301 displayed in step S107 of Figure 10. These correction recommendation screens 301 may be displayed and output on the display unit 200, for example, on a personal computer.

[0042] The correction recommendation screen 301 shown in Figure 11 may be an example of a case where, when recommending a correction to the correct label assigned to test data, the current correct label and the candidate label for correction are specified along with the explanatory variables of the test data to be corrected. The correction recommendation screen 301 shown in Figure 11 may have fields 301a to 301h. Field 301a may display the identification information of the test data to be corrected as "Test Data ID". Field 301b may display the correct label assigned to the test data to be corrected as "Current Label". Field 301c may display "Candidate Label for Correction" as a candidate for correction to the "Current Label" in field 301b. Field 301d may display the explanatory variables of the test data to be corrected as "Explanatory Variables". Field 301e may display events related to the correct label assigned to the test data to be corrected as "Related Events". Field 301f accepts text input of a label selected by the user of the learning data management system related to this issue, based on the candidate labels defined in "Correction Candidate Labels," and may display it as "Corrected Label." Field 301g accepts text input of the reason why the user selected the "Corrected Label," and may display it as "Reason for Correction." The user may select to correct the label attached to the test data to be corrected based on the information displayed on the correction recommendation screen 301. For example, the user may move the computer's pointer to the operation field displayed as "Confirm" in field 301h in Figure 11, and then click on the computer to accept and confirm the information displayed in "Corrected Label" and "Reason for Correction."

[0043] The embodiments of the present invention described above provide the following effects. (1) The learning data management system according to the present invention comprises: a test data management unit for managing test data to which labels are to be assigned; a classification model selection unit for selecting a classification model for selecting labels; a classification unit for calculating a degree of confidence indicating the need to correct the labels selected by the classification model; and a correction candidate calculation unit for recommending correction candidates for the labels when the degree of confidence exceeds a predetermined value.

[0044] The above configuration makes it possible to exclude the effects of conceptual drift contained in existing training data, selectively detect only label noise, and enable label correction.

[0045] (2) The system further includes an event information management unit that manages information about the events that generated the test data, and a classification model selection unit that selects a classification model based on the date and time the event occurred. Since conceptual drift is almost always caused by the occurrence of any event, this configuration makes it possible to more effectively exclude the effects of conceptual drift.

[0046] (3) The event information management unit manages the types of labels that affect each event, and the classification model selection unit selects a classification model that has been retrained with training data after the occurrence of the affected event. As a result, the classification model has been retrained to determine the labels associated with the event that occurred, making it possible to make more accurate determinations.

[0047] (4) The system further includes an event selection unit that, when multiple test data points are assigned the same label before and after the time of an event, determines that the type of event does not affect the label and excludes the event from the feedback target. This makes it possible to reduce the computation source.

[0048] (5) The event selection unit estimates the time of event occurrence based on the change points of the feature quantities. This makes it possible to select a classification method based on the estimated time of event occurrence, even if the time of event occurrence is not recorded in the event information beforehand.

[0049] (6) The correction candidate calculation unit recommends the correction to a person different from the person who originally assigned the label. This allows for objective evaluation by the worker who initially assigned the label, and is expected to improve the accuracy of the correction.

[0050] The technical scope of the present invention is not limited to the embodiments described above, and various modifications are included without departing from the main features of the invention. Therefore, the above-described embodiments are merely illustrative and should not be interpreted restrictively. Furthermore, it is possible to add, delete, or replace some of the components of each embodiment with other components, all of which remain within the scope of the present invention. [Explanation of Symbols]

[0051] 100: Learning data management server (learning data management system), 112: Learning unit, 122: Classification model selection unit, 123: Classification unit, 132: Event selection unit, 141: Correction candidate calculation unit

Claims

1. The test data management unit manages the test data to which labels are assigned, A classification model selection unit for selecting a classification model for selecting the aforementioned label, A classification unit that calculates the degree of confidence indicating the need to correct the label selected by the classification model, When the confidence level exceeds a predetermined value, a correction candidate calculation unit recommends correction candidates for the label, The event information management unit manages information related to the events that generated the aforementioned test data, Equipped with, The classification model selection unit selects the classification model based on the date and time the event occurred. A learning data management system characterized by the following features.

2. A learning data management system according to claim 1, The event information management unit manages the types of labels that affect each event for each type of event, The classification model selection unit selects a classification model that has been retrained using training data after the occurrence of the influential event. A learning data management system characterized by the following features.

3. A learning data management system according to claim 1, The system further includes an event selection unit that, when multiple test data points are assigned the same label around the time of an event, determines that the type of event does not affect the label and excludes the event from the feedback target. A learning data management system characterized by the following features.

4. A learning data management system according to claim 3, The event selection unit is a learning data management system that estimates the time of occurrence of the event based on the change points of the feature quantities.

5. A learning data management system according to Claim 1, The correction candidate calculation unit recommends the correction to a person different from the label assigner who assigned the label. A learning data management system characterized by the following features.

Images