Label accuracy improvement device, label accuracy improvement method, and program
The label accuracy improvement device and method address label errors in machine learning by refining models and labels through iterative processes, enhancing accuracy by reducing errors.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- KK TOSHIBA
- Filing Date
- 2022-11-29
- Publication Date
- 2026-06-22
AI Technical Summary
In machine learning, the assignment of labels by humans often results in errors due to the large volume of data, leading to incorrect answers.
A label accuracy improvement device and method that includes a control unit performing learning, determination, and label update processes to refine mathematical models and labels using labeled and unlabeled data, with conditions for updating based on second label errors and inference scores.
Improves the probability that labels indicate correct answers by iteratively updating mathematical models and labels, reducing errors and enhancing accuracy.
Smart Images
Figure 0007877185000001 
Figure 0007877185000002 
Figure 0007877185000003
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to a label accuracy improvement device, a label accuracy improvement method, and a program.
Background Art
[0002] In recent years, the technology of machine learning has been developing. In machine learning, there are cases where learning of a mathematical model is performed using data to which a label, which is a value indicating a correct answer, is assigned. In such cases, the label is often assigned by a person. However, since the amount of data used for learning is large, when a person assigns a label, there are cases where mistakes are made. Specifically, there are cases where the assigned label does not actually indicate the correct answer.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] The problem to be solved by the present invention is to provide a label accuracy improvement device, a label accuracy improvement method, and a program that can improve the probability that a label indicates a correct answer.
Means for Solving the Problems
[0005] The label accuracy improvement device of the embodiment has a control unit. The control unit performs unit processing. Unit processing includes a learning process, a determination process, and a label update process. The learning process is learning using labeled data as learning data, and the learning target is a mathematical model that estimates the labels to be assigned to unlabeled data based on unlabeled data which is the data to which labels are assigned from the learning data, and obtains the likelihood of the estimation result along with the estimation result, and the learning process updates the mathematical model to reduce the first label error, which is the difference between the estimated label of the learning target and the labels included in the learning data. The determination process determines whether a predetermined condition is met, which is a condition relating to a second label error, which is the difference between the result estimated by the mathematical model that has been trained based on the unlabeled data from the learning data and the labels included in the learning data, and an inference score, which is a value that indicates that the larger the value, the greater the likelihood obtained by the mathematical model. If the determination process determines that the determination condition is met, the label update process updates the labels included in the learning data based on the unlabeled data according to a predetermined rule. The determination condition includes the condition that the second label error is greater than a predetermined size, and the inference score is greater than a predetermined size. [Brief explanation of the drawing]
[0006] [Figure 1] An explanatory diagram illustrating the outline of the label accuracy improvement device according to the embodiment. [Figure 2] An explanatory diagram illustrating an example of the estimation results of the label estimation model in the embodiment. [Figure 3] A diagram showing an example of the hardware configuration of a label accuracy improvement device in an embodiment. [Figure 4] A flowchart showing an example of the processing flow performed by the label accuracy improvement device in the embodiment. [Modes for carrying out the invention]
[0007] The label accuracy improvement apparatus, label accuracy improvement method, and program of the embodiment will be described below with reference to the drawings.
[0008] Figure 1 is an explanatory diagram illustrating the outline of the label accuracy improvement device 1 according to an embodiment. The label accuracy improvement device 1 includes a control unit 11 which comprises a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program.
[0009] The control unit 11 executes a repetitive process. The repetitive process is a process that executes a unit process until a predetermined termination condition related to the repetitive process (hereinafter referred to as the "repetition termination condition") is met. The unit process includes a learning process and a data update process.
[0010] The training process is the process of updating the mathematical model being trained using training data. The training data used in the training process may consist of one or more sets of data. Specifically, the training data is labeled data.
[0011] The object of learning in the learning process is the label estimation model. The label estimation model is a mathematical model that estimates the labels to be assigned to the unlabeled data (hereinafter referred to as "unlabeled data") from the training data, and obtains the likelihood (probability) of that estimation along with the estimation result. In the learning process, the label estimation model is updated to reduce the difference between the labels estimated by the label estimation model and the labels included in the training data (hereinafter referred to as "first label error").
[0012] Likelihood may be defined to represent the probability that the estimated label is true, or it may be defined to represent the probability that the estimated label is not true. Alternatively, likelihood may be defined to represent both the probability that the estimated label is true and the probability that it is not true.
[0013] Learning in each unit process is performed until a predetermined condition for the termination of the learning process (hereinafter referred to as the "learning termination condition") is met. The learning termination condition may be, for example, that the change in the label estimation model due to updates is less than a predetermined change. The learning termination condition may also be, for example, that updates have been performed a predetermined number of times.
[0014] In the field of machine learning, the term "trained" is used. Using this term, a label estimation model at the point when the training termination condition is met is considered a trained label estimation model. However, it is only trained in the unit of processing at the time the training termination condition is determined. Therefore, a trained label estimation model can be further updated in subsequent unit processing.
[0015] The data update process includes a judgment process and a label update process. The judgment process determines whether or not predetermined conditions relating to the second label error and the inference score are met. The second label error is the difference between the label estimated by the trained label estimation model based on the unlabeled data in the training data and the label included in the training data.
[0016] The inference score is a value that indicates the magnitude of the likelihood obtained by the label estimation model; the larger the score, the greater the likelihood. For example, if the domain of likelihood is defined as -100 to +100, and the absolute value of a positive value indicates the probability that the estimation result is correct, and the absolute value of a negative value indicates the probability that the estimation result is incorrect, then the estimation score may be, for example, the absolute value of the likelihood. For example, if the domain of likelihood is defined as 0 to 100, then the estimation score may be, for example, the likelihood itself.
[0017] The criteria for judgment specifically include the condition that the second label error is greater than a predetermined size, and the condition that the inference score is greater than a predetermined size.
[0018] The determination condition may further include a condition that, for example, when there are a plurality of learning data used in the learning process, among the second label errors obtained for each learning data, it is within the Nth (N is a predetermined positive integer) from the larger one in size.
[0019] FIG. 2 is an explanatory diagram for explaining an example of the types of labels estimated by the label estimation model in the embodiment. In the example of FIG. 2, it is a case where a threshold value is predetermined for the likelihood, and the threshold value is expressed as OK score = 0. In FIG. 2, the OK score indicates the absolute value of the difference between the estimation score and the threshold value for an estimation score larger than the threshold value. In FIG. 2, the NG score indicates the absolute value of the difference between the estimation score and the threshold value for an estimation score below the threshold value. In FIG. 2, the OK label and the NG label indicate two types of labels used in learning.
[0020] Note that the description of FIG. 2 is an explanation by taking two-class classification as an example. The OK label is, for example, the label of class 1, and the NG label is the label of class 2. In learning, one of these labels is attached to each data, and they are used as labeled data. In the case of three-class classification, the classes are, for example, three classes: class 1 label, class 2 label, and class 3 label. Thus, the OK label and the NG label are terms specialized for two-class classification and are used for the sake of simplicity of explanation.
[0021] In FIG. 2, "the OK score is high but the label is NG" means that although the estimation score is larger than the threshold value, the second label error is larger than a predetermined size. In FIG. 2, "the NG score is high but the label is OK" means that although the estimation score is below the threshold value, the second label error is smaller than a predetermined size.
[0022] Returning to the description of FIG. 1. The label update process is a process of updating the label included in the learning data according to the label update rule based on the unlabeled data when it is determined by the determination process that the determination condition is satisfied.
[0023] The label update rule is a predetermined rule regarding label updates. For example, the label update rule is a rule for updating the labels included in the learning data to classification result labels. The classification result label is a label indicating the result of classifying the non-label data included in the learning data by a pre-trained classifier that classifies the classification target with a predetermined accuracy or higher.
[0024] In addition, when there are multiple pieces of learning data used in the learning process, the learning data that satisfies the determination condition is not necessarily all of the learning data used in the learning process. In such a case, the label update process may be executed only for the learning data that satisfies the determination condition, or may be executed for all the learning data regardless of whether the determination condition is satisfied.
[0025] As described above, the unit process includes a process for updating the mathematical model to be learned and a process for updating the labels included in the learning data used for learning. Therefore, by executing the unit process, the mathematical model to be learned and the labels included in the learning data are updated.
[0026] As described above, the repetition process is a process of repeating the unit process until the repetition end condition is satisfied. However, more specifically, in the unit processes after the second time in the repetition process, the updated learning data and mathematical model by the previous unit process are used instead of the learning data and mathematical model used in the previous unit process.
[0027] That is, in the next unit process after the execution of the i-th unit process, the learning data and mathematical model updated by the i-th unit process are used instead of the learning data and mathematical model before being updated by the i-th unit process. Here, i is an integer of 1 or more.
[0028] In addition, the repetition end condition is, for example, a condition that the unit process has been updated a predetermined number of times. The repetition end condition may be, for example, a condition that the second label error is smaller than a predetermined difference for all the learning data used in the learning process.
[0029] Figure 3 shows an example of the hardware configuration of the label accuracy improvement device 1 in the embodiment. As described above, the label accuracy improvement device 1 includes a control unit 11 which has a processor 91 such as a CPU and a memory 92 connected by a bus, and executes a program. The label accuracy improvement device 1 functions as a device comprising the control unit 11, input unit 12, communication unit 13, storage unit 14 and output unit 15 by executing the program.
[0030] More specifically, the processor 91 reads the program stored in the storage unit 14 and stores the read program in the memory 92. By executing the program stored in the memory 92, the processor 91 functions as a device comprising a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.
[0031] The control unit 11 controls the operation of various functional units of the label accuracy improvement device 1. For example, the control unit 11 performs repetitive processing.
[0032] The input unit 12 includes input devices such as a mouse, keyboard, and touch panel. The input unit 12 may also be configured as an interface for connecting these input devices to the label accuracy improvement device 1. The input unit 12 receives various types of information input to the label accuracy improvement device 1. For example, the input unit 12 receives instructions from the user to start a repetitive process. For example, the input unit 12 receives training data.
[0033] The communication unit 13 includes a communication interface for connecting the label accuracy improvement device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless connection. The external device is, for example, a device that transmits training data. The communication unit 13 acquires training data by communicating with the device that transmits training data.
[0034] The storage unit 14 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 14 stores various information related to the label accuracy improvement device 1. The storage unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The storage unit 14 stores various information generated by, for example, repetitive processing.
[0035] The output unit 15 outputs various types of information. The output unit 15 is comprised of a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may also be configured as an interface for connecting these display devices to the label accuracy improvement device 1. The output unit 15 outputs information input to, for example, the input unit 12 or the communication unit 13.
[0036] Figure 4 is a flowchart showing an example of the processing flow performed by the label accuracy improvement device 1 in the embodiment. The following explanation will use the case where the number of training data used in the learning process is M (where M is an integer of 1 or more) as an example.
[0037] The control unit 11 acquires M units of training data to be used in the training process (step S101). Next, the control unit 11 performs training using each of the acquired training data to obtain a trained label estimation model (step S102). In other words, the control unit 11 performs training and obtains a label estimation model.
[0038] Next, the control unit 11 executes the trained label estimation model on the unlabeled data within each training data (step S103). By executing step S103, the trained label estimation model estimates the labels to be assigned to the unlabeled data within each training data, and obtains the likelihood of the estimation results.
[0039] Next, the control unit 11 obtains the difference between the label estimated by the label estimation model trained in step S103 and the label included in the training data in step S103, for each training data (step S104). The training data in step S103 refers to the training data that includes the unlabeled data to which the trained label estimation model in step S103 is applied. The difference obtained in step S104 is the second label error.
[0040] Next, the control unit 11 performs data update processing (step S105). As described above, the data update processing includes a judgment process and a label update process. More specifically, in the data update processing, a judgment process is performed using the second label error obtained in step S104 and the likelihood obtained in step S103. That is, the control unit 11 uses the second label error obtained in step S104 and the likelihood obtained in step S103 to determine whether the judgment condition is met for each training data.
[0041] In the data update process, if the judgment condition is met, the label update process is executed. On the other hand, if the judgment condition is not met, the control unit 11 performs an action according to a predetermined rule during the data update process. If the predetermined rule is that the label update process is performed regardless of whether the judgment condition is met or not, the control unit 11 performs the label update process. If the predetermined rule is that the label update process is not performed if the judgment condition is not met, the control unit 11 does not perform the label update process.
[0042] The processing in steps S102 to S105 is an example of a unit process.
[0043] Following the data update process, the control unit 11 determines whether the repetition termination condition has been met (step S106). If the repetition termination condition is met (step S106: YES), the process ends.
[0044] On the other hand, if the iteration termination condition is not met (step S106: NO), the control unit 11 further executes unit processing using the training data and mathematical model updated by the unit processing in steps S102 to S105. That is, if the iteration termination condition is not met, the control unit 11 acquires each training data after the data update processing as training data to be used in the next training processing (step S107). Next, it returns to the processing in step S102.
[0045] The label accuracy improvement device 1 in this configured embodiment includes a control unit 11 that performs learning processing and data update processing. Specifically, the label accuracy improvement device 1 updates the label estimation model that estimates labels through the learning process, and updates the training data used to train the label estimation model by updating the labels through the data update process. Then, the label accuracy improvement device 1 further trains the label estimation model using training data that contains labels with even more accurate content. Naturally, as a result, the accuracy of the label estimation model's estimation improves.
[0046] Then, using the results of the improved label estimation model, the label accuracy improvement device 1 further updates the labels. In this way, the label accuracy improvement device 1 updates the initial labels in the labeled data to more accurate labels. Therefore, the label accuracy improvement device 1 can improve the probability that the labels are correct.
[0047] (modified version) Each training data acquired by the control unit 11 in step S101 is data to which a label estimated using a predetermined classifier has been assigned to each data point in a set of unlabeled data. In this case, since the training data is prepared using a classifier, the burden on the person required to prepare the training data is reduced compared to when a classifier is not used. Each training data acquired by the control unit 11 in step S101 is an example of training data used in the training of the first unit processing.
[0048] In step S101, each training data acquired by the control unit 11 is data to which the affiliation information is assigned as a label to each data point in the classification target set, which is a collection of unlabeled data. The affiliation information indicates which of the classifications resulting from a predetermined clustering for each classification target set the data belongs to.
[0049] Since clustering is a technique performed by the device, in such cases, training data is prepared using a classifier. Therefore, the human burden required to prepare the training data is reduced compared to when a classifier is not used. Note that each training data acquired by the control unit 11 in step S101 is an example of training data used in training in the first unit processing. Note that the predetermined clustering method may be, for example, the k-means method or the MeanShift method.
[0050] In the embodiments described above, the control unit 11 is assumed to be a software function unit, but it may also be a hardware function unit such as an LSI.
[0051] According to at least one embodiment described above, by having a control unit that performs learning processing and data update processing, the probability that a label indicates the correct answer can be improved.
[0052] The label accuracy improvement device 1 may be implemented using multiple information processing devices that are connected to each other via a network. In this case, the control unit 11 may be distributed and implemented across the multiple information processing devices.
[0053] All or part of the functions of the label accuracy improvement device 1 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.
[0054] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]
[0055] 1...Label accuracy improvement device, 11...Control unit, 12...Input unit, 13...Communication unit, 14...Storage unit, 15...Output unit, 91...Processor, 92...Memory
Claims
1. A learning process that performs learning using labeled data as training data, where the training target is a mathematical model that estimates the labels to be assigned to unlabeled data from the training data, which are the data to which labels are assigned, and obtains the likelihood of the estimation results along with the estimation results, and updates the mathematical model to reduce the first label error, which is the difference between the estimated labels of the training target and the labels included in the training data, A determination process that determines whether a predetermined condition is met regarding the second label error, which is the difference between the result estimated by the mathematical model trained on the unlabeled data among the training data and the labels included in the training data, and the inference score, which is a value that indicates that the larger the score, the greater the likelihood obtained by the mathematical model. If the determination process determines that the determination conditions are met, a label update process is performed to update the labels included in the training data according to a predetermined rule based on the unlabeled data. A control unit that performs unit processing including Equipped with, The determination condition includes the condition that the second label error is greater than a predetermined size, and the inference score is greater than a predetermined size, respectively. Label accuracy improvement device.
2. The aforementioned predetermined rule is to update the labels included in the training data to labels that indicate the results of classifying the unlabeled data included in the training data by a pre-trained classifier that classifies the objects to be classified with a predetermined accuracy or higher. The label accuracy improvement device according to claim 1.
3. Each training data used in the training process in the first unit processing is data to which a label estimated using a predetermined classifier has been assigned to each data point in a set of unlabeled data. The label accuracy improvement device according to claim 1.
4. Each training data used in the training process in the first unit processing is data to which each data in a set of unlabeled data is labeled with information indicating which of the classifications resulting from a predetermined clustering of the set each data belongs to. The label accuracy improvement device according to claim 1.
5. The control unit, after executing the unit processing, further executes the unit processing using the updated learning data and mathematical model after the unit processing, instead of the learning data and mathematical model before the unit processing was performed. The label accuracy improvement device according to claim 1.
6. A learning process that performs learning using labeled data as training data, where the training target is a mathematical model that estimates the labels to be assigned to unlabeled data from the training data, which are the data to which labels are assigned, and obtains the likelihood of the estimation results along with the estimation results, and updates the mathematical model to reduce the first label error, which is the difference between the estimated labels of the training target and the labels included in the training data, A determination process that determines whether a predetermined condition is met regarding the second label error, which is the difference between the result estimated by the mathematical model trained on the unlabeled data among the training data and the labels included in the training data, and the inference score, which is a value that indicates that the larger the score, the greater the likelihood obtained by the mathematical model. If the determination process determines that the determination conditions are met, a label update process is performed to update the labels included in the training data according to a predetermined rule based on the unlabeled data. A control step that performs unit processing including, It has, The determination condition includes the condition that the second label error is greater than a predetermined size, and the inference score is greater than a predetermined size, respectively. Methods for improving label accuracy.
7. A program for causing a computer to function as a label accuracy improvement device according to any one of claims 1 to 5.