Discrimination device and machine learning method

Abstract

The present invention provides a discrimination device and a machine learning method capable of detecting annotation errors. The discrimination device (1) of the present invention comprises: a sub-dataset extraction unit (34) which extracts a sub-learning data set for learning and a sub-verification data set for verification from a plurality of learning data to which labels are attached; a learning unit (110) which performs supervised learning based on the sub-learning data set and generates a learned model for discriminating labels based on the data of the above-mentioned object; a discrimination unit (120) which performs discrimination processing using the learned model for each learning data contained in the sub-verification data set; a verification result recording unit (40) which records the result of the discrimination processing in association with the learning data; and a true or false detection unit (42) which detects learning data that may have a label error attached based on the result of the discrimination processing recorded in association with each learning data.

Description

Technical Field

[0001] The present invention relates to a discriminating device and a machine learning method, and in particular to a discriminating device and a machine learning method for detecting annotation errors. Background Art

[0002] In the past, there have been operations such as performing an appearance inspection of an object such as a product manufactured in a factory by photographing the appearance of the object by a photographing mechanism and sorting the products based on the photographed images, and classifying normal products from defective products with scratches, defects, etc. (for example, Patent Documents 1 to 3, etc.). As a method for performing an appearance inspection of an object based on an image of the appearance of the object photographed, there is a method of machine learning using a machine learning machine using a deep neural network, etc. In the case of adopting such a method, as a pre-stage of learning by an appearance inspection system using images, an operation called annotation is performed. In annotation, an operation called annotation is performed in which a label indicating whether there is a defective position in the image is associated with each image data. In annotation, an operator confirms the images one by one and determines whether there is a defective position of the object in the image by visual observation.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-021756

[0006] Patent Document 2: Japanese Patent Application Publication No. 2005-052926

[0007] Patent Document 3: Japanese Patent Application Publication No. 2013-140090

[0008] Such annotations to images are usually done manually by humans, so when the object has obvious deformation, scratches, etc., it is possible to appropriately add a defective label, but subtle differences such as the position and size of the object's features (for example, the position and size of the holes opened on the product) when compared with qualified products are often overlooked, making it difficult to annotate them in a matching manner. In addition, due to human errors, it may be easy to regard the object as a qualified product even if it is obviously defective, and to regard a qualified product as a defective product.

[0009] Due to such human errors, annotations are not always consistent, making it impossible to learn a learner (learned model) that can accurately judge bad things. As a result, the accuracy of the inspection results of machine learning does not reach a practical level. In order to correct such errors, humans need to reconfirm all annotations through visual observation, but this becomes a very troublesome task. Such problems do not occur specifically in machine learning machines used for visual inspection of images, but the same problem occurs when machine learning is performed using learning data produced by human annotations. Summary of the invention

[0010] In this regard, an object of the present invention is to provide a discrimination device and a machine learning method capable of detecting annotation errors.

[0011] The present invention repeatedly performs discrimination processing based on labels of learning data and verification data extracted from a set of learning data, and performs statistical processing on the discrimination results, thereby detecting annotations with a high possibility of error, and prompting them to operators for correction. This method can improve the accuracy of annotations, thereby enabling machine learning machines to learn more accurately, and using the learned model generated by such learning, thereby improving the accuracy of object discrimination.

[0012] More specifically, annotation errors are detected by the following procedure: Procedure 1) A predetermined number or ratio of learning data is selected from the learning data set. The selection method is preferably a method that is less likely to cause bias, such as random selection.

[0013] Sequence 2) Set the set of learning data selected in Sequence 1 as a sub-learning data set.

[0014] Sequence 3) Set the remaining set of learning data not selected in Sequence 1 as a sub-validation data set.

[0015] Procedure 4) The learner is trained using the sub-training data set in the same manner as normal training.

[0016] Sequence 5) Use the learner generated in sequence 4 to perform discrimination on the sub-validation dataset.

[0017] Sequence 6) The number of times each learning data is judged in sequence 5 is increased by 1. If the judgment result is consistent with the annotation, the number of correct answers for the learning data is increased by 1.

[0018] Sequence 7) The selection method in Sequence 1 is changed, and Sequences 1 to 6 are performed a predetermined number of times.

[0019] Procedure 8) The value obtained by dividing the number of correct answers for each learning data by the number of discriminations is calculated as the probability that the annotation of the learning data is correct.

[0020] Sequence 9) The learning data whose probability calculated in Sequence 8 is lower than a predetermined threshold value are prompted to the user as learning data with the possibility of annotation errors, prompting the user to reconfirm whether the annotation is correct and make corrections if necessary.

[0021] Sequence 10) After correcting the annotation errors, use all the learning datasets for learning.

[0022] Furthermore, one embodiment of the present invention is a discrimination device that detects annotation errors, and the discrimination device comprises: a learning data storage unit that stores a learning data set composed of a plurality of learning data; a data acquisition unit that acquires data of an object to be discriminated and stores the data as learning data in the learning data storage unit; a label adding unit that adds labels to the learning data; a sub-dataset extraction unit that extracts a sub-learning data set for learning and a sub-verification data set for verification from the learning data set stored in the learning data storage unit; a learning unit that performs supervised learning based on the sub-learning data set to generate a learned model that discriminates labels based on the data of the object; a discrimination unit that performs discrimination processing using the learned model on each learning data contained in the sub-verification data set; and a verification result recording unit. A part, which records the result of the discrimination processing of each learning data by the above-mentioned discrimination part in the above-mentioned learning data storage part in association with the learning data; a correct / error detection part, which detects the learning data that has the possibility of the label attached by the above-mentioned label attachment part being wrong based on the result of the discrimination processing recorded in the above-mentioned learning data storage part in association with each learning data, the extraction of the sub-learning data set and the sub-verification data set by the above-mentioned sub-data set extraction part, the generation of the learning completion model by the above-mentioned learning part and the discrimination processing by the above-mentioned discrimination part are performed a predetermined number of times, the correct / error detection part calculates the probability that the label attached by the above-mentioned label attachment part is correct based on the result of the discrimination processing performed a predetermined number of times by the above-mentioned discrimination part, and detects the learning data that has the possibility of the label attached by the above-mentioned label attachment part being wrong based on the calculated probability that the label is correct.

[0023] Another embodiment of the present invention is a machine learning method for detecting annotation errors, wherein the machine learning method executes the following steps: a learning data storage step for storing a learning data set composed of a plurality of learning data; a first step for acquiring data of an object to be identified; a second step for attaching labels to the learning data acquired in the first step; a third step for extracting a sub-learning data set for learning and a sub-verification data set for verification from the set of learning data to which the labels are attached in the second step; a fourth step for performing supervised learning based on the sub-learning data set to generate a learning model for identifying labels based on the data of the object; and a fifth step for detecting the sub-verification data set contained in the sub-verification data set. Each learning data is subjected to discrimination processing using the above-mentioned learned model; in step 6, the result of the discrimination processing relative to each learning data in the above-mentioned step 5 is recorded in association with the learning data; in step 7, based on the result of the discrimination processing recorded in association with each learning data in the above-mentioned step 6, the learning data to which there is a possibility that the attached label is wrong is detected, and the above-mentioned steps 3 to 5 are executed a predetermined number of times, and the above-mentioned step 7 calculates the probability that the label attached to each learning data is correct based on the result of the discrimination processing executed a predetermined number of times in the above-mentioned 5 steps, and based on the calculated probability that the label is correct, the learning data to which there is a possibility that the attached label is wrong is detected.

[0024] The effects of the present invention are as follows.

[0025] According to the present invention, it is possible to automatically detect labels that may be erroneous in the annotation (labeling) of data on an object manually performed by an operator, correct the erroneous labels detected in this way, and thereby generate a completed learning model based on correct learning data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic hardware configuration diagram showing the main parts of a determination device including a machine learning device according to one embodiment.

[0027] Figure 2 This is a simplified functional block diagram of a discrimination device according to one embodiment.

[0028] Figure 3 FIG. 1 is a diagram illustrating a learning data set stored in the learning data storage unit.

[0029] Figure 4 This is a simplified flowchart of the processing executed by the determination device according to one embodiment. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0031] Figure 1 This is a schematic hardware configuration diagram showing the main parts of a determination device including a machine learning device according to Embodiment 1. In this embodiment, an example is described in which a machine learning method for detecting annotation errors is implemented in a determination device used for appearance inspection.

[0032] Figure 1 The hardware configuration diagram of the main part of the identification device of one embodiment of the present invention is shown briefly. The identification device 1 of this embodiment can be installed as a computer such as a personal computer installed in a factory, a unit computer for managing equipment installed in a factory, a host computer, an edge computer, a cloud server, etc. Figure 1 An example is shown in which the determination device 1 is installed as a personal computer installed in a factory.

[0033] The CPU 11 included in the identification device 1 of the present embodiment is a processor that controls the entire identification device 1. The CPU 11 reads the system program stored in the ROM 12 via the bus 20, and controls the entire identification device 1 according to the system program. Temporary calculation data, various data input by the operator via the input device 71, etc. are temporarily stored in the RAM 13.

[0034] The nonvolatile memory 14 is composed of, for example, a memory or SSD backed by a battery (not shown), and the storage state is maintained even when the power of the discrimination device 1 is turned off. The nonvolatile memory 14 stores a setting area for storing setting information of the operation of the discrimination device 1, a program or data input from the input device 71, and a learning data set read via an external storage device (not shown) or a network. The program or various data stored in the nonvolatile memory 14 can also be expanded to the RAM 13 when executed or used. In addition, the ROM 12 is pre-written with system programs including a known analysis program for analyzing the learning data set, a program for controlling communication with the machine learning device 100 described later, and the like.

[0035] The photographing device 4 is, for example, an electronic camera having a photographing element such as a CCD, and is a known light receiving device having a function of detecting a two-dimensional image by photographing on a photographing surface (CCD array surface). The photographing device 4 is, for example, mounted on the hand of a robot not shown in the figure, and the robot moves to a photographing position for photographing an object to be identified, and delivers image data obtained by photographing the object to the CPU 11 via the interface 19. The control of photographing the object by the photographing device 4 can also be performed by executing a program by the identification device 1. In addition, the control of photographing the object by the photographing device 4 can also be performed by control from a robot controller that controls the robot or other devices.

[0036] The display device 70 outputs and displays various data read from the memory, data obtained as a result of executing a program, etc., image data of an object captured by the image capturing device 4, data output from the machine learning device 100 described later, etc. via the interface 17. In addition, the input device 71 composed of a keyboard, a pointing device, etc. receives instructions, data, etc. based on the operation of the operator and delivers them to the CPU 11 via the interface 18.

[0037] The interface 21 is an interface for connecting the discriminator 1 to the machine learning device 100. The machine learning device 100 includes a processor 101 that controls the entire machine learning device 100, a ROM 102 that stores a system program, etc., a RAM 103 that temporarily stores various processes for machine learning, and a non-volatile memory 104 that is used to store learning models, etc. The machine learning device 100 can observe various information (for example, a learning data set, etc.) that can be obtained by the discriminator 1 via the interface 21. In addition, the discriminator 1 obtains the processing results output from the machine learning device 100 via the interface 21.

[0038] Figure 2 This is a schematic functional block diagram of the identification device 1 and the machine learning device 100 according to the first embodiment. Figure 2 The functional modules shown are Figure 1 This is realized by the CPU 11 included in the identification device 1 and the processor 101 of the machine learning device 100 executing the respective system programs to control the operations of the respective parts of the identification device 1 and the machine learning device 100 .

[0039] The discrimination device 1 of this embodiment includes: a data acquisition unit 30, a label attachment unit 32, a sub-dataset extraction unit 34, a verification result recording unit 40, and a correct error detection unit 42. In addition, the machine learning device 100 of the discrimination device 1 includes a learning unit 110 and a discrimination unit 120.

[0040] The data acquisition unit 30 acquires data of the object to be identified from the imaging device 4. The data acquisition unit 30 may also acquire data of the object to be identified via an external storage device or a network (not shown). In the present embodiment, the object to be identified is a product to be inspected for appearance, etc., so the data of the object acquired by the data acquisition unit 30 is image data. At this time, the data acquisition unit 30 may also perform pre-processing to convert the acquired data into a data format to be processed inside the identification device 1. For example, when the data of the object is an image as in the present embodiment, a process such as detecting the position and posture of the object in the image data by an image processing method known by Japanese Patent Application Laid-Open No. 2017-091079 and cutting out a partial image of a predetermined area at the detected position and posture may be performed. The data of the object acquired by the data acquisition unit 30 is stored as learning data in the learning data storage unit 50 which is secured as a storage area in the nonvolatile memory 14. In addition, for each learning data stored in the learning data storage unit 50, the number of identifications / number of correct answers may be stored in association. In this case, the number of determinations and the number of correct answers associated with the learning data are reset to 0 in advance at the stage of being stored in the learning data storage unit 50 by the data acquisition unit 30 .

[0041] The label adding unit 32 adds (annotates) labels to the learning data (data of the object) acquired by the data acquiring unit 30 and stored in the learning data storage unit 50, and stores the added labels in the learning data storage unit 50 in association with the learning data. The label adding unit 32 stores, for example, labels added to each learning data by an operator while observing the learning data displayed on the display device 70 and operating the input device 71 in association with the learning data in the learning data storage unit 50. The label adding unit 32 may also automatically add labels, for example, if the discrimination value calculated based on a mathematical algorithm for automatically discriminating the data of the object is above a predetermined first threshold value, the product becomes a qualified product, and if it is below a predetermined second threshold value, the product becomes a defective product. In addition, the result of automatic addition may also be displayed on the display device 70 for the operator to confirm or correct. The labels added by the label adding unit 32 can be, for example, two types of labels, qualified product / defective product, used when determining whether the object is a qualified product or a defective product during appearance inspection. However, in addition to these, it is also possible to add more than three types of labels (large / medium / small, type A / type B / ..., etc.) depending on the purpose of the determination by the determination device 1.

[0042] The sub-dataset extraction unit 34 extracts a predetermined number or a predetermined ratio of learning data from the plurality of learning data stored in the learning data storage unit 50 as sub-learning data sets, and extracts the other learning data as sub-verification data sets. It is preferable that each sub-dataset extracted by the sub-dataset extraction unit 34 is extracted in a manner that is unlikely to cause bias. For example, learning data as sub-learning data sets may be randomly extracted from the learning data stored in the learning data storage unit 50, or a statistical sample extraction method such as a systematic extraction method may be used. In addition, the sub-dataset extraction unit 34 may preferentially extract learning data with a small number of discriminations as described later as sub-verification data sets.

[0043] The learning unit 110 performs supervised learning using the sub-learning data set extracted by the sub-data set extraction unit 34, and generates (learns) a learned model for inferring a discrimination result (label) from the data of the object. The learning unit 110 of the present embodiment may be configured, for example, to use a neural network as a learning model, set the data of the object in each learning data contained in the sub-learning data set extracted by the sub-data set extraction unit 34 as input data, and perform supervised learning with the label as output data. In the case of such a configuration, as a learning model, a neural network having three layers of an input layer, an intermediate layer, and an output layer may be used, but it may also be configured to adopt a so-called deep learning method using a neural network forming more than three layers, thereby performing more efficient learning and inference. In the learning of the learning unit 110 of the present embodiment, for example, a method (error back propagation method) may be adopted in which the error between the calculated consistency and the attached label is back-propagated to the learning model to update the parameters of the learning model. In addition, the learning unit 110 may be configured to perform machine learning using SVM or other learning models, or may select an appropriate learning model in consideration of the characteristics of the data of the object to be discriminated by the discriminator 1. The learned model generated by the learning unit 110 is stored in the learning model storage unit 130 provided on the non-volatile memory 104, and is used for the inference processing based on the sub-validation data set by the discriminator 120.

[0044] The discriminating unit 120 discriminates (infers the discriminating result) each learning data included in the sub-verification data set extracted by the sub-data set extracting unit 34 using the learned model generated by the learning unit 110. The discriminating unit 120 discriminates the data of the object of the learning data as an input to the learned model stored in the learning model storage unit 130 for each learning data included in the sub-verification data set, and outputs the discriminating result as the result to the verification result recording unit 40.

[0045] The verification result recording unit 40 stores the results of the discrimination in association with the learning data included in the sub-verification data set, i.e., each learning data discriminated by the discrimination unit 120, in the learning data storage unit 50. The verification result recording unit 40 increases the number of discriminations of the learning data discriminated by the discrimination unit 120 by 1, and further increases the number of correct answers of the learning data by 1 when the discrimination result of the discrimination unit 120 is consistent with the label attached to the learning data by the label attaching unit 32.

[0046] The above-mentioned extraction process of the sub-learning data set and the sub-verification data set by the sub-data set extraction unit 34, the generation process of the learning model using the sub-learning data set by the learning unit 110, the discrimination process with respect to the sub-verification data set by the discrimination unit 120, and the recording process of the discrimination result of the discrimination unit 120 by the verification result recording unit 40 are repeated a predetermined number of times. The number of repetitions may also be determined based on the number of learning data stored in the learning data storage unit 50. In addition, the repetitive process may be repeated for all the learning data stored in the learning data storage unit 50 until the number of discriminations recorded by the verification result recording unit 40 becomes greater than the predetermined number.

[0047] Please note that in the number of repetitions, the sub-learning data set and the sub-verification data set extracted by the sub-data set extraction unit 34 become different data sets each time. In this way, the learning of the learning unit 110 and the discrimination of the discrimination unit 120 are performed in various combinations of sub-data sets. Then, as a result, the discrimination process is performed multiple times on each learning data, and the verification result recording unit 40 records the number of discriminations and the number of correct answers for each learning data.

[0048] The correctness-error detection unit 42 calculates the value obtained by dividing the number of correct answers by the number of judgments for each learning data stored in the learning data storage unit 50 as the probability that the label (annotation) attached to the learning data is correct. The correctness-error detection unit 42 may also display on the display device 70 the learning data whose probability of correct labels is lower than a predetermined threshold value, even if there is a possibility that the labels attached to the learning data are incorrect. The operator observes the display, confirms again whether the labels are correct, and corrects the labels as needed by operating the input device 71. For example, Figure 3 As illustrated, the probability that the label is correct calculated by the correct / incorrect detection unit may be stored in the learning data storage unit 50 in association with the learning data.

[0049] In the discrimination device 1 having such a structure, labels attached to the acquired learning data are discriminated as labels that may be incorrect based on the relationship with labels attached to other learning data. The operator can correct the labels based on the discrimination result. Then, the operator reconfirms and uses the set of learning data that is inferred to have been correctly labeled to regenerate the learned model of the learning unit 110. The learned model generated in this way is generated based on the correct annotation, so it can be expected that the learned model can be used to discriminate the object with higher accuracy.

[0050] Figure 4 This is a schematic flowchart of a process for detecting an annotation error by the determination device 1 according to the present embodiment.

[0051] [Step SA01 ] The data acquisition unit 30 acquires data of an object and stores the acquired data in the learning data storage unit 50 as learning data.

[0052] [Step SA02 ] The labeling unit 32 annotates (labels) the data of the object acquired in step SA01 based on the operation of the operator or the like, and stores the annotation in the learned data storage unit 50 in association with the learned data.

[0053] [Step SA03] It is determined whether the data of the object still exists. If the data of the object still exists, the process proceeds to step SA01, and if the data of the object does not exist, the process proceeds to step SA04.

[0054] [Step SA04 ] The sub-dataset extraction unit 34 extracts a sub-learning data set and a sub-verification data set from the set of learning data stored in the learning data storage unit 50 .

[0055] [Step SA05 ] The learning unit 110 performs supervised learning using the sub-learning data set extracted in step SA04 , and generates a learned model.

[0056] [Step SA06 ] The determination unit 120 performs determination processing using the learned model constructed in step SA05 on each piece of learning data included in the sub-validation data set extracted in step SA04 .

[0057] [Step SA07 ] The verification result recording unit 40 records the determination result in step SA06 in the learning data storage unit 50 as the number of determinations / number of correct answers for each learning data in association with the learning data.

[0058] [Step SA08] It is determined whether steps SA04 to SA07 have been repeated a predetermined number of times. If they have been repeated a predetermined number of times, the process proceeds to step SA09, and if they have not been repeated a predetermined number of times, the process proceeds to step SA04.

[0059] [Step SA09] The correctness / error detection unit 42 calculates the probability that a label attached to each piece of learning data is correct based on the number of judgments / number of correct answers of the learning data stored in the learning data storage unit 50, extracts learning data to which there is a possibility that an incorrect label is attached based on the calculated probability that the label is correct, and displays the learning data on the display device 70.

[0060] [Step SA10 ] The label adding unit 32 corrects the label added to the learning data based on the operator's operation.

[0061] As a variation of the discrimination device 1 of the present embodiment, the discrimination device 1 may also be configured to display on the display device 70 a situation where it is difficult to correctly discriminate labels in the given learning data or the learning model used for learning / discrimination, that is, the difficulty of discriminating the learning data set. The judgment of the difficulty can be performed based on the probability that the labels attached to each learning data are correct, calculated by the correct / incorrect detection unit 42. For example, the correct / incorrect detection unit 42 may also perform statistical processing on the probability that the attached labels are correct in the set of learning data stored in the learning data storage unit 50, and judge the situation where it is difficult to correctly discriminate labels in the given learning data or the learning model used for learning / discrimination, that is, the difficulty of discriminating the learning data set, based on the statistical distribution, when the average probability of the correctness of the labels attached to the learning data set is significantly low, when the dispersion value is high, etc.

[0062] As another variation of the discrimination device 1 of this embodiment, the label attached to the object to be discriminated by the discrimination device 1 may be set as a continuous value or a group of continuous values ​​such as coordinate values, angles, components, etc., instead of the classification to which the object belongs (discrete values ​​such as qualified products / defective products). When the label is set as a continuous value or a group of continuous values, the learned model generated by the learning unit 110 outputs a continuous value (numerical value) or a group of continuous values ​​using the data of the object as input, and the discrimination unit 120 outputs a continuous value (numerical value) or a group of continuous values ​​as an inference result.

[0063] At this time, by comparing the distance between the label attached to the learning data and the inferred value of the label output by the discrimination unit 120 with a predetermined threshold, it is possible to determine whether the label attached to the learning data is a correct answer. Alternatively, the probability that the label is correct can be calculated based on the calculated distance. As an example, for each learning data stored in the learning data storage unit 50, the difference cumulative value is stored in association with the number of correct answers, so that whenever the discrimination unit 120 performs discrimination processing on each learning data contained in the sub-verification data set, the verification result recording unit 40 accumulates the absolute value of the difference between the inferred value of the label output as the discrimination result of the discrimination unit 120 and the label attached to the learning data by the label attachment unit 32, or the distance between the inferred value of the label output as the discrimination result of the discrimination unit 120 and the label attached to the learning data by the label attachment unit 32 as a difference cumulative value. Then, the correctness / error detection unit 42 only needs to calculate the probability that the label attached to each learning data stored in the learning data storage unit 50 is correct based on the value obtained by dividing the differential cumulative value by the number of discriminations (for example, based on the distribution of the values ​​obtained by dividing the differential cumulative value of each learning data by the number of discriminations, the probability that the label is correct is calculated by a statistical method of a known probability distribution, etc.).

[0064] As mentioned above, although embodiment of this invention was described, this invention is not limited only to the example of the said embodiment, It can implement in various forms by adding appropriate changes.

[0065] For example, the learning algorithm executed by the machine learning device 100 , the calculation algorithm executed by the machine learning device 100 , the control algorithm executed by the determination device 1 , and the like are not limited to the above-mentioned algorithms, and various algorithms can be adopted.

[0066] In the above-described embodiment, the determination device 1 and the machine learning device 100 are described as devices having different CPUs (processors), but the machine learning device 100 may be implemented by the CPU 11 included in the determination device 1 and a system program stored in the ROM 12 .

[0067] In the above-mentioned embodiment, an example is shown in which image data of an object used in appearance inspection is used as the data of the object, but the error detection method of annotation of the present invention can be applied to other data besides image data, such as waveform data of physical quantities such as current / voltage and pressure obtained from the object, sound data of the sound emitted by the object, etc. Even when arbitrary data is used as the object, it can be appropriately applied when the operator annotates (attaches labels) based on his / her own feelings.

[0068] Description of Reference Numerals

[0069] 1-Discrimination device,

[0070] 4- Camera device,

[0071] 11-CPU,

[0072] 12-ROM,

[0073] 13-RAM,

[0074] 14- Non-volatile memory,

[0075] 17, 18, 19 - interfaces,

[0076] 20-bus,

[0077] 21-Interface,

[0078] 30-Data acquisition department,

[0079] 32-label attachment part,

[0080] 34-Sub-dataset extraction unit,

[0081] 40-Verification result recording department,

[0082] 42-correct error detection unit,

[0083] 70- display device,

[0084] 71- input device,

[0085] 100-machine learning device,

[0086] 101-processor,

[0087] 102-ROM,

[0088] 103-RAM,

[0089] 104 - non-volatile memory,

[0090] 110-Study Department,

[0091] 120-Discrimination Department,

[0092] 130-Learning model storage unit.

Claims

1. A discriminating device for detecting errors in annotations, It is characterized in that The identification device comprises: A learning data storage unit storing a learning data set composed of a plurality of learning data; a data acquisition unit that acquires data of an object to be identified and stores the data as learning data in the learning data storage unit; a label adding unit that adds labels to the learning data; a sub-dataset extracting unit, which extracts a sub-learning data set for learning and a sub-verification data set for verification from the learning data set stored in the learning data storage unit; a learning unit that performs supervised learning based on the sub-learning data set and generates a learned model that discriminates labels based on the data of the object; a discriminating unit that performs a discriminating process using the learned model on each learning data included in the sub-validation data set; a determination result recording unit for recording a result of a determination process performed by the determination unit on each learning data in association with the learning data in the learning data storage unit; as well as a true or false detection unit for detecting the learning data that may contain a label error added by the label adding unit based on the result of the discrimination process recorded in the learning data storage unit in association with each learning data, The extraction of the sub-learning data set and the sub-verification data set by the sub-data set extraction unit, the generation of the learned model by the learning unit, and the determination processing by the determination unit are performed a predetermined number of times, The correctness error detection unit calculates the probability that the label attached by the label attachment unit is correct based on the result of the discrimination processing performed by the discrimination unit a predetermined number of times, and detects the learning data that has the possibility that the label attached by the label attachment unit is incorrect based on the calculated probability that the label is correct, The correctness-error detection unit presents the operator with learning data for which the calculated probability of the label being correct is lower than a predetermined threshold value.

2. The identification device according to claim 1, characterized in that: The difficulty of distinguishing the learning data set is determined based on the statistical distribution of the probability that the labels of the learning data contained in the learning data set are correct.

3. A discrimination device, characterized in that: The identification device comprises: An extraction unit, which extracts a sub-learning dataset for learning and a sub-validation dataset for validation from the dataset based on the labeled data; A discriminating unit, which performs discriminating processing on each data contained in the sub-verification data set based on the sub-learning data set; as well as a detection unit that detects data indicating whether there is a possibility that the attached label is erroneous based on the result of the discrimination process, The extraction process of the extraction unit and the determination process of the determination unit are performed a predetermined number of times, The detection unit presents data indicating that the calculated probability that the label is correct is lower than a predetermined threshold value to the operator.

4. The identification device according to claim 3, characterized in that: The difficulty of distinguishing the data set is determined based on the statistical distribution of the probability that the labels contained in the data set are correct.

5. A machine learning method for detecting errors in annotations, It is characterized in that The machine learning method performs the following steps: A learning data storage step, in which a learning data set consisting of a plurality of learning data is stored; A first step, in which data of an object to be identified is acquired; A second step, in which a label is added to the learning data acquired in the first step; Step 3, in which a sub-learning data set for learning and a sub-validation data set for validation are extracted from the set of learning data to which the labels are added in step 2; Step 4, in which supervised learning based on the sub-learning data set is performed to generate a learned model that discriminates labels based on the data of the object; Step 5, in which a discrimination process using the learned model is performed on each learning data included in the sub-validation data set; Step 6, in which the result of the discrimination process on each learning data in step 5 is recorded in association with the learning data; Step 7, in which, based on the result of the discrimination processing recorded in association with each learning data in step 6, learning data having a possibility of having an attached label error is detected, The third to fifth steps are executed a predetermined number of times. In the step 7, the probability that the label attached to each learning data is correct is calculated based on the result of the discrimination processing performed a predetermined number of times in the step 5, and based on the calculated probability that the label is correct, the learning data that may have an incorrect label attached is detected.

Images