Machine learning device, method, program, and system
The machine learning device enhances accuracy by classifying and annotating data in two stages, reducing manual effort and improving training data quality.
Patent Information
- Application Number
- JP2024154790
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-11-19
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Conventional supervised machine learning requires significant human effort for annotating training data, leading to low accuracy and inefficiency in creating appropriate training data.
A machine learning device that classifies data into classifiable and unclassifiable categories using a first learning model, adds annotations to classifiable data, classifies unclassified data using a second learning model, acquires and adds annotations to unclassified data, and updates both learning models based on classification results and acquired annotations.
Improves the accuracy of machine learning by reducing the need for manual annotation and enhancing the precision of training data creation.
Smart Images

Figure 2024164320000001_ABST
Abstract
Description
[Technical field]
[0001] This application claims priority from basic application No. 2019-208576, filed on November 19, 2019, with the Japan Patent Office, the entire contents of which are incorporated herein by reference.
[0002] The present invention relates to a machine learning device, a method, a program, and a system. [Background technology]
[0003] Traditionally, supervised learning in machine learning uses a large amount of training data. For example, a person known as a data scientist creates training data by adding annotations (also called labels) to data (also called annotation), and machine learning is performed using the training data to generate a learning model. If the training data is inappropriate, the accuracy of the learning model will be low, so it is desirable to create appropriate training data.
[0004] Classification, a machine learning technique, classifies input data into two categories (also called binary classification, two-value classification, or two-class classification) or into three or more categories (also called multi-class classification). Data for creating training data for such classification includes data for which it is possible to determine with high certainty which annotations should be added, and data for which it is possible to determine with only low certainty which annotations should be added.
[0005] For example, in Patent Document 1, a teacher data creation device acquires one of positive evaluation, negative evaluation, and ignore evaluation as the label evaluation for each label to create teacher data. Then, a learning device adjusts the weight coefficient of the intermediate layer so that the recognition score of the positive evaluation or negative evaluation label approaches the correct answer score of the positive evaluation or negative evaluation, and prevents the recognition score of the ignore evaluation label from affecting the adjustment of the weight coefficient of the intermediate layer (paragraph
[0010] of Patent Document 1). [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2018-18537 A Summary of the Invention [Problem to be solved by the invention]
[0007] However, in Patent Document 1, a human annotator must assign an evaluation to every label, and creating a large amount of training data requires a lot of manpower, making it difficult to improve the accuracy of machine learning.
[0008] Therefore, an object of one embodiment of the present invention is to improve the accuracy of machine learning. [Means for solving the problem]
[0009] The present invention has the following configuration. [1] A first classification unit that classifies data into classifiable data and unclassified data based on a first learning model; a first annotation unit for annotating the classifiable data; a second classification unit that classifies the unclassified data based on a second learning model; an annotation acquisition unit that acquires an annotation to be added to the unknown classification data; a second annotation unit that annotation-adds the unclassified data; a second learning model update unit that updates the second learning model based on the classification result by the second classification unit and the unclassified data to which the annotations are added based on the annotations acquired by the annotation acquisition unit; A machine learning device equipped with the above. [2] The machine learning device described in [1], further comprising a first learning model update unit that updates the first learning model based on the unclassified data to which the annotation is added based at least on the classification result by the second classification unit. [3] The machine learning device described in [1] or [2], wherein the annotation is an annotation for classifying the data into two classes. [4] The data is a particle image; The machine learning device according to any one of [1] to [3], wherein the annotation is an annotation indicating that the shape of a particle included in the particle image satisfies a predetermined criterion, or an annotation indicating that the shape of a particle included in the particle image does not satisfy a predetermined criterion. [5] The machine learning device described in any of [1] to [4], wherein the annotation acquisition unit acquires only annotations to be added to unclassified data for which the probability that the image is an OK particle image classified by the second classification unit is less than a predetermined value and the probability that the image is an NG particle image is less than a predetermined value. [6] the data is a whole particle image of an image of an object; The machine learning device according to any one of [1] to [5], wherein the first assigning unit notifies a ratio of OK particle images to all particle images, a ratio of NG particle images to all particle images, and a ratio of the unknown classification data to all particle images. [7] Classifying the data into classifiable data and unclassified data based on a first learning model; annotating the classifiable data; classifying the unclassified data based on a second learning model; obtaining an annotation to be added to the unclassified data; annotating the unclassified data; updating the second learning model based on the unclassified data to which the annotations have been added based on a result of the classification of the unclassified data and an annotation to be added to the unclassified data; The method includes: [8] Computer a first classification unit that classifies data into classifiable data and unclassified data based on a first learning model; a first annotation unit for annotating the classifiable data; a second classification unit that classifies the unclassified data based on a second learning model; an annotation acquisition unit that acquires an annotation to be added to the unknown classification data; a second annotation unit that annotation-adds the unclassified data; a second learning model update unit that updates the second learning model based on the classification result by the second classification unit and the unclassified data to which the annotation is added based on the annotation acquired by the annotation acquisition unit; A program to function as a [9] A system including a machine learning device and a user terminal, a first classification unit that classifies data into classifiable data and unclassified data based on a first learning model; a first annotation unit for annotating the classifiable data; a second classification unit that classifies the unclassified data based on a second learning model; an annotation acquisition unit that acquires an annotation to be added to the unknown classification data; a second annotation unit that annotation-adds the unclassified data; a second learning model update unit that updates the second learning model based on the classification result by the second classification unit and the unclassified data to which the annotations are added based on the annotations acquired by the annotation acquisition unit; A system equipped with Effect of the Invention
[0010] In one embodiment of the present invention, the accuracy of machine learning can be improved. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a diagram showing an overall system configuration including a machine learning device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating a hardware configuration of a machine learning device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing functional blocks of a machine learning device according to an embodiment of the present invention. [Figure 4]4 is an example of an image of an object stored in an object image storage unit according to an embodiment of the present invention. [Diagram 5] 4 is an example of a particle image stored in a particle image storage unit according to an embodiment of the present invention. [Figure 6] FIG. 2 is a data flow diagram showing the flow of data according to one embodiment of the present invention. [Figure 7] 1 is a flowchart showing the flow of a teacher data creation process according to one embodiment of the present invention. [Figure 8] 13 is a flowchart showing the flow of a process for updating a first learning model according to one embodiment of the present invention. [Figure 9] 10 is a flowchart showing the flow of inference processing of data classification results according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.
[0013] In this specification, a case will be described in which image data is used as training data, but the present invention can be applied to cases in which any data, such as text data, audio data, etc., is used.
[0014] In addition, the present specification describes a case where a particle image of an aluminum oxide particle contained in an aluminum oxide powder or the like is used as the teacher data, but the present invention can be applied to a case where a particle image of a particle of any substance is used. In addition, the present specification describes a case where data (particle image) is classified into two (also called binary classification, two-value classification, or two-class classification), but the present invention can be applied to a case where data (particle image) is classified into three or more (also called multi-class classification). In addition, the present specification describes a case where an image obtained by an optical microscope is used as the teacher data, but the present invention can be applied to a case where an image obtained by any device such as an optical microscope or a scanning electron microscope (SEM) is used.
[0015] The following describes an embodiment in which training data is created (<Embodiment 1> and <Embodiment 2> below) and an embodiment in which classification results of input data are inferred (<Embodiment 3> below).
[0016] <Embodiment 1> <System configuration> FIG. 1 is a diagram showing an overall system configuration including a machine learning device 102 according to an embodiment of the present invention. The machine learning device 102 can be used in a system (quality inspection system 100) for performing quality inspection. The quality inspection system 100 can include an optical microscope 101, a machine learning device 102, and a user terminal 103. The machine learning device 102 acquires an image captured by the optical microscope 101 connected to the machine learning device 102. The machine learning device 102 also transmits and receives data to and from the user terminal 103 via an arbitrary network 104. Note that data transmission and reception between the machine learning device 102 and the optical microscope 101 or the user terminal 103 may be performed via a storage medium such as a semiconductor memory described later. Each of these will be described below.
[0017] The optical microscope 101 photographs an object (for example, aluminum oxide particles contained in aluminum oxide powder, etc.). The optical microscope 101 may include an imaging device such as a digital camera and a storage device for storing an image of the object. The optical microscope 101 may send the image of the object obtained by photographing to a machine learning device 102 connected to the optical microscope 101. The microscope included in the optical microscope 101 may be a reflection microscope or a transmission microscope. The optical microscope 101 may include light sources such as an ultra-high pressure mercury lamp, a xenon lamp, LEDs of various colors including the three primary colors, ultraviolet LEDs, and laser light, and the image may be observed using bright-field observation, dark-field observation, phase contrast observation, differential interference observation, polarized light observation, fluorescence observation, or other observation methods.
[0018] The machine learning device 102 is a device for creating teacher data by adding annotations (also called labels) to data (also called annotations). Specifically, the machine learning device 102 classifies data into classifiable data and unclassified data based on a first learning model. The machine learning device 102 also classifies the unclassified data based on a second learning model. The machine learning device 102 also updates the second learning model by reflecting corrections from the user terminal 103. The machine learning device 102 is composed of, for example, one or more computers. The machine learning device 102 will be described in detail later with reference to FIG. 3.
[0019] The user terminal 103 is a terminal used by a person who corrects classification based on the second learning model of the machine learning device 102. Specifically, the user terminal 103 receives unclassified data from the machine learning device 102 and displays the data on the user terminal 103 or on a display device (not shown) connected to the user terminal 103. The user terminal 103 also transmits annotation data input to the user terminal 103 to the machine learning device 102. The user terminal 103 is, for example, a computer such as a personal computer.
[0020] In this specification, the machine learning device 102 and the user terminal 103 are described as separate computers, but the machine learning device 102 and the user terminal 103 may be implemented as a single computer. Furthermore, the machine learning device 102 may have some of the functions of the user terminal 103, and the user terminal 103 may have some of the functions of the machine learning device 102.
[0021] <Hardware configuration of the machine learning device 102> 2 is a diagram showing an example of a hardware configuration of a machine learning device 102 according to an embodiment of the present invention. The machine learning device 102 has a central processing unit (CPU) 1, a read only memory (ROM) 2, and a random access memory (RAM) 3. The CPU 1, the ROM 2, and the RAM 3 form a so-called computer.
[0022] Moreover, the machine learning device 102 may further include a GPU (Graphics Processing Unit) 4, an auxiliary storage device 5, an I / F (Interface) device 6, and a drive device 7. The hardware components of the machine learning device 102 are connected to each other via a bus 8.
[0023] The CPU 1 is a computing device that executes various programs installed in the auxiliary storage device 5 .
[0024] The ROM 2 is a non-volatile memory. The ROM 2 functions as a main storage device that stores various programs, data, etc. required for the CPU 1 to execute various programs installed in the auxiliary storage device 5. Specifically, the ROM 2 functions as a main storage device that stores boot programs such as a BIOS (Basic Input / Output System) and an EFI (Extensible Firmware Interface).
[0025] The RAM 3 is a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 3 functions as a main storage device that provides a working area in which various programs installed in the auxiliary storage device 5 are expanded when the CPU 1 executes them.
[0026] The GPU4 is a computing device specialized for image processing.
[0027] The auxiliary storage device 5 is an auxiliary storage device that stores various programs and information used when the various programs are executed.
[0028] The I / F device 6 is a communication device for communicating with the optical microscope 101 and the user terminal 103 .
[0029] The drive device 7 is a device for setting the storage medium 9. The storage medium 9 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The storage medium 9 may also include semiconductor memories that record information electrically, such as ROM, flash memory, etc.
[0030] The various programs to be installed in the auxiliary storage device 5 are installed, for example, by setting the distributed storage medium 9 in the drive device 7 and reading out the various programs recorded in the storage medium 9 by the drive device 7. Alternatively, the various programs to be installed in the auxiliary storage device 5 may be installed by being downloaded from a network via the I / F device 6.
[0031] <Function blocks of machine learning device 102> FIG. 3 is a diagram showing functional blocks of the machine learning device 102 according to an embodiment of the present invention. The machine learning device 102 includes an object image acquisition unit 301, an object image storage unit 302, a particle image extraction unit 303, a particle image storage unit 304, a first classification unit 305, a first assignment unit 306, a first learning model storage unit 308, a second classification unit 309, a second assignment unit 310, a second learning model update unit 311, a second learning model storage unit 312, an annotation acquisition unit 313, and a classification unknown data storage unit 314. The machine learning device 102 may further include a first learning model update unit 307. In addition, the machine learning device 102 functions as the object image acquisition unit 301, the particle image extraction unit 303, the first classification unit 305, the first assignment unit 306, the second classification unit 309, the second assignment unit 310, the second learning model update unit 311, and the annotation acquisition unit 313 by executing a program. By executing a program, the machine learning device 102 can further function as a first learning model update unit 307. Each of these will be described below.
[0032] The object image acquiring section 301 acquires an image of an object captured by the optical microscope 101 or the like. The object image acquiring section 301 also stores the acquired image of the object in the object image storage section 302.
[0033] The object image storage section 302 stores the image of the object acquired by the object image acquisition section 301. Hereinafter, the image of the object stored in the object image storage section 302 will be described with reference to FIG.
[0034] 4 is an example of an object image 400 stored in the object image storage unit 302 according to an embodiment of the present invention. For example, the object image 400 is an image of aluminum oxide particles contained in aluminum oxide powder or the like. As shown in FIG. 4, the object image 400 includes images of a plurality of particles.
[0035] Returning to the explanation of Fig. 3, the particle image extraction unit 303 extracts images of particles from the image of the object in the object image storage unit 302. In addition, the particle image extraction unit 303 stores the extracted images of particles in the particle image storage unit 304.
[0036] For example, the particle image extraction unit 303 binarizes the image of the object to generate a mask image. Furthermore, the particle image extraction unit 303 extracts (also called crop) an image of a particle that has no other particles around it from the image of the object based on the mask image.
[0037] As described below, in one embodiment of the present invention, an image of a particle (hereinafter also referred to as a particle image) included in an image of an object is used as data for creating training data. That is, training data is created by adding annotations to the particle image. In addition, the following describes a case where data (particle image) is classified into two types, OK particle image and NG particle image (also referred to as binary classification, two-value classification, or two-class classification), but the present invention can also be applied to a case where data (particle image) is classified into three or more types (also referred to as multi-class classification).
[0038] The particle image storage unit 304 stores the particle images extracted by the particle image extraction unit 303. Hereinafter, the particle images stored in the particle image storage unit 304 will be described with reference to FIG.
[0039] 5 is an example of a particle image 500 stored in the particle image storage unit 304 according to an embodiment of the present invention. As will be described in detail later, the particle image 500 includes classifiable data (OK particle image 501, NG particle image 503) to which the first annotation unit 306 can add annotations, and unknown classification data 502 to which the first annotation unit 306 cannot add annotations.
[0040] Here, an OK particle image and an NG particle image will be described. In this specification, a particle image of a particle whose shape included in the particle image meets a predetermined standard (hereinafter also referred to as an OK particle) is called an OK particle image. Also, a particle image of a particle whose shape included in the particle image does not meet a predetermined standard (hereinafter also referred to as an NG particle) is called an NG particle image.
[0041] For example, an OK particle image is an image that includes particles whose circularity is equal to or greater than a threshold value. Here, circularity refers to "the degree of deviation of a circular shape from a geometrically correct circle" as defined in JIS B0621-1984 "Definition and Display of Geometric Deviation". For example, an NG particle image is an image that includes particles whose circularity is lower than a threshold value. Note that the criteria that the particle shape must satisfy are not limited to circularity, and may include cases where the particle shape is an ellipse, etc., and criteria configured with feature quantities such as the particle area, the particle's major axis, minor axis, circle equivalent diameter, Feret's diameter, etc., or the particle's perimeter may be used.
[0042] Returning to the description of FIG. 3, the first classification unit 305 classifies data (e.g., particle images) into classifiable data (e.g., OK particle images, NG particle images) and classification-unknown data. Specifically, the first classification unit 305 uses the first learning model stored in the first learning model storage unit 308 to output the certainty (also called confidence, reliability, or probability) that the particle image is an OK particle image and the certainty that the particle image is an NG particle image. Furthermore, the first classification unit 305 classifies the particle image into an OK particle image if the certainty that the particle image is an OK particle image is equal to or greater than a predetermined value. Furthermore, the first classification unit 305 classifies the particle image into an NG particle image if the certainty that the particle image is an NG particle image is equal to or greater than a predetermined value. Furthermore, the first classification unit 305 classifies a particle image that does not correspond to either an OK particle image or an NG particle image (i.e., the certainty that the particle image is an OK particle image is less than a predetermined value and the certainty that the particle image is an NG particle image is less than a predetermined value) into classification-unknown data.
[0043] The first learning model storage unit 308 stores a first learning model. The first learning model is a model for classifying a particle by machine learning. For example, the first learning model is a model for outputting, when a particle image is input, a probability that the particle image is an OK particle image and a probability that the particle image is an NG particle image.
[0044] The first annotation adding unit 306 adds annotations to the classifiable data (e.g., OK particle images, NG particle images) classified by the first classification unit 305 to create teacher data. Specifically, the first annotation adding unit 306 associates the OK particle images classified by the first classification unit 305 with an annotation (i.e., an annotation indicating that the particles included in the particle image are OK particles) and stores them in the particle image storage unit 304. Also, the first annotation adding unit 306 associates the NG particle images classified by the first classification unit 305 with an annotation (i.e., an annotation indicating that the particles included in the particle image are NG particles) and stores them in the particle image storage unit 304.
[0045] Note that the first annotation adding unit 306 may be configured to associate and store, in the object image storage unit 302, for each object image in the object image storage unit 302, the ratio of OK particle images to all particle images of the object image, the ratio of NG particle images to all particle images, and the ratio of unclassified data to all particle images. Also, the first annotation adding unit 306 may be configured to notify an arbitrary terminal such as the user terminal 103 of the ratio of OK particle images to all particle images of each object image, the ratio of NG particle images to all particle images, and the ratio of unclassified data to all particle images.
[0046] The first learning model update unit 307 will be described in <Embodiment 2>.
[0047] The second classification unit 309 classifies the unclassified data classified by the first classification unit 305.
[0048] The second learning model storage unit 312 stores a second learning model. The second learning model is a model for class classification generated by machine learning.
[0049] Hereinafter, the case of classifying unclassified data into <<OK particle images and NG particle images>> and the case of classifying into <<OK particle images, NG particle images, and unclassified data>> will be described separately.
[0050] <<Classify into OK particle images and NG particle images>> The second classification unit 309 classifies the unclassified data into OK particle images and NG particle images using the second learning model stored in the second learning model storage unit 312.
[0051] When classifying into <<OK particle images and NG particle images>>, the second learning model is a model for outputting whether the unclassified data is an OK particle image or an NG particle image when the unclassified data is input.
[0052] <<Classifying into OK particle images, NG particle images, and unclassified data>> The second classification unit 309 classifies the unclassified data into OK particle images, NG particle images, and unclassified data using the second learning model stored in the second learning model storage unit 312. Specifically, the second classification unit 309 outputs the probability (also referred to as confidence, reliability, or probability) that the unclassified data is an OK particle image and the probability that it is an NG particle image using the second learning model. Also, when the probability that the unclassified data is an OK particle image is equal to or greater than a predetermined value, the second classification unit 309 classifies the unclassified data as an OK particle image. Also, when the probability that the unclassified data is an NG particle image is equal to or greater than a predetermined value, the second classification unit 309 classifies the unclassified data as an NG particle image. Also, the second classification unit 309 classifies unclassified data that does not correspond to either an OK particle image or an NG particle image (that is, the probability that it is an OK particle image is less than a predetermined value and the probability that it is an NG particle image is less than a predetermined value) as unclassified data.
[0053] When classifying into <<OK particle images, NG particle images, and unclassified data>>, the second learning model is a model for outputting the probability that the unclassified data is an OK particle image and the probability that it is an NG particle image when the unclassified data is input.
[0054] The annotation acquisition unit 313 acquires the annotation data to be assigned to the unclassified data. Specifically, the annotation acquisition unit 313 transmits the unclassified data (for example, a particle image) to the user terminal 103. Note that the annotation acquisition unit 313 may be configured to transmit the information on the classification result (for example, whether it is an OK particle image or an NG particle image, the accuracy, etc.) classified by the second classification unit 309 to the user terminal 103 together with the unclassified data (for example, a particle image). Further, the annotation acquisition unit 313 receives the annotation data input to the user terminal 103 from the user terminal 103. The person operating the user terminal 103 visually observes the unclassified data (for example, a particle image) and determines the annotation to be assigned to the unclassified data.
[0055] Note that when classifying the above unclassified data into <<OK particle images, NG particle images, and unclassified data>>, the annotation acquisition unit 313 may be configured to acquire only the annotation data to be assigned to the unclassified data for which the probability of being an OK particle image is less than a predetermined value and the probability of being an NG particle image is less than a predetermined value.
[0056] The second assignment unit 310 assigns an annotation to the unclassified data based on the classification result (for example, whether it is an OK particle image or an NG particle image) classified by the second classification unit 309 and the annotation acquired by the annotation acquisition unit 313. Specifically, when the classification result classified by the second classification unit 309 and the annotation acquired by the annotation acquisition unit 313 are the same, the second assignment unit 310 assigns that annotation to the unclassified data. Further, when the classification result classified by the second classification unit 309 and the annotation acquired by the annotation acquisition unit 313 are different, the second assignment unit 310 assigns the annotation acquired by the annotation acquisition unit 313 to the unclassified data. Also, the second assignment unit 310 associates the unclassified data with the assigned annotation and stores it in the unclassified data storage unit 314.
[0057] In the unclassified data storage unit 314, the unclassified data and the annotation assigned by the second assignment unit 310 are stored in an associated manner.
[0058] The second learning model update unit 311 updates the second learning model in the second learning model storage unit 312 based on the unclassified data annotated by the second annotation unit 310. Specifically, the second learning model update unit 311 updates the second learning model by performing machine learning on the unclassified data annotated by the second annotation unit 310 using it as teacher data.
[0059] In this way, in <Embodiment 1>, since the second learning model for classifying unclassified data is updated, the unclassified data can be classified more accurately. Also, when classifying the above unclassified data into <<OK particle images, NG particle images, and unclassified data>>, the annotation acquisition unit 313 acquires only the annotation data to be assigned to unclassified data where the probability of being an OK particle image is less than a predetermined value and the probability of being an NG particle image is less than a predetermined value. As the second learning model is updated, the learning accuracy of the second learning model improves and the unclassifiable data decreases, so the data for which a person has to visually judge the annotation to be assigned can be reduced.
[0060] <Embodiment 2> Hereinafter, <Embodiment 2> will be described. Note that the differences from <Embodiment 1> will be mainly described.
[0061] In <Embodiment 2>, the machine learning device 102 includes the first learning model update unit 307 of FIG. 3. Hereinafter, the first learning model update unit 307 will be described.
[0062] The first learning model update unit 307 updates the first learning model in the first learning model storage unit 308 based on the unclassified data annotated by the second annotation unit 310. Note that the first learning model update unit 307 can also be configured to update the first learning model when instructed to update the first learning model from an arbitrary terminal such as the user terminal 103.
[0063] Specifically, the first learning model update unit 307 acquires the unknown classification data to which the second attachment unit 310 has added annotations, and adds it to the particle image storage unit 304. The first learning model update unit 307 also updates the first learning model by performing machine learning on the data to which the first attachment unit 306 has added annotations (e.g., OK particle images and NG particle images) and the unknown classification data to which the second attachment unit 310 has added annotations in the particle image storage unit 304, as teacher data.
[0064] <<Automating updates to the first learning model>> The first learning model update unit 307 can be configured to update the first learning model without human intervention. Specifically, the second annotation unit 310 annotations the classification-unknown data based only on the classification-unknown data that the second classification unit 309 judges to have a certainty that the image is an OK particle image or higher, and the classification-unknown data that the second classification unit 309 judges to have a certainty that the image is an NG particle image or higher (i.e., data other than the unclassifiable data) (i.e., the annotation acquired by the annotation acquisition unit 313 is not used). The first learning model update unit 307 updates the first learning model by machine learning the data (e.g., OK particle images and NG particle images) annotated by the first annotation unit 306 in the particle image storage unit 304 and the data annotated by the second annotation unit 310 other than the unclassifiable data as teacher data.
[0065] <<Display the percentage of unclassified data>> As described in <Embodiment 1>, the first assigning unit 306 can be configured to associate and store the ratio of OK particle images to all particle images of the image of the object, the ratio of NG particle images to all particle images, and the ratio of unknown classification data to all particle images for each image of the object in the object image storage unit 302. The first assigning unit 306 can also be configured to notify an arbitrary terminal such as the user terminal 103 of the ratio of OK particle images to all particle images of the image of the object, the ratio of NG particle images to all particle images, and the ratio of unknown classification data to all particle images for each image of the object. Therefore, the fact that the amount of unknown classification data decreases as the first learning model is updated is presented to an arbitrary terminal such as the user terminal 103.
[0066] In this way, in the second embodiment, the first learning model for classifying data (e.g., particle images) is updated, so that the data (e.g., particle images) can be classified more accurately. In addition, as the first learning model is updated, the amount of unclassified data decreases, so that the amount of data that needs to be visually judged by a human to add annotations can be reduced.
[0067] <Embodiment 3> Hereinafter, the third embodiment will be described, focusing mainly on the differences from the first and second embodiments.
[0068] In the third embodiment, the machine learning device 102 infers a classification result of input data. Below, classification using only the first learning model and two-stage classification using the first learning model and the second learning model will be described separately.
[0069] <<Classification using only the first learning model>> The first classification unit 305 acquires input data (e.g., particle images) and classifies the input data into classifiable data (e.g., OK particle images, NG particle images) and classification-unknown data using the first learning model in the first learning model storage unit 308. The first classification unit 305 notifies an arbitrary terminal such as the user terminal 103 of the classification result.
[0070] <<Two-step classification of the first learning model and the second learning model>> The first classification unit 305 acquires input data (e.g., particle images) and classifies the data into classifiable data (e.g., OK particle images, NG particle images) and classification-unknown data using the first learning model in the first learning model storage unit 308. The second classification unit 309 classifies the classification-unknown data (e.g., into OK particle images and NG particle images). The second classification unit 309 notifies an arbitrary terminal such as the user terminal 103 of the classification result. The classification result to be notified may be only the result of the two-stage classification, or the classification results of the first classification unit 305 and the second classification unit 309 may be shown separately.
[0071] <Data flow> Fig. 6 is a data flow diagram showing the flow of data according to an embodiment of the present invention. As shown in Fig. 6, data (e.g., particle images) are classified into classifiable data (e.g., OK particle images, NG particle images) and classification-unknown data by a first learning model. Then, the classification-unknown data is classified (e.g., into OK particle images and NG particle images) by a second learning model. Then, the classification result by the second learning model is manually corrected.
[0072] <Processing method> Each process will be described below with reference to FIGS.
[0073] FIG. 7 is a flowchart showing the flow of a teacher data creation process according to one embodiment of the present invention.
[0074] In step 11 (S11), the object image acquisition unit 301 acquires an image of the object captured by the optical microscope 101 or the like.
[0075] In step 12 (S12), the particle image extraction unit 303 extracts particle images from the image of the object acquired in S11.
[0076] In step 13 (S13), the first classification unit 305 classifies the data (eg, the particle images extracted in S12) into classifiable data (eg, OK particle images, NG particle images) and classification-unknown data.
[0077] In step 14 (S14), the first annotation unit 306 creates training data by adding annotations to the classifiable data (for example, OK particle images, NG particle images) classified in S13.
[0078] In step 15 (S15), the second classification unit 309 classifies the unknown classification data classified in S13.
[0079] In step 16 (S16), the second annotation unit 310 annotations the unknown classification data based on the classification result obtained in S15 (e.g., whether the image is an OK particle image or an NG particle image) and the annotation acquired by the annotation acquisition unit 313.
[0080] In step 17 (S17), the second learning model update unit 311 updates the second learning model in the second learning model storage unit 312 based on the unknown classification data to which annotations were added in S16.
[0081] FIG. 8 is a flowchart showing the flow of the update process of the first learning model according to one embodiment of the present invention.
[0082] In step 21 (S21), the first learning model update unit 307 acquires the unknown classification data to which the second annotation unit 310 has added annotations.
[0083] In step 22 (S22), the first learning model update unit 307 adds the unknown classification data acquired in S21 to the particle image storage unit 304.
[0084] In step 23 (S23), the first learning model update unit 307 updates the first learning model by machine learning the data annotated by the first annotation unit 306 in the particle image storage unit 304 (e.g., OK particle images and NG particle images) and the unknown classification data added in S22 as teacher data.
[0085] FIG. 9 is a flowchart showing the flow of inference processing of data classification results according to one embodiment of the present invention.
[0086] In step 31 (S31), the first classification unit 305 acquires input data (for example, a particle image).
[0087] In step 32 (S32), the first classification unit 305 classifies the data acquired in S31 into classifiable data (e.g., OK particle images, NG particle images) and unknown classification data using the first learning model in the first learning model storage unit 308.
[0088] In step 33 (S33), the first classification unit 305 notifies an arbitrary terminal such as the user terminal 103 of the classification result inferred in S32.
[0089] As described above, the learning model may be classified into two stages, a first learning model and a second learning model.
[0090] <Effects> In one embodiment of the present invention, the machine learning device 102 classifies data into "classifiable data" and "unclassified data", and then classifies the "unclassified data". The classification result of the "unclassified data" is then manually corrected. In this manner, in one embodiment of the present invention, classification is performed in two stages using a first learning model and a second learning model, and the classification using the second learning model is specialized in classifying only "unclassified data", thereby improving the accuracy of classification of data for which it is only possible to determine with low certainty which annotations should be added.
[0091] The present invention is not limited to the configurations shown in the above embodiments, combinations with other elements, etc. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form. [Explanation of symbols]
[0092] 100 Quality Inspection System 101 Optical microscope 102 Machine Learning Device 103 User terminal 104 Network 301 Object image acquisition unit 302 Object image storage unit 303 Particle Image Extraction Unit 304 Particle Image Storage Unit 305 First Classification Department 306 First Grant Department 307 First Learning Model Update Unit 308 First Learning Model Storage Unit 309 Second Classification Department 310 Second grant part 311 Second Learning Model Update Unit 312 Second Learning Model Storage Unit 313 Annotation Acquisition Unit 314 Unclassified Data Storage
Claims
[Claim 1] a first classification unit that classifies data into classifiable data and unclassified data based on a first learning model; a first annotation unit for annotating the classifiable data; a second classification unit that classifies the unclassified data based on a second learning model; an annotation acquisition unit that acquires an annotation to be added to the unknown classification data; a second annotation unit that annotation-adds the unclassified data; a second learning model update unit that updates the second learning model based on the classification result by the second classification unit and the unclassified data to which the annotations are added based on the annotations acquired by the annotation acquisition unit; A machine learning device equipped with the above.
Citation Information
Patent Citations
Threshold change device
JP2018018537A
