Machine learning apparatus, method, program, and system

Abstract

Accuracy of machine learning can be improved. Provided are: a first classification unit that classifies data into classifiable data and unknown category data according to a first learning model; a first addition unit that adds a label to the classifiable data; a second classification unit that classifies the unknown category data according to a second learning model; a label acquisition unit that acquires a label to be added to the unknown category data; a second addition unit that adds a label to the unknown category data; and a second learning model update unit that updates the second learning model according to the unknown category data to which the label is added, the label being added to the unknown category data according to a result of classification by the second classification unit and a label acquired by the label acquisition unit.

Description

Technical Field

[0001] This application claims priority to basic application No. 2019-208576 filed with the Japan Patent Office on November 19, 2019, the contents of which are incorporated herein by reference in their entirety.

[0002] This invention relates to machine learning apparatus, methods, programs, and systems. Background Technology

[0003] In existing technologies, a large amount of training data has been used for supervised learning (also known as teacher-guided learning) in machine learning. For example, training data is created by assigning (also known as adding) labels (annotation or annotation) to the data by people also known as data experts, and machine learning can be performed using this training data to generate a learning model. However, if the training data is inappropriate, the accuracy of the learning model will decrease, so it is necessary to create appropriate training data.

[0004] In class classification, a machine learning method, input data is classified into two categories (also known as binary classification, dichotomous classification, or two-class classification) or three or more categories (also known as multi-class classification). The training data used to generate such class classifications falls into two categories: data that can determine which label to assign with high accuracy and data that can only determine which label to add with low accuracy.

[0005] For example, in Patent Document 1, the training data generation device generates training data by acquiring any one of positive evaluations, negative evaluations, and neglected evaluations as labels for each label. Furthermore, the learning device adjusts the weight coefficients (factors) of the intermediate layer in a manner that makes the recognition score of the positive or negative evaluation label close to the ground truth score of the positive or negative evaluation, and also ensures that the adjustment of the intermediate layer weight factors is unaffected by the recognition score of the neglected evaluation label (paragraph

[0010] of Patent Document 1).

[0006] [Cited Documents]

[0007] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 2018-18537 Summary of the Invention

[0009] <Technical problems to be solved>

[0010] However, in patent document 1, human annotators must perform the operation of assigning evaluations to all labels, and a large amount of manpower is required to generate a large amount of training data. Therefore, it is difficult to improve the accuracy of machine learning.

[0011] Therefore, one embodiment of the present invention aims to improve the accuracy of machine learning.

[0012] <Technical Solution>

[0013] The present invention has the following structure.

[0014] [1] A machine learning device, comprising:

[0015] The first classification department classifies the data into classifiable data and unknown category data according to the first learning model;

[0016] The first adding section adds labels to the classifiable data;

[0017] The second classification department classifies the unknown category data according to the second learning model;

[0018] The tag acquisition unit acquires tags that should be added to the unknown category data.

[0019] The second adding unit adds labels to the unknown category data; and

[0020] The second learning model update unit updates the second learning model based on the unknown category data with the added labels, wherein the labels are added to the unknown category data based on the classification results of the second classification unit and the labels obtained by the label acquisition unit.

[0021] [2] The machine learning device as described in [1] further comprises:

[0022] The first learning model update unit updates the first learning model based on the unknown category data with the added labels, wherein the labels are added to the unknown category data based at least on the results of the second classification partial classification.

[0023] [3] The machine learning apparatus as described in [1] or [2], wherein,

[0024] The labels are used to classify the data into two categories.

[0025] [4] A machine learning apparatus as described in any one of [1] to [3], wherein,

[0026] The data is a particle image.

[0027] The label is a label indicating that the shape of the particles in the particle image meets a predetermined benchmark (also known as a standard), or a label indicating that the shape of the particles in the particle image does not meet the predetermined benchmark.

[0028] [5] A machine learning apparatus as described in any one of [1] to [4], wherein,

[0029] The tag acquisition unit only acquires tags that should be added to the following unknown category data: the accuracy of OK particle images after the second classification part of the unknown category data is less than a predetermined value, and the accuracy of NG particle images after the second classification part of the unknown category data is less than a predetermined value.

[0030] [6] A machine learning apparatus as described in any one of [1] to [5], wherein,

[0031] The data is a full-grained image of the object (also called the object itself or the object object).

[0032] The first adding unit notifies the ratio of OK particle image to the full particle image, the ratio of NG particle image to the full particle image, and the ratio of unknown category data to the full particle image.

[0033] [7] A method comprising:

[0034] The steps to classify data into categorizable data and unknown category data based on the first learning model;

[0035] The steps to add labels to the categorizable data are as follows:

[0036] The steps of classifying the unknown category data according to the second learning model;

[0037] The steps to obtain the labels that should be added to the unknown category data;

[0038] The steps of adding labels to the unknown category data; and

[0039] The step of updating the second learning model based on the unknown category data with the added labels, wherein the labels are added to the unknown category data based on the classification result of the unknown category data and the labels to be added to the unknown category data.

[0040] [8] A program for enabling a computer to function as the following parts:

[0041] The first classification department classifies the data into classifiable data and unknown category data according to the first learning model;

[0042] The first adding section adds labels to the classifiable data;

[0043] The second classification department classifies the unknown category data according to the second learning model;

[0044] The tag acquisition unit acquires tags that should be added to the unknown category data.

[0045] The second adding unit adds labels to the unknown category data; and

[0046] The second learning model update unit updates the second learning model based on the unknown category data with the added labels, wherein the labels are added to the unknown category data based on the classification results of the second classification unit and the labels obtained by the label acquisition unit.

[0047] [9] A system comprising a machine learning device and a user terminal, wherein,

[0048] The machine learning device includes:

[0049] The first classification department classifies the data into classifiable data and unknown category data according to the first learning model;

[0050] The first adding section adds labels to the classifiable data;

[0051] The second classification department classifies the unknown category data according to the second learning model;

[0052] The tag acquisition unit acquires tags that should be added to the unknown category data.

[0053] The second adding unit adds labels to the unknown category data; and

[0054] The second learning model update unit updates the second learning model based on the unknown category data with the added labels, wherein the labels are added to the unknown category data based on the classification results of the second classification unit and the labels obtained by the label acquisition unit.

[0055] <Beneficial Effects>

[0056] One embodiment of the present invention can improve the accuracy of machine learning. Attached Figure Description

[0057] [ Figure 1 [A schematic diagram of the overall system structure of a machine learning device according to one embodiment of the present invention.]

[0058] [ Figure 2 [A schematic diagram of the hardware structure of a machine learning device according to one embodiment of the present invention.]

[0059] [ Figure 3 [A schematic diagram of the functional blocks of a machine learning device according to one embodiment of the present invention.]

[0060] [ Figure 4 An example of an object image stored in the object image storage unit according to one embodiment of the present invention.

[0061] [ Figure 5 An example of a particle image stored in a particle image storage unit according to an embodiment of the present invention.

[0062] [ Figure 6 This is a data flow diagram representing the flow of data according to one embodiment of the present invention.

[0063] [ Figure 7 [ ] represents a flowchart of the training data generation and processing flow according to one embodiment of the present invention.

[0064] [ Figure 8 [ ] This is a flowchart illustrating the update process of the first learning model according to an embodiment of the present invention.

[0065] [ Figure 9 [This is a flowchart illustrating the reasoning process of data classification results according to one embodiment of the present invention.] Detailed Implementation

[0066] The embodiments will now be described in conjunction with the accompanying drawings. It should be noted that in this specification and the accompanying drawings, constituent elements having substantially the same functional configuration are given the same reference numerals, thereby omitting redundant descriptions.

[0067] It should be noted that this specification describes the use of image data as training data, but the present invention can also be applied to the use of any data such as text data and audio data.

[0068] Furthermore, this specification describes the use of particle images containing alumina particles, such as alumina powder, as training data; however, the present invention can also be applied to particle images using particles of any substance. Additionally, this specification describes the classification of data (particle images) into two categories (also known as binary classification, dichotomous classification, or two-class classification); however, the present invention can also be applied to the classification of data (particle images) into three or more categories (also known as multi-class classification). Furthermore, this specification describes the use of images obtained by an optical microscope as training data; however, the present invention can also be applied to the use of images obtained by any device other than an optical microscope, such as a scanning electron microscope (SEM).

[0069] The following describes the implementation methods for generating training data (hereinafter referred to as <Implementation Method 1> and <Implementation Method 2>) and the implementation method for reasoning about the classification results of the input data (hereinafter referred to as <Implementation Method 3>).

[0070] <Implementation Method 1>

[0071] <System Structure>

[0072] Figure 1 This is a schematic diagram of the overall system structure of a machine learning device 102 according to one embodiment of the present invention. The machine learning device 102 can be used in a system for quality inspection (quality inspection system 100). The quality inspection system 100 may include an optical microscope 101, the machine learning device 102, and a user terminal 103. The machine learning device 102 acquires images captured by the optical microscope from the optical microscope 101, which is connected to the machine learning device 102. Furthermore, the machine learning device 102 can transmit and receive data with the user terminal 103 via any network 104. It should be noted that the transmission and reception of data between the machine learning device 102 and the optical microscope 101 or the user terminal 103 can also be performed using a storage medium such as a semiconductor memory, which will be described later. These will be explained separately below.

[0073] The optical microscope 101 photographs an object (e.g., alumina particles contained in alumina powder). The optical microscope 101 may be equipped with a photographing device such as a digital camera and a storage device for saving the photographed object image. Furthermore, the optical microscope 101 can also send the photographed object image to a machine learning device 102 connected to the optical microscope 101. The microscope included in the optical microscope 101 may be a reflection microscope or a transmission microscope. In addition, the optical microscope 101 may also be equipped with a light source such as an ultra-high pressure mercury lamp, a xenon lamp, LEDs of various colors including the three primary colors (also called three primary colors), ultraviolet LEDs, laser beams, etc. Furthermore, as an image observation method, observation methods such as field observation, dark-field observation, phase difference observation, differential interference observation, polarized light observation, and fluorescence observation can be used.

[0074] Machine learning device 102 is a device for creating training data by adding labels (also called annotations) to data. Specifically, machine learning device 102 classifies data into categorizable data and data of unknown categories (i.e., data of unknown categories (also called unclassifiable data)) according to a first learning model. Furthermore, machine learning device 102 classifies data of unknown categories according to a second learning model. Additionally, machine learning device 102 can also reflect modifications from user terminal 103, thereby updating the second learning model. Machine learning device 102 may consist of, for example, one or more computers. (This will be discussed later.) Figure 3 The machine learning device 102 will be described in detail.

[0075] User terminal 103 is a terminal used by a person to modify the classification based on the second learning model of machine learning device 102. Specifically, user terminal 103 receives unknown category data from machine learning device 102 and displays it on user terminal 103 or on a display device (not shown) connected to user terminal 103. In addition, user terminal 103 also sends data of tags input to user terminal 103 to machine learning device 102. User terminal 103 is, for example, a computer such as a personal computer.

[0076] It should be noted that in this specification, the machine learning device 102 and the user terminal 103 are described as separate computers, but the machine learning device 102 and the user terminal 103 may also be implemented (installed) on the same computer. In addition, the machine learning device 102 may have some of the functions of the user terminal 103, or the user terminal 103 may have some of the functions of the machine learning device 102.

[0077] <Hardware Structure of Machine Learning Device 102>

[0078] Figure 2 This is a schematic diagram illustrating an example of the hardware structure of a machine learning device 102 according to one embodiment of the present invention. The machine learning device 102 includes a CPU (Central Processing Unit) 1, a ROM (Read Only Memory) 2, and a RAM (Random Access Memory). The CPU 1, ROM 2, and RAM 3 constitute a so-called computer.

[0079] In addition, the machine learning device 102 may also include a GPU (Graphics Processing Unit) 4, an auxiliary storage device 5, an I / F (Interface) device 6, and a driver device 7. It should be noted that the various hardware components of the machine learning device 102 are interconnected via a bus 8.

[0080] CPU1 is a computing device that executes various programs installed in auxiliary storage device 5.

[0081] ROM2 is a non-volatile memory. ROM2 functions as the main storage device for storing various programs and data required by the CPU1 when executing various programs installed in auxiliary storage device 5. Specifically, ROM2 functions as the main storage device for storing boot programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface).

[0082] RAM3 is a volatile memory such as DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory). RAM3 functions as the main storage device for providing the working area expanded when the various programs installed in the auxiliary storage device 5 are executed by the CPU 1.

[0083] GPU4 is a computing device dedicated to image processing.

[0084] Auxiliary storage device 5 is an auxiliary storage device used to store various programs and the information used when the various programs are executed.

[0085] I / F device 6 is a communication device used to communicate with optical microscope 101 and user terminal 103.

[0086] The drive device 7 is a means for placing (or mounting) the storage medium 9. The storage medium 9 referred to here includes media such as CD-ROMs, floppy disks, and magneto-optical disks, which record information optically, electrically, or magnetically. Furthermore, the storage medium 9 may also include semiconductor memories that record information electrically, such as ROMs and flash memory.

[0087] It should be noted that the various programs installed in the auxiliary storage device 5 can be installed, for example, by inserting the distributed storage medium 9 into the drive device 7, and having the drive device 7 read the various programs recorded in the storage medium 9. Alternatively, the various programs installed in the auxiliary storage device 5 can also be installed by downloading them from the network via the I / F device 6.

[0088] <Functional blocks of machine learning device 102>

[0089] Figure 3This is a schematic diagram of the functional blocks of a machine learning device 102 according to one embodiment of the present invention. The machine learning device 102 includes an object image acquisition unit 301, an object image storage unit 302, a particle image extraction unit 303, a particle image storage unit 304, a first classification unit 305, a first addition unit 306, a first learning model storage unit 308, a second classification unit 309, a second addition unit 310, a second learning model update unit 311, a second learning model storage unit 312, a label acquisition unit 313, and an unknown category data storage unit 314. The machine learning device 102 may also include a first learning model update unit 307. Furthermore, the machine learning device 102 can function as the object image acquisition unit 301, the particle image extraction unit 303, the first classification unit 305, the first addition unit 306, the second classification unit 309, the second addition unit 310, the second learning model update unit 311, and the label acquisition unit 313 by executing a program. Additionally, the machine learning device 102 can also function as the first learning model update unit 307 by executing a program. The following sections will explain each of them.

[0090] The object image acquisition unit 301 acquires object images captured by the optical microscope 101 or the like. Furthermore, the object image acquisition unit 301 can also save the acquired object images to the object image storage unit 302.

[0091] The object image storage unit 302 stores the object image acquired by the object image acquisition unit 301. Hereinafter, in conjunction with... Figure 4 The object image stored in the object image storage unit 302 is described.

[0092] Figure 4 This is an example of an object image 400 stored in the object image storage unit 302 according to one embodiment of the present invention. For example, the object image 400 is an image obtained by photographing alumina particles contained in alumina powder or the like. Figure 4 As shown, object image 400 includes an image of multiple (plural) particles.

[0093] return Figure 3 The particle image extraction unit 303 extracts particle images from the object image in the object image storage unit 302. Furthermore, the particle image extraction unit 303 can also save the extracted particle images to the particle image storage unit 304.

[0094] For example, the particle image extraction unit 303 binarizes the object image to generate a mask image. Furthermore, the particle image extraction unit 303 can also extract (also called cropping or trimming) images of particles that are not surrounded by other particles from the object image based on the mask image.

[0095] It should be noted that, as described below, in one embodiment of the present invention, the image of particles contained in the object image (hereinafter also referred to as particle image) is used as data to create training data. In other words, training data can be generated by adding labels to the particle image. Furthermore, the following description addresses the case where the data (particle image) is classified into two categories: OK particle image and NG particle image (also referred to as binary classification, binary classification, or two-class classification), but the present invention can also be applied to the case where the data (particle image) is classified into three or more categories (also referred to as multi-class classification).

[0096] The particle image storage unit 304 stores the particle image extracted by the particle image extraction unit 303. Below, in conjunction with... Figure 5 The particle images stored in the particle image storage unit 304 will be explained.

[0097] Figure 5 This is an example of a particle image 500 stored in a particle image storage unit 304 according to an embodiment of the present invention. As described below, the particle image 500 includes two types of data: one type is categorizable data (OK particle image 501 and NG particle image 503) that can be tagged by the first addition unit 306, and the other type is unknown category data 502 that cannot be tagged by the first addition unit 306.

[0098] Here, OK particle images and NG particle images are explained. In this specification, a particle image containing particles whose shapes meet a predetermined standard (hereinafter also referred to as OK particles) is called an OK particle image. Furthermore, a particle image containing particles whose shapes do not meet the predetermined standard (hereinafter also referred to as NG particles) is called an NG particle image.

[0099] For example, an OK particle image is an image containing particles whose roundness (also called true roundness) is above a threshold. Here, roundness refers to "the degree to which the cross-section of a circular object approximates a theoretical circle," as defined in JIS B0621-1984 "Definition and Display of Geometric Deviations." For example, an NG particle image is an image containing particles whose roundness is below a threshold. It should be noted that the criteria for particle shape are not limited to roundness, as it can also include cases where the particle shape is elliptical, etc. Therefore, a criterion composed of characteristic quantities such as particle area, major axis, minor axis, equivalent circle diameter, ferret diameter, and perimeter can also be used.

[0100] return Figure 3The first classification unit 305 classifies data (e.g., particle images) into classifiable data (e.g., OK particle images and NG particle images) and data of unknown category. Specifically, the first classification unit 305 uses the first learning model stored in the first learning model storage unit 308 to output the accuracy (also called confidence or reliability) of a particle image as an OK particle image and the accuracy of a particle image as an NG particle image. Furthermore, if the accuracy of a particle image as an OK particle image is greater than or equal to a predetermined value, the first classification unit 305 classifies the particle image as an OK particle image. Similarly, if the accuracy of a particle image as an NG particle image is greater than or equal to a predetermined value, the first classification unit 305 classifies the particle image as an NG particle image. Additionally, the first classification unit 305 classifies particle images that are neither OK nor NG particle images (i.e., the accuracy of the particle image as an OK particle image is less than a predetermined value, and the accuracy of the particle image as an NG particle image is also less than a predetermined value) as data of unknown category.

[0101] The first learning model is stored in the first learning model storage unit 308. The first learning model is a model used to classify categories generated by machine learning. For example, the first learning model is a model used to output the accuracy of a particle image as an OK particle image and the accuracy of a particle image as an NG particle image when the particle image is input.

[0102] The first adding unit 306 adds labels to the classifiable data (e.g., OK particle images and NG particle images) classified by the first classification unit 305, thereby generating training data. Specifically, the first adding unit 306 saves the OK particle images classified by the first classification unit 305 along with labels (i.e., labels indicating that the particles contained in the particle image are OK particles) to the particle image saving unit 304. Furthermore, the first adding unit 306 saves the NG particle images classified by the first classification unit 305 along with labels (i.e., labels indicating that the particles contained in the particle image are NG particles) to the particle image saving unit 304.

[0103] It should be noted that the first addition unit 306 may also be configured to store the ratios of each object image and OK particle image to the full particle image (i.e., the image of all particles) of the object image, the ratio of NG particle image to the full particle image, and the ratio of unknown category data to the full particle image in association with the object image storage unit 302. Furthermore, the first addition unit 306 may also be configured to notify any terminal, such as the user terminal 103, of the ratios of each object image, OK particle image to the full particle image of the object image, NG particle image to the full particle image, and unknown category data to the full particle image.

[0104] The first learning model update unit 307 will be described in <Embodiment 2>.

[0105] The second classification section 309 classifies the unknown category data after classification by the first classification section 305.

[0106] The second learning model is stored in the second learning model storage unit 312. The second learning model is a model used to classify categories generated by machine learning.

[0107] The following sections will explain the cases of classifying unknown category data into OK grain images and NG grain images, and the cases of classifying unknown category data into OK grain images, NG grain images, and unknown category data.

[0108] <<Classified as OK grain images and NG grain images>>

[0109] The second classification unit 309 uses the second learning model stored in the second learning model storage unit 312 to classify the unknown category data into OK particle images and NG particle images.

[0110] In the case of <<classifying into OK particle images and NG particle images>>, the second learning model is a model used to output whether the unknown category data is an OK particle image or an NG particle image when the unknown category data is input.

[0111] <<Classified into OK grain images, NG grain images, and data of unknown category>>

[0112] The second classification unit 309 uses the second learning model stored in the second learning model storage unit 312 to classify unknown category data into OK particle images, NG particle images, and unknown category data. Specifically, the second classification unit 309 uses the second learning model to output the accuracy (also called confidence, reliability, or probability) of unknown category data being OK particle images and the accuracy of unknown category data being NG particle images. Furthermore, if the accuracy of unknown category data being OK particle images is greater than or equal to a predetermined value, the second classification unit 309 classifies the unknown category data as OK particle images. Similarly, if the accuracy of unknown category data being NG particle images is greater than or equal to a predetermined value, the second classification unit 309 classifies the unknown category data as NG particle images. Additionally, the second classification unit 309 classifies unknown category data that belongs neither to OK particle images nor NG particle images (i.e., the accuracy of unknown category data being OK particle images is less than a predetermined value, and the accuracy of unknown category data being NG particle images is also less than a predetermined value) as unknown category data.

[0113] In the case of <<classifying data into OK particle images, NG particle images, and unknown category data>>, the second learning model is a model that outputs the accuracy of the unknown category data as OK particle images and the accuracy of the unknown category data as NG particle images when the unknown category data is input.

[0114] The tag acquisition unit 313 acquires tag data to be added to the unknown category data. Specifically, the tag acquisition unit 313 sends unknown category data (e.g., a particle image) to the user terminal 103. It should be noted that the tag acquisition unit 313 may also be configured to send the unknown category data (e.g., a particle image) and information about the classification result after classification by the second classification unit 309 (e.g., whether it is an OK or NG particle image, accuracy, etc.) to the user terminal 103. Furthermore, the tag acquisition unit 313 can also receive tag data input to the user terminal 103. The person operating the user terminal 103 can judge the tags to be added to the unknown category data by observing (visually) the unknown category data (e.g., a particle image).

[0115] It should be noted that when the above-mentioned unknown category data is classified into OK particle images, NG particle images and unknown category data, the tag acquisition unit 313 may also be configured to acquire only the data of the tags that should be added to the unknown category data, that is, the accuracy of the unknown category data being OK particle images is less than a predetermined value, and the accuracy of the unknown category data being NG particle images is also less than a predetermined value.

[0116] The second adding unit 310 adds labels to the unknown category data based on the classification result after classification by the second classification unit 309 (e.g., whether it is an OK particle image or an NG particle image) and the labels obtained by the label acquisition unit 313. Specifically, if the classification result after classification by the second classification unit 309 is the same as the label obtained by the label acquisition unit 313, the second adding unit 310 adds the label to the unknown category data. Furthermore, if the classification result after classification by the second classification unit 309 is different from the label obtained by the label acquisition unit 313, the second adding unit 310 adds the label obtained by the label acquisition unit 313 to the unknown category data. In addition, the second adding unit 310 can also save the unknown category data and the added labels in association to the unknown category data storage unit 314.

[0117] The unknown category data storage unit 314 stores unknown category data and tags added by the second addition unit 310 in association.

[0118] The second learning model update unit 311 updates the second learning model in the second learning model storage unit 312 based on the unknown category data labeled by the second addition unit 310. Specifically, the second learning model update unit 311 can update the second learning model by performing machine learning using the unknown category data labeled by the second addition unit 310 as training data.

[0119] As described above, in <Embodiment 1>, since the second learning model used for classifying unknown category data can be updated, the unknown category data can be classified more accurately. Furthermore, when the aforementioned unknown category data is classified as <<OK particle images, NG particle images, and unknown category data>>, the label acquisition unit 313 is configured to acquire only the data for which labels should be added to the unknown category data, i.e., the accuracy of the unknown category data being OK particle images is less than a predetermined value, and the accuracy of the unknown category data being NG particle images is also less than a predetermined value. Accordingly, with the update of the second learning model, the learning accuracy of the second learning model can be improved, and the amount of data that cannot be classified will decrease. Therefore, the amount of data that needs to be judged based on human visual judgment of the added labels can be reduced.

[0120] <Implementation Method 2>

[0121] The following describes <Effective Implementation 2>. It should be noted that this description mainly focuses on the parts that differ from <Effective Implementation 1>.

[0122] In <Embodiment 2>, the machine learning device 102 includes Figure 3 The first learning model update unit 307 is shown below. The first learning model update unit 307 will be described below.

[0123] The first learning model update unit 307 updates the first learning model in the first learning model storage unit 308 based on the unknown category data tagged by the second addition unit 310. It should be noted that the first learning model update unit 307 may also be configured to update the first learning model after receiving an instruction to update the first learning model from any terminal such as the user terminal 103.

[0124] Specifically, the first learning model update unit 307 acquires the unknown category data that has been labeled by the second addition unit 310 and appends it to the particle image storage unit 304. Furthermore, the first learning model update unit 307 can update the first learning model by performing machine learning on the data labeled by the first addition unit 306 (e.g., OK particle images and NG particle images) and the unknown category data labeled by the second addition unit 310 in the particle image storage unit 304 as training data.

[0125] <<Automatic Updates of the First Learning Model>>

[0126] The first learning model update unit 307 can also be configured to update the first learning model without human intervention. Specifically, the second addition unit 310 adds labels to the unknown category data only based on the following unknown category data: unknown category data whose accuracy is greater than or equal to a predetermined value as OK particle images as determined by the second classification unit 309, and unknown category data whose accuracy is greater than or equal to a predetermined value as NG particle images as determined by the second classification unit 309 (i.e., data other than unclassifiable data). (i.e., without using the labels obtained by the label acquisition unit 313). The first learning model update unit 307 can update the first learning model by using the data in the particle image storage unit 304 that has been labeled by the first addition unit 306 (e.g., OK particle images and NG particle images) and the data other than the unclassifiable data that has been labeled by the second addition unit 310 as training data to perform machine learning.

[0127] <<Display of the ratio of unknown category data>>

[0128] As described in <Embodiment 1>, the first addition unit 306 may also be configured to store each object image in the object image storage unit 302 in association with the ratio of the OK particle image to the full particle image of the object image, the ratio of the NG particle image to the full particle image, and the ratio of unknown category data to the full particle image. Furthermore, the first addition unit 306 may also be configured to notify any terminal, such as the user terminal 103, of the ratio of each object image, the OK particle image to the full particle image of the object image, the NG particle image to the full particle image, and the ratio of unknown category data to the full particle image. For this purpose, a prompt can be given on any terminal, such as the user terminal 103, indicating that the amount of unknown category data decreases as the first learning model is updated.

[0129] As described above, in <Embodiment 2>, since the first learning model used for classifying data (e.g., particle images) can be updated, the data (e.g., particle images) can be classified more accurately. Furthermore, as the first learning model is updated, the amount of data with unknown categories decreases, thus reducing the amount of data that needs to be judged based on human visual perception of added labels.

[0130] <Implementation Method 3>

[0131] The following describes <Embodiment 3>. It should be noted that this description mainly focuses on the parts that differ from <Embodiment 1> and <Embodiment 2>.

[0132] In <Embodiment 3>, the machine learning device 102 infers the classification results of the input data. The following will describe classification based solely on the first learning model and two-stage classification based on both the first and second learning models.

[0133] <<Classification based solely on the first learning model>>

[0134] The first classification unit 305 acquires input data (e.g., particle images) and classifies it into classifiable data (e.g., OK particle images and NG particle images) and unknown category data using the first learning model stored in the first learning model storage unit 308. The first classification unit 305 may also notify any terminal, such as the user terminal 103, of the classification results.

[0135] <<Two-stage classification based on first and second learning models>>

[0136] The first classification unit 305 acquires input data (e.g., particle images) and classifies it into classifiable data (e.g., OK particle images and NG particle images) and unknown category data using the first learning model stored in the first learning model storage unit 308. The second classification unit 309 classifies the unknown category data (e.g., classifies it into OK particle images and NG particle images). The second classification unit 309 may also notify any terminal, such as the user terminal 103, of the classification results. The notified classification results may be only the results of the two-stage classification, or they may be results representing the classification results of the first classification unit 305 and the second classification unit 309, respectively.

[0137] <Data Flow>

[0138] Figure 6 This is a data flow diagram illustrating the flow of data according to one embodiment of the present invention. For example... Figure 6 As shown, data (e.g., particle images) is classified by a first learning model into classifiable data (e.g., OK particle images and NG particle images) and data of unknown category. Then, the data of unknown category is classified by a second learning model (e.g., classified as OK particle images and NG particle images). Afterwards, the classification results of the second learning model are modified by a human.

[0139] <Processing Method>

[0140] Below, in conjunction with Figures 7-9 Each process is explained.

[0141] Figure 7 This is a flowchart illustrating the training data generation process of one embodiment of the present invention.

[0142] In step 11 (S11), the object image acquisition unit 301 acquires an object image captured by the optical microscope 101 or the like.

[0143] In step 12 (S12), the particle image extraction unit 303 extracts a particle image from the object image obtained in S11.

[0144] In step 13 (S13), the first classification unit 305 classifies the data (e.g., the particle images extracted in S12) into classifiable data (e.g., OK particle images and NG particle images) and unknown category data.

[0145] In step 14 (S14), the first adding unit 306 adds labels to the classified data (e.g., OK particle images and NG particle images) that have been classified in S13, thereby creating training data.

[0146] In step 15 (S15), the second classification unit 309 classifies the unknown category data that was classified in S13.

[0147] In step 16 (S16), the second adding unit 310 adds labels to the unknown category data based on the classification result after classification in S15 (e.g., whether it is an OK particle image or an NG particle image) and the labels obtained by the label acquisition unit 313.

[0148] In step 17 (S17), the second learning model update unit 311 updates the second learning model in the second learning model storage unit 312 based on the unknown category data with added labels in S16.

[0149] Figure 8 This is a flowchart illustrating the update process of the first learning model according to one embodiment of the present invention.

[0150] In step 21 (S21), the first learning model update unit 307 acquires the unknown category data that has been labeled by the second addition unit 310.

[0151] In step 22 (S22), the first learning model update unit 307 appends the unknown category data obtained in S21 to the particle image storage unit 304.

[0152] In step 23 (S23), the first learning model update unit 307 updates the first learning model by using the data labeled by the first addition unit 306 in the particle image storage unit 304 (e.g., OK particle images and NG particle images) and the unknown category data added in S22 as training data to perform machine learning.

[0153] Figure 9 This is a flowchart illustrating the reasoning process of data classification results according to one embodiment of the present invention.

[0154] In step 31 (S31), the first classification unit 305 acquires input data (e.g., particle image).

[0155] In step 32 (S32), the first classification unit 305 uses the first learning model in the first learning model storage unit 308 to classify the data obtained in S31 into classifiable data (e.g., OK particle images and NG particle images) and unknown category data.

[0156] In step 33 (S33), the first classification unit 305 notifies any terminal, such as the user terminal 103, of the classification result inferred in S32.

[0157] It should be noted that, as described above, this embodiment can also be configured as a two-stage classification based on a first learning model and a second learning model.

[0158] <Effect>

[0159] In one embodiment of the present invention, the machine learning device 102 classifies data into "classifiable data" and "unknown category data," and then classifies the "unknown category data." Afterwards, a human modifies the classification result of the "unknown category data." Therefore, in one embodiment of the present invention, a two-stage classification based on a first learning model and a second learning model is performed, and the classification based on the second learning model only targets "unknown category data," thus improving the classification accuracy when classifying data for which label should be added with only low accuracy.

[0160] It should be noted that the present invention is not limited to the configurations listed in the above embodiments, combinations with other elements, or the configurations shown herein. Modifications can be made in these aspects without departing from the spirit of the invention, and the specific details can be determined according to its application.

[0161] Explanation of reference numerals in the attached figures:

[0162] 100 Quality Inspection System

[0163] 101 Optical Microscope

[0164] 102 Machine Learning Devices

[0165] 103 User Terminal

[0166] 104 Network

[0167] 301 Object Image Acquisition Unit

[0168] 302 Object Image Storage Unit

[0169] 303 Particle Image Extraction Unit

[0170] 304 Particle Image Storage Unit

[0171] 305 First Classification Section

[0172] 306 First Addition Department

[0173] 307 First Learning Model Update Department

[0174] 308 First Learning Model Storage Department

[0175] 309 Second Classification Section

[0176] 310 Second Addition Section

[0177] 311 Second Learning Model Update Department

[0178] 312 Second Learning Model Storage Department

[0179] 313 Tag Acquisition Department

[0180] 314 Unknown Category Data Storage Department.

Claims

1. A machine learning device for a quality inspection system, comprising: The first classification department classifies the data into classifiable data and unknown category data according to the first learning model, wherein the data is a particle image; The first adding unit adds labels to the classifiable data. The labels are either labels indicating that the shape of the particles in the particle image meets a predetermined criterion, or labels indicating that the shape of the particles in the particle image does not meet the predetermined criterion. The second classification department classifies the unknown category data according to the second learning model; The tag acquisition unit acquires tags that should be added to the unknown category data. The second adding section adds labels to the unknown category data; and The second learning model update unit updates the second learning model based on the unknown category data with the added labels. The labels are added to the unknown category data based on the classification results of the second classification part and the labels obtained by the label acquisition unit. The second labeling unit adds labels only to the data in the unknown category data, excluding unclassifiable data. The data in the unknown category data, excluding unclassifiable data, are those unknown category data whose accuracy as an OK grain image is greater than or equal to a predetermined value, as determined by the second classification unit, and those unknown category data whose accuracy as an NG grain image is greater than or equal to a predetermined value. The machine learning device further includes a first learning model updating unit, which updates the first learning model by performing machine learning on data other than unclassifiable data from the data labeled by the first adding unit and the unknown category data labeled by the second adding unit.

2. The machine learning apparatus as described in claim 1, wherein, The labels are used to classify the data into two categories.

3. The machine learning apparatus as described in claim 1 or 2, wherein, The tag acquisition unit only acquires tags that should be added to the following unknown category data: the accuracy of OK particle images after the second classification part of the unknown category data is less than a predetermined value, and the accuracy of NG particle images after the second classification part of the unknown category data is less than a predetermined value.

4. The machine learning apparatus as described in claim 1 or 2, wherein, The data is a full-grained image of the object. The first adding unit notifies the ratio of the OK particle image to the full particle image, the ratio of the NG particle image to the full particle image, and the ratio of the unknown category data to the full particle image.

5. A machine learning method for a quality inspection system, comprising: The first classification step involves classifying the data into classifiable data and unknown category data based on the first learning model, wherein the data is a granular image. The first addition step is to add labels to the classifiable data. The labels are either labels indicating that the shape of the particles in the particle image meets a predetermined benchmark, or labels indicating that the shape of the particles in the particle image does not meet the predetermined benchmark. The second classification step involves classifying the unknown category data according to the second learning model; The tag acquisition step involves obtaining the tags that should be added to the unknown category data; The second step involves adding labels to the unknown category data. and The second learning model update step involves updating the second learning model based on the unknown category data with the added labels, wherein the labels are added to the unknown category data based on the classification results of the unknown category data and the labels that should be added to the unknown category data. The second labeling step only adds labels to the data in the unknown category data, excluding unclassifiable data. The data in the unknown category data, excluding unclassifiable data, are those identified by the second classification step as either OK grain images with an accuracy greater than or equal to a predetermined value, or NG grain images with an accuracy greater than or equal to a predetermined value. The machine learning method further includes a first learning model update step, which updates the first learning model by performing machine learning on data other than unclassifiable data from the data labeled in the first addition step and the unknown category data labeled in the second addition step, as training data.

6. A program product for use in a quality inspection system, which functions as follows: The first classification department classifies the data into classifiable data and unknown category data according to the first learning model, wherein the data is a particle image; The first adding unit adds labels to the classifiable data. The labels are either labels indicating that the shape of the particles in the particle image meets a predetermined criterion, or labels indicating that the shape of the particles in the particle image does not meet the predetermined criterion. The second classification department classifies the unknown category data according to the second learning model; The tag acquisition unit acquires tags that should be added to the unknown category data. The second adding section adds labels to the unknown category data; and The second learning model update unit updates the second learning model based on the unknown category data with the added labels. The labels are added to the unknown category data based on the classification results of the second classification part and the labels obtained by the label acquisition unit. The second labeling unit adds labels only to the data in the unknown category data, excluding unclassifiable data. The data in the unknown category data, excluding unclassifiable data, are those unknown category data whose accuracy as an OK grain image is greater than or equal to a predetermined value, as determined by the second classification unit, and those unknown category data whose accuracy as an NG grain image is greater than or equal to a predetermined value. The program product also enables the computer to function as a first learning model update unit, which updates the first learning model by performing machine learning on data other than unclassifiable data from the data labeled by the first addition unit and the unknown category data labeled by the second addition unit.

7. A quality inspection system, comprising a machine learning device and a user terminal, wherein, The machine learning device includes: The first classification department classifies the data into classifiable data and unknown category data according to the first learning model, wherein the data is a particle image; The first adding unit adds labels to the classifiable data. The labels are either labels indicating that the shape of the particles in the particle image meets a predetermined criterion, or labels indicating that the shape of the particles in the particle image does not meet the predetermined criterion. The second classification department classifies the unknown category data according to the second learning model; The tag acquisition unit acquires tags that should be added to the unknown category data. The second adding section adds labels to the unknown category data; and The second learning model update unit updates the second learning model based on the unknown category data with the added labels. The labels are added to the unknown category data based on the classification results of the second classification part and the labels obtained by the label acquisition unit. The second labeling unit adds labels only to the data in the unknown category data, excluding unclassifiable data. The data in the unknown category data, excluding unclassifiable data, are those unknown category data whose accuracy as an OK grain image is greater than or equal to a predetermined value, as determined by the second classification unit, and those unknown category data whose accuracy as an NG grain image is greater than or equal to a predetermined value. The machine learning device further includes a first learning model updating unit, which updates the first learning model by performing machine learning on data other than unclassifiable data from the data labeled by the first adding unit and the unknown category data labeled by the second adding unit.

Images