Classification processing program, classification processing apparatus, and classification processing method

Abstract

To accurately classify a plurality of data to be classified.SOLUTION: First input data 302 having no correct answer label and second input data 303 having a known correct answer label are input to a machine learning model 201 that classifies input data into any of a plurality of classes, and deterioration of the machine learning model 201 is determined based on an output result for the second input data 303 and the correct answer label among output results output from the machine learning model 201.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a classification processing program, a classification processing device, and a classification processing method. [Background technology]

[0002] Deep learning (DL) models, which are an example of machine learning models, are sometimes used to classify data. DL models are neural networks generated by DL.

[0003] In classification processing, a DL model is used to estimate the class to which the data to be classified belongs. However, due to environmental changes during the operation of the DL model, data with trends different from the training data used to generate the DL model may appear as the classification target.

[0004] For example, if the data to be classified are images, the shooting environment can change due to factors such as deterioration of lighting, dust on the camera lens, etc., resulting in images with trends that differ from those of the training data. Such changes in data trends are called data drift.

[0005] Because data drift can occur, it is desirable to update a DL model by performing DL again, even while the model is in operation. However, because the data to be classified that occurs during operation does not have correct labels attached, it is difficult to perform effective machine learning as is. Therefore, High Durability Learning (HDL) technology is used to adapt DL models to data drift during operation.

[0006] Regarding HDL technology, machine learning programs that can suppress deterioration in the accuracy of machine learning models are known (see, for example, Patent Document 1 and Non-Patent Document 1). Machine learning programs that shorten the update time of machine learning models are also known (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2023 / 067782 Brochure [Patent Document 2] Japanese Patent Publication No. 2022-105916 [Non-patent literature]

[0008] [Non-Patent Document 1] H. Kingetsu et al., “MULTI-STEP TEST-TIME ADAPTATION WITH ENTROPY MINIMIZATION AND PSEUDO-LABELING”, 2022 IEEE International Conference on Image Processing (ICIP), pages 4153-4157, 2022. Summary of the Invention [Problem to be solved by the invention]

[0009] When a DL model is in operation, imbalanced data may occur as a classification target. Imbalanced data refers to a data set in which the number of data belonging to each class is significantly biased when the number of classes into which the data is classified is known. When the data set to be classified is imbalanced data, the accuracy of the classification results obtained by the DL model updated using the HDL technology of Patent Document 1 is likely to decrease.

[0010] This problem is not limited to DL models updated using the HDL technology of Patent Document 1, but occurs in classification processes using various machine learning models.

[0011] In one aspect, the present invention aims to accurately classify a plurality of data to be classified. [Means for solving the problem]

[0012] For this reason, this classification processing program inputs first input data that does not have a correct label and second input data whose correct label is known into a machine learning model that classifies input data into one of multiple classes, and causes a computer to execute a process of determining deterioration of the machine learning model based on the output result for the second input data and the correct label among the output results output from the machine learning model. [Effects of the Invention]

[0013] According to one embodiment, a plurality of data to be classified can be classified with high accuracy. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 10 is a diagram illustrating an example of imbalance data. [Figure 2] FIG. 1 is a diagram schematically illustrating a configuration of a classification processing device according to a related art. [Figure 3] FIG. 10 is a diagram illustrating information stored in a storage unit of a classification processing device according to a related art technique. [Figure 4] FIG. 1 is a diagram schematically illustrating a configuration of a classification processing device according to an embodiment. [Figure 5] 10 is a flowchart illustrating an example of a training process in the classification processing device according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of classification processing in the classification processing device according to an embodiment. [Figure 7] FIG. 1 is a diagram illustrating data processed in a classification processing device according to an embodiment. [Figure 8] FIG. 1 is a diagram illustrating an example of the configuration of a defective product detection system. [Figure 9] 10 is a flowchart illustrating an example of processing by a classification processing device in a defective product detection system using image inspection AI (Artificial Intelligence). [Figure 10]1 is a block diagram showing an example of the hardware (HW) configuration of a computer that realizes the functions of a classification processing device according to an embodiment and a classification processing device of related art. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the classification processing program, classification processing device, and classification processing method will be described with reference to the drawings. However, the embodiments shown below are merely examples, and are not intended to exclude the application of various modifications and techniques not explicitly stated in the embodiments. In other words, the present embodiment can be implemented with various modifications (such as combining the embodiment and each modification) within the scope of its intent. Furthermore, each figure does not intend to include only the components shown in the figure, but can also include other functions, etc.

[0016] (A) Related technologies 1 shows an example of imbalanced data. In this example, the DL model classifies the data to be classified into one of classes #1 to #4. Therefore, the number of classes to be classified into is four.

[0017] Data set 112 is a set of data to be classified that occurs when the DL model is operated, and includes data 111-1 to data 111-10. Data 111-1 and data 111-2 belong to class #1, and data 111-3 to data 111-10 belong to class #3, but no correct answer label is added to data 111-1 to data 111-10. In this case, the class ratio of data set 112 is 2:0:8:0, and therefore data set 112 is imbalanced data.

[0018] The expected classification result for the DL model is that data 111-1 and data 111-2 are classified into class #1, and data 111-3 to data 111-10 are classified into class #3.

[0019] However, a common characteristic of many machine learning algorithms is that even when a DL model is updated using data to be classified during operation, the updated DL model tends to classify the data to be classified so as to approximate the class ratio of the set of training data. Therefore, if the class ratio of the set of data to be classified is significantly different from the class ratio of the set of training data, even in classification processing using the HDL technology of Patent Document 1, some data may be classified into classes different from the correct answer.

[0020] 1, data 111-1 and data 111-2 belonging to class #1 are classified into class #1. On the other hand, data 111-3 and data 111-4 belonging to class #3 are classified into class #2, data 111-5 to data 111-8 belonging to class #3 are classified into class #3, and data 111-9 and data 111-10 belonging to class #3 are classified into class #4. Therefore, data 111-3, data 111-4, data 111-9, and data 111-10 are classified into classes different from the correct answer.

[0021] In this way, when the data to be classified is imbalanced data, the accuracy of the classification result is likely to decrease using the HDL technique of Patent Document 1. In particular, when the number of data items belonging to any class is zero, the accuracy of the classification result decreases significantly.

[0022] Fig. 2 is a diagram showing a schematic configuration of a classification processing device 1 according to related art, and Fig. 3 is a diagram showing information stored in a storage unit 107 of the classification processing device 1.

[0023] The classification processing device 1 illustrated in FIG. 2 has functions as an input processing unit 101, an additional data generating unit 102, a model generating unit 103, a classification and tuning unit 104, an additional data removing unit 105, and an output processing unit 106.

[0024] The classification processing device 1 operates in a training phase (training mode) and an operation phase (operation mode). In the operation phase, the classification processing device 1 classifies data to be classified into one of a known number of classes. The data to be classified may be an image, an audio, or a text.

[0025] The storage unit 107 stores a training data set 301, a classification target data set 302, a classification model 201, an augmented data set 304, and classification results 305 and 306.

[0026] The training data set 301 includes multiple training data. Each training data is assigned a correct label indicating the correct class. The classification target data set 302 includes multiple data to be classified. Each data to be classified does not have a correct label assigned. The data included in the classification target data set 302 can be referred to as input data.

[0027] The input processing unit 101 processes input data (input data). For example, the input processing unit 101 checks whether the input data is data to be classified. The input processing unit 101 stores the input data to be classified in the data set 302 to be classified in the storage unit 107.

[0028] In the training phase, the model generation unit 103 generates a classification model 201 by performing a first machine learning process using a training data set 301. The classification model 201 is an example of a machine learning model that classifies input data into one of multiple classes.

[0029] The model generation unit 103 stores the generated classification model 201 in the storage unit 107. The generated classification model 201 is a trained machine learning model.

[0030] The machine learning model may be a neural network, a logistic regression, a support vector machine, a decision tree, a random forest, etc. The machine learning model may be a DL model.

[0031] On the other hand, in the operation phase, the additional data generation unit 102 may generate the additional data 303 by, for example, selecting one or more pieces of training data for each class from the training data included in the training data set 301 and deleting the correct label from the selected training data. Thus, the additional data 303 belonging to each of the multiple classes is generated. Then, the additional data generation unit 102 generates the augmented data set 304 by adding the additional data 303 to the data to be classified (input data) included in the data set to be classified 302. The augmented data set 304 includes the input data and the additional data 303.

[0032] The input data included in the expanded data set 304 is an example of first input data that does not have a correct label. The additional data 303 included in the expanded data set 304 is an example of second input data that has a known correct label (class). The additional data generation unit 102 stores the generated expanded data set 304 in the storage unit 107.

[0033] By selecting training data whose classes are known, a desired number of training data can be selected for each class, and therefore additional data 303 belonging to each of multiple classes can be efficiently generated. The additional data generation unit 102 may select additional data 303 from data whose classes are known other than training data.

[0034] The additional data generation unit 102 may generate additional data 303 in a number corresponding to the number of input data (number of input data). For example, the additional data generation unit 102 may generate additional data 303 in a number that is a predetermined ratio R to the number of input data for each of all classes. For example, when the number of input data=100 and the ratio R=20%, the additional data generation unit 102 may create 20 pieces of additional data 303 for each class. The ratio R used to determine the number of additional data 303 for each class may be referred to as the additional data ratio R. It is desirable that the additional data ratio R be a ratio that is less affected by accuracy degradation for both good and bad products.

[0035] The classification and tuning unit 104 updates the classification model 201 by performing second machine learning using the augmented dataset 304. For example, online machine learning using HDL technology is used as the second machine learning. The second machine learning may be the online machine learning described in Patent Document 1.

[0036] The classification / tuning unit 104 inputs input data (first input data) and additional data 303 (second input data) to the classification model 201.

[0037] The classification and tuning unit 104 updates the classification model 201 through machine learning using the input data (first input data) and the additional data 303 (second input data).

[0038] The classification / tuning unit 104 uses the updated classification model 201 to classify each piece of data to be classified and each piece of additional data 303 included in the expanded data set 304 into one of the classes, thereby generating classification results 305 and storing them in the storage unit 107. The classification results 305 include the classification results for each piece of data to be classified and the classification results for each piece of additional data 303.

[0039] By updating the classification model 201 using the augmented data set 304 and classifying each piece of data using the updated classification model 201, the data can be classified with high accuracy even if the data to be classified has data drift.

[0040] The additional data removal unit 105 extracts the classification results of each data to be classified by removing the classification results of each additional data 303 from the classification result 305, thereby generating classification result 306 and storing it in the storage unit 107. The classification result 306 includes only the classification results of each data to be classified. The classification result 306 is an example of a first output result for input data (first input data) among the data output from the classification model 201.

[0041] The output processing unit 106 outputs the classification result 306 as the classification result of the classification model 201. The output processing unit 106 may also output the determination result of the classification model 201 as to whether it has deteriorated.

[0042] 1 , when classifying data to be classified that is included in a data set to be classified 302, a portion of the training data whose class is known for each class is added to an augmented data set 304 as additional data 303. Therefore, even if the data set to be classified 302 is imbalanced data, the class ratio of the augmented data set 304 can be made closer to the class ratio of the training data set 301.

[0043] By inputting an augmented data set 304 having a class ratio close to that of the training data set 301 into the classification model 201 and classifying each piece of data, the likelihood that each piece of data will be classified into the correct class increases. Therefore, even if the class ratio of the data set to be classified 302 is significantly different from the class ratio of the training data set 301, the data to be classified can be classified with high accuracy.

[0044] 1, additional data 303 is added to an imbalanced classification target data set 302 to create an expanded data set 304. However, if the proportion of data for which the correct answer is known is increased as the additional data 303, the characteristics of the input data in the classification target data set 302 will be diluted by the additional data 303, which may result in a decrease in classification accuracy for the classification target data set 302.

[0045] On the other hand, if the ratio of adding data with known correct answers as additional data 303 is reduced, the classification / tuning unit 104 may classify an imbalanced data set 302 to be classified and update the classification model 201 based on this data set 302 to be classified, which may cause degradation of the classification model 201.

[0046] Even if the classification model 201 is actually deteriorating, the data set 302 to be classified usually does not have a correct label, so it is not possible to check in real time during operation whether the classification accuracy of the classification model 201 has deteriorated.

[0047] Therefore, in the classification processing device 1a as an example of this embodiment, even when imbalanced input data or drift data is input during operation, the classification model 201 is able to correctly restore accuracy without degradation, thereby achieving accurate classification of multiple data to be classified.

[0048] (B) Description of the embodiment FIG. 4 is a diagram schematically showing the configuration of a classification processing device 1a according to an embodiment.

[0049] The classification processing device 1a illustrated in Fig. 4 is the classification processing device 1 of the related art (see Fig. 2) equipped with a model degradation determination unit 401 and a model recovery unit 402, and other parts are configured in the same way as the classification processing device 1 of the related art. Note that in the figure, the same reference numerals as those already described indicate similar parts, and therefore their explanation will be omitted.

[0050] The model degradation determination unit 401 determines whether the classification model 201 has deteriorated based on the classification results of each additional data 303 removed from the classification result 305 by the additional data removal unit 105 and its correct label.

[0051] The model degradation determination unit 401 determines the degradation of the classification model 201 based on the output result for the additional data 303 (second input data) and the correct label (class) from the output results (classification results 305) output from the classification model 201.

[0052] The model degradation determination unit 401 may, for example, calculate the classification accuracy for the additional data 303, and if this classification accuracy is less than a threshold, determine that degradation has occurred in the classification model 201. The classification accuracy may be, for example, a rate of correct answers.

[0053] Note that the method of determining whether or not the classification model 201 has deteriorated by the model deterioration determination unit 401 is not limited to this, and various modifications can be made. For example, the model deterioration determination unit 401 may compare the calculated classification accuracy for the additional data 303 with the classification accuracy calculated last time, and determine that deterioration of the classification model 201 has occurred if the classification accuracy has decreased by a predetermined threshold or more compared to the classification accuracy calculated last time.

[0054] The model recovery unit 402 executes recovery processing for the classification model 201 when the model degradation determination unit 401 determines that the classification model 201 has deteriorated.

[0055] The recovery process may recover the classification model 201 by, for example, restoring the classification model 201 to the state before the degradation was determined. That is, the model recovery unit 402 may restore the classification model 201 to the state before the classification / tuning unit 104 updated it (pre-update).

[0056] Therefore, in the classification processing device 1a, it is desirable that every time the classification / tuning unit 104 updates the classification model 201, the classification model 201 before the update is stored in the storage unit 107 or the like.

[0057] Note that the method of recovery processing of the classification model 201 by the model recovery unit 402 is not limited to this. For example, the number of additional data 303 in the augmented data set 304 may be increased, and the classification model 201 may be re-trained using this augmented data set 304. For example, if the previous additional data ratio R was 5%, the additional data ratio R may be set to 20%.

[0058] By increasing the additional data 303 in the augmented data set 304 to eliminate the imbalance, and then using this augmented data set 304 to retrain the classification model 201, the degraded classification model 201 can be restored. This method of restoring the classification model 201 by increasing the additional data 303 is particularly effective when the same input data is processed again.

[0059] (C) Operation An example of training processing in the classification processing device 1a according to the embodiment configured as described above will be described with reference to the flowchart (steps S1 to S2) shown in Fig. 5. This processing is carried out in the training phase.

[0060] The model generation unit 103 inputs a plurality of training data sets included in the training data set 301 into the pre-training machine learning model (step S1), and generates the classification model 201 by performing a first machine learning process (step S2).

[0061] Next, an example of classification processing in the classification processing device 1a according to one embodiment will be described in accordance with the flowchart (steps S11 to S21) shown in Fig. 6 with reference to Fig. 7. Fig. 7 is a diagram illustrating data processed in the classification processing device 1a. The processing shown below is carried out in the operation phase.

[0062] 7 illustrates processing using a four-class classification model 201, which classifies data into one of classes #1 to #4. In addition, in FIG. 7, symbol A describes the processing of the input processing unit 101 and the additional data generation unit 102, symbol B describes the processing of the classification / tuning unit 104, symbol C describes the processing of the additional data removal unit 105, and symbol D describes the processing of the model degradation determination unit 401.

[0063] The input processing unit 101 checks whether a plurality of data items to be classified have been input (step S11).

[0064] When multiple data to be classified are input (see the YES route in step S11), the input processing unit 101 generates a classification target data set 302 including the data and stores it in the storage unit 107 (step S12). In the example shown in Fig. 7, the input data is indicated by dashed lines, and 10 pieces of input data are shown (see symbol P1). These multiple input data correspond to the classification target data set 302.

[0065] The additional data generation unit 102 determines the number M of additional data 303 belonging to each class based on the number of input data and the ratio R (step S13). Specifically, the additional data generation unit 102 determines M so that the number M of additional data relative to the number of input data becomes the ratio R. In the example shown in Fig. 7, the number of input data is 10 and the ratio R is 20%, so the additional data generation unit 102 determines the number M of additional data 303 belonging to each class to be 2.

[0066] The additional data generation unit 102 selects M pieces of training data for each class from the training data included in the training data set 301, and generates additional data 303 by deleting the correct answer label from the selected training data (step S14).

[0067] In the example shown by symbol A in Fig. 7, the additional data generation unit 102 adds data with known correct answers at a ratio of 20% of the number of input data (10 items) to each class. That is, the additional data generation unit 102 generates two (M=2) pieces of data (additional data 303) with known correct answers for each class (see symbol P2). In Fig. 7, the additional data 303 is indicated by a solid line.

[0068] The additional data generation unit 102 generates an expanded data set 304 by adding additional data 303 to the data to be classified included in the data set to be classified 302 (step S15). In the example shown by symbol A in Fig. 7, eight additional data 303 are added to ten input data, resulting in an expanded data set 304 having 18 data.

[0069] At this time, the additional data generating unit 102 also assigns identification information to each piece of data in the expanded data set 304 to distinguish between the data to be classified and the additional data 303 .

[0070] The classification and tuning unit 104 updates the classification model 201 by performing a second machine learning using the expanded data set 304 (step S16). Then, the classification and tuning unit 104 generates the classification result 305 by classifying each piece of data included in the expanded data set 304 into one of the classes using the updated classification model 201 (step S17).

[0071] In the example shown by symbol B in Figure 7, two input data and two additional data 303 are classified into class #1, one input data and three additional data 303 are classified into class #2, five input data are classified into class #3, and two input data and three additional data 303 are classified into class #4 (see symbol P3).

[0072] Thereafter, the additional data removal unit 105 extracts the classification results of each data to be classified by removing the classification results of each additional data 303 from the classification result 305, and generates the classification result 306 (step S18; see symbol P4). The output unit 416 outputs the classification result 306 (step S19).

[0073] The model degradation determination unit 401 determines whether the classification model 201 has deteriorated based on the classification results of each additional data 303 removed from the classification result 305 by the additional data removal unit 105 and its correct label (step S20).

[0074] In the example shown by symbol D in Fig. 7, two additional data 303 that should be classified into class #3 are misclassified into class #2 and class #4. That is, six of the eight additional data 303 are correctly classified, so the model degradation determination unit 401 calculates the classification accuracy (correct rate) of the classification model 201 as 0.75 (= 6 / 8). The model degradation determination unit 401 compares this calculated classification accuracy with a preset threshold (e.g., 0.9), and if the classification accuracy is less than the threshold, determines that degradation of the classification model 201 has occurred.

[0075] If the model degradation determination unit 401 determines that the classification model 201 has deteriorated (see the YES route in step S20), the model recovery unit 402 recovers the classification model 201 (step S21). For example, the model recovery unit 402 reads out from the storage unit 107 the classification model 201 before the update (previous version) that has not been determined to have deteriorated, and replaces the classification model 201 with it.

[0076] Thereafter, the process returns to step S11, and the classification processing device 1a repeats the processes from step S11 onwards.

[0077] On the other hand, if the model degradation determination unit 401 determines that the classification model 201 has not deteriorated (see the NO route from step S20), the process returns to step S11, and the classification processing device 1a repeats the processes from step S11 onwards.

[0078] If the input of data to be classified stops (see the NO route in step S11), the classification processing device 1a ends the process.

[0079] The classification processing device 1a configured as described above may be applied to, for example, a defective product detection system using image inspection AI (Artificial Intelligence).

[0080] FIG. 8 is a diagram illustrating an example of the configuration of the defective product detection system 2.

[0081] The defective product detection system 2 illustrated in FIG. 8 includes an image inspection AI 21 and a high-durability learning processing unit 22.

[0082] Images (input image group) of a plurality of inspection objects (for example, bolts) are input to the image inspection AI 21. That is, in the defective product detection system 2, the input data is data of the inspection objects.

[0083] The image inspection AI 21 inspects each input image and outputs inspection result information for each image. The inspection result information may include a judgment result indicating whether the inspection object is a good product or a defective product. The image inspection AI 21 inputs the inspection results, which associate the image with the inspection result information, to the high-durability learning processing unit 22.

[0084] The high-resistance learning processing unit 22 determines (diagnoses) whether the image inspection results obtained by the image inspection AI 21 are correct. The high-resistance learning processing unit 22 outputs accuracy information (accuracy diagnosis result) as a diagnosis result. The accuracy information output by the high-resistance learning processing unit 22 may be referred to as image inspection AI accuracy information.

[0085] The high-resistance learning processing unit 22 includes the functions of the classification processing device 1a described above.

[0086] For example, the high-robustness learning processing unit 22 functions as the model generation unit 103, and performs machine learning (generation) of a classification model 201 based on image data using multiple training images. In this defective product detection system 2, the classification model 201 performs two-class classification to classify input image data into either a good product or a defective product. The multiple image data correspond to a training data set 301.

[0087] After the generation of the classification model 201 (machine learning), the operation of the defective product detection system 2 begins, and the high-robust learning processing unit 22 functions as the classification processing device 1a, classifying each of the multiple input images into either a good product or a defective product. The multiple input image data corresponds to the target data set 302.

[0088] The high-robustness learning processing unit 22 may compare the inspection result of the image inspection AI21 with the classification result of the classification model 201 to determine whether the inspection result of the image inspection AI21 is correct.

[0089] An example of processing by the classification processing device 1a in the defective product detection system 2 using the image inspection AI 21 will be described with reference to the flowchart (steps S31 to S35) shown in FIG.

[0090] In the training phase, the classification model 201 is normally trained (machine learned) in advance, assuming that the ratio of non-defective products to defective products is 90:10. The flowchart shown in Fig. 9 illustrates the processing in the operation phase.

[0091] In step S31, imbalanced input data is input. In the example shown in Fig. 9, 100 pieces (images) of image data of non-defective products are input as input data, and 0 pieces (images) of image data of defective products are input as input data. Hereinafter, image data of non-defective products may be referred to as non-defective product images, and image data of defective products may be referred to as defective product images.

[0092] The additional data generation unit 102 generates additional data 303 according to the additional data ratio R, without knowing the correct answer for the input data. The additional data ratio R is a ratio at which the impact of accuracy degradation is small for both good and defective products. In this example, the additional data ratio R=5%. The additional data generation unit 102 generates five additional data 303 for each class. That is, the additional data generation unit 102 generates five good product images and five defective product images.

[0093] The expanded data set 304 contains 110 pieces of image data, which is a combination of the input data and the additional data 303, so although the actual correct answers are unknown, it contains 105 good product images and 5 bad product images.

[0094] In step S32, the classification / tuning unit 104 uses the updated classification model 201 to classify each data to be classified and each additional data 303 included in the expanded data set 304 into one of the classes, thereby generating a classification result 305.

[0095] The classification model 201 classifies 100 pieces of input data (input images) into 98 good products and 2 bad products, due to the DL model's characteristic of classifying according to the ratios at the time of learning.

[0096] Furthermore, for the additional data 303, the classification model 201 classifies only three of the five items that are actually good as good products, and classifies the remaining two as defective products. Furthermore, of the five items that are actually defective, all (five items) are classified as defective products.

[0097] That is, the classification model 201 classifies the 110 data in the augmented data set 304 into 101 good products and 9 bad products.

[0098] In step S33, the classification / tuning unit 104 tunes the classification model 201. In this tuning, since the additional data 303 is insufficient, it can be said that the classification model 201 is corrected so that the inspection object can be more easily classified as a defective product than with a conventional classification model.

[0099] Of the 10 pieces of additional data 303 with correct labels, 8 were correctly classified, so the classification accuracy is 0.8. Here, if the threshold is 0.9, the model degradation determination unit 401 determines that the classification accuracy is less than the threshold, and therefore that degradation of the classification model 201 has occurred. If the model degradation determination unit 401 determines that the classification model 201 has deteriorated, the model recovery unit 402 executes recovery processing for the classification model 201.

[0100] In step S34, new imbalanced input data is input. In the example shown in Fig. 9, it is assumed that 90 pieces of image data (images) of non-defective products and 10 pieces of image data (images) of defective products are input as input data.

[0101] The additional data generation unit 102 generates additional data 303 according to the additional data ratio R=5% without knowing the correctness of the input data. That is, five good product images and five defective product images are generated. In this example, the same input data as in step S31 will not be reprocessed, so the additional data ratio R is not changed.

[0102] The expanded data set 304 contains 110 pieces of image data, which is a combination of the input data and the additional data 303, so although the actual correct answers are unknown, it contains 95 good product images and 15 bad product images.

[0103] In step S35, the classification / tuning unit 104 uses the restored classification model 201 to classify each data to be classified and each additional data 303 included in the enriched data set 304 into one of the classes, thereby generating a classification result 305.

[0104] Here, because classification model 201 has recovered, for input data with the ratio (ratio at the time of learning; in this case, good products:defective products = 90:10) as originally expected for classification model 201, classification model 201 correctly classifies 100 pieces of input data (input images) into 90 good products and 10 defective products. In other words, classification model 201 classifies all 90 good products included in the 100 pieces of input data as good products, and also classifies all 10 defective products included in the 100 pieces of input data as defective products.

[0105] The classification model 201 also correctly classifies the additional data 303. That is, for the additional data 303, all five non-defective items are classified as non-defective items, and all five defective items are classified as defective items.

[0106] In addition, when the classification processing device 1 of the related art shown in Figure 2 is applied to a defective product detection system using image inspection AI instead of the classification processing device 1a, the model recovery unit 402 will not recover the classification model 201 even if the classification model 201 deteriorates.

[0107] Therefore, when the same processing as that shown in FIG. 9 is performed, the same processing is performed in steps S31 and S32, but the classification accuracy is lower in step S35 than in the classification processing device 1a.

[0108] That is, for input data with the ratio (ratio at the time of learning; in this case, good products:defective products = 90:10) as initially assumed for classification model 201, of 100 pieces of input data (input images), of the 90 that are actually good products, for example, 89 are classified as good products and 1 is classified as defective. Also, of the 10 that are actually defective products, all (10) are classified as defective.

[0109] For the additional data 303, four of the five items that are actually good are classified as good and one is classified as defective. Of the five items that are actually defective, all (five items) are classified as defective.

[0110] As described above, the classification accuracy of the classification processing device 1 of the related art may be lower than that of the classification processing device 1a of the present invention.

[0111] (D) Effects As described above, in the classification processing device 1a according to one embodiment, the model degradation determination unit 401 determines whether or not the classification model 201 has deteriorated, based on the classification results of each additional data 303 removed from the classification result 305 by the additional data removal unit 105 and its correct label. If the model degradation determination unit 401 determines that the classification model 201 has deteriorated, the model recovery unit 402 executes recovery processing for the classification model 201.

[0112] As a result, even if the classification model 201 deteriorates during the operation phase due to imbalanced input data, data drift, or other reasons, the model deterioration determination unit 401 detects the deterioration of the classification model 201, and the model recovery unit 402 recovers the classification model 201. Therefore, high classification accuracy can be maintained.

[0113] The model degradation determination unit 401 determines whether or not the classification model 201 has deteriorated based on the classification results of each additional data 303 removed from the classification result 305 by the additional data removal unit 105 and its correct label, thereby making it possible to determine in real time during operation whether or not the classification accuracy has deteriorated.

[0114] If the amount of additional data 303 generated by the additional data generation unit 102 is too large, the features of the input data will be diluted by the additional data 303, which may cause the classification model 201 to deteriorate and reduce the classification accuracy of the input operational data. On the other hand, if the amount of additional data 303 is too small, updating the classification model with imbalanced input data may cause the classification model 201 to deteriorate. Even in these cases, if the model deterioration determination unit 401 determines that the classification model 201 has deteriorated, the model recovery unit 402 can perform recovery processing on the classification model 201, thereby maintaining high classification accuracy.

[0115] The additional data generation unit 102 generates additional data 303 and adds it to the classification target data set 302 to generate an expanded data set 304, and the classification / tuning unit 104 classifies each of the classification target data and each of the additional data 303 included in the expanded data set 304 into one of the classes using the classification model 201. This makes it possible to improve classification accuracy even if the classification target data set 302 is imbalanced data.

[0116] (E) Other 10 is a block diagram showing an example of the hardware (HW) configuration of a computer 10 that realizes the functions of the classification processing device 1a according to one embodiment and the classification processing device 1 of the related art. When multiple computers are used as HW resources that realize the functions of the classification processing device 1a, each computer may have the HW configuration exemplified in FIG.

[0117] As shown in FIG. 10, the computer 10 may, as a HW configuration, illustratively include a processor 10a, a GPU (Graphics Processing Unit) 10b, a memory 10c, a storage device 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a reading unit 10g.

[0118] The processor 10a is an example of a processing unit that performs various controls and calculations, and is a control unit that executes various processes. The processor 10a may be connected to each block in the computer 10 via a bus 10j so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multi-core processor having multiple processor cores, or a configuration having multiple multi-core processors.

[0119] The processor 10a may be, for example, an integrated circuit (IC) such as a CPU, MPU, APU, DSP, ASIC, or FPGA. Note that the processor 10a may be a combination of two or more of these integrated circuits. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, APU is an abbreviation for Accelerated Processing Unit, DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0120] The GPU 10b may be, for example, an accelerator such as a general-purpose computing on graphics processing unit (GPGPU). The GPU 10b may also be used to control screen display on an output device such as a monitor in the IO unit 10f. The GPU 10b may be configured as an accelerator that executes machine learning processing and inference (prediction) processing using a machine learning model. It can be said that the GPU 10b has higher processing performance in machine learning processing and inference processing than the CPU 10a.

[0121] The memory 10c is an example of HW that stores various types of data, programs, and other information. The memory 10c may be, for example, a volatile memory such as a dynamic random access memory (DRAM) or a non-volatile memory such as a persistent memory (PM), or both.

[0122] The storage device 10d is an example of HW that stores various types of data, programs, and other information. Examples of the storage device 10d include various storage devices such as magnetic disk devices such as HDDs (Hard Disk Drives), semiconductor drive devices such as SSDs (Solid State Drives), and nonvolatile memories. Examples of nonvolatile memories include flash memories, SCMs (Storage Class Memory), and ROMs (Read Only Memory). The storage unit 107 described above may be realized by the storage device 10d, or may be realized by a database (not shown), and can be implemented in various modified forms.

[0123] The storage device 10d may store a program 10h (classification processing program) that realizes all or part of the various functions of the computer 10.

[0124] For example, the processor 10a of the classification processing device 1a can implement the functions of the classification processing device 1a described above by loading the program 10h stored in the storage device 10d into the memory 10c and executing it. The processor 10a is an example of a control unit that implements the functions of the input processing unit 101, the additional data generation unit 102, the model generation unit 103, the classification / tuning unit 104, the additional data removal unit 105, the output processing unit 106, the model degradation determination unit 401, and the model recovery unit 402.

[0125] The IF unit 10e is an example of a communication IF that controls the connection and communication between the computer 10 and other computers. For example, the IF unit 10e may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support either or both of wireless and wired communication methods.

[0126] The program 10h may be downloaded to the computer 10 from a network via the IF unit 10e and stored in the storage device 10d.

[0127] The IO unit 10f may include one or both of an input device and an output device. Examples of input devices include a keyboard, a mouse, and a touch panel. Examples of output devices include a monitor, a projector, and a printer. The IO unit 10f may also include a touch panel or the like that combines an input device and an output device. The output device may be connected to the GPU 10b. The IO unit 10f may also be an input device or an output device of another information processing device connected to the computer 10 via SSH (Secure Shell) or the like.

[0128] The reading unit 10g is an example of a reader that reads data or program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 10i may store the program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the storage device 10d.

[0129] Examples of the recording medium 10i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

[0130] The above-described HW configuration of the computer 10 is an example. Therefore, the HW in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or the HW may be added or deleted as needed.

[0131] The configurations and processes of this embodiment can be selected as needed, or can be combined as appropriate.

[0132] The disclosed technology is not limited to the above-described embodiment, and can be implemented in various modifications without departing from the spirit of the present embodiment.

[0133] For example, FIG. 8 shows an example in which the classification processing device 1a is applied to a defective product detection system 2 using an image inspection AI 21, but this is not limited to this and various modifications can be made.

[0134] In the above-described embodiment, in the operation phase, the additional data generation unit 102 selects one or more training data for each class from the training data included in the training data set 301, and generates the additional data 303 by deleting the correct label from the selected training data, and the classification / tuning unit 104 updates the classification model 201 by machine learning using the input data and the additional data 303, but the present invention is not limited to this.

[0135] For example, the classification and tuning unit 104 may select one or more training data for each class from the training data included in the training data set 301, and process the selected training data while ignoring the correct labels, thereby realizing processing similar to that of generating the additional data 303. In other words, the classification and tuning unit 104 may include the function of the additional data generation unit 102, and various modifications can be made to the implementation.

[0136] Furthermore, the above disclosure will enable those skilled in the art to implement and manufacture the present embodiment.

[0137] (F) Notes The following additional notes are provided regarding the above-described embodiments.

[0138] (Appendix 1) A machine learning model that classifies input data into one of multiple classes is input with first input data that does not have a correct answer label and second input data whose correct answer label is known, Determining deterioration of the machine learning model based on the output result for the second input data and the correct label among the output results output from the machine learning model. A classification processing program that causes a computer to execute processing.

[0139] (Appendix 2) When it is determined that the machine learning model has deteriorated, the machine learning model is restored. 2. The classification processing program according to claim 1, wherein the processing is executed by the computer.

[0140] (Appendix 3) Among the data output from the machine learning model, a first output result for the first input data is output as a classification result of the machine learning model. 3. The classification processing program according to claim 1 or 2, characterized in that the processing is executed by the computer.

[0141] (Appendix 4) Outputting the result of the determination of the deterioration of the machine learning model 2. The classification processing program according to claim 1, wherein the processing is executed by the computer.

[0142] (Appendix 5) The process of inputting into the machine learning model includes: and inputting the second input data prepared for each of the plurality of classes into the machine learning model. 2. The classification processing program according to claim 1,

[0143] (Appendix 6) updating the machine learning model by machine learning using the first input data and the second input data; causing the computer to execute a process; The process of recovering the machine learning model includes: and a process of restoring the updated machine learning model to the machine learning model before the update. 3. The classification processing program according to claim 2,

[0144] (Appendix 7) The process of recovering the machine learning model includes: Increasing the number of second input data for which the correct answer label is known 3. The classification processing program according to claim 2, further comprising:

[0145] (Appendix 8) the input data is data to be inspected, the plurality of classes includes good products and bad products; The computer is provided in a defective product detection system in which the machine learning model classifies the inspection object into either a good product or a defective product based on data of the inspection object. 2. The classification processing program according to claim 1,

[0146] (Appendix 9) A machine learning model that classifies input data into one of multiple classes is input with first input data that does not have a correct answer label and second input data whose correct answer label is known, Determining deterioration of the machine learning model based on the output result for the second input data and the correct label among the output results output from the machine learning model. A classification processing device comprising a control unit that executes processing.

[0147] (Appendix 10) The control unit When it is determined that the machine learning model has deteriorated, the machine learning model is restored. 10. The classification processing device according to claim 9, wherein the classification processing device executes a process.

[0148] (Appendix 11) The control unit Among the data output from the machine learning model, a first output result for the first input data is output as a classification result of the machine learning model. 11. The classification processing device according to claim 9 or 10, which executes processing.

[0149] (Appendix 12) The control unit Outputting the result of the determination of the deterioration of the machine learning model 10. The classification processing device according to claim 9, wherein the classification processing device executes a process.

[0150] (Appendix 13) The process of inputting into the machine learning model includes: and inputting the second input data prepared for each of the plurality of classes into the machine learning model. 10. The classification processing device according to claim 9,

[0151] (Appendix 14) The control unit updating the machine learning model by machine learning using the first input data and the second input data; Execute the process, The process of recovering the machine learning model includes: and a process of restoring the updated machine learning model to the machine learning model before the update. 9. The classification processing device according to claim 8,

[0152] (Appendix 15) The process of recovering the machine learning model includes: Increasing the number of second input data for which the correct answer label is known 11. The classification processing device according to claim 10, further comprising:

[0153] (Appendix 16) the input data is data to be inspected, the plurality of classes includes good products and bad products; The control unit is provided in a defective product detection system in which the machine learning model classifies the inspection object into either a good product or a defective product based on data of the inspection object. 10. The classification processing device according to claim 9,

[0154] (Appendix 17) A machine learning model that classifies input data into one of multiple classes is input with first input data that does not have a correct answer label and second input data whose correct answer label is known, Determining deterioration of the machine learning model based on the output result for the second input data and the correct label among the output results output from the machine learning model. A classification processing method characterized in that the processing is performed by a computer.

[0155] (Appendix 18) When it is determined that the machine learning model has deteriorated, the machine learning model is restored. 18. The classification processing method according to claim 17, wherein the processing is performed by the computer.

[0156] (Appendix 19) Among the data output from the machine learning model, a first output result for the first input data is output as a classification result of the machine learning model. 19. The classification processing method according to claim 17 or 18, wherein the processing is executed by the computer.

[0157] (Appendix 20) Outputting the result of the determination of the deterioration of the machine learning model 18. The classification processing method according to claim 17, wherein the processing is performed by the computer.

[0158] (Appendix 21) The process of inputting into the machine learning model includes: and inputting the second input data prepared for each of the plurality of classes into the machine learning model. 18. The classification processing method according to claim 17,

[0159] (Appendix 22) updating the machine learning model by machine learning using the first input data and the second input data; The processing is executed by the computer, The process of recovering the machine learning model includes: and a process of restoring the updated machine learning model to the machine learning model before the update. 19. The classification processing method according to claim 18,

[0160] (Appendix 23) The process of recovering the machine learning model includes: Increasing the number of second input data for which the correct answer label is known 19. The classification processing method according to claim 18, further comprising:

[0161] (Appendix 24) the input data is data to be inspected, the plurality of classes includes good products and bad products; The computer is provided in a defective product detection system in which the machine learning model classifies the inspection object into either a good product or a defective product based on data of the inspection object. 18. The classification processing method according to claim 17, [Explanation of symbols]

[0162] 1,1a Classification processing device 2. Defective product detection system 10. Computers 10a processor 10b GPU 10c memory 10d storage section 10e IF section 10f IO section 10g reading unit 10-hour program 10i recording media 10j bus 21 Image Inspection AI 22 High-resistance learning processing unit 101 Input processing section 102 Additional data generation unit 103 Model Generation Unit 104 Classification and Tuning Section 105 Additional Data Removal Unit 106 Output Processing Unit 107 Storage section 111 Input Data 112 Data Set 511 Additional Data 201 Classification Model 301 Training Data Set 302 Data set to be classified 303 Additional Data 304 Expanded Data Set 305,306 classification results 401 Model Deterioration Judgment Unit 402 Model Recovery Department

Claims

1. A machine learning model that classifies input data into one of a plurality of classes is provided with first input data that does not have a correct answer label and second input data whose correct answer label is known; Deterioration of the machine learning model is determined based on the output result for the second input data and the correct label among the output results output from the machine learning model. A classification processing program that causes a computer to execute processing.

2. When it is determined that the machine learning model has deteriorated, the machine learning model is restored.

2. The classification processing program according to claim 1, wherein the classification processing program is executed by the computer.

3. Among the data output from the machine learning model, a first output result for the first input data is output as a classification result of the machine learning model.

3. The classification processing program according to claim 1, wherein the classification processing program is executed by the computer.

4. Outputting the result of the determination of the deterioration of the machine learning model 2. The classification processing program according to claim 1, wherein the classification processing program is executed by the computer.

5. The process of inputting into the machine learning model includes: and inputting the second input data prepared for each of the plurality of classes into the machine learning model.

2. The classification processing program according to claim 1.

6. updating the machine learning model by machine learning using the first input data and the second input data; causing the computer to execute a process; The process of recovering the machine learning model includes: and a process of restoring the updated machine learning model to the machine learning model before the update.

3. The classification processing program according to claim 2.

7. The process of recovering the machine learning model includes: Increasing the number of second input data whose correct labels are known 3. The classification processing program according to claim 2, further comprising a process for:

8. the input data is data to be inspected, the plurality of classes includes good products and bad products; A defective product detection system in which the machine learning model classifies the inspection object into either a good product or a defective product based on data of the inspection object. The computer is provided in 2. The classification processing program according to claim 1.

9. A machine learning model that classifies input data into one of a plurality of classes is provided with first input data that does not have a correct answer label and second input data whose correct answer label is known; Deterioration of the machine learning model is determined based on the output result for the second input data and the correct label among the output results output from the machine learning model. A classification processing device comprising a control unit that executes processing.

10. A machine learning model that classifies input data into one of a plurality of classes is provided with first input data that does not have a correct answer label and second input data whose correct answer label is known; Deterioration of the machine learning model is determined based on the output result for the second input data and the correct label among the output results output from the machine learning model. A classification processing method characterized in that the processing is performed by a computer.

Images