Management device, inspection system, management method, and program
The management device and method address the challenge of detecting class distribution changes using Kullback-Leibler divergence and similarity measures to update the model, enhancing prediction accuracy and preventing missed retraining.
Patent Information
- Application Number
- JP2021185990
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-11-15
- Publication Date
- 2026-01-21
- Estimated Expiration
- 2041-11-15
AI Technical Summary
Existing technologies struggle to reliably detect changes in the distribution of classes classified by a learning model, leading to missed opportunities for retraining and inaccurate predictions due to concept drift.
A management device and method that utilize indices such as Kullback-Leibler divergence and similarity measures to compare confidence level distributions across time periods, determining whether to update a trained model based on these comparisons to adapt to changing data distributions.
Enhances the reliability of detecting changes in class distributions, preventing missed retraining opportunities and improving prediction accuracy by adapting the model to concept drift.
Smart Images
Figure 0007803090000002 
Figure 0007803090000003 
Figure 0007803090000004
Abstract
Description
[Technical Field]
[0001] The present invention relates to a management device, an inspection system, a management method, and a program. [Background technology]
[0002] Non-Patent Document 1 discloses a technology for detecting concept drift using the Kullback-Leibler divergence between the output of a classifier for reference data at the start of operation and the output of the classifier for data during operation. Patent Document 1 also discloses extracting data candidates that belong to a new group different from multiple groups from collected data. [Prior art document] [Patent documents] [Non-Patent Document 1] Tsutomu Ishida and five others, Proceedings of the 34th National Conference of the Japanese Society for Artificial Intelligence, "Evaluation of Concept Drift Detection Methods for Unlabeled Operational Data" [Patent Document 1] Patent No. 6865901 Summary of the Invention [Problem to be solved by the invention]
[0003] It is desirable to more reliably detect changes in the distribution of classes into which a data set is classified by a learning model. [Means for solving the problem]
[0004] A management device according to one aspect of the present invention may be a management device that manages updating of a trained model in a classification device that receives state data indicating the state of an object as input and outputs a class into which the state of the object is classified and a confidence level for the classified class in accordance with a trained model that classifies the state of the object into multiple classes. The management device may include a derivation unit that derives, for each class, a distribution of confidence levels for classification results of a group of state data in a first time period and a second time period. The management device may include a determination unit that determines whether to update the trained model based on a first comparison result based on a first index between a group of state data in a first time period in which the confidence level of the classification result is equal to or greater than a threshold and a group of state data in a second time period in which the confidence level of the classification result is equal to or greater than the threshold, and a second comparison result based on a second index different from the first index between a group of state data in a first time period in which the confidence level of the classification result is less than the threshold and a group of state data in a second time period in which the confidence level of the classification result is less than the threshold.
[0005] The first index may be the similarity between the distribution of the certainty of the classification results by class of a group of state data for a first period in which the certainty of the classification results is equal to or greater than a threshold, and the distribution of the certainty of the classification results by class of a group of state data for a second period in which the certainty of the classification results is equal to or greater than a threshold.
[0006] The first index may be the similarity between a normalized distribution of the certainty of the classification result for a group of state data for a first period in which the certainty of the classification result is equal to or greater than a threshold, and a normalized distribution of the certainty of the classification result for a group of state data for a second period in which the certainty of the classification result is equal to or greater than a threshold, respectively.
[0007] The first index may be the Kullback-Leibler divergence between the normalized class-by-class distribution of the confidence of the group of state data in the first period for which the confidence of the classification result is equal to or greater than a threshold, and the normalized class-by-class distribution of the confidence of the group of state data in the second period for which the confidence of the classification result is equal to or greater than a threshold.
[0008] The second indicator may be the difference between the ratio of the total number of status data groups in the first period for which the certainty of the classification result is less than a threshold to the total number of status data groups in the first period, and the ratio of the total number of status data groups in the second period for which the certainty of the classification result is less than a threshold to the total number of status data groups in the second period.
[0009] A management device according to one aspect of the present invention may be a management device that manages updating of a trained model in a classification device that receives state data indicating the state of an object as input and outputs a class into which the state of the object is classified and a confidence level for the classified class in accordance with a trained model that classifies the state of the object into multiple classes. The management device may include a derivation unit that derives, for each class, a distribution of confidence levels for classification results of multiple state data groups in a first time period and a second time period. The management device may include a determination unit that determines whether to update the trained model based on a first comparison result for each class between a normalized distribution of confidence levels for each class of state data groups in a first time period in which the confidence levels of the classification results are equal to or greater than a threshold and a normalized distribution of confidence levels for each class of state data groups in a second time period in which the confidence levels of the classification results are equal to or greater than the threshold.
[0010] The determination unit may determine whether or not to update the trained model based on a first comparison result for each class using Kullback-Leibler divergence between a normalized distribution for each class of the certainty of the state data group for a first time period in which the certainty of the classification result is equal to or greater than a threshold, and a normalized distribution for each class of the certainty of the state data group for a second time period in which the certainty of the classification result is equal to or greater than a threshold.
[0011] The determination unit may determine that the trained model should be updated if the ratio of the total number of state data groups in the second period for which the confidence level of the classification result is less than a threshold to the total number of state data groups in the second period is equal to or greater than a reference value.
[0012] The management device may further include a receiving unit that receives the threshold setting.
[0013] An inspection system according to one aspect of the present invention includes the management device and the classification device, and inspects objects based on classification results.
[0014] A management method according to one aspect of the present invention may be a management method for managing updates to a trained model in a classification device that receives input state data indicating the state of an object and outputs a class into which the state of the object is classified and a confidence level for the classified class in accordance with a trained model that classifies the state of the object into a plurality of classes. The management method may include a step of deriving a distribution of confidence levels for classification results of a group of state data in a first time period and a second time period for each class. The management method may include a step of determining whether to update the trained model based on a first comparison result, based on a first index, between a group of state data in the first time period in which the confidence level of the classification result is equal to or greater than a threshold, and a group of state data in the second time period in which the confidence level of the classification result is equal to or greater than the threshold, and a second comparison result, based on a second index different from the first index, between a group of state data in the first time period in which the confidence level of the classification result is less than the threshold, and a group of state data in the second time period in which the confidence level of the classification result is less than the threshold.
[0015] A management method according to one aspect of the present invention may be a management method for managing updating of a trained model in a classification device that receives state data indicating the state of an object as input and outputs a class into which the state of the object is classified and a confidence level for the classified class in accordance with a trained model that classifies the state of the object into a plurality of classes. The management method may include a step of deriving, for each class, a distribution of confidence levels for classification results of a plurality of state data groups in a first time period and a second time period. The management method may include a step of determining whether to update the trained model based on a first comparison result for each class between a normalized distribution of confidence levels for each class of state data groups in a first time period in which the confidence levels of the classification results are equal to or greater than a threshold and a normalized distribution of confidence levels for each class of state data groups in a second time period in which the confidence levels of the classification results are equal to or greater than the threshold.
[0016] A program according to one aspect of the present invention may cause a computer to function as a management device that manages updating of a trained model in a classification device that receives state data indicating the state of an object as input and outputs a class into which the state of the object is classified and a confidence level for the classified class according to a trained model that classifies the state of the object into multiple classes. The program may cause the computer to function as a derivation unit that derives, for each class, a distribution of confidence levels for classification results of state data groups in a first time period and a second time period. The program may cause the computer to function as a determination unit that determines whether to update the trained model based on a first comparison result based on a first index between a state data group in a first time period in which the confidence level of the classification result is equal to or greater than a threshold and a state data group in a second time period in which the confidence level of the classification result is equal to or greater than the threshold, and a second comparison result based on a second index different from the first index between a state data group in a first time period in which the confidence level of the classification result is less than the threshold and a state data group in a second time period in which the confidence level of the classification result is less than the threshold.
[0017] A program according to one aspect of the present invention may cause a computer to function as a management device that manages updating of a trained model in a classification device that receives state data indicating the state of an object as input and outputs a class into which the state of the object is classified and a confidence level for the classified class in accordance with a trained model that classifies the state of the object into multiple classes. The program may cause the computer to function as a derivation unit that derives, for each class, a distribution of confidence levels of classification results for multiple state data groups in a first time period and a second time period. The program may cause the computer to function as a determination unit that determines whether to update the trained model based on a first comparison result for each class between a normalized distribution of confidence levels for each class of state data groups in a first time period in which the confidence levels of the classification results are equal to or greater than a threshold and a normalized distribution of confidence levels for each class of state data groups in a second time period in which the confidence levels of the classification results are equal to or greater than a threshold.
[0018] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions. [Brief explanation of the drawings]
[0019] [Figure 1] FIG. 2 is a diagram illustrating an example of functional blocks of an inspection system. [Figure 2] FIG. 2 is a diagram showing a part of the operation flow of the inspection system 10. [Figure 3] FIG. 10 is a diagram showing the first half of the comparison determination performed by the determination unit. [Figure 4] FIG. 10 is a diagram showing the second half of the comparison and determination performed by the determination unit. [Figure 5] FIG. 10 is a diagram showing time series data of fluctuations in the determination results of the certainty distribution in an area where the certainty is smaller than a threshold value. [Figure 6] FIG. 10 is a diagram showing time series data of fluctuations in the determination results of the certainty distribution in an area where the certainty is equal to or greater than a threshold. [Figure 7] FIG. 2 illustrates an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION
[0020] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0021] FIG. 1 is a diagram showing an example of functional blocks of an inspection system 10 according to this embodiment. The inspection system 10 includes an inspection device 100 and a sensor 200. The sensor 200 detects the state of an object to be inspected. The sensor 200 may be an imaging device that captures an image of the object. The sensor 200 may be a sensor other than an imaging device, as long as it detects the desired state of the object, such as a temperature sensor that measures the temperature of the object or a distance sensor that measures the distance to the object.
[0022] The inspection device 100 determines the state of the object based on the state data indicating the state of the object from the sensor 200, and outputs the determination result of the state of the object. The inspection device 100 includes a collection unit 102, a derivation unit 104, a determination unit 106, a classification unit 110, a storage unit 120, and a display unit 130.
[0023] The collection unit 102 collects status data indicating the status of the object detected by the sensor 200. The collection unit 102 may collect image data indicating the status of the object captured by the sensor 200 as the status data.
[0024] The classification unit 110 receives state data indicating the state of the object according to a trained model that classifies the state of the object into a plurality of classes, and outputs the class into which the state of the object has been classified and the confidence level of the classified class as inspection results. The classes may be predetermined for each type of abnormal pattern of the object that can be distinguished based on the state data.
[0025] The collection unit 102 collects combinations of object state data and object classes as training data and stores the training data in the storage unit 120. The classification unit 110 generates a trained model that predicts the object class by performing machine learning according to a supervised learning algorithm using the object state data as an explanatory variable and the object class as a target variable, and stores the trained model in the storage unit 120. The trained model indicates the relationship between the object state data and the object class. The supervised learning algorithm may be any type of algorithm, such as a neural network, a support vector machine, multiple regression analysis, or a decision tree.
[0026] The classification unit 110 derives a certainty of a class into which the state data of the object is classified according to a predetermined judgment criterion. The certainty indicates the degree of reliability of a judgment result according to a trained model. The classification unit 110 may derive the certainty such that the greater the similarity between the state data of the object to be derived and the state data of the object indicated in the teacher data, the higher the certainty. The classification unit 110 may derive the certainty as the similarity between image data of the object to be derived and image data of the object indicated in the teacher data. The classification unit 110 may determine the class of the teacher data with the highest certainty among the certainties of the state data of the object to be derived and the state data of the object indicated in the teacher data of each class as the class of the state data of the object to be derived.
[0027] The inspection device 100 may, for example, determine an abnormal pattern that occurs in a product during the product inspection process according to a trained model. However, after starting to use the trained model for determination, the nature of the input condition data may change due to the emergence of a new abnormal pattern, etc. This phenomenon is called concept drift. If the inspection device 100 predicts an abnormal pattern in the condition data using a trained model that has experienced concept drift, the predicted result may be unexpected. Therefore, it is preferable to detect the occurrence of concept drift early while performing determination using the trained model and retrain the trained model.
[0028] Condition data that has been classified by the trained model into one of the abnormal pattern classes with low confidence can be said to be condition data whose characteristics differ from any of the abnormal patterns in the trained training data. It is likely that a certain amount of such condition data will occur in the manufacturing process.
[0029] Therefore, it is conceivable to use a change in the distribution shape of the confidence factors as an index for determining whether or not to retrain a trained model, as in Non-Patent Document 1. However, when monitoring changes in the overall shape of the distribution of the confidence factors to determine whether or not to retrain, differences in the magnitude of the confidence factors may result in differences in the change in the distribution shape when comparing a set of state data included in a reference distribution with a set of state data included in a distribution to be monitored.
[0030] The status data group included in the reference distribution is, for example, a status data group classified according to a trained model in a first period in the initial stage of operation. The status data group included in the distribution of the monitored object is, for example, a status data group classified according to a trained model in a second period after the first period in the initial stage of operation.
[0031] For example, while there are approximately the same number of low-certainty state data groups included in the reference distribution and the low-certainty state data groups included in the distribution of the monitored object, the distribution shape of the high-certainty state data groups included in the reference distribution and the distribution shape of the high-certainty state data groups included in the distribution of the monitored object may be similar, and the number of high-certainty state data groups included in the distribution of the monitored object may be fewer or more than the number of high-certainty state data groups included in the reference distribution.Even if the properties of the state data do not change, the number of state data groups classified into the corresponding class may fluctuate depending on the timing of the second period.
[0032] In such a case, when monitoring changes in the overall shape of the certainty distribution and determining whether or not re-learning is necessary, it is determined that re-learning is necessary. However, even if the number of high certainty state data groups included in the distribution to be monitored is smaller or larger than the number of high certainty state data groups included in the reference distribution, re-learning is not necessary as long as the distribution shape of the high certainty state data groups included in the reference distribution and the distribution shape of the high certainty state data groups included in the distribution to be monitored are similar.
[0033] Furthermore, when extracting data candidates belonging to a new group different from multiple groups from collected data, as in Patent Document 1, it becomes possible to detect the occurrence of a new anomaly that has characteristics different from known anomalies and notify the need for re-learning. However, with a method like Patent Document 1, it is not possible to detect changes in the distribution of known anomaly classes when the properties of known anomalies change, and it may not be possible to detect the need for re-learning.
[0034] Therefore, in this embodiment, changes in the distribution of classes into which state data groups are classified according to a trained model are detected more reliably, thereby preventing missed opportunities to retrain the trained model.
[0035] The derivation unit 104 derives a distribution of confidence levels of classification results of the status data group for each class in the first period and the second period. The first period may be an initial period from the start of classifying the status data group according to the trained model to a predetermined time. In other words, the first period may be a period from the start of inspecting the object to a predetermined time. The second period may be a predetermined period after the first period during the operation of classifying the status data group according to the trained model. In other words, the second period may be a predetermined period after the first period during the operation of inspecting the object. The second period may be the same period as the first period. The first period and the second period may be one day, one week, one month, or the like.
[0036] The derivation unit 104 may derive a histogram for each class, which indicates the distribution of the certainty of the classification results of the state data groups in the first period and the second period. The histogram indicates the number of state data groups for each certainty.
[0037] The determination unit 106 determines whether or not to update the trained model based on the distribution of the confidence levels of the classification results of the state data group in the first period and the second period. The determination unit 106 may determine whether or not to update the trained model based on a first comparison result between the distribution of the confidence levels of the state data group in the first period, in which the confidence levels of the classification results are equal to or greater than a first threshold, and the distribution of the confidence levels of the state data group in the second period, in which the confidence levels of the classification results are equal to or greater than the first threshold.
[0038] The determination unit 106 may determine whether or not to update the trained model based on a first comparison result based on a first index between a group of state data from a first period in which the certainty of the classification result is equal to or greater than a first threshold and a group of state data from a second period in which the certainty of the classification result is equal to or greater than the first threshold, and a second comparison result based on a second index different from the first index between a group of state data from a first period in which the certainty of the classification result is less than the first threshold and a group of state data from a second period in which the certainty of the classification result is less than the first threshold.
[0039] The first index may be a similarity between a distribution of certainty levels for each class of a group of state data for a first period in which the certainty level of the classification result is equal to or greater than a first threshold and a distribution of certainty levels for each class of a group of state data for a second period in which the certainty level of the classification result is equal to or greater than the first threshold. The first index may be a similarity between a histogram of certainty levels for each class of a group of state data for a first period in which the certainty level of the classification result is equal to or greater than the first threshold and a histogram of certainty levels for each class of a group of state data for a second period in which the certainty level of the classification result is equal to or greater than the first threshold. The similarity may be a degree of overlap between the histograms. The determination unit 106 may determine that the trained model should be updated if the similarity between the distribution of certainty levels for a class of a group of state data for a first period in which the certainty level of the classification result is equal to or greater than the first threshold and the distribution of certainty levels for that class of a group of state data for a second period in which the certainty level of the classification result is equal to or greater than the first threshold is smaller than a reference value.
[0040] The first index may be a similarity between a normalized distribution of certainty for each class of a group of state data for a first period in which the certainty of the classification result is equal to or greater than a first threshold, and a normalized distribution of certainty for each class of a group of state data for a second period in which the certainty of the classification result is equal to or greater than the first threshold. The first index may be a similarity between each distribution normalized with the total number of certainty for each class of a group of state data for a first period in which the certainty of the classification result is equal to or greater than the first threshold set to 1, and each distribution normalized with the total number of certainty for each class of a group of state data for a second period in which the certainty of the classification result is equal to or greater than the first threshold set to 1. The first threshold may be a lower limit value of a range of the mean value ± standard deviation σ of the normal distribution of certainty.
[0041] The first index may be the Kullback-Leibler divergence between a normalized distribution of the confidence levels of a group of state data for a first period in which the confidence levels of the classification results are equal to or greater than a first threshold, and a normalized distribution of the confidence levels of a group of state data for a second period in which the confidence levels of the classification results are equal to or greater than the first threshold. The determination unit 106 may determine that the trained model should be updated if the Kullback-Leibler divergence of a certain class is smaller than a reference value.
[0042] The second indicator may be the difference between the ratio of the total number of state data groups in the first period for which the confidence level of the classification result is less than the first threshold to the total number of state data groups in the first period, and the ratio of the total number of state data groups in the second period for which the confidence level of the classification result is less than the first threshold to the total number of state data groups in the second period. The determination unit 106 may determine that the trained model should be updated if the difference between the ratios is greater than a reference value. Focusing only on the second period, the determination unit 106 may determine that the trained model should be updated if the ratio of the total number of state data groups in the second period for which the confidence level of the classification result is less than the first threshold to the total number of state data groups in the second period is equal to or greater than a reference value.
[0043] If the judgment unit 106 determines that the similarity between the state data group for the first period and the state data group for the second period is equal to or less than a threshold based on the first index or the second index, that is, that the state data group for the second period has changed with respect to the state data group for the first period, the judgment unit 106 may cause the display unit 130 to display an alert indicating that the trained model should be updated.
[0044] 2 shows a part of the operation flow of the inspection system 10. The collection unit 102 collects status data of the object via the sensor 200. The reference data group 310 is the status data group 31 collected in a first period. The monitored data group 311 is the status data group 31 collected in a second period.
[0045] The classification unit 110 classifies the reference data group 310 and the monitored data group 311 into their respective classes using the trained model 410, labels them, and obtains a prediction result 411 (41). The prediction result 411 indicates the confidence level of each class of the state data and the predicted class, i.e., the predicted label of the class with the highest confidence level among the confidence levels of the classes.
[0046] The derivation unit 104 divides each of the reference data group 310 and the monitored data group 311 into predicted labels according to the prediction results 411, and derives a confidence distribution 421 for each predicted label (42).
[0047] 3 shows the first half of the comparison and determination performed by the determination unit 106. The determination unit 106 derives the number of state data groups whose confidence level is smaller than a predetermined confidence level threshold for each confidence level distribution 4211a of each predicted label obtained from the reference data set 310 and each confidence level distribution 4211b of each predicted label obtained from the monitored data set 311, and records the number of state data groups whose confidence level is smaller than a predetermined confidence level threshold in the storage unit 120.
[0048] Furthermore, the determination unit 106 derives the total number of state data groups smaller than the threshold for all predicted labels and the total number of state data groups for all predicted labels for each of the reference data group 310 and the monitoring target data group 311. Then, the determination unit 106 derives the ratio of the total number of state data groups smaller than the threshold for all predicted labels to the total number of state data groups for all predicted labels. That is, the determination unit 106 derives a ratio 4212a of reference data groups smaller than the threshold and a ratio 4212b of monitoring target data groups smaller than the threshold.
[0049] Next, the determination unit 106 calculates the difference between the proportion 4212a of the reference data group that is smaller than the threshold and the proportion 4212b of the monitoring target data group that is smaller than the threshold (4213). The determination unit 106 determines whether or not re-learning of the trained model is necessary based on the difference (4214). If the difference is equal to or greater than the reference value for determining whether or not re-learning is necessary, the determination unit 106 determines that re-learning of the trained model is necessary due to a change in the distribution of classes of the state data group.
[0050] A state data group whose confidence level is smaller than the threshold is a state data group whose characteristics are dissimilar to the abnormal pattern shown in the training data. By the process shown in Figure 3, the determination unit 106 detects changes in the characteristics of the state data group whose characteristics are dissimilar to the abnormal pattern shown in the training data, and determines whether or not re-learning of the trained model is necessary.
[0051] The confidence threshold is a common value for the same predicted label in both the reference data set 310 and the monitored data set 311. However, the confidence threshold may be a different value for each predicted label. For example, if the distribution of confidence in the reference data set 310 is a normal distribution, the confidence threshold may be the lower limit of the range of the mean value ± standard deviation σ of the normal distribution.
[0052] 4 shows the second half of the comparison and determination performed by the determination unit 106. The determination unit 106 calculates (4222) the change between the distribution shape of the portion of the confidence distribution 4221a of the predicted label A obtained from the reference data group 310 where the confidence is equal to or greater than a threshold and the distribution shape of the portion of the confidence distribution 4221b of the predicted label A obtained from the monitored data group 311 where the confidence is equal to or greater than a threshold. That is, the determination unit 106 monitors the change in the portion of the confidence distribution 4221b of the monitored data group 311 where the confidence is equal to or greater than a threshold relative to the portion of the confidence distribution 4221a of the reference data group 310 where the confidence is equal to or greater than a threshold, determines whether retraining of the trained model is necessary, and outputs the determination result 4223. The determination result 4223 indicates the degree of change in the distribution shape for each predicted label.
[0053] The determination unit 106 may use Kullback-Leibler divergence as an index for quantifying a change in the distribution shape to determine the degree of similarity between the confidence distribution 4221a of the reference data group 310 and the confidence distribution 4221b of the monitored data group 311. That is, the determination unit 106 may determine whether or not there is a change in the confidence distribution 4221b of the monitored data group 311 relative to the confidence distribution 4221a of the reference data group 310, according to the Kullback-Leibler divergence. If the Kullback-Leibler divergence is smaller than a reference value, the determination unit 106 may determine that the trained model should be updated.
[0054] The Kullback-Leibler divergence applied to a discretely quantized distribution such as a histogram is given by:
number
[0055] "i" indicates the number of each bin in the histogram. "p" and "q" correspond to the histogram of the reference data group and the histogram of the monitored data group, respectively. "p(i)" and "q(i)" indicate the data density in each bin after normalization so that the number of data in the entire histogram becomes 1. In the region where the certainty of the histogram is equal to or greater than a threshold, the determination unit 106 normalizes the number of data and then compares the distribution shapes. This allows the determination unit 106 to detect changes in the properties of known abnormal patterns regardless of fluctuations in the number of data.
[0056] 5 and 6 show examples of time series data of changes in the fluctuation of the determination result by the determination unit 106. FIG. 5 shows time series data of fluctuations in the determination result of the certainty distribution in an area where the certainty is smaller than a threshold. FIG. 6 shows time series data of fluctuations in the determination result of the certainty distribution in an area where the certainty is equal to or larger than a threshold. The determination unit 106 may determine whether or not to issue an alert indicating that re-learning is required, according to the time series data of fluctuations in the determination result of the certainty distribution in an area where the certainty is smaller than a threshold, shown in FIG. 5. Furthermore, the determination unit 106 may determine whether or not to issue an alert indicating that re-learning is required, according to the time series data of fluctuations in the determination result of the certainty distribution in an area where the certainty is equal to or larger than a threshold, shown in FIG.
[0057] The determination unit 106 determines whether re-learning is necessary using different indices for different levels of certainty, and determines whether re-learning is necessary for each predicted label in areas where the certainty is equal to or greater than a threshold. This allows the inspection system 10 to accurately determine whether re-learning is necessary and, if necessary, to notify the user of an alert indicating that re-learning is necessary.
[0058] The display unit 130 may display a message according to the determination result of the determination unit 106. When the determination unit 106 determines that re-learning is necessary for the region where the certainty is equal to or greater than the threshold, the display unit 130 may display time-series data of fluctuations in the determination result of the certainty distribution in the region where the certainty is equal to or greater than the threshold. When the determination unit 106 determines that re-learning is necessary for the region where the certainty is equal to or greater than the threshold, the display unit 130 may display state data of each object where the certainty is equal to or greater than the threshold. When the determination unit 106 determines that re-learning is necessary for the region where the certainty is equal to or greater than the threshold, the display unit 130 may sequentially switch between displaying state data of each object where the certainty is equal to or greater than the threshold.
[0059] Furthermore, when the determination unit 106 determines that re-learning is necessary for an area where the certainty factor is smaller than the threshold, the display unit 130 may display time-series data of fluctuations in the determination results of the certainty factor distribution in the area where the certainty factor is smaller than the threshold. When the determination unit 106 determines that re-learning is necessary for an area where the certainty factor is smaller than the threshold, the display unit 130 may display state data of each object where the certainty factor is smaller than the threshold.
[0060] In this way, by presenting the operator with at least one of the time series data and status data that caused the alert indicating the need for re-learning, the final decision on whether or not re-learning is necessary can be left to the operator.
[0061] Furthermore, the inspection device 100 may further include a receiving unit 108 that receives, from an operator, the setting of a threshold value used to determine whether relearning is necessary. For example, if the detection sensitivity of the relearning alert is higher than necessary, the operator can take measures such as raising the threshold value, thereby enabling the relearning alert to be issued at a more optimal timing.
[0062] 7 illustrates an example of a computer 1200 in which aspects of the present invention may be embodied, in whole or in part. A program installed on the computer 1200 may cause the computer 1200 to perform operations associated with an apparatus according to an embodiment of the present invention or to function as one or more “parts” of the apparatus. Alternatively, the program may cause the computer 1200 to perform the operations or one or more “parts” of the apparatus. The program may cause the computer 1200 to perform a process or steps of a process according to an embodiment of the present invention. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.
[0063] The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are interconnected by a host controller 1210. The computer 1200 also includes a communication interface 1222 and an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit.
[0064] The communication interface 1222 communicates with other electronic devices via a network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores a boot program executed by the computer 1200 upon activation and / or programs dependent on the computer's hardware. The programs may be provided via a computer-readable recording medium such as a CD-ROM, USB memory, or IC card, or via a network. The programs may be installed in the RAM 1214 or the ROM 1230, which are also examples of computer-readable recording media, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and establishes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.
[0065] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.
[0066] The CPU 1212 may also cause all or a necessary portion of a file or database stored on an external recording medium such as a USB memory to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.
[0067] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.
[0068] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.
[0069] A computer-readable medium may include any tangible device capable of storing instructions that are executed by an appropriate device. As a result, the computer-readable medium with instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated circuit card, etc.
[0070] The computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. The source code or object code includes conventional procedural programming languages. The conventional procedural programming languages may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and the “C” programming language or similar programming languages. The computer-readable instructions may be provided to a processor or programmable circuitry of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus locally or over a wide-area network (WAN) such as a local area network (LAN), the Internet, etc. The processor or programmable circuitry may execute the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
[0071] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.
[0072] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]
[0073] 10 Inspection System 31 Status Data Group 100 Inspection equipment 102 Collection Department 104 Derivation part 106 Judgment section 108 Reception 110 Classification Department 120 Storage section 130 Display section 200 sensors 1200 Computer 1210 host controller 1212 CPU 1214 RAM 1220 Input / Output Controller 1222 communication interface 1230 ROM
Claims
1. A management device that manages updates of a trained model in a classification device that receives state data indicating a state of an object as input and outputs a class into which the state of the object is classified and a confidence level of the classified class, according to a trained model that classifies the state of the object into a plurality of classes, a derivation unit that derives a distribution of certainty factors of classification results of the state data group in the first period and the second period for each class; a determination unit that determines whether or not the trained model should be updated based on a first comparison result based on a first index between a state data group of the first period in which the certainty of the classification result is equal to or greater than a threshold and a state data group of the second period in which the certainty of the classification result is equal to or greater than the threshold, and a second comparison result based on a second index different from the first index between a state data group of the first period in which the certainty of the classification result is less than the threshold and a state data group of the second period in which the certainty of the classification result is less than the threshold; A management device comprising:
2. 2. The management device according to claim 1, wherein the first index is a similarity between a distribution of certainty by class of the group of status data for the first period in which the certainty of the classification result is equal to or greater than a threshold, and a distribution of certainty by class of the group of status data for the second period in which the certainty of the classification result is equal to or greater than a threshold.
3. 2. The management device according to claim 1, wherein the first index is a similarity between a normalized distribution of the certainty of the classification result for the group of status data for the first period for which the certainty of the classification result is equal to or greater than a threshold, and a normalized distribution of the certainty of the classification result for the group of status data for the second period for which the certainty of the classification result is equal to or greater than a threshold.
4. 2. The management device according to claim 1, wherein the first index is a Kullback-Leibler divergence of a normalized class-by-class distribution of the certainty of the state data group for the first period in which the certainty of the classification result is equal to or greater than a threshold, and a normalized class-by-class distribution of the certainty of the state data group for the second period in which the certainty of the classification result is equal to or greater than a threshold.
5. 5. The management device according to claim 1, wherein the second index is the difference between the ratio of the total number of status data groups in the first period whose classification result confidence is less than the threshold value to the total number of status data groups in the first period, and the ratio of the total number of status data groups in the second period whose classification result confidence is less than the threshold value to the total number of status data groups in the second period.
6. A management device that manages updates of a trained model in a classification device that receives state data indicating a state of an object as input and outputs a class into which the state of the object is classified and a confidence level of the classified class, according to a trained model that classifies the state of the object into a plurality of classes, a derivation unit that derives a distribution of certainty factors of classification results of a plurality of state data groups in a first period and a second period for each class; a determination unit that determines whether or not the trained model should be updated based on a first comparison result for each class between a normalized distribution of the certainty of the state data group for the first period, in which the certainty of the classification result is equal to or greater than a threshold, and a normalized distribution of the certainty of the state data group for the second period, in which the certainty of the classification result is equal to or greater than a threshold; A management device comprising:
7. 7. The management device according to claim 6, wherein the determination unit determines whether to update the trained model based on the first comparison result for each class using Kullback-Leibler divergence between a normalized distribution for each class of the certainty of the state data group for the first time period in which the certainty of the classification result is equal to or greater than a threshold and a normalized distribution for each class of the certainty of the state data group for the second time period in which the certainty of the classification result is equal to or greater than a threshold.
8. 8. The management device according to claim 6, wherein the determination unit determines that the trained model should be updated when a ratio of the total number of state data groups in the second period for which the certainty of the classification result is smaller than the threshold to the total number of state data groups in the second period is equal to or greater than a reference value.
9. The management device according to claim 1 , further comprising a receiving unit that receives the setting of the threshold value.
10. 10. An inspection system comprising: the management device according to claim 1; and the classification device; and inspecting an object based on a classification result.
11. A management method in which a management device manages updates of a trained model in a classification device that receives state data indicating a state of an object as input and outputs a class into which the state of the object is classified and a confidence level of the classified class, according to a trained model that classifies the state of the object into a plurality of classes, a step in which the management device derives a distribution of certainty of classification results of the status data group in the first period and the second period for each class; a step in which the management device determines whether or not to update the trained model based on a first comparison result based on a first index between a state data group of the first period in which the certainty of the classification result is equal to or greater than a threshold and a state data group of the second period in which the certainty of the classification result is equal to or greater than a threshold, and a second comparison result based on a second index different from the first index between a state data group of the first period in which the certainty of the classification result is less than the threshold and a state data group of the second period in which the certainty of the classification result is less than the threshold; A management method comprising:
12. A management method in which a management device manages updates of a trained model in a classification device that receives state data indicating a state of an object as input and outputs a class into which the state of the object is classified and a confidence level of the classified class, according to a trained model that classifies the state of the object into a plurality of classes, a step in which the management device derives a distribution of certainty factors of classification results of a plurality of status data groups in a first period and a second period for each class; a step in which the management device determines whether or not to update the trained model based on a first comparison result for each class between a normalized distribution of the certainty of the state data group for the first period in which the certainty of the classification result is equal to or greater than a threshold and a normalized distribution of the certainty of the state data group for the second period in which the certainty of the classification result is equal to or greater than a threshold; A management method comprising:
13. A program for causing a computer to function as a management device that manages updating of a trained model in a classification device that receives state data indicating a state of an object and outputs a class into which the state of the object is classified and a confidence level of the classified class, in accordance with a trained model that classifies the state of the object into a plurality of classes, a derivation unit that derives a distribution of certainty factors of classification results of the state data group in the first period and the second period for each class; a determination unit that determines whether or not the trained model should be updated based on a first comparison result based on a first index between a state data group of the first period in which the certainty of the classification result is equal to or greater than a threshold and a state data group of the second period in which the certainty of the classification result is equal to or greater than the threshold, and a second comparison result based on a second index different from the first index between a state data group of the first period in which the certainty of the classification result is less than the threshold and a state data group of the second period in which the certainty of the classification result is less than the threshold; and causing the computer to function.
14. A program for causing a computer to function as a management device that manages updating of a trained model in a classification device that receives state data indicating a state of an object and outputs a class into which the state of the object is classified and a confidence level of the classified class, in accordance with a trained model that classifies the state of the object into a plurality of classes, a derivation unit that derives a distribution of certainty factors of classification results of a plurality of state data groups in a first period and a second period for each class; a determination unit that determines whether or not the trained model should be updated based on a first comparison result for each class between a normalized distribution of the certainty of the state data group for the first period, in which the certainty of the classification result is equal to or greater than a threshold, and a normalized distribution of the certainty of the state data group for the second period, in which the certainty of the classification result is equal to or greater than a threshold; and causing the computer to function.