Model update device, model update method and program

Abstract

To provide a technology that can achieve a model update capable of appropriately dealing with a fall in model accuracy.SOLUTION: A model update device according to one embodiment of the present disclosure has: a first calculation unit that calculates an inter-distribution distance in a feature amount space between a plurality of data sets including a learning data set used in parameter learning of a machine learning model; a second calculation unit that calculates an expression representing a relation of the inter-distribution distance with a second index value representing a degree of fall in a first index value, using the inter-distribution distance and the first index value; a third calculation unit that calculates an object inter-distribution distance in a feature amount space between an operation data set obtained after starting an operation and the learning data set; a fourth calculation unit that calculates the second index value about the operation data set, using the object inter-distribution distance and the expression; a determination unit that determines whether the second index value exceeds a threshold; and an update unit that, when it is determined that the second index value does not exceed the threshold, updates the parameter of the machine learning model by a teacherless learning method, using the operation data set.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a model updating device, a model updating method, and a program. [Background technology]

[0002] Classification of product defect patterns and the like is currently being carried out using classification models created using machine learning technology. However, after the classification model is put into operation, its accuracy may decrease due to drifts such as concept drift and data drift. In response to this, methods of updating models to deal with the decrease in model accuracy due to drift are known (e.g., Patent Document 1, Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2023 / 013024 [Non-patent literature]

[0004] [Non-Patent Document 1] Yoshihiro Okawa, Kenichi Kobayashi, "Recent research trends and their considerations on unsupervised adaptation techniques to data changes," Proceedings of the 36th National Conference of the Japanese Society for Artificial Intelligence Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, model updating does not necessarily improve the accuracy of the model. For example, the method described in Patent Document 1 uses a detection model that inputs a feature and detects whether the feature is a feature of training data or operational data, but the accuracy of the model does not necessarily improve depending on the performance of this detection model. Also, for example, the method described in Non-Patent Document 1 uses unsupervised domain adaptation, but the accuracy of the model does not necessarily improve depending on the data used for unsupervised domain adaptation.

[0006] The present disclosure has been made in consideration of the above points, and provides a technology that can realize model updating that can appropriately deal with a decrease in model accuracy. [Means for solving the problem]

[0007] A model updating device according to an aspect of the present disclosure includes a first calculation unit that calculates a distribution distance in a feature space between a plurality of datasets including a training dataset used in parameter learning of a machine learning model, a second calculation unit that uses the distribution distance calculated by the first calculation unit and a first index value that represents a model accuracy of the machine learning model for each of the plurality of datasets to calculate an equation that represents a relationship between the distribution distance and a second index value that represents a degree of decrease in the first index value for the training dataset, and an operation dataset obtained after operation of the machine learning model is started, and the training dataset. a third calculation unit that calculates a distribution distance in a feature space between a target distribution and a target set as a target distribution distance; a fourth calculation unit that calculates the second index value for the operational dataset using the target distribution distance calculated by the third calculation unit and the formula; a determination unit that determines whether the second index value calculated by the fourth calculation unit exceeds a predetermined threshold value; and an update unit that updates parameters of the machine learning model by an unsupervised learning method using the operational dataset when it is determined that the second index value calculated by the fourth calculation unit does not exceed the threshold value. Effect of the Invention

[0008] A technique is provided that can realize model updating that can appropriately deal with a decrease in model accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of the classification device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of the classification device according to the present embodiment. [Diagram 3] 10 is a flowchart illustrating an example of a classification process according to the embodiment. [Figure 4] 11 is a flowchart illustrating an example of a model update process according to the present embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a linear approximation formula representing the relationship between the distribution distance and the accuracy rate drop point. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An embodiment of the present invention will be described below. In the following embodiment, assuming a case where data to be classified is classified using a classification model created using machine learning technology, a classification device 10 will be described that can perform model updating capable of appropriately dealing with a decrease in accuracy of the classification model due to the occurrence of drift, etc. The drift includes concept drift and data drift.

[0011] <Example of Hardware Configuration of Classification Device 10> An example of a hardware configuration of a classification device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the classification device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0012] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc.

[0013] The external I / F 103 is an interface with an external device such as a recording medium 103a. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0014] The communication I / F 104 is an interface for connecting the classification device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The processor 108 is various types of arithmetic devices such as a central processing unit (CPU) or a graphic processing unit (GPU).

[0015] 1 is merely an example, and the hardware configuration of the classification device 10 is not limited thereto. For example, the classification device 10 may have multiple auxiliary storage devices 107 and multiple processors 108, may not have some of the hardware shown in the figure, or may have various types of hardware other than the hardware shown in the figure.

[0016] <Example of functional configuration of classification device 10> An example of the functional configuration of the classification device 10 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the classification device 10 according to this embodiment includes a data acquisition unit 201, a feature extraction unit 202, a classification unit 203, a feature / classification result storage unit 204, a comparison operation unit 205, a performance inference unit 206, an update determination unit 207, a model update unit 208, and a display control unit 209. Each of these units is realized, for example, by a process in which one or more programs installed in the classification device 10 are executed by the processor 108 or the like. Furthermore, the classification device 10 according to this embodiment includes an operation database 210 and a reference database 211. Each of these databases is realized, for example, by a storage area of ​​the auxiliary storage device 107 or the like.

[0017] At least one of the operational database 210 and the reference database 211 may be realized by, for example, a storage area of ​​a storage device communicatively connected to the classification device 10.

[0018] The data acquisition unit 201 acquires data to be classified given to the classification device 10 as operational data. The data acquisition unit 201 also stores the operational data in the operational database 210. Here, the data to be classified refers to any data to be classified by the classification device 10. A typical example of data to be classified is image data of a product that is the subject of a visual inspection. However, this is only one example, and the data to be classified may be any data acquired by measuring, photographing, or the like, an object.

[0019] The feature extraction unit 202 extracts features from the operation data acquired by the data acquisition unit 201. Hereinafter, the features extracted from the operation data are also referred to as "operation data features." Note that a feature is information that represents the characteristics of data, and is expressed by a predetermined n-dimensional vector. Also, the n-dimensional space represented by the values ​​that the feature can take is called a feature space.

[0020] The classification unit 203 classifies the operational data features extracted by the feature extraction unit 202 into a plurality of predetermined classes. That is, the classification unit 203 receives the operational data features as input and calculates a label indicating to which of the plurality of classes the operational data features belong. Hereinafter, the predetermined number of classes is K, and each class is {1, , K}.

[0021] Here, the feature extraction unit 202 and the classification unit 203 configure the classification model M. The classification model M is a machine learning model including a neural network or the like. For example, the feature extraction unit 202 is realized by an intermediate layer immediately before the input layer to the output layer of the classification model M, and the classification unit 203 is realized by the output layer. For example, the output layer may be a linear transformation layer having a sigmoid function as an activation function when performing binary classification (two-class classification) using the classification model M, or a linear transformation layer having a softmax function as an activation function when performing multi-class classification using the classification model M. The input layer of the classification model M may adopt any configuration depending on the format of the operational data, and the intermediate layer may also adopt any configuration.

[0022] The classification model M is, for example, a data set that is given divided into three data sets, a training data set, a validation data set, and a test data set, and is trained in advance using the training data set, the validation data set, and the test data set. The training data set is a data set for learning (updating) the learnable parameters (hereinafter also simply referred to as parameters) of the classification model M. The validation data set is a data set for verifying and adjusting the hyperparameters, etc. of the classification model M. The test data set is a data set for evaluating the generalization performance, etc. of the classification model M.

[0023] Below, the training data set is X 1 ={(x i ,y i )|i=1,...,N 1}, the validation dataset is X 2 ={(x i ,yi )|i=1,...,N 2}, test dataset X 3 ={(x i ,y i )|i=1,...,N 3} where x i ∈X 1 is the data used for parameter learning of the classification model M, hereafter referred to as the learning data. Similarly, x i ∈X 2 is the data used to verify and adjust the hyperparameters of the classification model M, and will be referred to as the verification data hereafter. Similarly, x i ∈X 3 is data used to evaluate the generalization performance of the classification model M, and will be referred to as test data below. i is the training data, validation data, or test data x i is the label (correct label) that indicates which class N belongs to. 1 , N 2 and N 3 are the number of training data, the number of validation data, and the number of test data, respectively. Note that the correct answer label may be called teacher data or teacher label, etc.

[0024] Furthermore, in the following, the feature extraction unit 202 extracts the learning data x i The features extracted from are called training data features, and the set of training data features is called Z 1 ={z i |i=1,...,N 1 Similarly, the feature extraction unit 202 extracts the verification data x i The features extracted from are called validation data features, and the validation data feature set is called Z 2 ={z i |i=1,...,N 2} Similarly, the test data x i The features extracted from are called test data features, and the set of test data features is called Z 3 ={z i |i=1,...,N 3} where z i ∈Z1 is the training data feature, z i ∈Z 2 is the validation data feature, z i ∈Z 3 are the test data features.

[0025] The feature amount / classification result storage unit 204 stores the operation data feature amount extracted by the feature amount extraction unit 202 and the label calculated by the classification unit 203 in the operation database 210 .

[0026] The comparison calculation unit 205 calculates the distribution distance between the operation data feature amount and the training data feature amount (hereinafter referred to as the target distribution distance). The comparison calculation unit 205 also calculates the distribution distance between the training data feature amount (hereinafter referred to as the first distribution distance), the distribution distance between the training data feature amount and the verification data feature amount (hereinafter referred to as the second distribution distance), and the distribution distance between the training data feature amount and the test data feature amount (hereinafter referred to as the third distribution distance).

[0027] The distribution distance between the operation data feature values ​​and the training data feature values ​​can also be referred to as the distribution distance in the feature space between the operation data and the training data. Similarly, the distribution distance between the training data feature values, the distribution distance between the training data feature values ​​and the validation data feature values, and the distribution distance between the training data feature values ​​and the test data feature values ​​can also be referred to as the distribution distance in the feature space between the training data, the training data and the validation data, and the training data and the test data, respectively.

[0028] The performance inference unit 206 calculates the first inter-distribution distance, the second inter-distribution distance, the third inter-distribution distance, and the learning data set X 1 The accuracy rate of the classification model M for the validation dataset X 2 The accuracy rate of the classification model M for the test data X 3and the accuracy rate of classification model M for the above, to calculate a linear approximation formula for inferring the accuracy degradation of classification model M. Furthermore, performance inference unit 206 uses the target distribution distance and the linear approximation formula to calculate an accuracy rate degradation point (hereinafter referred to as a target accuracy rate degradation point) as an index value indicating the degree to which the accuracy of classification model M has declined.

[0029] The accuracy rate is a value that represents the proportion of correctly classified data among the data included in a certain data set when each data included in the data set is classified by the classification model M. The accuracy rate is often expressed as a value between 0 and 1, but in the following, the accuracy rate is calculated by multiplying the accuracy rate expressed as a value between 0 and 1 by 100, and the accuracy rate is expressed as a percentage (%).

[0030] The update determination unit 207 uses the target accuracy rate decline point calculated by the performance estimation unit 206 and a predetermined threshold to determine the update method for the classification model M. That is, if the target accuracy rate decline point does not exceed the threshold, the update determination unit 207 determines to update the classification model M by automatic model update, and if the target accuracy rate decline point exceeds the threshold, the update determination unit 207 determines to update the classification model M by manual model update.

[0031] When the update determination unit 207 determines that the classification model M should be updated by automatic model update, the model update unit 208 uses the operational data stored in the operational database 210 to update the parameters of the classification model M by an unsupervised learning technique (e.g., unsupervised domain adaptation, etc.).

[0032] When the update determination unit 207 determines that the classification model M is to be updated through a manual model update, the display control unit 209 displays information indicating that a manual model update is necessary (manual model update alert) on the display device 102. This allows the user to know that a manual model update is necessary, and therefore, for example, it becomes possible to assign labels to all or part of the operation data stored in the operation database 210, and then use the operation data and its labels to update the parameters of the classification model M through a supervised learning technique.

[0033] In addition, the display control unit 209 may display at least one of information such as the classification result by the classification unit 203, the target accuracy rate reduction point calculated by the performance inference unit 206, the judgment result by the update judgment unit 207, the update result by the model update unit 208, etc.

[0034] The display control unit 209 may display the above-mentioned various pieces of information on a display of a terminal device communicatively connected to the classification device 10, for example.

[0035] The operation database 210 stores operation data, operation data feature quantities, and labels. i (i=1,...,N 4 ), and the feature extraction unit 202 extracts the operational data x i The features extracted from z i , the label calculated by the classification unit 203 is denoted by y i Then, the operational database 210 contains the operational data set X 4 ={x i |i=1,...,N 4} and the operational data feature set Z 4 ={z i |i=1,...,N 4} and the classification result set Y 4 ={y i |i=1,...,N 4} is stored. Here, N 4 is the number of operational data.

[0036] The reference database 211 stores the mean and standard deviation of the training data feature values ​​for each class, the mean and standard deviation of the validation data feature values ​​for each class, and the mean and standard deviation of the test data feature values ​​for each class. i ∈Z 1 The mean and standard deviation of the validation data feature z i ∈Z 2 The mean and standard deviation of the test data feature z i ∈Z 3 The mean and standard deviation of the training data feature z i ∈Z 1 The average of μ 1,k , standard deviation is σ 1,k Similarly, the validation data feature z i ∈Z 2 The average of μ 2,k , standard deviation is σ 2,k Similarly, the test data feature z i ∈Z 3 The average of μ 3,k , standard deviation is σ 3,k Let us assume that.

[0037] In addition, the reference database 211 includes a learning data set X 1 The accuracy rate of the classification model M for the validation dataset X 2 The accuracy rate of the classification model M for the test data X 3 The accuracy rate of the classification model M for the training data set X 1 The accuracy rate of classification model M for ACC 1 , validation dataset X 2 The accuracy rate of classification model M for ACC 2 , test data X 3 The accuracy rate of classification model M for ACC 3 Let us assume that.

[0038] Here, the accuracy rate ACC 1 , ACC 2 and ACC 3 is calculated by the comparison calculation unit 205 after parameter learning of the classification model M, and is stored in the reference database 211. Similarly, the above average μ 1,k , μ 2,k , μ 3,k and variance σ 1,k , σ 2,k , σ 3,k is calculated by the comparison calculation unit 205 after parameter learning of the classification model M, for example, and is stored in the reference database 211.

[0039] The average and standard deviation of the training data feature amounts, the validation data feature amounts, and the test data feature amounts are examples of statistics of the feature amounts, and statistics other than the average and the standard deviation may be used depending on the type of distribution distance calculated by the comparison calculation unit 205.

[0040] <Classification process> The classification process according to this embodiment will be described below with reference to Fig. 3. In the following, it is assumed that data to be classified is provided to the classification device 10.

[0041] First, the data acquiring unit 201 acquires the given data to be classified as operational data (step S101). Hereinafter, the operational data acquired in this step is referred to as x.

[0042] Next, the data acquisition unit 201 stores the operation data x acquired in the above step S101 in the operation database 210 (step S102).

[0043] Next, the feature extraction unit 202 extracts a feature from the operation data x acquired in the above step S101 (step S103). Hereinafter, the operation data feature extracted in this step is referred to as z.

[0044] Next, the classification unit 203 calculates a label from the operational data feature amount z extracted in the above step S103 (step S104). Hereinafter, the label calculated in this step is referred to as y.

[0045] Next, the feature amount / classification result storage unit 204 stores the operation data feature amount z extracted in the above step S103 and the label y calculated in the above step S104 in the operation database 210 (step S105).

[0046] Finally, the display control unit 209 displays the label y calculated in step S104 (i.e., the classification result of the operational data x) on the display device 102 (step S106). This allows the user to know the classification result of the operational data x.

[0047] <Model update process> The model update process according to this embodiment will be described below with reference to FIG. 4. In the following, the operation database 210 includes an operation data set X 4 and the operational data feature set Z 4 And the classification result set Y 4 The execution timing of the model update process can be set arbitrarily, and may be, for example, executed at predetermined time intervals.

[0048] The comparison operation unit 205 compares the operational data feature quantity z i ∈Z 4 and the training data feature value z i ∈Z 1 The distribution distance between is calculated as the target distribution distance (step S201). For example, if the distribution of the feature amounts classified into each class can be assumed to be a normal distribution, the comparison operation unit 205 may calculate the target distribution distance by the following steps 1-1 to 1-3.

[0049] Step 1-1: The comparison operation unit 205 compares the operational data feature set Z 4 and the classification result set Y 4For each class, the operational data feature value z i ∈Z 4 The mean and standard deviation of the operational data feature z i ∈Z 4 The average of μ 4,k , standard deviation is σ 4,k Let us assume that.

[0050] Step 1-2: The comparison operation unit 205 compares the average μ 4,k and standard deviation σ 4,k and the average μ 1,k and standard deviation σ 1,k In other words, the Kullback-Leibler divergence for class k is calculated by KL k Then, the comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k as follows: k are calculated respectively.

[0051]

number

[0052] Step 1-3: The comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k calculated in the above step 1-2. k That is, the comparison operation unit 205 calculates the sum KL of the Kullback-Leibler divergence for each class k calculated in the above step 1-2 as the distance between the target distributions. 1 +···+KL Kis calculated as the distance between the target distributions. Hereinafter, the distance between the target distributions is referred to as KL.

[0053] Next, the comparison operation unit 205 compares the learning data feature quantity z i ∈Z 1 The comparison operation unit 205 calculates the inter-distribution distance between the first and second distributions as the first inter-distribution distance (step S202). For example, if the distribution of the feature amounts classified into each class can be assumed to be a normal distribution, the comparison operation unit 205 may calculate the first inter-distribution distance by the following steps 2-1 and 2-2.

[0054] Step 2-1: The comparison operation unit 205 compares the average μ 1,k and standard deviation σ 1,k The Kullback-Leibler divergence for class k is calculated using the above formula. In other words, the Kullback-Leibler divergence for class k is calculated using the following formula: k (1) Then, the comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k as follows: k (1) are calculated respectively.

[0055]

number

[0056] Step 2-2: The comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k calculated in the above step 2-1. k (1) That is, the comparison operation unit 205 calculates the sum KL of the Kullback-Leibler divergence for each class k calculated in the above step 2-1 as the first inter-distribution distance. 1 (1) +···+KL K (1)is calculated as the first distribution distance. Hereinafter, the first distribution distance is referred to as KL (1) Let us assume that.

[0057] Next, the comparison operation unit 205 compares the learning data feature quantity z i ∈Z 1 and the validation data feature value z i ∈Z 2 and the second distribution distance is calculated as the second distribution distance (step S203). For example, if the distribution of the feature amounts classified into each class can be assumed to be a normal distribution, the comparison operation unit 205 may calculate the second distribution distance by the following steps 3-1 and 3-2.

[0058] Step 3-1: The comparison operation unit 205 compares the average μ 1,k and standard deviation σ 1,k and the average μ 2,k and standard deviation σ 2,k The Kullback-Leibler divergence for class k is calculated using the above formula. In other words, the Kullback-Leibler divergence for class k is calculated using the following formula: k (2) Then, the comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k as follows: k (2) are calculated respectively.

[0059]

number

[0060] Step 3-2: The comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k calculated in the above step 3-1. k (2)That is, the comparison operation unit 205 calculates the sum KL of the Kullback-Leibler divergence for each class k calculated in the above step 3-1 as the second inter-distribution distance. 1 (2) +···+KL K (2) is calculated as the second distribution distance. Hereinafter, the second distribution distance is referred to as KL (2) Let us assume that.

[0061] Next, the comparison operation unit 205 compares the learning data feature quantity z i ∈Z 1 and the test data feature z i ∈Z 3 and the distribution distance between them is calculated as the third distribution distance (step S204). For example, if the distribution of the feature amounts classified into each class can be assumed to be a normal distribution, the comparison operation unit 205 may calculate the third distribution distance by the following steps 4-1 and 4-2.

[0062] Step 4-1: The comparison operation unit 205 compares the average μ 1,k and standard deviation σ 1,k and the average μ 3,k and standard deviation σ 3,k The Kullback-Leibler divergence for class k is calculated using the above formula. In other words, the Kullback-Leibler divergence for class k is calculated using the following formula: k (3) Then, the comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k as follows: k (3) are calculated respectively.

[0063]

number

[0064] Step 4-2: The comparison operation unit 205 calculates the Kullback-Leibler divergence KL for each class k calculated in the above step 4-1. k (3) That is, the comparison operation unit 205 calculates the sum KL of the Kullback-Leibler divergence for each class k calculated in the above step 4-1 as the third inter-distribution distance. 1 (3) +···+KL K (3) is calculated as the third distribution distance. Hereafter, the third distribution distance is called KL (3) Let us assume that.

[0065] Next, the performance inference unit 206 calculates the first inter-distribution distance KL (1) , the second distribution distance KL (2) and the third distribution distance KL (3) and the accuracy rate ACC stored in the standard database 211 1 , ACC 2 and ACC 3 Using the above, a linear approximation formula expressing the relationship between the distribution distance and the accuracy rate drop point is calculated (step S205). For example, the performance inference unit 206 may calculate the linear approximation formula by the following steps 5-1 and 5-2.

[0066] Step 5-1: Training data set X 1 Training dataset X for 1 The point at which the accuracy rate drops is Q 1 , training data set X 1 Validation dataset X for 2 The point at which the accuracy rate drops is Q 2 , training data set X 1 Test dataset X for 3 The point at which the accuracy rate drops is Q 3 In this case, the performance inference unit 206 calculates Q 1 =ACC 1 -ACC 1 =0, Q 2 =ACC1 -ACC 2 , Q 3 =ACC 1 -ACC 3 Point Q: 1 , Q 2 and Q 3 are calculated respectively.

[0067] In addition, the learning dataset X 1 The point at which the accuracy rate of a certain dataset drops for ACC is ACC 1 -It can be calculated using ACC.

[0068] Step 5-2: The performance inference unit 206 calculates P on an xy plane with the inter-distribution distance on the x-axis and the accuracy rate drop point on the y-axis. 1 :=(KL (1) ,Q 1 ), P 2 :=(KL (2) ,Q 2 ), P 3 :=(KL (3) ,Q 3 ) is calculated as a linear approximation equation. This can be calculated, for example, by the least squares method. Hereinafter, the regression line is represented as L, and the linear approximation equation is represented as L:y=ax.

[0069] The performance estimation unit 206 may set the linear approximation equation as L':y=βax using the correction coefficient β>0 determined in advance in the above step 5-2. Hereinafter, L':y=βax is referred to as the corrected linear approximation equation, and it is assumed that the corrected linear approximation equation L' is calculated in the above step 5-2.

[0070] Here, the relationship between the linear approximation formula L and the corrected linear approximation formula L' is shown in FIG. 5. As shown in FIG. 5, the linear approximation formula L is P 1 , P 2 and P 3 On the other hand, the corrected linear approximation equation L' represents a line obtained by multiplying the slope a of the regression line represented by the linear approximation equation L by β>0.

[0071] Next, the performance inference unit 206 uses the corrected linear approximation formula calculated in the above step S205 to calculate the target accuracy rate drop point from the target distribution distance calculated in the above step S201 (step S206). That is, the performance inference unit 206 calculates the target accuracy rate drop point by, for example, y=βa×KL. This target accuracy rate drop point indicates how much the accuracy of the classification model M has decreased since learning.

[0072] Next, the update determination unit 207 determines whether the target accuracy rate drop point calculated in the above step S206 exceeds a predetermined threshold th (step S207). Here, as shown in Fig. 5, the region where the target accuracy rate point is less than the threshold th is the automatic model update region, and the region where the target accuracy rate point is equal to or greater than the threshold th is the manual model update alert region. Therefore, when the target accuracy rate drop point exceeds the threshold th, it means that it has been determined that the classification model M should be updated by manual model update, and when the target accuracy rate drop point is equal to or less than the threshold th, it means that it has been determined that the classification model M should be updated by automatic model update.

[0073] The reason why we decided to update the classification model M by manual model updating when the target accuracy rate drop point exceeds the threshold th is that when the accuracy of the classification model M has significantly decreased due to drifts such as concept drift and data drift, model updating by unsupervised domain adaptation or the like may further decrease the model accuracy.

[0074] If it is determined in step S207 that the target accuracy rate drop point exceeds the threshold th, the display control unit 209 displays a manual model update alert on the display device 102 (step S208). This allows the user to know that a manual model update is necessary, and therefore, for example, it becomes possible to assign labels to all or part of the operation data stored in the operation database 210, and then use the operation data and its labels to update the parameters of the classification model M by a supervised learning technique.

[0075] On the other hand, if it is not determined in step S207 that the target accuracy rate drop point exceeds the threshold th, the model update unit 208 updates the parameters of the classification model M by an unsupervised learning method (e.g., unsupervised domain adaptation, etc.) using the operation data stored in the operation database 210 (step S209). This makes it possible to adapt the classification model M to the domain after drift has occurred, and the accuracy of the classification model M can be improved.

[0076] <Modification> Variation 1 In the above embodiment, in step S205 of FIG. 1 , P 2 and P 3 The regression line was calculated as a linear approximation from the above equation. 1 , P 2 and P 3 In addition, other points may be used.

[0077] For example, a correct answer label is manually assigned to a portion of the operational data stored in the operational database 210, and the accuracy rate of the operational data to which the correct answer label is assigned is compared with the learning data set X 1 4, the accuracy rate drop point is calculated from this accuracy rate, and the point represented by the accuracy rate drop point and the distribution distance may be used to calculate the linear approximation formula. This will result in labeling costs, but it will be possible to obtain a more accurate linear approximation formula. On the other hand, although the accuracy of the linear approximation formula will decrease, for example, P 1 , P 2 and P 3 Two of these points may be used. 1 , P 2 and P 3 One or two of the above and P 1 , P 2 and P 3 Points other than the above may also be used.

[0078] Variation 2 In the above embodiment, the equation representing the regression line is calculated as a linear approximation equation in step S205 in Fig. 4, but it is not limited to linear approximation and may be nonlinear approximation. That is, an equation representing a regression curve may be calculated in step S205 in Fig. 4.

[0079] Variation 3 The process of step S201 in Fig. 4 may be executed, for example, after the processes of steps S202 to S205 in Fig. 4. Also, for example, the processes of steps S202 to S205 in Fig. 4 may be executed in advance before the model update process is executed, and only the processes of steps S201 and steps S206 to S209 may be executed in the model update process.

[0080] <Summary> As described above, the classification device 10 according to this embodiment calculates the accuracy rate drop point, which indicates the degree to which the model accuracy has decreased, from the distribution distance between the operation data obtained after the start of operation of the classification model M and the learning data used in parameter learning of the classification model M. Then, when the accuracy rate drop point is large, the classification device 10 according to this embodiment notifies the user by an alert that the classification model M should be updated manually, and when the accuracy rate drop point is small, automatically updates the classification model M by unsupervised domain adaptation or the like.

[0081] In this way, when the deterioration of the model accuracy due to the drift is small, the model accuracy can be improved by automatically adapting it to the domain after the drift occurs. On the other hand, when the deterioration of the model accuracy due to the drift is large, the classification model M can be retrained by a supervised learning method involving labeling by the user in order to prevent further deterioration of the model accuracy.

[0082] Therefore, according to the classification device 10 of this embodiment, it is possible to appropriately perform model updating according to the degree of accuracy degradation of the classification model M. Furthermore, when the degree of accuracy degradation of the classification model M is small, model updating is performed automatically, so that it is possible to reduce the number of times the user performs labeling, and it is also possible to reduce the burden on the user. Furthermore, when the degree of accuracy degradation of the classification model M is large, a manual model update alert is notified to the user, so that it is also possible to monitor for a large degradation of the model accuracy.

[0083] The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications, changes, combinations with known technologies, etc. are possible without departing from the scope of the claims. [Explanation of symbols]

[0084] 10 Classification device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Data Acquisition Department 202 Feature Extraction Unit 203 Classification Department 204 Feature and classification result storage 205 Comparison operation section 206 Performance inference section 207 Update determination section 208 Model Update Department 209 Display control section 210 Operational Database 211 Reference Database M Classification Model

Claims

1. a first calculation unit that calculates a distribution distance in a feature space between a plurality of data sets including a training data set used in parameter learning of a machine learning model; a second calculation unit that calculates an equation expressing a relationship between the distribution distance calculated by the first calculation unit and a second index value expressing a degree of decrease in the first index value for the learning dataset, using the distribution distance calculated by the first calculation unit and a first index value expressing a model accuracy of the machine learning model for each of the plurality of datasets; a third calculation unit that calculates a distribution distance in a feature space between an operation dataset obtained after the start of operation of the machine learning model and the learning dataset as a target distribution distance; a fourth calculation unit that calculates the second index value regarding the operation data set by using the target distribution distance calculated by the third calculation unit and the formula; a determination unit that determines whether the second index value calculated by the fourth calculation unit exceeds a predetermined threshold value; an update unit that updates parameters of the machine learning model by an unsupervised learning method using the operational dataset when it is determined that the second index value calculated by the fourth calculation unit does not exceed the threshold; A model updating device having the following:

2. 2. The model updating device according to claim 1, further comprising: a notification unit that, when it is determined that the second index value calculated by the fourth calculation unit exceeds the threshold value, labels the operational data included in the operational dataset and notifies an alert indicating that the machine learning model should be re-trained by a supervised learning technique.

3. The plurality of datasets include a validation dataset used to adjust hyperparameters of the machine learning model, and a test dataset used to evaluate generalization performance of the machine learning model; The first calculation unit, Calculating a distribution distance in a feature space between the learning data sets as a first distribution distance, a distribution distance in a feature space between the learning data set and the validation data set as a second distribution distance, and a distribution distance in a feature space between the learning data set and the test data set as a third distribution distance, The second calculation unit is 3. The model updating device according to claim 1 or 2, wherein the equation is calculated using the first inter-distribution distance, the second inter-distribution distance, the third inter-distribution distance, the first index value for the training dataset, the first index value for the validation dataset, and the first index value for the test dataset.

4. The second calculation unit is Calculating a value obtained by subtracting the first index value for the learning data set from the first index value for the learning data set as the second index value for the learning data set, a value obtained by subtracting the first index value for the validation data set from the first index value for the learning data set as the second index value for the validation data set, and a value obtained by subtracting the first index value for the test data set from the first index value for the learning data set as the second index value for the test data set, 4. The model updating device according to claim 3, wherein the equation is calculated as a regression equation approximated by a point represented by a pair of the first distribution distance and the second index value for the learning data set, a point represented by a pair of the second distribution distance and the second index value for the validation data set, and a point represented by a pair of the third distribution distance and the second index value for the test data set.

5. The first index value is a correct answer rate, The model updating device according to claim 4 , wherein the second index value is an accuracy rate decline point that indicates a degree of decline in the accuracy rate for the learning data set.

6. A first calculation step of calculating a distribution distance in a feature space between a plurality of data sets including a training data set used in parameter learning of a machine learning model; a second calculation step of calculating an equation expressing a relationship between the distribution distance calculated by the first calculation step and a second index value expressing a degree of decrease in the first index value for the learning dataset, using the distribution distance calculated by the first calculation step and a first index value expressing a model accuracy of the machine learning model for each of the plurality of datasets; A third calculation step of calculating a distribution distance in a feature space between an operation dataset obtained after the start of operation of the machine learning model and the learning dataset as a target distribution distance; a fourth calculation step of calculating the second index value for the operation data set by using the distance between the target distributions calculated by the third calculation step and the formula; a determination step of determining whether the second index value calculated by the fourth calculation step exceeds a predetermined threshold value; an update step of updating parameters of the machine learning model by an unsupervised learning technique using the operational dataset when it is determined that the second index value calculated by the fourth calculation step does not exceed the threshold value; The computer performs the model update method.

7. A first calculation step of calculating a distribution distance in a feature space between a plurality of data sets including a training data set used in parameter learning of a machine learning model; a second calculation step of calculating an equation expressing a relationship between the distribution distance calculated by the first calculation step and a second index value expressing a degree of decrease in the first index value for the learning dataset, using the distribution distance calculated by the first calculation step and a first index value expressing a model accuracy of the machine learning model for each of the plurality of datasets; A third calculation step of calculating a distribution distance in a feature space between an operation dataset obtained after the start of operation of the machine learning model and the learning dataset as a target distribution distance; a fourth calculation step of calculating the second index value for the operation data set by using the distance between the target distributions calculated by the third calculation step and the formula; a determination step of determining whether the second index value calculated by the fourth calculation step exceeds a predetermined threshold value; an update step of updating parameters of the machine learning model by an unsupervised learning technique using the operational dataset when it is determined that the second index value calculated by the fourth calculation step does not exceed the threshold value; A program that causes a computer to execute the following.

Images