Machine learning program, machine learning method, and information processing device

The machine learning program detects accuracy deterioration by monitoring centroid position and variance changes in data clusters, addressing the delay in detecting performance declines and enabling timely retraining.

JP7585915B2Active Publication Date: 2024-11-19FUJITSU LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2021048770
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-23
Publication Date
2024-11-19
Estimated Expiration
2041-03-23

AI Technical Summary

Technical Problem

Existing techniques fail to detect signs of accuracy deterioration in machine learning models until concept drift occurs, leading to unaddressed declines in model performance.

Method used

A machine learning program that calculates the centroid position and variance of data clusters in the feature space, comparing these values with past data to detect signs of accuracy deterioration before they become significant.

Benefits of technology

Enables early detection of accuracy deterioration in machine learning models, allowing for timely retraining to maintain model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585915000003
    Figure 0007585915000003
  • Figure 0007585915000004
    Figure 0007585915000004
  • Figure 0007585915000005
    Figure 0007585915000005
Patent Text Reader

Abstract

To provide an information processor capable of detecting signs of accuracy deterioration in a machine learning model.SOLUTION: The information processor calculates at least one value out of a centroid position of a cluster and an amount of dispersion depending on distribution of multiple data based on a feature value that is obtained by inputting multiple data into a machine learning model. The information processor compares at least one value and at least a past value corresponding to at least one value that is obtained by inputting data that has been previously input into the machine learning model, and depending on the comparison result, detects accuracy deterioration of the machine learning model.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to training machine learning models. [Background technology]

[0002] In various fields, machine learning models are increasingly being introduced for data judgment and classification functions, etc. Machine learning models judge and classify data based on training data that was machine-learned during system development, so if concept drift occurs during system operation, in which the tendency of input data changes, the accuracy of the machine learning model will deteriorate.

[0003] For this reason, various techniques for detecting accuracy degradation of a machine learning model are known. For example, a technique is known in which the distribution of input data is monitored using the feature space of a machine learning model, and accuracy degradation of the machine learning model is detected when the distribution of the input data crosses the decision boundary of the machine learning model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-113119 A [Patent Document 2] Special Publication No. 2005-523534 [Patent Document 3] US Patent Application Publication No. 2009 / 0106178 [Patent Document 4] US Patent Application Publication No. 2018 / 0285691 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it is difficult for the above-mentioned techniques to detect a sign of accuracy degradation of a machine learning model. For example, the above-mentioned techniques cannot detect accuracy degradation until concept drift actually occurs and the distribution of input data crosses the decision boundary, so there may be a period of time in which the accuracy degradation is left unattended.

[0006] In one aspect, an object of the present invention is to provide a machine learning program, a machine learning method, and an information processing device that are capable of detecting signs of deterioration in the accuracy of a machine learning model. [Means for solving the problem]

[0007] In a first proposal, the machine learning program causes a computer to execute a process of calculating at least one value of a cluster's center of gravity position and variance amount according to a distribution of a plurality of data based on a feature amount obtained by inputting a plurality of data into a machine learning model. The machine learning program causes a computer to execute a process of comparing the at least one value with a past value corresponding to the at least one value obtained by inputting data previously input into the machine learning model, and detecting a deterioration in accuracy of the machine learning model according to a result of the comparison. Effect of the Invention

[0008] According to one embodiment, it is possible to detect signs of deterioration in the accuracy of a machine learning model. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an information processing device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining the reference technique 1. In FIG. [Diagram 3] FIG. 3 is a diagram for explaining problem 1 of the reference technique 1. In FIG. [Figure 4] FIG. 4 is a diagram for explaining problem 2 of Reference Technique 1. In FIG. [Diagram 5] FIG. 5 is a functional block diagram of the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating the state of the generated machine learning model. [Figure 7] FIG. 7 is a diagram for explaining detection of a sign of accuracy deterioration. [Figure 8] FIG. 8 is a flowchart showing the flow of the process of detecting a sign of accuracy degradation. [Figure 9] FIG. 9 is a diagram for explaining the verification contents. [Figure 10] FIG. 10 is a diagram for explaining the verification results of the center of gravity position. [Figure 11] FIG. 11 is a diagram for explaining the verification results of the amount of dispersion. [Figure 12] FIG. 12 is a diagram for explaining detection of a sign of accuracy deterioration of a specific cluster. [Figure 13] FIG. 13 is a diagram for explaining the convergence determination. [Figure 14] FIG. 14 is a diagram illustrating retraining of a machine learning model. [Figure 15] FIG. 15 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the machine learning program, the machine learning method, and the information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. In addition, each embodiment can be appropriately combined within a range that does not cause inconsistency.

[0011] Fig. 1 is a diagram illustrating an information processing device 10 according to a first embodiment. The information processing device 10 illustrated in Fig. 1 is an example of a computer device that detects a sign of deterioration in accuracy of a machine learning model due to concept drift of input data or the like.

[0012] Here, a reference technique for detecting deterioration in accuracy of a machine learning model will be described. FIG. 2 is a diagram for explaining Reference Technique 1. Here, a two-dimensional feature is used as the feature of input data generated by the machine learning model, but the number of dimensions is not limited. As shown in FIG. 2, in Reference Technique 1, in the development stage, a machine learning model is generated so that input data can be classified into positive examples (low risk) or negative examples (high risk). That is, a machine learning model is generated in a state where the distribution of input data can be clearly classified by a decision boundary in a two-dimensional feature space.

[0013] In the operation phase, when the same type of input data as in the development phase is input, the machine learning model can accurately classify the input data, so the classification accuracy does not fall below the threshold, i.e., the feature values ​​do not cross the decision boundary. After that, a change in the trend of the input data occurs, and the distribution shifts to the upper right (lower right) in the feature space, crossing the decision boundary. At this time, the classification accuracy of the machine learning model falls below the threshold, and it is determined that the accuracy has deteriorated. In other words, the occurrence of concept drift requires classification of input data that is different from that during machine learning, and the classification accuracy deteriorates.

[0014] Then, in Reference Technology 1, a machine learning model that can classify input data into positive examples (low risk) or negative examples (high risk) is generated by retraining the machine learning model with degraded classification accuracy. In other words, a machine learning model with a regenerated decision boundary is generated.

[0015] In this way, in Reference Technology 1, the timing for retraining the machine learning model is identified based on whether the classification accuracy falls below a threshold (whether the feature value crosses the decision boundary), but another method is to detect signs of accuracy degradation by clustering the feature space. For example, a method is known that detects a sign of accuracy degradation when the distance between the center of class A, which is a cluster of positive examples, and the center of class B, which is a cluster of negative examples, in the feature space is greater than or equal to a threshold value from the time of development (learning). However, with this method, the distance between the centers may not change even if concept drift occurs, and a sign of accuracy degradation may be left unattended for a period of time.

[0016] FIG. 3 is a diagram explaining problem 1 of Reference Technology 1. As shown in FIG. 3, even if the decision boundary is not crossed, the variation in the distribution of the feature amount may change, and in this case, the accuracy does not decrease, so the administrator, etc., does not notice the deterioration in accuracy and leaves the deterioration unattended. That is, since Reference Technology 1 only looks at the cluster center distance, when the distribution becomes uneven, the deterioration in accuracy cannot be detected unless the accuracy deteriorates until the decision boundary is crossed, and the deterioration is left unattended. In other words, since only the accuracy is monitored, phenomena other than the crossing of the decision boundary cannot be detected even if they occur.

[0017] Furthermore, in Reference Technology 1, for data that moves apart in the feature space during the operation stage, there is no change in the classification accuracy of the machine learning model, so accuracy degradation cannot be detected. Figure 4 is a diagram explaining problem 2 of Reference Technology 1. As shown in Figure 4, in Reference Technology 1, accuracy degradation cannot be detected unless the decision boundary is crossed, so when class A, which is a cluster of positive examples, and class B, which is a cluster of negative examples, move apart, it cannot be determined that there is accuracy degradation.

[0018] Therefore, the information processing device 10 of Example 1 detects signs of deterioration in the accuracy of the machine learning model by monitoring not only the density of features and the central distance, but also the "center of gravity position of the cluster" and the "variance amount of the cluster" in the feature space.

[0019] Specifically, as shown in Fig. 1, the information processing device 10 generates a machine learning model using training data. Then, when the training is completed, the information processing device 10 generates a class A of positive examples and a class B of negative examples in the feature space of the machine learning model using the features of the input data. Then, the information processing device 10 acquires and holds the center of gravity position A1 of the positive example class A and the variance amount A2 of the class A.

[0020] Thereafter, the information processing device 10 classifies the input data using the generated machine learning model. At this time, the information processing device 10 acquires the centroid position A1' of class A of the positive example in the feature space and the variance amount A2' of class A. Then, the information processing device 10 detects a sign of accuracy degradation based on the movement distance from the centroid position A1 to the centroid position A1', the difference between the variance amount A2 and the variance amount A2', and the like. In this way, the information processing device 10 can detect a sign of accuracy degradation before the accuracy degradation occurs.

[0021] Fig. 5 is a functional block diagram showing a functional configuration of the information processing device 10 according to the first embodiment. As shown in Fig. 5, the information processing device 10 includes a communication unit 11, a storage unit 12, and a control unit 20. Note that the information processing device 10 is not limited to the illustrated one, and may include a display unit, etc.

[0022] The communication unit 11 executes communication with other devices. For example, the communication unit 11 receives input data. The communication unit 11 also receives various instructions and data from an administrator terminal used by an administrator, and transmits the result of the detection of a sign of accuracy degradation to the administrator terminal.

[0023] The storage unit 12 stores various data and programs executed by the control unit 20. For example, the storage unit 12 stores a training data DB 13, a machine learning model 14, and the like.

[0024] The training data DB 13 is a database that stores multiple training data used for machine learning of the machine learning model. For example, the training data DB 13 may be labeled training data for supervised learning to which a label that is correct answer information is assigned, or may be unlabeled training data for unsupervised learning, and can be arbitrarily selected according to the machine learning method of the machine learning model 14. For example, when generating a machine learning model 14 such as image classification, the training data DB 13 stores image data to which a label "dog" is assigned and image data to which a label "cat" is assigned as training data.

[0025] The machine learning model 14 is a model generated by machine learning. For example, the machine learning model 14 is a model using a deep neural network (DNN) or the like, and can employ a neural network or other machine learning algorithms. The machine learning model 14 may be generated by another device.

[0026] The control unit 20 is a processing unit that controls the entire information processing device 10. For example, the control unit 20 includes a generating unit 21, a classifying unit 22, a detecting unit 23, and a retraining unit 24.

[0027] The generation unit 21 generates the machine learning model 14. For example, the generation unit 21 performs optimization of various parameters of the machine learning model 14 through machine learning using each training data stored in the training data DB 13, thereby generating the machine learning model 14.

[0028] Fig. 6 is a diagram illustrating the state of the generated machine learning model 14. As shown in (a) of Fig. 6, when the machine learning model 14 is generated, the generation unit 21 calculates at least one value of the center of gravity position and the amount of variance of the cluster according to the distribution of a plurality of training data based on the feature amount obtained by inputting a plurality of training data to the machine learning model 14.

[0029] For example, in a feature space that is a distribution of features generated from training data by the machine learning model 14 in the machine learning process, a decision boundary (classification boundary) that can classify positive examples (low risk) and negative examples (high risk) is determined. In the following explanation, the boundary may be simplified as shown in (b) of FIG. 6. Note that, although an example in which positive examples are low risk and negative examples are high risk is illustrated here, this is merely an example, and other classifications may also be applied.

[0030] Here, the generation unit 21 calculates or acquires information on the feature space of the generated machine learning model 14. For example, the generation unit 21 executes clustering on the feature space of the machine learning model 14 to generate a class A of positive examples and a class B of negative examples. Then, the generation unit 21 calculates a center of gravity A1 and a variance amount A2 of the positive example class A and stores them in the storage unit 12, and calculates a center of gravity B1 and a variance amount B2 of the negative example class B and stores them in the storage unit 12. Note that the generation unit 21 can also generate the feature space by acquiring features from the machine learning model 14.

[0031] The classification unit 22 executes classification of input data. For example, when the classification unit 22 acquires input data, the classification unit 22 inputs the input data to the machine learning model 14 and classifies the input data into a positive example or a negative example according to the output of the machine learning model 14. The classification unit 22 can store the classification result in the storage unit 12 and display it on a display or the like.

[0032] The detection unit 23 detects a sign of accuracy degradation of the machine learning model 14. Specifically, when the classification unit 22 classifies the input data, the detection unit 23, like the generation unit 21, acquires features generated from the input data by the machine learning model 14 and performs clustering in a feature space that is a distribution of the features. Then, the detection unit 23 calculates the center of gravity or variance of the positive example or negative example class in the feature space. Thereafter, the detection unit 23 detects a sign of accuracy degradation of the machine learning model 14 by comparing the center of gravity or variance of the positive example or negative example class in the feature space at the start of operation with the center of gravity or variance of the positive example or negative example class in the feature space after the start of operation.

[0033] Fig. 7 is a diagram for explaining detection of a sign of deterioration in accuracy. Fig. 7(a) shows the clustering result of the feature space at the start of operation, and Fig. 7(b) and (c) show the clustering result after the start of operation. As shown in Fig. 7(a), at the start of operation, class A and class B are clearly distinguished from each other by the decision boundary in the feature space. The detection unit 23 acquires the centroid position A1 and the variance amount A2 of the positive example class A at this time from the storage unit 12, etc.

[0034] Thereafter, when the detection unit 23 detects that the center of gravity position of class A after the start of operation has moved from A1 to A1' as shown in (b) of Fig. 7, it calculates the amount of change from A1 to A1'. Then, when the amount of change is equal to or greater than a threshold, the detection unit 23 detects a sign of accuracy deterioration.

[0035] Furthermore, when the detection unit 23 detects that the amount of dispersion of class A after the start of operation has changed from A2 to A2' as shown in (c) of Fig. 7, it calculates the amount of change (difference) between A2 and A2'. Then, when the difference is equal to or greater than a threshold, the detection unit 23 detects a sign of accuracy deterioration.

[0036] Although an example of detecting a sign of accuracy degradation using the amount of change in the center of gravity position and the amount of change in the amount of variance in class A of positive examples has been described here, the detection can be performed by a similar process using the center of gravity position of class B of negative examples. Furthermore, the detection can be performed every time input data is classified, when a predetermined number of pieces of input data are classified, after a predetermined time has elapsed since the start of operation, when an administrator or the like sets a schedule, when machine learning converges, or at any other timing.

[0037] When a sign of accuracy degradation is detected, the retraining unit 24 executes retraining of the machine learning model 14. For example, when a sign of accuracy degradation is detected by the detection unit 23, the retraining unit 24 executes retraining of the machine learning model 14 using training data for retraining, and updates various parameters of the machine learning model 14.

[0038] In this case, if the retraining unit 24 detects signs of accuracy degradation for class A, which is a positive example, it can also retrain the machine learning model 14 using only the retraining training data that has been assigned a label corresponding to class A.

[0039] The training data for retraining can be stored in advance in the storage unit 12. The retraining unit 24 can also acquire the training data for retraining through input from the administrator, etc. by transmitting a request for the training data for retraining to the administrator, etc., or displaying it on a display, etc. The retraining unit 24 can also acquire the corresponding training data for retraining by accessing a server, etc., that stores the training data for retraining.

[0040] Fig. 8 is a flowchart showing the flow of the process of detecting a sign of accuracy deterioration. Here, an example will be described in which the machine learning model 14 has already been generated. As shown in Fig. 8, when an instruction to start processing is given (S101: Yes), the classification unit 22 acquires the center of gravity position and the amount of variance of the feature amount generated by the generation unit 21 after the completion of machine learning (at the start of operation) (S102).

[0041] Next, the classification unit 22 acquires input data (S103), inputs the data to the machine learning model 14, and executes classification (S104). Then, the detection unit 23 calculates the center of gravity of the feature (S105), and calculates the variance of the feature (S106). Note that S105 and S106 may be performed in any order.

[0042] Here, if the amount of change in the center of gravity position from the start of operation after the start of operation is less than the threshold value (S107: No), or if the amount of change in the amount of variance from the start of operation after the start of operation is less than the threshold value (S108: No), the detection unit 23 determines that there are no signs of deterioration in accuracy, and S103 and subsequent steps are repeated.

[0043] On the other hand, if the amount of change in the center of gravity position from the start of operation to the start of operation is equal to or greater than the threshold (S107: Yes), or if the amount of change in the amount of variance from the start of operation to the start of operation is equal to or greater than the threshold (S108: Yes), the detection unit 23 detects a sign of accuracy deterioration (S109). After that, the retraining unit 24 executes retraining of the machine learning model 14 (S110).

[0044] As described above, the information processing device 10 not only monitors the accuracy of whether or not the decision boundary is crossed, but also monitors the "center of gravity position of the class" and the "variance amount of the class" in the feature space, thereby being able to detect changes in distribution before the accuracy decreases and to perform prediction of deterioration.

[0045] Here, the verification result according to the first embodiment will be described. FIG. 9 is a diagram for explaining the verification content. Here, an image classification model is generated as the machine learning model 14, noise is added to the data set used for training, and it is confirmed how the variance and the centroid coordinates (centroid position) change. As the data set, CIFAR10, which is 60,000 color photos with labels such as vehicles and animals, is adopted, and the batch size is set to 20. In other words, 20 pieces of image data are randomly read from CIFAR10, inference is performed on the 20 images, and 20 points are plotted in the feature space.

[0046] In addition, the calculation formula of CoG is adopted for the center of gravity coordinate. Specifically, the information processing device 10 extracts only the first component and the second component from the 10-dimensional data, and plots 20 outputs. The information processing device 10 uses formula (1) as a formula that does not take mass into account to calculate the average value of n first components (x) and the average value of n second components (y) as the center of gravity position.

[0047]

number

[0048] The amount of variance (variance value) is calculated using a formula for variance. Specifically, the information processing device 10 reduces the dimension from 10 to 2-dimensional data by principal component analysis (PCA), and calculates the amount of variance from 20 plots. For example, the information processing device 10 calculates the average value of the standard deviation of the X coordinate and the standard deviation of the Y coordinate using formula (2).

[0049]

number

[0050] 9, the information processing device 10 generates an image classification model using training data. Next, the information processing device 10 generates input data including noise data obtained by performing left-right inversion, black window insertion, grayscale, contrast change, distortion addition, etc. on the training data. Then, the information processing device 10 inputs the input data to the image classification model and classifies the input data.

[0051] Here, the information processing device 10 converts the output result from the torch.tensor type to the ndarray type, converts the ndarray type to the pd.DataFrame type, and then extracts only the first and second components from the 10-dimensional data. After that, the information processing device 10 calculates, as output (center of gravity coordinates, variance amount), the center of gravity coordinates and variance amount of the feature amount distribution in the feature space when there is noise, and the center of gravity coordinates and variance amount of the feature amount distribution in the feature space when there is no noise.

[0052] Fig. 10 is a diagram explaining the verification result of the center of gravity position. As shown in Fig. 10, the center of gravity position of the source image corresponding to the training data without noise is compared with the center of gravity position of the noise data with the image rotated, the center of gravity position of the noise data with the black window inserted, the center of gravity position of the noise data changed to grayscale, the center of gravity position of the noise data with the left and right inverted, the center of gravity position of the noise data with the image rotated, and the center of gravity position of the noise data with the illumination changed. The comparison results show that the center of gravity position when there is no noise does not match the center of gravity position when there is noise, which shows that the center of gravity position changes depending on the presence or absence of noise, and it was verified that the comparison of the center of gravity positions is effective in detecting the signs of deterioration of accuracy.

[0053] Figure 11 is a diagram explaining the verification results of the amount of variance. As shown in Figure 11, it was verified that when the noise intensity was increased by two stages by inserting black windows (Random Erasing), the variance increased as the accuracy decreased. As a result, it was verified that the comparison of the amount of variance is effective in detecting the signs of accuracy deterioration.

[0054] Moreover, by executing the method according to the first embodiment, it is possible to detect the deterioration of the accuracy of a specific class even when the number of classes is large. In the above-mentioned reference technology, in the case of a 100-class classification problem, even if the accuracy of only one specific class decreases and the accuracy rate of that class becomes 0%, the overall accuracy is only affected by about 1% at most. Therefore, the overall accuracy decreases by only 1%, and the accuracy monitoring is also within the margin of error, so the deterioration of the accuracy of a specific class cannot be detected.

[0055] In contrast, the information processing device 10 according to the first embodiment monitors the change in the position of the center of gravity of the distribution of each class and the change in the amount of variance, so that it is possible to detect the deterioration of the accuracy of a specific class even in a classification problem with a large number of classes. FIG. 12 is a diagram for explaining the detection of a sign of deterioration of the accuracy of a specific cluster. As shown in FIG. 12, when the information processing device 10 detects a change in the amount of variance in class A and a change in the center of gravity movement in class C in a four-class classification problem, it is possible to detect the occurrence of a sign of deterioration of the accuracy only for class A and class C. In this way, the information processing device 10 can monitor the change in the center of gravity movement and the amount of variance for each class, so that it is possible to detect which class has experienced deterioration of the accuracy. In addition, the information processing device 10 can perform retraining only for a specific class in which a sign of deterioration of the accuracy has been detected, so that it is possible to suppress unnecessary retraining and shorten the time for retraining.

[0056] Furthermore, the information processing device 10 can further improve accuracy by appropriately setting the batch size, which is the number of samples, when calculating the center of gravity and variance of a class (cluster). Since the optimal batch size for calculating each value varies depending on the data subject, an example will be described in which the information processing device 10 calculates the center of gravity and variance for each batch size and simultaneously determines the batch size based on the converged point.

[0057] FIG. 13 is a diagram for explaining convergence judgment. FIG. 13 shows a convergence graph by batch size when the noise of the black window in the above verification is P=0.5. The horizontal axis is the batch size, and the vertical axis is the calculation result of "centroid position / variance amount". As shown in FIG. 13, the condition for judging convergence is the change ratio before and after each step, and it is judged to have converged when "the ratio before and after one step is 0.8 or more" continues for 50 batches or more. In the example of FIG. 13, the information processing device 10 determines that the vicinity of 1000 batches is optimal, and executes judgment of accuracy deterioration based on the centroid position and variance amount from the point when the processing of 1000 batches is completed.

[0058] It is also known that training using training data and accuracy verification using verification data are repeated to generate the machine learning model 14. Here, an example will be described in which a highly accurate machine learning model 14 is generated by retraining only a specific class during training, even when the machine learning has progressed to a certain extent.

[0059] Fig. 14 is a diagram illustrating retraining of the machine learning model 14. As shown in Fig. 14, the information processing device 10 executes training of the machine learning model using training data to generate the machine learning model 14 in a state where training has progressed to a certain degree (S201).

[0060] Next, the information processing device 10 inputs the verification data to the machine learning model 14, and executes accuracy verification of the machine learning model 14 (S202). Note that the accuracy verification can be performed using a general method.

[0061] Then, the information processing device 10 stores the output data of the machine learning model 14 for the verification data (S203), and determines convergence based on the batch size using the center of gravity position and the amount of variance (S204).

[0062] Thereafter, when it is determined that convergence has occurred due to the batch size, the information processing device 10 determines whether or not retraining is necessary according to the amount of change in the center of gravity position of each class in the feature space and the amount of variance (S205).

[0063] For example, when the information processing device 10 detects a class in which the change in the center of gravity position is greater than or equal to a threshold value using the method of Example 1, when it detects a class in which the change in the variance is greater than or equal to a threshold value, or when it detects a class in which the change in the center of gravity position and the change in the variance are both greater than or equal to a threshold value, it determines that retraining is necessary.

[0064] Here, when the information processing device 10 determines that retraining is not necessary (S205: No), the process ends. On the other hand, when the information processing device 10 determines that retraining is necessary (S205: Yes), the information processing device 10 determines control parameters for retraining (S206) and executes retraining of the machine learning model 14 (S207).

[0065] For example, the information processing device 10 performs weight change, control change, training change, narrowing down of training, etc. as a control parameter. Specifically, the information processing device 10 performs weight change by biasing only the training data of a specific class in which accuracy degradation has occurred, and can perform retraining while suppressing accuracy degradation while maintaining the overall control.

[0066] Furthermore, the information processing device 10 can execute retraining only when it is determined that retraining is necessary based on the importance (class type) of a specific class in which accuracy degradation has occurred, as a control change, and suppress retraining when it is determined that retraining is not necessary. As a result, the information processing device 10 can execute appropriate training according to the necessity of retraining. Note that the importance of each class and the importance of each class type can be defined in advance before machine learning.

[0067] Furthermore, when a specific class in which accuracy deterioration has occurred is a class that does not require retraining, the information processing device 10 can determine that the specific class is blank data and discard the training data of the class as a training change. As a result, the information processing device 10 can reduce the training time while maintaining the accuracy of the machine learning model 14 by suppressing unnecessary training.

[0068] Furthermore, the information processing device 10 can narrow down the training by using training data corresponding to only a specific class in which accuracy degradation has occurred. As a result, the information processing device 10 can perform efficient retraining. Note that, for the above-mentioned retraining, data for retraining different from that used during training can be used.

[0069] The data examples, numerical examples, thresholds, number of classes, number of dimensions of the feature space, specific examples, and the like used in the above embodiments are merely examples and can be changed arbitrarily. For example, the information processing device 10 can monitor only one of the center of gravity position or the amount of variance, or can monitor both. A class is an example of a cluster. As training data or input data, image data, voice data, time-series data, and the like can be used, and the machine learning model 14 can also be used for image classification and various analyses. As the machine learning model 14, a model such as a DNN (Deep Neural Network) or a CNN (Convolution Neural Network) can be adopted.

[0070] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0071] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure. In other words, all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0072] Furthermore, each processing function performed by each device may be realized, in whole or in part, by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0073] Fig. 15 is a diagram for explaining an example of a hardware configuration. As shown in Fig. 15, an information processing device 10 includes a communication device 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 15 are connected to each other via a bus or the like.

[0074] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0075] The processor 10d reads out a program that executes the same processes as the respective processing units shown in FIG. 5 from the HDD 10b or the like and loads it in the memory 10c, thereby operating a process that executes each function described in FIG. 5 or the like. For example, this process executes a function similar to that of each processing unit possessed by the information processing device 10. Specifically, the processor 10d reads out a program having functions similar to those of the generation unit 21, the classification unit 22, the detection unit 23, the retraining unit 24, or the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processes as those of the generation unit 21, the classification unit 22, the detection unit 23, the retraining unit 24, or the like.

[0076] In this way, the information processing device 10 operates as an information processing device that executes a machine learning method by reading and executing a program. The information processing device 10 can also realize functions similar to those of the above-mentioned embodiment by reading the above program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 10. For example, the present invention can be similarly applied to a case where another computer or server executes a program, or a case where these execute a program in cooperation with each other.

[0077] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer. [Explanation of symbols]

[0078] 10. Information processing device 11 Communications Department 12 Storage section 13 Training Data DB 14 Machine Learning Models 20 Control section 21 Generation part 22 Classification Department 23 Detection unit 24 Retraining Department

Claims

1. Calculating at least one of a center of gravity position and a variance amount of a cluster according to a distribution of the plurality of data based on a feature amount obtained by inputting the plurality of data into a machine learning model; comparing the at least one value with a past value corresponding to the at least one value obtained by inputting data previously input to the machine learning model; Detecting a deterioration in accuracy of the machine learning model according to a result of the comparison. A machine learning program that causes a computer to carry out processing.

2. Controlling retraining of the machine learning model based on the cluster in which a deterioration in accuracy of the machine learning model is detected. The machine learning program according to claim 1 , further causing the computer to execute a process.

3. The calculation process includes: generating a plurality of clusters according to a distribution of the plurality of data based on the feature amount; Calculating at least one of the center of gravity position and the variance amount for each of the plurality of clusters; The comparing process includes: For each of the plurality of clusters, comparing the at least one value with a previous value corresponding to the at least one value; The detecting process includes: detecting a deterioration in accuracy of the machine learning model for each of the plurality of clusters in accordance with a result of the comparison; The machine learning program according to claim 2 , comprising a process.

4. The detecting process includes: detecting a cluster in which a difference between the at least one value and a past value corresponding to the at least one value is equal to or greater than a threshold value, from among the plurality of clusters; The control process includes: retraining the machine learning model using training data corresponding to the detected clusters. The machine learning program according to claim 3 , comprising a process.

5. The detecting process includes: detecting a cluster in which a difference between the at least one value and a past value corresponding to the at least one value is equal to or greater than a threshold value, from among the plurality of clusters; The control process includes: When the importance of the detected cluster is equal to or greater than a threshold, retraining the machine learning model using training data corresponding to the cluster is performed, and when the importance of the cluster is less than a threshold, retraining the machine learning model is suppressed. The machine learning program according to claim 3 , comprising a process.

6. The calculation process includes: Calculating both the centroid position and the variance of the cluster; The comparing process includes: calculating a first comparison result obtained by comparing the center of gravity position of the cluster with the past value, and a second comparison result obtained by comparing the variance amount of the cluster with the past value; The detecting process includes: detecting a deterioration in accuracy of the machine learning model when the first comparison result is equal to or greater than a threshold and the second comparison result is equal to or greater than a threshold; The machine learning program according to claim 1 , comprising a process.

7. The calculation process includes: Calculating both the centroid position and the variance of the cluster; The comparing process includes: After a batch size has elapsed in which a value obtained by dividing the centroid position by a variance amount satisfies a predetermined condition, a comparison is performed between the at least one value and a past value corresponding to the at least one value; The detecting process includes: detecting a deterioration in accuracy of the machine learning model according to a result of the previous comparison; The machine learning program according to claim 1 , comprising a process.

8. Calculating at least one of a center of gravity position and a variance amount of a cluster according to a distribution of the plurality of data based on a feature amount obtained by inputting the plurality of data into a machine learning model; comparing the at least one value with a past value corresponding to the at least one value obtained by inputting data previously input to the machine learning model; Detecting a deterioration in accuracy of the machine learning model according to a result of the comparison. A machine learning method in which processing is performed by a computer.

9. Calculating at least one of a center of gravity position and a variance amount of a cluster according to a distribution of the plurality of data based on a feature amount obtained by inputting the plurality of data into a machine learning model; comparing the at least one value with a past value corresponding to the at least one value obtained by inputting data previously input to the machine learning model; Detecting a deterioration in accuracy of the machine learning model according to a result of the comparison. An information processing device including a control unit that executes processing.

Citation Information

Patent Citations

  • Automatic neural net model generation and maintenance

    JP2005523534A

  • Monitoring system, monitoring method and program

    JP2020113119A

  • Computer-Implemented Systems And Methods For Updating Predictive Models

    US20090106178A1

  • Automated and customized post-production release review of a model

    US20180285691A1