Information processing device, information processing method, and program

Abstract

An information processing device 100 according to the present disclosure comprises: a classification unit 121 that classifies a machine learning model on the basis of preset characteristics of a plurality of machine learning models; a creation unit 122 that creates, for each classification, a monitoring target machine learning model that is a machine learning model to be monitored within the classification on the basis of machine learning models included in the classification; and a monitoring unit 123 that monitors, for each classification, a change in prediction accuracy of the monitoring target machine learning model. This makes it possible to support user decision-making by using predictions made by machine learning models.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] In recent years, machine learning models have been used in various situations to make predictions from data. However, the predictive accuracy of machine learning models can deteriorate due to factors such as changes in the trends of data input to the machine learning models. Therefore, in order to maintain the predictive accuracy of the machine learning models, retraining of the machine learning models is necessary. For example, Patent Literature 1 describes detecting deterioration in the accuracy of a machine learning model and retraining the machine learning model with deteriorated accuracy.

[0003] International Publication No. 2021 / 084623

[0004] However, in a situation where a large number of machine learning models are used, relearning all of the machine learning models requires a great deal of effort and cost. Furthermore, as described in Patent Literature 1, in order to detect a deterioration in the prediction accuracy of a machine learning model, all of the machine learning models must be monitored, which requires a great deal of effort and cost. Thus, there arises a problem in that maintaining the prediction accuracy of a machine learning model requires a great deal of effort and cost.

[0005] Therefore, the purpose of the present disclosure is to solve the above-mentioned problem that maintaining the predictive accuracy of a machine learning model requires a great deal of effort and cost.

[0006] An information processing device according to an embodiment of the present disclosure includes: a classification unit that classifies machine learning models based on preset characteristics of a plurality of machine learning models; a creation unit that creates, for each classification, a monitored machine learning model that is a machine learning model to be monitored for that classification, based on the machine learning models included in that classification; and a monitoring unit that monitors, for each classification, changes in prediction accuracy of the monitored machine learning model. An information processing method according to an embodiment of the present disclosure includes: classifying the machine learning models based on preset characteristics of a plurality of machine learning models; creating, for each classification, a monitored machine learning model that is a machine learning model to be monitored for that classification, based on the machine learning models included in that classification; and monitoring, for each classification, changes in prediction accuracy of the monitored machine learning model. A program according to an embodiment of the present disclosure includes: causing a computer to execute processes of: classifying the machine learning models based on preset characteristics of a plurality of machine learning models; creating, for each classification, a monitored machine learning model that is a machine learning model to be monitored for that classification, based on the machine learning models included in that classification; and monitoring, for each classification, changes in prediction accuracy of the monitored machine learning model.

[0007] By being configured as described above, the present disclosure can reduce the effort and cost required to maintain the predictive accuracy of a machine learning model.

[0008] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a diagram showing a state of processing by an information processing device according to the present disclosure. FIG. 3 is a diagram showing a state of processing by an information processing device according to the present disclosure. FIG. 4 is a diagram showing a state of processing by an information processing device according to the present disclosure. FIG. 5 is a flowchart showing processing operations of an information processing device according to the present disclosure. FIG. 6 is a block diagram showing a hardware configuration of an information processing device according to the present disclosure. FIG. 7 is a block diagram showing a configuration of an information processing device according to the present disclosure.

[0009] First Embodiment A first embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any embodiment.

[0010] [Configuration] In this embodiment, the information processing device 10 monitors the deterioration of the prediction accuracy of a machine learning model. Furthermore, the information processing device 10 re-learns the machine learning model in accordance with the monitoring results. Here, in this embodiment, the machine learning model to be monitored and re-learned is a prediction model that inputs data acquired from a predetermined event to be predicted and outputs a prediction result. As an example, the machine learning model is a model that predicts sales (e.g., sales volume) of products sold in a store based on various data such as date and time and weather. In this case, a machine learning model is used for each product. However, the machine learning model may be a model that performs any prediction.

[0011] The information processing device 10 is composed of one or more information processing devices each including a calculation device and a storage device. As shown in FIG. 1 , the information processing device 10 includes an analysis unit 11, a clustering unit 12, a monitoring target setting unit 13, a monitoring unit 14, and a learning unit 15. The functions of the analysis unit 11, the clustering unit 12, the monitoring target setting unit 13, the monitoring unit 14, and the learning unit 15 can be realized by the calculation device executing a program for realizing each function stored in the storage device. The information processing device 10 also includes a learning data storage unit 16, an operational data storage unit 17, and a model storage unit 18. The learning data storage unit 16, the operational data storage unit 17, and the model storage unit 18 are each configured using a storage device. Each component will be described in detail below.

[0012] The training data storage unit 16 stores training data used for machine learning of a machine learning model. The training data consists of pairs of explanatory variables and target variables obtained from a predetermined event. As will be described later, the training data is used when training the machine learning model and when analyzing the characteristics of the machine learning model.

[0013] The operational data storage unit 17 stores operational data that is input to the machine learning model and used for prediction. The operational data consists of pairs of explanatory variables and objective variables acquired from a predetermined event. When the machine learning model is in operation, the explanatory variables are input to the machine learning model, and the machine learning model outputs a prediction result. When the machine learning model is monitored, the objective variables of the operational data are used and compared with the prediction result when the explanatory variables are input to the machine learning model. Note that the operational data is different from the above-mentioned learning data, and is data acquired from the event to be predicted after the learning data is acquired. However, the operational data may be the same data as the learning data, or may be part of the learning data.

[0014] The model storage unit 18 stores multiple machine learning models that have been machine-learned using the training data stored in the training data storage unit 16. Specifically, the internal parameters of the machine learning models have been adjusted by machine learning so that a prediction result output in response to an input of an explanatory variable, which is training data, approaches a target variable, which is paired training data. In this case, the model storage unit 18 may store multiple machine learning models that have already been generated, or may store a machine learning model that has been generated by machine learning using training data by the learning unit 15, which will be described later. As an example, the model storage unit 18 stores multiple machine learning models denoted by symbols A to I, as indicated by reference symbol F1 in FIG. 2 . However, the number of machine learning models is not limited to the number shown in the figure.

[0015] The analysis unit 11 (classification unit) analyzes the characteristics of multiple machine learning models stored in the model storage unit 18. In this embodiment, the analysis unit 11 considers a portion of the training data (e.g., evaluation data) as simulated time-series operational data and examines changes in the prediction accuracy of the machine learning data over time. In particular, the analysis unit 11 examines changes in the decline in the prediction accuracy of the machine learning data over time, i.e., the deterioration trend of the prediction accuracy. In this case, the analysis unit 11 analyzes the deterioration trend by, for example, normalizing the prediction accuracy at each time point using the initial accuracy of the machine learning model and representing the graph. As a result, the analysis unit 11 can obtain a graph of the deterioration trend of each machine learning model A to I, as shown by symbol F2 in FIG. 2.

[0016] The characteristic of the machine learning model analyzed by the analysis unit 11 is not limited to the aforementioned tendency of deterioration in prediction accuracy. The characteristic of the machine learning model analyzed by the analysis unit 11 may be, for example, the prediction accuracy itself, or any index that can be extracted from the machine learning model, such as precision or recall when the machine learning model performs a classification task.

[0017] The clustering unit 12 (classification unit) classifies machine learning models based on the deterioration trend of prediction accuracy, which is a characteristic of each machine learning model. Specifically, the clustering unit 12 clusters the deterioration trends of multiple machine learning models into multiple clusters (classifications) without a teacher using a preset clustering algorithm. As a result, multiple machine learning models are clustered into multiple clusters according to the characteristics of the deterioration trend of prediction accuracy. Note that the clustering unit 12 performs clustering using a clustering algorithm such as the k-means method, for example, but any method may be used to perform clustering. Furthermore, at this time, the clustering unit 12 may extract feature quantities (acc1, acc2, acc3) at each time (t1, t2, t3) of the deterioration trend of the prediction accuracy of machine learning model A, as shown in FIG. 4, and perform clustering using these feature quantities. Note that the clustering unit 12 may perform clustering based on any value of the characteristic of the analyzed machine learning model.

[0018] Reference symbol F3 in FIG. 2 shows an example of clustering machine learning models by the clustering unit 12. In this example, the clustering unit 12 clusters multiple machine learning models A to I into three clusters X, Y, and Z. Specifically, machine learning models A, D, and E are classified into cluster X, machine learning models B, G, and H are classified into cluster Y, and machine learning models C, F, and I are classified into cluster Z. In this case, machine learning models with a characteristic of relatively good prediction accuracy degradation over time are clustered into cluster X, machine learning models with a characteristic of a deterioration trend that deteriorates suddenly from the beginning of time are clustered into cluster Y, and machine learning models with a deterioration trend that gradually deteriorates over time are clustered into cluster Z. In this way, the clustering unit 12 performs clustering so that machine learning models with a common characteristic of a deterioration trend in prediction accuracy belong to the same cluster. Note that the clustering based on the characteristic of a deterioration trend in prediction accuracy as described above is just one example, and clustering may be performed based on any characteristic of a deterioration trend.

[0019] Although the above example illustrates a case in which the clustering unit 12 performs clustering based on characteristics such as a deterioration trend in the prediction accuracy of the machine learning model, clustering may also be performed by adding other data to the characteristics of the machine learning model. For example, clustering may be performed by adding, to the deterioration trend in the prediction accuracy, feature quantities of the explanatory variables or statistical values ​​(e.g., time change of the average value) of the target variables of the learning data (operational data) used in analyzing the deterioration trend in the prediction accuracy.

[0020] 2, the clustering unit 12 performs clustering so that the number of machine learning models belonging to each cluster is equal or nearly equal, that is, so that the difference in the number of machine learning models belonging to each cluster is small. However, the clustering unit 12 is not necessarily limited to performing clustering so that the number of machine learning models belonging to each cluster is nearly equal, and may perform clustering so that any number of machine learning models belongs to each cluster.

[0021] Then, the clustering unit 12 stores information about the clusters and machine learning models obtained by the clustering as described above in the model storage unit 18.

[0022] The monitoring target setting unit 13 (creation unit) creates and sets a machine learning model to be monitored (monitored machine learning model) for each cluster based on the machine learning models included in the cluster. Note that here, the machine learning model to be monitored is referred to as a canary model. For example, the monitoring target setting unit 13 selects, for each cluster, one of the machine learning models belonging to the cluster as the canary model. At this time, as an example, the monitoring target setting unit 13 selects, for each cluster, the machine learning model that deteriorates most quickly over time as the canary model among the machine learning models belonging to the cluster. As a result, as shown by reference symbol F4 in FIG. 3 , machine learning model E is selected as canary model Xc for cluster X, machine learning model H is selected as canary model Yc for cluster Y, and machine learning model C is selected as canary model Zc for cluster Z. In this way, the monitoring target setting unit 13 selects, as the canary model, the machine learning model that deteriorates most quickly over time in each cluster, i.e., the model with the poorest degradation resistance, which represents the characteristic of not deteriorating prediction accuracy over time.

[0023] In the above description, the monitoring target setting unit 13 selects the canary model with the poorest value for an index such as the resistance to degradation of prediction accuracy. However, the selection is not limited to selecting the single index with the poorest value. The monitoring target setting unit 13 may select multiple canary models with the lowest n index values, or may select a model that satisfies a preset criterion. Furthermore, in the above description, the monitoring target setting unit 13 selects a canary model using an index related to prediction accuracy, such as the resistance to degradation of the prediction accuracy of the machine learning model, as an index representing the characteristics of the machine learning model. However, any index may be used to select a canary model. For example, the monitoring target setting unit 13 may select a canary model using another index related to the prediction accuracy of the machine learning model. For example, when the machine learning model handles a classification task, values ​​such as precision or recall may be used as indices, and the model with the poorest such value or that meets a preset criterion may be selected as the canary model.

[0024] Furthermore, the monitoring target setting unit 13 is not limited to selecting a canary model from the machine learning models belonging to each cluster as described above, but may also generate a new machine learning model to serve as a canary model based on the characteristics of the machine learning models belonging to each cluster. For example, the monitoring target setting unit 13 may use an index related to prediction accuracy, such as the aforementioned tendency for prediction accuracy to deteriorate, as a characteristic of the machine learning models belonging to the cluster, and generate a machine learning model having characteristics that further lower the tendency for deterioration, i.e., the deterioration resistance performance.

[0025] Here, an example of generating a new canary model will be described with reference to FIG. 5 . As an example, when generating a new canary model for cluster Z, a machine learning model having characteristics represented by a graph of degradation resistance performance that is even lower than the envelope of a graph (dotted line) showing the degradation resistance performance, which is the time-dependent degradation trend of the prediction accuracy of all machine learning models C, F, and I belonging to cluster Z, is generated and designated as canary model Zc. That is, a machine learning model that satisfies the following formula (1) is generated by machine learning and designated as canary model Zc: Accuracy(Zc)<min(Accuracy(C), Accuracy(F), Accuracy(I)) (1) Note that, in order to generate a canary model that can detect accuracy degradation earlier, the following formula (2) or (3) may be used instead of the above formula (1). Accuracy (Zc) < min (Accuracy (C), Accuracy (F), Accuracy (I)) - e2 (2) Accuracy (Zc) < min (Accuracy (C), Accuracy (F), Accuracy (I)) * e3 (3) Here, parameters e2 and e3 are predetermined constants that are values ​​that adjust how early deterioration occurs. In other words, the larger e2 is and the smaller e3 is, the earlier detection becomes possible. The parameters satisfy e2 > 0, 0 < e3 < 1.

[0026] Note that the monitoring target setting unit 13 is not limited to generating a new canary model by the above-described method, and may generate a new canary model by any method. In other words, the monitoring target setting unit 13 may generate a new canary model based on other indicators related to the prediction accuracy of the machine learning model.

[0027] Then, the monitoring target setting unit 13 stores the canary model selected or generated for each cluster as described above in the model storage unit 18.

[0028] The monitoring unit 14 (monitoring unit) monitors changes in the prediction accuracy of the canary models Xc, Yc, and Zc set as described above. At this time, all machine learning models A to I indicated by reference symbol F4 in FIG. 3 are operated using the operational data stored in the operational data storage unit 17, but the monitoring unit 14 monitors changes in prediction accuracy only for the canary models Xc, Yc, and Zc. In other words, the monitoring unit 14 monitors changes in prediction accuracy over time due to the error between the prediction, which is the output when the explanatory variables of the operational data are input to the canary models Xc, Yc, and Zc, and the objective variable of the operational data. The monitoring unit 14 then monitors declines in the prediction accuracy of the canary models and identifies canary models that have reached a predetermined state, such as when the prediction accuracy has declined to a predetermined standard. For example, the monitoring unit 14 identifies a canary model when the prediction accuracy of the canary model has declined by a predetermined percentage relative to the initial accuracy or when the prediction accuracy has declined to a predetermined value. However, the monitoring unit 14 may identify the canary model being monitored using any standard.

[0029] As described above, when a canary model whose prediction accuracy has decreased due to monitoring is identified, the learning unit 15 (relearning unit) retrains the machine learning models included in the cluster to which the identified canary model belongs. As an example, as shown by reference symbol F5 in FIG. 3 , when a canary model Yc in cluster Y is identified as having decreased prediction accuracy, the learning unit 15 performs machine learning on all machine learning models B, G, and H belonging to cluster Y using operational data. In this case, the learning unit 15 is not limited to retraining all machine learning models B, G, and H in cluster Y to which the identified canary model Yc belongs, but may retrain some of the machine learning models in cluster Y. For example, only the machine learning model H corresponding to canary model Yc may be retrained, or the prediction accuracy of the other machine learning models B and G may also be examined, and some lower-ranked machine learning models with low prediction accuracy may be retrained.

[0030] Then, the learning unit 15 stores the re-learned machine learning model in the model storage unit 18 as described above.

[0031] Furthermore, after a machine learning model is retrained, each unit of the information processing device 10 performs the same processing as described above on all machine learning models, including the retrained machine learning model. To this end, the analysis unit 11 analyzes the deterioration trend in prediction accuracy for existing machine learning models and the retrained machine learning model, and the clustering unit 12 performs new clustering of the machine learning models based on the deterioration trend. The monitoring target setting unit 13 then selects or creates a canary model for each new cluster, the monitoring unit 14 monitors the canary models, and the learning unit 15 retrains the machine learning model of the new cluster to which the canary model with degraded accuracy belongs.

[0032] 6 shows an example of processing for machine learning models after relearning. As described above, when a deterioration in accuracy of machine learning model H, which is canary model Yc belonging to cluster Y before relearning, is detected and identified, machine learning models B, G, and H belonging to cluster Y are re-learned, and machine learning models B', G', and H' are generated. After relearning, all machine learning models are again analyzed for a deterioration trend in prediction accuracy and clustered. As a result, machine learning models A, B', D, and E belong to cluster X, machine learning models F and H' belong to cluster Y, and machine learning models C, G', and I belong to cluster Z, which may result in a change in the cluster configuration after relearning. After that, canary models are set for each cluster, and machine learning model D is set as canary models Xc, Yc, and Zc in cluster X, machine learning model H' in cluster Y, and machine learning model G' in cluster Z, respectively.

[0033] [Operation] Next, the operation of the information processing device 10 described above will be described.

[0034] The information processing device 10 first stores multiple machine learning models A to I that have been trained using training data (step S1 in FIG. 7, F1 in FIG. 2). The information processing device 10 may train and store machine learning models using training data, or may store machine learning models that have been input from outside.

[0035] Next, the information processing device 10 analyzes the characteristics of the multiple machine learning models A to I (step S2 in FIG. 7, symbol F2 in FIG. 2). For example, the information processing device 10 regards a portion of the learning data (e.g., evaluation data) as simulated time-series operational data and analyzes the deterioration trend, which is the temporal change in the prediction accuracy of the machine learning data.

[0036] Next, the information processing device 10 clusters the machine learning models A to I based on the deterioration trend of the prediction accuracy of each machine learning model (step S3 in FIG. 7, F3 in FIG. 2). In the example of FIG. 2, the machine learning models A to I are clustered into three clusters X, Y, and Z.

[0037] Next, the information processing device 10 sets a canary model, which is a machine learning model to be monitored, for each cluster (step S4 in FIG. 7 , F4 in FIG. 3 ). For example, the information processing device 10 selects, for each cluster, the machine learning model whose accuracy deteriorates the fastest among the machine learning models belonging to the cluster as the canary model. In the example of FIG. 3 , machine learning model E is selected as canary model Xc for cluster X, machine learning model H is selected as canary model Yc for cluster Y, and machine learning model C is selected as canary model Zc for cluster Z.

[0038] The information processing device 10 may generate and set a new canary model, which is a machine learning model to be monitored, for each cluster. For example, as shown in Fig. 5, a machine learning model having a characteristic that the deterioration tendency is worse (lower) than the deterioration tendencies of all machine learning models belonging to the cluster may be generated as a new canary model.

[0039] Next, the information processing device 10 monitors changes in the prediction accuracy of the canary models Xc, Yc, and Zc during operation using the operational data (step S5 in FIG. 7 , F4 in FIG. 3 ). Then, when the information processing device 10 detects a deterioration in the prediction accuracy of the canary models Xc, Yc, and Zc (Yes in step S6 in FIG. 7 ), it retrains the machine learning models included in the cluster to which the detected canary model belongs (step S7 in FIG. 7 , F5 in FIG. 3 ). For example, as shown in FIG. 3 , if the prediction accuracy of the canary model Yc in cluster Y decreases, all of the machine learning models B, G, and H or some of the machine learning models belonging to cluster Y are machine-learned using the operational data. This updates the machine learning models.

[0040] The information processing device 10 then repeats the above-described process for all machine learning models, including the retrained machine learning model. Specifically, the information processing device 10 analyzes the trend of deterioration in prediction accuracy for all machine learning models using operational data (step S8 in FIG. 7 ), and performs new clustering of the machine learning models based on the deterioration trend (step S3 in FIG. 7 ). The information processing device 10 then selects or creates a canary model for each new cluster, monitors the canary model, and retrains the machine learning model for the new cluster to which the canary model with degraded accuracy belongs (steps S4 to S7 in FIG. 7 ).

[0041] As described above, the information processing device 10 in this embodiment clusters multiple machine learning models and monitors a canary model set for each cluster. Therefore, there is no need to monitor the deterioration of the prediction accuracy of all machine learning models, which reduces the effort and cost involved in monitoring. Furthermore, if the prediction accuracy of a monitored canary model deteriorates, the machine learning model of the cluster to which the monitored canary model belongs is retrained. This eliminates the need to retrain all machine learning models, which reduces the effort and cost involved in retraining. As a result, it is possible to maintain and improve the prediction accuracy of machine learning models while reducing effort and cost.

[0042] Here, an application example of the information processing device 10 of this embodiment will be described in the medical and healthcare fields. For example, assume that the machine learning model is a model that predicts the number of days until hospitalization for each symptom A, B, ... I, and that there are multiple machine learning models A to I, as described above. In this case, the information processing device 10 clusters the multiple machine learning models A to I based on the tendency of deterioration in prediction accuracy, as described above, and monitors the canary models set for each cluster, thereby enabling retraining of the machine learning model of the cluster to which the degraded canary model belongs. This allows the accuracy of the machine learning models to be maintained or improved, allowing for accurate prediction of the number of days until hospitalization for each symptom, thereby supporting decision-making by doctors and others.

[0043] Second Embodiment Next, a second embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the configuration of the information processing device described in the above embodiment. Note that Figures 8 and 9 are diagrams for explaining the configuration, and these drawings may be relevant to any of the embodiments.

[0044] First, the hardware configuration of the information processing device 100 will be described with reference to Fig. 8. The information processing device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, for example: CPU (Central Processing Unit) 101 (arithmetic unit); ROM (Read Only Memory) 102 (storage device); RAM (Random Access Memory) 103 (storage device); programs 104 loaded into RAM 103; storage device 105 storing programs 104; drive device 106 for reading and writing data from and to a storage medium 110 external to the information processing device; communication interface 107 for connecting to a communication network 111 external to the information processing device; input / output interface 108 for inputting and outputting data; and bus 109 for connecting the various components.

[0045] 8 shows an example of the hardware configuration of the information processing device 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with only a part of the above-described configuration, such as excluding the drive device 106. Furthermore, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof, instead of the above-described CPU.

[0046] The information processing device 100 can then configure and include the classification unit 121, creation unit 122, and monitoring unit 123 shown in FIG. 9 by having the CPU 101 acquire and execute the program group 104. The program group 104 is stored in advance in the storage device 105 or the ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may also be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, with the drive device 106 reading out the program and supplying it to the CPU 101. However, the classification unit 121, creation unit 122, and monitoring unit 123 described above may also be configured using dedicated electronic circuits for realizing such means.

[0047] The classification unit 121 classifies the machine learning models based on preset characteristics of the multiple machine learning models. The creation unit 122 creates, for each classification, a monitored machine learning model that is a machine learning model to be monitored in the classification based on the machine learning models included in the classification. The monitoring unit 123 monitors changes in the prediction accuracy of the monitored machine learning model for each classification.

[0048] With the above-described configuration, the present disclosure categorizes multiple machine learning models and monitors the machine learning models to be monitored that are set for each category. This eliminates the need to monitor all machine learning models, thereby reducing the effort and cost required for monitoring. Furthermore, by monitoring the targets to be monitored that are set for each category in this manner, it is possible to retrain only the machine learning models that belong to a specific category. As a result, it is possible to maintain and improve the prediction accuracy of the machine learning models while reducing effort and cost.

[0049] In addition, at least one or more of the functions of the above-mentioned classification unit 121, creation unit 122, and monitoring unit 123 may be executed by an information processing device installed and connected anywhere on the network, that is, they may be executed by so-called cloud computing.

[0050] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-RWs, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program can also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0051] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each of the above-described embodiments can be combined with other embodiments as appropriate.

[0052] <Supplementary Notes> Some or all of the above embodiments may also be described as in the following supplementary notes. Below, an outline of the configurations of an information processing device, an information processing method, and a program according to the present disclosure will be described. However, the present disclosure is not limited to the following configurations. (Supplementary Note 1) An information processing device comprising: a classification unit that classifies machine learning models based on preset characteristics of the multiple machine learning models; a creation unit that creates, for each classification, a monitored machine learning model that is a machine learning model to be monitored in the classification based on the machine learning models included in the classification; and a monitoring unit that monitors, for each classification, changes in the prediction accuracy of the monitored machine learning model. (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the classification unit classifies the machine learning models based on the prediction accuracy of the machine learning models as the characteristic of the machine learning models. (Supplementary Note 3) The information processing device according to Supplementary Note 1, wherein the classification unit classifies the machine learning models based on a tendency for deterioration in the prediction accuracy of the machine learning models as the characteristic of the machine learning models. (Supplementary Note 4) The information processing device according to Supplementary Note 1, wherein the creation unit creates the monitored machine learning model for each classification based on the characteristics of the machine learning models included in the classification. (Supplementary Note 5) The information processing device according to Supplementary Note 2, wherein the creation unit creates the monitored machine learning model for each classification based on a predetermined index related to the prediction accuracy of the machine learning models included in the classification. (Supplementary Note 6) The information processing device according to Supplementary Note 5, wherein the creation unit selects, for each classification, the machine learning model whose index is lowest according to a predetermined criterion, and sets this as the monitored machine learning model. (Supplementary Note 7) The information processing device according to Supplementary Note 5, wherein the creation unit generates, for each classification, a machine learning model whose index is even lower according to a predetermined criterion than the index of the machine learning model included in the classification, and sets this as the monitored machine learning model.(Supplementary Note 8) The information processing device according to Supplementary Note 1, comprising a re-learning unit that, by monitoring changes in the prediction accuracy of the monitored machine learning model, re-learns the machine learning models included in a classification to which the monitored machine learning model whose prediction accuracy has changed to a predetermined situation belongs. (Supplementary Note 9) The information processing device according to Supplementary Note 8, wherein the re-learning unit, by monitoring changes in the prediction accuracy of the monitored machine learning model, re-learns the machine learning models included in a classification to which the monitored machine learning model whose deterioration in prediction accuracy meets a predetermined criterion belongs. (Supplementary Note 10) The information processing device according to Supplementary Note 8, wherein the classifier classifies the machine learning models based on the characteristics of a plurality of machine learning models including the re-learned machine learning model. (Supplementary Note 11) An information processing method that classifies the machine learning models based on predetermined characteristics of a plurality of machine learning models, creates, for each classification, a monitored machine learning model that is a machine learning model to be monitored in that classification based on the machine learning models included in that classification, and monitors changes in the prediction accuracy of the monitored machine learning models for each classification. (Supplementary Note 12) The information processing method according to Supplementary Note 11, wherein the machine learning models are classified based on the predictive accuracy of the machine learning models as the characteristic of the machine learning models. (Supplementary Note 12.1) The information processing method according to Supplementary Note 11, wherein the machine learning models are classified based on a tendency for deterioration in the predictive accuracy of the machine learning models as the characteristic of the machine learning models. (Supplementary Note 13) The information processing method according to Supplementary Note 11, wherein, for each classification, the monitored machine learning model is created based on the characteristics of the machine learning models included in the classification. (Supplementary Note 14) The information processing method according to Supplementary Note 12, wherein, for each classification, the monitored machine learning model is created based on a predetermined index related to the predictive accuracy of the machine learning models included in the classification.(Supplementary Note 14.1) The information processing method according to Supplementary Note 14, comprising: selecting, for each classification, the machine learning model whose index is lowest according to a predetermined criterion among the machine learning models included in that classification, and setting this as the supervised machine learning model. (Supplementary Note 14.2) The information processing method according to Supplementary Note 14, comprising: generating, for each classification, a machine learning model whose index is even lower according to a predetermined criterion than the index of the machine learning models included in that classification, and setting this as the supervised machine learning model. (Supplementary Note 15) The information processing method according to Supplementary Note 11, comprising: monitoring changes in prediction accuracy of the supervised machine learning model, and re-training the machine learning model included in a classification to which the supervised machine learning model whose prediction accuracy has changed to a predetermined situation belongs. (Supplementary Note 15.1) The information processing method according to Supplementary Note 15, comprising: monitoring changes in prediction accuracy of the supervised machine learning model, and re-training the machine learning model included in a classification to which the supervised machine learning model whose deterioration in prediction accuracy meets a predetermined criterion belongs. (Supplementary Note 16) The information processing method according to Supplementary Note 15, comprising classifying the machine learning models based on the characteristics of a plurality of the machine learning models including the retrained machine learning model. (Supplementary Note 17) A computer-readable storage medium storing a program causing a computer to execute processes of: classifying the machine learning models based on preset characteristics of a plurality of the machine learning models, creating, for each classification, a monitored machine learning model that is a machine learning model to be monitored in that classification based on the machine learning models included in that classification, and monitoring, for each classification, changes in the prediction accuracy of the monitored machine learning model.

[0053] REFERENCE SIGNS LIST 10 Information processing device 11 Analysis unit 12 Clustering unit 13 Monitoring target setting unit 14 Monitoring unit 15 Learning unit 16 Learning data storage unit 17 Operational data storage unit 18 Model storage unit 100 Information processing device 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Classification unit 122 Creation unit 123 Monitoring unit

Claims

1. An information processing device comprising: a classification unit that classifies multiple machine learning models based on predetermined characteristics of the models; a creation unit that creates, for each classification, a monitored machine learning model that is the machine learning model to be monitored in that classification based on the machine learning models included in that classification; and a monitoring unit that monitors, for each classification, changes in the predictive accuracy of the monitored machine learning model.

2. An information processing device according to claim 1, wherein the classification unit classifies the machine learning models based on the predictive accuracy of the machine learning models as the characteristics of the machine learning models.

3. An information processing device according to claim 1, wherein the classification unit classifies the machine learning models based on a tendency for the prediction accuracy of the machine learning models to deteriorate as the characteristic of the machine learning models.

4. An information processing device according to claim 1, wherein the creation unit creates the monitored machine learning model for each classification based on the characteristics of the machine learning models included in that classification.

5. An information processing device according to claim 2, wherein the creation unit creates the monitored machine learning model for each classification based on a predetermined index relating to the prediction accuracy of the machine learning model included in that classification.

6. An information processing device according to claim 5, wherein the creation unit selects, for each classification, the machine learning model whose index is the poorest according to a preset criterion, and sets the selected machine learning model as the machine learning model to be monitored.

7. An information processing device according to claim 5, wherein the creation unit generates, for each classification, a machine learning model whose index is lower than the index of the machine learning model included in that classification according to a predetermined standard, and sets the machine learning model as the monitored machine learning model.

8. An information processing device according to claim 1, comprising a re-learning unit that monitors changes in the prediction accuracy of the monitored machine learning model and re-learns the machine learning model included in a classification to which the monitored machine learning model whose prediction accuracy has changed to a predetermined situation belongs, by monitoring changes in the prediction accuracy of the monitored machine learning model.

9. An information processing device according to claim 8, wherein the re-learning unit monitors changes in the prediction accuracy of the monitored machine learning model, and re-learns the machine learning model included in a classification to which the monitored machine learning model belongs, the deterioration in prediction accuracy of which meets a preset criterion.

10. An information processing device according to claim 8, wherein the classification unit classifies the machine learning models based on the characteristics of the plurality of machine learning models including the retrained machine learning model.

11. An information processing method comprising: categorizing a plurality of machine learning models based on predetermined characteristics of the machine learning models; for each classification, creating a monitored machine learning model that is a machine learning model to be monitored in that classification based on the machine learning models included in that classification; and monitoring changes in the predictive accuracy of the monitored machine learning model for each classification.

12. An information processing method according to claim 11, wherein the machine learning models are classified based on the predictive accuracy of the machine learning models as the characteristic of the machine learning models.

13. An information processing method according to claim 11, wherein the supervised machine learning model is created for each classification based on the characteristics of the machine learning models included in that classification.

14. An information processing method according to claim 12, wherein the monitored machine learning model is created for each classification based on a predetermined index relating to the prediction accuracy of the machine learning model included in that classification.

15. An information processing method according to claim 11, comprising monitoring changes in the prediction accuracy of the monitored machine learning model, and re-training the machine learning model included in a classification to which the monitored machine learning model whose prediction accuracy has changed to a predetermined situation belongs.

16. The information processing method according to claim 15, further comprising classifying the machine learning models based on the characteristics of the plurality of machine learning models, including the retrained machine learning model.

17. A computer-readable storage medium storing a program that causes a computer to execute the following processes: categorizing a plurality of machine learning models based on predetermined characteristics of the models; for each classification, creating a monitored machine learning model that is a machine learning model to be monitored in that classification based on the machine learning models included in that classification; and monitoring changes in the predictive accuracy of the monitored machine learning model for each classification.

Images