Information processing apparatus, information processing method, and information processing program

Abstract

To improve analysis capability with which a generation model analyzes a situation change of a prediction model accompanying an operation.SOLUTION: The information processing apparatus includes a first recording unit, a control unit, and a second recording unit. The first recording unit records history information related to a development process of the prediction model. The control unit causes the generation model to output first evaluation information indicating an analysis result by analyzing a situation of the prediction model learned according to the development process on the basis of the history information. The second recording part records the first evaluation information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] A method has been proposed that can appropriately estimate the accuracy of a prediction model, focusing on the fact that a large difference between actual measured values ​​and predicted values ​​does not necessarily mean that the prediction accuracy of the prediction model has deteriorated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6708204 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional techniques do not necessarily improve the analytical ability of the generative model to analyze changes in the situation of the predictive model that occur during operation.

[0005] Therefore, the present invention proposes an information processing device, an information processing method, and an information processing program that can improve the analytical ability of a generative model to analyze changes in the situation of a predictive model that occur as the model is operated. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention comprises a first recording unit that records historical information relating to the development process of a predictive model, a control unit that causes a generative model to analyze the status of the predictive model learned in accordance with the development process based on the historical information and output first evaluation information indicating the analysis results, and a second recording unit that records the first evaluation information. [Effects of the Invention]

[0007] According to one aspect of the embodiment, the analytical capability of the generative model to analyze changes in the status of the predictive model as it is operated can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a situation in which the information processing device is put to practical use. [Figure 3] FIG. 3 is a flowchart showing the operation procedure at the development stage. [Figure 4] FIG. 4 is a flowchart showing the operation procedure in the operation stage. [Figure 5] FIG. 5 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 10 according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0011] (Embodiment) 1. Introduction In the development stage of a predictive model that uses past time series data to predict future time series data, the predictive performance of the predictive model is evaluated after learning. If the target predictive performance is achieved, the system moves to the operation stage, where it operates using the predictive model. However, if a problem occurs after operation begins, such as the predictive model not being able to demonstrate sufficient predictive performance, investigating the cause of the problem requires a lot of work and effort.

[0012] Therefore, the inventors of the present invention have devised a method in which a generative model performs an analysis of a predictive model based on historical information related to the development process of the predictive model and records the information on the analysis results output by the generative model during the development stage. When the predictive model enters the operational stage, the generative model analyzes changes in the status of the predictive model that accompany operation based on the information on the analysis results. This proposed technology of the present invention improves the analytical ability of the generative model to analyze changes in the status of the predictive model that accompany operation. For example, the proposed technology of the present invention enables the generative model to provide enhanced trend analysis and problem explanations for the predictive model during the operational stage.

[0013] To give a more specific example, according to the proposed technology of the present invention, for example, if a difference in predictive performance occurs between a predictive model in the development stage and a predictive model in the operational stage, the generative model can be made to respond to the cause of the difference, and the content of the response can be made more comprehensive. Furthermore, by enriching the content of the response in this way, for example, an operator can easily understand the cause of the performance degradation of the predictive model and take measures, thereby providing the advantage of reducing the work time and man-hours required for the response work to restore performance degradation.

[0014] 2. Functional configuration of information processing device First, an overview of an information processing device according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of the information processing device according to the embodiment.

[0015] 1, the information processing device 10 includes a data acquisition unit 11, a prediction model generation unit 12, a first recording unit 13, a control unit 14, and a second recording unit 15. As shown in FIG. 1, the information processing device 10 may be configured as a system including a model development system 10-1 used to develop a prediction model, and a model operation system 10-2 used to operate the prediction model.

[0016] First, the processes performed in the learning stage and the development stage will be described. In the learning stage, the model development system 10-1 develops a predictive model M11 using a dataset. The development process for developing the predictive model M11 is divided into a learning stage and an evaluation stage.

[0017] The learning process also includes a training process in which the prediction model M11 learns feature values ​​and trends in correct answers, and a verification process in which the prediction model M11 is repeatedly changed in hyperparameters while checking the prediction accuracy of the prediction model M11.

[0018] In the training step, the model development system 10-1 inputs learning data into the prediction model M11, thereby adjusting parameters (e.g., weights and biases) for the prediction model M11. In the verification step, the model development system 10-1 calculates prediction accuracy based on the prediction results output by the prediction model M11 when validation data is input to the prediction model M11. The model development system 10-1 then accepts, from a programmer or the like, an operation to adjust hyperparameters (e.g., the number of neurons, the number of layers, the number of learning iterations, and the learning coefficient) while checking the prediction accuracy, and executes the training step again using the accepted content. In this way, in the learning step, the training step and verification step are repeated until the desired prediction accuracy is achieved.

[0019] In the evaluation process, the model development system 10-1 evaluates the prediction performance based on the prediction results output by the prediction model M11 when test data is input into the prediction model M11 (a trained prediction model M11 that has been trained through the learning process).

[0020] According to the above, in the development stage, the process proceeds in the order of training process → verification process → evaluation process, and the timing when the training data and the verification data are used is during learning when the training process is executed. On the other hand, the timing when the test data is used is during evaluation when the evaluation process is executed. In addition, the training dataset may be divided into a training dataset and a verification dataset. In addition, a test dataset may be prepared separately from the training dataset and the verification dataset.

[0021] The model development system 10-1 may deploy the trained prediction model M11 that has achieved the desired prediction performance to a production environment (live environment). Specifically, the model development system 10-1 may deploy the trained prediction model M11 so that it is actually operated in the model operation system 10-2. The trained prediction model M11 deployed in this manner is referred to as the prediction model M12.

[0022] Therefore, in the operation stage, the model operation system 10-2 inputs operation data to the prediction model M12 and controls the provision of services based on the prediction results output by the prediction model M12. Below, each processing unit of the information processing device 10 will be described separately for the development stage and the operation stage.

[0023] <Development stage> In the development stage, for example, a data science team performs development work using the model development system 10-1.

[0024] (Data Acquisition Unit 11) In the development stage, the data acquisition unit 11 acquires a dataset. The dataset may be composed of a set of training data, a set of validation data, and a set of test data. The data acquisition unit 11 may also output the acquired dataset to the prediction model generation unit 12.

[0025] (Prediction model generation unit 12) The prediction model generation unit 12 executes a development process in which the prediction model M11 is trained using a dataset to generate a prediction model M11 that can achieve a desired prediction performance. As described above, the development process is divided into a learning process and an evaluation process, and the learning process includes a training process and a verification process. The processing then proceeds in the order of the training process, verification process, and evaluation process.

[0026] For example, in the training step, the prediction model generation unit 12 inputs learning data into the prediction model M11, thereby adjusting parameters (e.g., weights and biases) for the prediction model M11. In the verification step, the prediction model generation unit 12 calculates prediction accuracy based on the prediction result output by the prediction model M11 when verification data is input to the prediction model M11, and then executes the training step again using hyperparameters adjusted based on the prediction accuracy. In other words, the prediction model generation unit 12 repeats the training step and the verification step until the prediction model M11 achieves the desired prediction accuracy.

[0027] In the evaluation step, the prediction model generation unit 12 evaluates the prediction performance based on the prediction result output by the prediction model M11 when test data is input to the prediction model M11. For example, the prediction model generation unit 12 may calculate an index value (e.g., a model evaluation index value such as a precision rate, an accuracy rate, or a recall rate) for evaluating the prediction performance based on the prediction result.

[0028] (First recording unit 13) The first recording unit 13 records history information related to the development process of the prediction model M11 in the history information DB1. For example, the first recording unit 13 may record history information indicating the content of the learning process of the development process as the history information in the history information DB1.

[0029] Here, the history information indicating the content of the learning process may include learning data used to train the prediction model M11, validation data used to adjust the hyperparameters of the prediction model M11, and information on prediction results obtained using the validation data and based on the prediction results of the prediction model M11. The information based on the prediction results may be the predicted values ​​themselves as the prediction results, or information on prediction accuracy calculated based on the predicted values. Furthermore, the history information indicating the content of the learning process may also include adjustment histories of weights and biases and adjustment histories of hyperparameters.

[0030] Furthermore, the first recording section 13 may record, as the history information, history information indicating the contents of the evaluation process in the development process in the history information DB1.

[0031] The history information indicating the contents of the evaluation process may include test data used in the performance evaluation of the prediction model M11 and information on prediction results obtained using the test data and based on the prediction results of the prediction model M11. The information based on the prediction results may be the predicted values ​​themselves as the prediction results, or model evaluation index values ​​calculated based on the predicted values.

[0032] (Control unit 14) The control unit 14 causes the generation model M2 to analyze the status of the trained prediction model M11 trained according to the development process. Specifically, the control unit 14 causes the generation model M2 to perform an analysis process of analyzing the status of the prediction model M11 based on the history information stored in the history information DB1, and to output first evaluation information indicating the analysis results. For example, the control unit 14 causes the generation model M2 to analyze the prediction performance of the prediction model M11 by providing the generation model M2 with a command (prompt) instructing the generation model M2 to analyze information related to the prediction performance of the prediction model M11 based on the history information.

[0033] Here, the information regarding the predictive performance of the predictive model M11 may be the status of the predictive model M11 in the training process or the validation process, a graph summarizing the prediction results, considerations regarding the predictive performance, etc. Furthermore, the status of the predictive model M11 in the training process or the validation process may be, for example, a portion of the thought process that led the predictive model M11 to output such a prediction result (for example, a black-box portion that is the weighting criteria used by the predictive model M11). The generative model M2 analyzes such information regarding the predictive performance in response to the prompt and outputs first evaluation information indicating the analysis results. The first evaluation information may be composed of text or a graph indicating the analysis results.

[0034] Furthermore, the prompt may be created by, for example, an operator of the model development system 10-1, or may be automatically created by the control unit 14 in accordance with predetermined conditions. An example of a prompt might be something like, "You are an excellent analyst. Use the history information stored in the history information DB1 to analyze the predictive performance of the predictive model M11 and create an evaluation report of the analysis results."

[0035] By using the historical information according to the embodiment, the generative model M2 can perform advanced analysis of the query, thereby enriching the explanation for the query, i.e., the content of the first evaluation information.

[0036] The control unit 14 may input the history information stored in the history information DB1 to the generative model M2 in advance as learning data so that the generative model M2 can analyze information related to prediction performance. The generative model M2 may be trained based on a language model that performs natural language processing, such as a Generative Pre-trained Transformer (GPT) or a Bidirectional Encoder Representations from Transformers (BERT).

[0037] (Second recording unit 15) The second recording unit 15 records the first evaluation information in the evaluation information DB 2. For example, the second recording unit 15 acquires the first evaluation information from the generative model M2 and records the acquired first evaluation information in the evaluation information DB 2.

[0038] <Operational Phase> In the operation stage, for example, an IT production team performs practical work using the model operation system 10-2.

[0039] (Data Acquisition Unit 11) In the operation stage, the data acquisition unit 11 acquires operation data. The operation data may be actual measurement data measured in an actual operating environment. The data acquisition unit 11 may input the acquired operation data into a prediction model M12 to output future prediction values.

[0040] (First recording unit 13) The first recording unit 13 may store actual prediction results, which are predicted values ​​output by the prediction model M12, in the history information DB1. As a result, the history information DB1 further stores history information indicating the contents of each process in the development stage and actual prediction results. Therefore, for example, the generative model M2 can track information that is useful for investigating the cause of a performance problem with the prediction model M12 by referring to the history information DB1.

[0041] (Control unit 14) The prediction model M12 deteriorates over time, and its performance will decline unless regular attention is paid and appropriate measures are taken. Regular maintenance is necessary to ensure high-quality performance, but the performance of the prediction model M12 is affected by various factors. For example, the performance of the prediction model M12 is likely to depend not only on the structure of the prediction model M12, but also on the data and tuning status, and the status of regular updates and relearning (retraining). Therefore, it is impossible to ensure high-quality performance without properly investigating the cause of performance degradation, but performing this work manually is time-consuming and costly. Therefore, the control unit 14 effectively utilizes the generation model M2 to analyze the status of the prediction model M12.

[0042] For example, the control unit 14 outputs second evaluation information indicating the analysis results by having the generation model M2 analyze the predictive performance at a predetermined timing associated with the operation of the predictive model M12 based on historical information and first evaluation information as the status of the predictive model M12.

[0043] The control unit 14 may cause the generative model M2 to perform analysis at a predetermined cycle, or may cause the generative model M2 to perform analysis when the performance of the prediction model M12 deteriorates.

[0044] For example, the control unit 14 may detect whether the prediction performance of the prediction model M12 has deteriorated by comparing a predicted value by the prediction model M12 with an actual measurement value corresponding to the predicted value. When the control unit 14 detects that the prediction performance of the prediction model M12 has deteriorated, the control unit 14 may analyze information about the prediction performance at the time when the prediction performance has deteriorated based on the history information and the first evaluation information.

[0045] For example, when the prediction performance of the prediction model M12 deteriorates, the control unit 14 can recognize that a difference in performance has occurred between the prediction model M11, which is a prediction model at the development stage, and the prediction model M12. In such a case, the control unit 14 can analyze the factors that caused the difference based on the history information and the first evaluation information as information related to the prediction performance. For example, the control unit 14 can cause the generation model M2 to analyze the factors that caused the difference by providing the generation model M2 with a command (prompt) that instructs the generation model M2 to analyze the factors that caused the difference by referring to the history information DB1, and that includes the first evaluation information as a condition for the analysis.

[0046] The generative model M2 analyzes information about the predictive performance of the predictive model M12 in response to the prompts and outputs second evaluation information indicating the analysis results. For example, the generative model M2 can analyze the causes of deterioration by comparing the historical information in the historical information DB1 with the actual prediction results to track evaluation indicators such as drift and deterioration throughout a series of processes. In addition, because performance issues with the predictive model M12 may be caused by changes in the data or data source, the generative model M2 can also track indicators such as data size, update frequency, region, category, and type by comparing the historical information in the historical information DB1 with the actual prediction results.

[0047] The second evaluation information may be composed of text or graphs showing the analysis results. An example of a prompt might be, "Analysis results such as XX were obtained regarding the performance of the predictive model. Please analyze the cause of the performance degradation based on the information obtained from these analysis results."

[0048] (Second recording unit 15) The second recording unit 15 records the second evaluation information in the evaluation information DB 2. Specifically, the second recording unit 15 records the second evaluation information in the evaluation information DB 2 so that it can be used in future analyses together with the first evaluation information. As a result, the control unit 14 can provide the generative model M2 with a prompt that includes not only the first evaluation information but also the second evaluation information as conditions for analysis, thereby allowing the generative model M2 to output a better explanation.

[0049] 3. Operation Procedure of Information Processing Device The operation procedure of the information processing device 10 will be described based on a practical example of the information processing device 10. Prior to the description of the operation procedure, an example of a situation in which the information processing device 10 is put to practical use will be described with reference to Fig. 2. For example, the information processing device 10 can be implemented as a control device that performs vibration management control of production equipment using IoT data obtained from IoT devices installed at predetermined locations.

[0050] The model development system 10-1 portion of such a control device may be placed, for example, in a laboratory where a data science team or the like is located, and is connected to an IoT device DV1 via a network NW. The IoT device DV1 includes a sensor SN1 or is connected to the sensor SN1, and transmits sensor data detected by the sensor SN1 with respect to an object O1 to the model development system 10-1 as IoT data.

[0051] The model development system 10-1 can use this IoT data as a data set for developing the predictive model M11. In this example, the training data in the data set can include structure and weight data of the device, vibration data, equipment current data, temperature and humidity data, and vibration isolation device setting data. The verification data in the data set can include vibration data, equipment current data, temperature and humidity data, and vibration isolation device setting data.

[0052] A vibration isolation apparatus is, for example, a vibration isolation system introduced as a vibration countermeasure in a vibration environment. For example, a vibration isolation apparatus is installed in a semiconductor manufacturing facility. Such a vibration isolation apparatus includes a coil spring, a base on which a device to be subjected to vibration suppression is placed, a sensor that detects vibrations of the base, an actuator that generates minute vibrations on the base, and a controller that drives the actuator based on a detection signal from the sensor. Therefore, the vibration isolation apparatus setting data may include setting data for these devices included in the vibration isolation apparatus, sensor data, control values ​​by the controller, etc.

[0053] On the other hand, the model operation system 10-2 portion of the control device may be placed in a control room where an IT production team or the like is located, and is connected to an IoT device DV2 via a network NW. The IoT device DV2 includes a sensor SN2 or is connected to the sensor SN2, and transmits sensor data detected by the sensor SN2 with respect to the object O2 to the model operation system 10-2 as IoT data.

[0054] The model operation system 10-2 can use this IoT data as operation data for prediction by the prediction model M12.

[0055] The IoT device DV2 may be, for example, a vibration isolation system for a precision device placed in a customer's environment, and the model operation system 10-2 may remotely control the IoT device DV2 based on the prediction results of the prediction model M12. Also, as shown in Fig. 2, the model operation system 10-2 may include a service system SV that provides services related to the IoT device DV2 to customers.

[0056] For example, suppose that the generative model M2 analyzes the cause of a performance degradation of the predictive model M12 and outputs second evaluation information that suggests a failure of the IoT device DV2, rather than a performance degradation of the predictive model M12. In such a case, the service system SV can provide a service of notifying the customer of the failure of the IoT device DV2, the cause of the failure, and countermeasures.

[0057] In addition, when the information processing device 10 is a control device that performs vibration management control of production equipment, the object O1 (object O2) may be production equipment (e.g., semiconductor production equipment) in which a vibration isolation device is installed, and the sensor SN1 may be a vibration sensor provided in the vibration isolation device. Furthermore, the prediction model M11 (prediction model M12) may be a prediction model that predicts vibrations in the production equipment. Furthermore, the service system SV may be a system that actively controls the vibration isolation device using vibration predictions of the production equipment.

[0058] <Operation procedure during development> Fig. 3 is a flowchart showing the operation procedure at the development stage. The step numbers shown in Fig. 3 correspond to the step numbers written on the development stage side of Fig. 2. Fig. 3 shows the operation procedure of the model development system 10-1 part of the information processing device 10.

[0059] First, the prediction model generation unit 12 executes a learning process (step S11) in which training and verification of the prediction model M11 are repeated using the data set (IoT data) acquired by the data acquisition unit 11. In the training process, learning data from the data set is used, and in the verification process, verification data from the data set is used.

[0060] If the prediction model M11 achieves the target prediction accuracy, the prediction model generation unit 12 executes an evaluation step to evaluate the general-purpose performance of the prediction model at the current time (step S12). In the evaluation step, test data from the dataset is used.

[0061] The first recording unit 13 records the history information indicating the content of the learning process and the history information indicating the content of the evaluation process in the history information BD1 (step S13).

[0062] Next, the control unit 14 inputs the historical information stored in the historical information BD1 into the generative model M2 as learning data (step S14).

[0063] Furthermore, the control unit 14 inputs a prompt to the generation model M2 to instruct the generation model M2 to analyze the prediction performance of the prediction model M11 (step S15). Specifically, in response to access from the prediction model M11, the control unit 14 creates a prompt inquiring about information related to the prediction performance of the prediction model M11, and inputs the created prompt to the generation model M2.

[0064] In response to the prompt, the generative model M2 analyzes information about the prediction performance of the prediction model M11 and outputs first evaluation information indicating the analysis result. Therefore, the second recording unit 15 records the first evaluation information indicating the analysis result in the evaluation information DB2 (step S16).

[0065] <Operational procedure during operation> Fig. 4 is a flowchart showing the operation procedure in the operation phase. The step numbers shown in Fig. 4 correspond to the step numbers written on the operation phase side of Fig. 2. Fig. 4 shows the operation procedure of the model operation system 10-2 part of the information processing device 10.

[0066] First, the data acquisition unit 11 executes an operation process using a prediction model (step S21). In the operation process, the data acquisition unit 11 inputs the acquired operation data (IoT data) into the prediction model M12, thereby outputting a future prediction value.

[0067] The first recording unit 13 acquires the predicted values ​​output by the prediction model M12 and stores them in the history information DB1 (step S22). Every time the prediction model M12 outputs a predicted value, the first recording unit 13 stores the output predicted value in the history information DB1. When an actual measurement value corresponding to each predicted value is obtained, the first recording unit 13 may associate the actual measurement value with the predicted value and store it in the history information DB1. The first recording unit 13 may input the predicted value to a service system SV so that a service corresponding to the predicted value is realized.

[0068] When a sufficient number of predicted values ​​have been accumulated, the control unit 14 refers to the history information DB1 and determines whether the prediction performance of the prediction model M12 has deteriorated (step S23). For example, the control unit 14 determines whether the prediction performance of the prediction model M12 has deteriorated by comparing the predicted value by the prediction model M12 with the actual measurement value corresponding to the predicted value.

[0069] If the prediction performance of the prediction model M12 has not deteriorated (step S23; No), the control unit 14 returns the process to step S21.

[0070] On the other hand, when the control unit 14 determines that the prediction performance of the prediction model M12 has deteriorated (step S23; Yes), the control unit 14 creates a prompt that instructs the control unit 14 to analyze the cause of the performance difference between the prediction model M11 and the prediction model M12 based on the first evaluation information, and inputs the created prompt to the generation model M2 (step S24). Specifically, the control unit 14 creates a prompt inquiring about the cause of the deterioration of the prediction performance of the prediction model M12, and inputs the created prompt to the generation model M2.

[0071] In addition, the control unit 14 may calculate the difference in performance between the prediction model M11 and the prediction model M12, and if the difference becomes larger than a reference value (i.e., if the prediction performance of the prediction model M12 has significantly deteriorated compared to the prediction model M11), it may output an alert to have the IT production team create a prompt.

[0072] The generative model M2 accesses the model operation system 10-2 to perform analysis in response to the prompt. In response to the access by the generative model M2, the control unit 14 acquires the history information stored in the history information DB1 and the actual prediction results accumulated in the history information DB1, and inputs these to the generative model M2 (step S25). Based on the input information, the generative model M2 analyzes information related to the prediction performance defined by the prompt, and outputs second evaluation information indicating the analysis results.

[0073] The second recording unit 15 records the second evaluation information in the evaluation information DB2 (step S26). Specifically, the second recording unit 15 records the second evaluation information in the evaluation information DB2 so that the second evaluation information can be used together with the first evaluation information for future analysis.

[0074] 4. Other Embodiments In the above embodiment, an example has been shown in which the information processing device 10 is implemented as a system including a model development system 10-1 and a model operation system 10-2. However, the information processing device 10 may also be implemented as a device independent of the model development system 10-1 and the model operation system 10-2. In this case, the information processing device 10 is connected to the model development system 10-1 and the model operation system 10-2 so as to be able to communicate with each other.

[0075] Furthermore, the information processing device 10 does not necessarily have to have the history information DB 1 and the evaluation information DB 2. For example, the history information DB 1 and the evaluation information DB 2 may be implemented in an external device different from the information processing device 10.

[0076] In the above embodiment, the information processing device 10 is a control device that performs vibration management control of production equipment. However, there is no limitation on the type of control device that the information processing device 10 is implemented as. For example, the information processing device 10 may be a control device that performs indoor air conditioning management control.

[0077] [5. Hardware Configuration] The information processing device 10 according to the embodiment may be realized, for example, by a computer 1000 configured as shown in Fig. 5. Fig. 5 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 10 according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0078] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0079] The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0080] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. The CPU 1100 also outputs generated data to the output device via the input / output interface 1600.

[0081] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0082] For example, when the computer 1000 functions as the information processing device 10 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to implement each processing unit of the information processing device 10. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0083] [6. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0084] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0085] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0086] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0087] 10. Information processing equipment 10-1 Model Development System 10-2 Model Operation System 11 Data Acquisition Section 12 Prediction model generation unit 13 First Recording Section 14 Control Unit 15 Second Recording Section DB1 History Information DB2 evaluation information

Claims

1. a first recording unit that records history information related to the development process of the predictive model; a control unit that causes a generative model to analyze a status of the predictive model learned according to the development process based on the history information, and outputs first evaluation information indicating an analysis result; a second recording unit that records the first evaluation information; An information processing device comprising:

2. the first recording unit records, as the history information, history information indicating the content of a learning process within the development process; The history information indicating the contents of the learning process includes learning data used to learn the prediction model, verification data used to tune the prediction model, and information based on a prediction result by the prediction model obtained using the verification data. The information processing device according to claim 1 .

3. the first recording unit records, as the history information, history information indicating the content of an evaluation process in the development process; The history information indicating the contents of the evaluation step includes test data used in performance evaluation of the prediction model and information based on a prediction result by the prediction model obtained using the test data. The information processing device according to claim 1 .

4. the control unit causes the generative model to analyze the predictive performance of the predictive model by providing the generative model with a directive that instructs the generative model to analyze information about the predictive performance of the predictive model based on the history information. The information processing device according to claim 1 .

5. In the development stage of developing the predictive model, the first recording unit records the history information; the control unit causes the generative model to output the first evaluation information; the second recording unit records the first evaluation information; The history information and the first evaluation information are recorded in a database as information to be used for analyzing the status of the developed prediction model when the developed prediction model reaches an operation stage in which it is operated. The information processing device according to claim 1 .

6. In an operation stage of operating the prediction model, the control unit causes the generative model to analyze, based on the history information and the first evaluation information, prediction performance at a predetermined timing associated with the operation of the developed prediction model as a status of the developed prediction model, and outputs second evaluation information indicating an analysis result; The information processing device according to claim 5 .

7. When it is detected that the prediction performance of the developed prediction model has deteriorated based on a comparison between a prediction result of the developed prediction model and an actual result, the control unit analyzes information about the prediction performance at the current time when the prediction performance has deteriorated based on the history information and the first evaluation information. The information processing device according to claim 6 .

8. when a performance difference occurs between the prediction model at the development stage and the developed prediction model due to a decrease in prediction performance of the developed prediction model, the control unit analyzes factors causing the difference as information regarding the prediction performance based on the history information and the first evaluation information. The information processing device according to claim 7 .

9. the control unit causes the generative model to analyze a factor causing the difference based on the history information, by providing the generative model with an instruction statement including the first evaluation information as an analysis condition. The information processing device according to claim 8 .

10. The second recording unit records the second evaluation information together with the first evaluation information so that the second evaluation information can be used for future analysis. The information processing device according to claim 6 .

11. An information processing method executed by an information processing device, a first recording step of recording history information relating to the development process of the predictive model; a control step of causing a generative model to analyze the status of the predictive model learned according to the development step based on the history information, and outputting first evaluation information indicating the analysis result; a second recording step of recording the first evaluation information; An information processing method including:

12. a first recording step of recording historical information relating to the development process of the predictive model; a control procedure for causing a generative model to analyze a status of the predictive model learned according to the development process based on the history information, and outputting first evaluation information indicating an analysis result; a second recording step of recording the first evaluation information; An information processing program that causes a computer to execute the above.

Images