Information processing device, information processing method, and information processing program

Abstract

To improve analysis capability with which a generation model analyzes a situation change of a prediction model accompanying an operation.SOLUTION: The information processing apparatus includes a first recording unit, a control unit, and a second recording unit. The first recording unit records history information related to a development process of the prediction model. The control unit causes the generation model to output first evaluation information indicating an analysis result by analyzing a situation of the prediction model learned according to the development process on the basis of the history information. The second recording part records the first evaluation information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program. [Background Art]

[0002] A technique capable of appropriately estimating the accuracy of a prediction model has been proposed, focusing on the fact that a large difference between an actually measured value and a predicted value does not necessarily mean that the prediction accuracy of the prediction model has deteriorated. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent No. 6708204 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, the above-described conventional technique does not necessarily improve the analysis capability of a generative model that analyzes a change in situation of a prediction model accompanying operation of the prediction model.

[0005] Accordingly, the present invention proposes an information processing apparatus, an information processing method, and an information processing program capable of improving the analysis capability of a generative model that analyzes a change in situation of a prediction model accompanying operation of the prediction model. [Means for Solving the Problem]

[0006] An information processing apparatus according to an aspect of the present invention includes: a first recording unit that records history information related to a development process of a prediction model; a control unit that causes a generative model to output first evaluation information indicating an analysis result by causing the generative model to analyze a situation of the prediction model learned in accordance with the development process based on the history information; and a second recording unit that records the first evaluation information. [Effect of the Invention]

[0007] According to one embodiment, the analytical capability of the generative model in analyzing changes in the status of the predictive model during operation can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing the functional configuration of an information processing apparatus according to an embodiment. [Figure 2] Figure 2 illustrates an example of a scenario in which an information processing device is put into practical use. [Figure 3] Figure 3 is a flowchart showing the operation procedure during the development phase. [Figure 4] Figure 4 is a flowchart showing the operating procedure during the operational phase. [Figure 5] Figure 5 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device 10 according to the embodiment. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The one or more embodiments (including examples, modifications, and applications) described below can each be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in appropriate combination with at least some of the other embodiments. These embodiments may contain novel features that differ from each other. Therefore, these embodiments may contribute to solving different objectives or problems and may produce different effects.

[0011] (Embodiment) [1. Introduction] In the development phase of a predictive model that uses past time series data to forecast future time series data, the predictive performance is evaluated after the model has been trained. If the target predictive performance is achieved, the system using the predictive model is put into operation. However, if problems arise after the start of operation, such as the predictive model not being able to perform adequately, investigating the cause of the problem requires a lot of work time and effort.

[0012] Therefore, the inventors of the present invention have arrived at a method in which, during the development phase, the generative model is made to perform an analysis of the predictive model based on historical information related to the development process of the predictive model, and the information of the analysis results output by the generative model is recorded. Then, when the predictive model reaches the operational phase, the generative model is made to analyze the changes in the status of the predictive model that occur during operation, based on the information of the analysis results. According to the proposed technology of the present invention, it becomes possible to improve the analytical ability of the generative model in analyzing changes in the status of the predictive model that occur during operation. For example, according to the proposed technology of the present invention, it becomes possible to enhance the generative model's analysis of trends and explanations of problems in the predictive model during the operational phase.

[0013] To give a more specific example, according to the proposed technology of the present invention, if there is a difference in prediction performance between the prediction model in the development phase and the prediction model in the operation phase, the generative model can be made to respond with the factors that caused that difference, and the content of that response can be made more comprehensive. Furthermore, by making the content of the response more comprehensive in this way, for example, operators can easily identify the cause of the decline in the performance of the prediction model and take countermeasures, resulting in the benefit of reducing the time and effort required for corrective work to restore performance.

[0014] [2. Functional Configuration of Information Processing Equipment] First, an overview of the information processing apparatus according to one embodiment of the present invention will be described using Figure 1. Figure 1 is a block diagram showing the functional configuration of the information processing apparatus according to the embodiment.

[0015] As shown in FIG. 1, an information processing apparatus 10 includes a data acquisition unit 11, a prediction model generation unit 12, a first recording unit 13, a control unit 14, and a second recording unit 15. Note that the information processing apparatus 10 may be configured as a system including a model development system 10-1 used for developing a prediction model and a model operation system 10-2 used for operating the prediction model, as shown in FIG. 1.

[0016] First, processing performed in a learning phase and processing performed in a development phase will be described. In the learning phase, the model development system 10-1 develops a prediction model M11 using a dataset. A development process for developing the prediction model M11 is divided into a learning process and an evaluation process.

[0017] The learning process includes a training process of causing the prediction model M11 to learn feature amounts and tendencies of correct answers, and a verification process of causing repeated changes to hyperparameters while checking the prediction accuracy of the prediction model M11.

[0018] In the training process, the model development system 10-1 inputs training data to the prediction model M11, thereby causing the prediction model M11 to adjust parameters (for example, weights and biases). In the verification process, the model development system 10-1 calculates prediction accuracy based on a prediction result output by the prediction model M11 when verification data is input to the prediction model M11. Then, the model development system 10-1 receives an operation of adjusting hyperparameters (for example, the number of neurons, the number of layers, the number of learning iterations, a learning coefficient) from a programmer or the like while confirming prediction accuracy, and re-executes the training process according to the received content. As described above, in the learning process, the training process and the verification process are repeated until a target prediction accuracy is achieved.

[0019] In the evaluation process, the model development system 10-1 evaluates prediction performance based on a prediction result output by the prediction model M11 when test data is input to the prediction model M11 (a trained prediction model M11 that has completed learning in the learning process).

[0020] According to the above, in the development stage, processing proceeds in the order of training process → verification process → evaluation process, and the timing at which learning data and verification data are used is the learning time when the learning process is executed. On the other hand, the timing at which test data is used is the evaluation time when the evaluation process is executed. Further, the learning data set may be divided into a learning data set and a verification data set. Also, a test data set may be prepared separately from the learning data set and the verification data set.

[0021] Note that the model development system 10-1 may deploy the trained prediction model M11 that has achieved the target prediction performance to a production environment (actual operating environment). Specifically, the model development system 10-1 may deploy the trained prediction model M11 so that it is actually operated in the model operation system 10-2. The trained prediction model M11 deployed in this manner is referred to as prediction model M12.

[0022] Therefore, in the operation stage, the model operation system 10-2 inputs operation data to the prediction model M12, and performs control such that a service based on the prediction result output by the prediction model M12 is provided. Hereinafter, each processing unit included in the information processing apparatus 10 will be described separately for the development stage and the operation stage.

[0023] <Development Stage> In the development stage, for example, development work using the model development system 10-1 is performed by a data science team.

[0024] (Data Acquisition Unit 11) In the development stage, the data acquisition unit 11 acquires a data set. The data set may be composed of a set of learning data, a set of verification data, and a set of test data. Further, the data acquisition unit 11 may output the acquired data set to the prediction model generation unit 12.

[0025] (Prediction Model Generation Unit 12) The prediction model generation unit 12 performs a development process in which it generates a prediction model M11 that can achieve the desired prediction performance by training the prediction model M11 using the dataset. As described above, the development process is divided into a learning process and an evaluation process, and the learning process includes a training process and a validation process. The processes then proceed in the order of training process, validation process, and evaluation process.

[0026] For example, in the training phase, the prediction model generation unit 12 inputs training data into the prediction model M11, causing the prediction model M11 to adjust its parameters (e.g., weights and biases). In the validation phase, the prediction model generation unit 12 inputs validation data into the prediction model M11 and calculates the prediction accuracy based on the prediction results output by the prediction model M11. Based on this prediction accuracy, the unit 12 then executes the training phase again using the adjusted hyperparameters. In other words, the prediction model generation unit 12 repeats the training phase and the validation phase until the prediction model M11 achieves the desired prediction accuracy.

[0027] Furthermore, in the evaluation process, the prediction model generation unit 12 evaluates the prediction performance based on the prediction results output by the prediction model M11 when test data is input to the prediction model M11. For example, the prediction model generation unit 12 may calculate indicator values ​​for evaluating prediction performance (e.g., model evaluation indicator values ​​such as precision, accuracy, and recall) based on the prediction results.

[0028] (First Record Section 13) The first recording unit 13 records historical information related to the development process of the prediction model M11 in the history information DB1. For example, the first recording unit 13 may record historical information indicating the content of the learning process within the development process in the history information DB1 as historical information.

[0029] Here, the historical information indicating the content of the learning process may include the training data used to train the prediction model M11, the validation data used to adjust the hyperparameters of the prediction model M11, and the prediction results obtained using the validation data, which are based on the prediction results of the prediction model M11. The information based on the prediction results may include the predicted values ​​themselves as prediction results, or information on the prediction accuracy calculated based on the predicted values. Furthermore, the historical information indicating the content of the learning process may also include the history of weight and bias adjustments and the history of hyperparameter adjustments.

[0030] Furthermore, the first recording unit 13 may record historical information in the historical information DB1, which indicates the content of the evaluation process among the development processes, as historical information.

[0031] The historical information indicating the content of the evaluation process may include test data used to evaluate the performance of the prediction model M11, and prediction results obtained using the test data, which are based on the prediction results of the prediction model M11. Information based on prediction results may include the predicted values ​​themselves as prediction results, or model evaluation index values ​​calculated based on the predicted values.

[0032] (Control Unit 14) The control unit 14 instructs the generation model M2 to analyze the status of the trained prediction model M11, which has been trained according to the development process. Specifically, the control unit 14 instructs the generation model M2 to perform an analysis process that analyzes the status of the prediction model M11 based on the history information stored in the history information DB1, and also outputs first evaluation information that shows the analysis results. For example, the control unit 14 instructs the generation model M2 to analyze the prediction performance of the prediction model M11 by giving it an instruction statement (prompt) that tells it to analyze information regarding the prediction performance of the prediction model M11 based on the history information.

[0033] Here, information regarding the predictive performance of the predictive model M11 may include the status of the predictive model M11 during the training and validation phases, graphs summarizing the prediction results, and considerations regarding the predictive performance. Furthermore, the status of the predictive model M11 during the training and validation phases may include, for example, the thought process behind why the predictive model M11 arrived at such a prediction result (e.g., the weighting criteria used by the predictive model M11, which are a black box). The generative model M2 analyzes this information regarding predictive performance according to prompts and outputs first evaluation information showing the analysis results. The first evaluation information may consist of text and graphs showing the analysis results.

[0034] Furthermore, prompts may be created, for example, by an operator of the model development system 10-1, or a configuration may be adopted in which the control unit 14 automatically creates them according to predetermined conditions. An example of a prompt might be, "You are an excellent analyst. Use the historical information stored in the historical information DB1 to analyze the predictive performance of the predictive model M11 and create an evaluation report of the analysis results."

[0035] The generative model M2 can perform advanced analysis of queries by using the historical information according to the embodiment, and as a result, it becomes possible to enrich the explanation of the query, i.e., the content of the first evaluation information.

[0036] Furthermore, the control unit 14 may pre-input historical information stored in the historical information DB1 as training data into the generative model M2 so that the generative model M2 can analyze information regarding its prediction performance. In addition, the generative model M2 may be trained based on a language model that performs natural language processing, such as GPT (Generative Pre-trained Transformer) or BERT (Bidirectional Encoder Representations from Transformers).

[0037] (Second Record Section 15) The second recording unit 15 records the first evaluation information in the evaluation information DB2. For example, the second recording unit 15 obtains the first evaluation information from the generation model M2 and records the obtained first evaluation information in the evaluation information DB2.

[0038] <Operational Phase> During the operational phase, for example, the IT production team will perform practical tasks using the model operational system 10-2.

[0039] (Data acquisition unit 11) During the operational phase, the data acquisition unit 11 acquires operational data. This operational data may be actual measured data obtained in the actual operating environment. The data acquisition unit 11 may also input the acquired operational data into the prediction model M12 to output future predicted values.

[0040] (First Record Section 13) The first recording unit 13 may store the actual prediction results, which are the predicted values ​​output by the prediction model M12, in the history information DB1. As a result, the history information DB1 will further store history information and actual prediction results that show the content of each process during the development stage. Therefore, for example, the generation model M2 can refer to the history information DB1 to track information that is useful for investigating the cause of performance problems in the prediction model M12.

[0041] (Control Unit 14) The predictive model M12 degrades with use, and its performance will decline unless it is regularly monitored and properly addressed. While regular maintenance is necessary to ensure high-quality performance, the performance of the predictive model M12 is influenced by various factors. For example, the performance of the predictive model M12 is thought to depend not only on its structure but also on the data, tuning status, and the status of regular updates and retraining. Therefore, it is impossible to ensure high-quality performance without properly investigating the cause of performance degradation, but performing this task manually is time-consuming and costly. To address this, the control unit 14 utilizes the generative model M2 to analyze the status of the predictive model M12.

[0042] For example, the control unit 14 outputs second evaluation information showing the analysis results by having the generation model M2 analyze the prediction performance at a predetermined timing associated with the operation of the prediction model M12, based on historical information and first evaluation information, as a status of the prediction model M12.

[0043] The control unit 14 may perform analysis on the generation model M2 at predetermined intervals, or it may perform analysis on the generation model M2 when the performance of the prediction model M12 deteriorates.

[0044] For example, the control unit 14 may detect whether the prediction performance of the prediction model M12 has deteriorated by comparing the predicted value from the prediction model M12 with the measured value corresponding to the predicted value. If the control unit 14 detects that the prediction performance of the prediction model M12 has deteriorated, it may have the control unit 14 analyze information regarding the prediction performance at the time the performance deteriorated, based on historical information and first evaluation information.

[0045] For example, if the prediction performance of prediction model M12 deteriorates, the control unit 14 can recognize that a performance difference has occurred between prediction model M11, which was the prediction model used during the development phase, and prediction model M12. In such cases, the control unit 14 can have the generation model M2 analyze the factors that caused the difference based on historical information and first evaluation information as information regarding prediction performance. For example, the control unit 14 may have the generation model M2 analyze the factors that caused the difference by providing an instruction statement (prompt) to the generation model M2 that instructs it to refer to the historical information DB1 and analyze the factors that caused the difference, and that includes the first evaluation information as a condition for analysis.

[0046] The generative model M2, following prompts, analyzes information regarding the prediction performance of the predictive model M12 and outputs a second evaluation information showing the analysis results. For example, by comparing the historical information in the historical information DB1 with the actual prediction results, the generative model M2 can analyze the causes of degradation by tracking evaluation indicators such as drift and degradation throughout the series of processes. Also, since the performance problems of the predictive model M12 may be due to changes in data or data sources, the generative model M2 can also track indicators such as data size, update frequency, region, category, and type by comparing the historical information in the historical information DB1 with the actual prediction results.

[0047] The second evaluation information may consist of text and graphs showing the analysis results. An example of a prompt might be: "The following analysis results were obtained regarding the performance of the predictive model. Please analyze the cause of the performance degradation based on this information."

[0048] (Second Record Section 15) The second recording unit 15 records the second evaluation information in the evaluation information DB2. Specifically, the second recording unit 15 records the second evaluation information in the evaluation information DB2 so that it can be used in future analyses along with the first evaluation information. As a result, the control unit 14 can provide the generation model M2 with instructions (prompts) that include not only the first evaluation information but also the second evaluation information as analysis conditions, thereby enabling the generation model M2 to output better explanations.

[0049] [3. Operating Procedure of Information Processing Device] The operating procedure of the information processing device 10 will be explained based on a practical example of the information processing device 10. Prior to explaining the operating procedure, an example of a scenario in which the information processing device 10 is used in practice will be explained using Figure 2. For example, the information processing device 10 can be implemented as a control device that performs vibration management control of production equipment using IoT data obtained from an IoT device installed in a predetermined location.

[0050] The model development system 10-1 portion of such a control device may be located, for example, in a laboratory where a data science team is located, and is connected to the IoT device DV1 via a network NW. The IoT device DV1 is equipped with or connected to a sensor SN1, and transmits the sensor data detected by the sensor SN1 on the target object O1 to the model development system 10-1 as IoT data.

[0051] The model development system 10-1 can use this IoT data as a dataset for developing the predictive model M11. In this example, the training data from the dataset can include equipment structure and weight data, vibration data, equipment current data, temperature and humidity data, vibration isolation device setting data, etc. The validation data from the dataset can include vibration data, equipment current data, temperature and humidity data, vibration isolation device setting data, etc.

[0052] A vibration isolation device is, for example, a vibration isolation system introduced as a vibration countermeasure in a vibrating environment. For example, a vibration isolation device may be installed in a semiconductor manufacturing facility. Such a vibration isolation device includes a coil spring, a surface plate on which the equipment to be vibration suppressed is placed, a sensor that detects vibrations of the surface plate, an actuator that causes minute vibrations of the surface plate, and a controller that drives the actuator based on the sensor's detection signal. Therefore, the vibration isolation device setting data may include setting data for these components of the vibration isolation device, sensor data, control values ​​from the controller, etc.

[0053] On the other hand, the model operation system 10-2 portion of the control device may be located in a control room where, for example, the IT production team is located, and is connected to the IoT device DV2 via a network NW. The IoT device DV2 is equipped with or connected to a sensor SN2, and transmits sensor data detected by the sensor SN2 on the target object O2 as IoT data to the model operation system 10-2.

[0054] The model operation system 10-2 can use this IoT data as operational data for predictions by the prediction model M12.

[0055] The IoT device DV2 may be, for example, a vibration isolation system for precision equipment placed in the customer's usage environment, and the model operation system 10-2 may remotely control the IoT device DV2 based on the prediction results of the prediction model M12. Furthermore, as shown in Figure 2, the model operation system 10-2 may include a service system SV that provides services related to the IoT device DV2 to the customer.

[0056] For example, suppose the generative model M2 analyzes the cause of the performance degradation in the predictive model M12 and outputs a second evaluation information suggesting a failure in the IoT device DV2, rather than a performance degradation in the predictive model M12. In such a case, the service system SV can provide a service to notify the customer of the failure of the IoT device DV2, as well as the cause of the failure and countermeasures.

[0057] Furthermore, if the information processing device 10 is a control device that performs vibration management control of production equipment, the object O1 (object O2) may be production equipment with a vibration isolation device installed (for example, semiconductor production equipment), and the sensor SN1 may be a vibration sensor provided by the vibration isolation device. Also, the prediction model M11 (prediction model M12) may be a prediction model that predicts the vibration of the production equipment. In addition, the service system SV may be a system that actively controls the vibration isolation device using the vibration prediction of the production equipment.

[0058] <Operating procedures during the development phase> Figure 3 is a flowchart showing the operation procedure during the development phase. The step numbers shown in Figure 3 correspond to the step numbers indicated on the development phase side of Figure 2. Note that Figure 3 shows the operation procedure of the model development system 10-1 part of the information processing device 10.

[0059] First, the prediction model generation unit 12 uses the dataset (IoT data) acquired by the data acquisition unit 11 to perform a learning process that repeatedly trains and validates the prediction model M11 (step S11). In the training process, the training data from the dataset is used, and in the validation process, the validation data from the dataset is used.

[0060] If the prediction model generation unit 12 has achieved the target prediction accuracy, it performs an evaluation step (step S12) to evaluate the general performance of the prediction model at this point. The evaluation step uses the test data from the dataset.

[0061] The first recording unit 13 records history information indicating the content of the learning process and history information indicating the content of the evaluation process in the history information BD1 (step S13).

[0062] Next, the control unit 14 inputs the history information stored in the history information BD1 as training data to the generation model M2 (step S14).

[0063] Furthermore, the control unit 14 inputs a prompt to the generation model M2 instructing it to analyze the prediction performance of the prediction model M11 (step S15). Specifically, in response to an access from the prediction model M11, the control unit 14 creates a prompt that queries for information regarding the prediction performance of the prediction model M11, and inputs the created prompt to the generation model M2.

[0064] The generative model M2, following prompts, analyzes information regarding the prediction performance of the prediction model M11 and outputs first evaluation information showing the analysis results. Therefore, the second recording unit 15 records the first evaluation information showing the analysis results in the evaluation information DB2 (step S16).

[0065] <Operating Procedures> Figure 4 is a flowchart of the operation procedure during the operational phase. The step numbers shown in Figure 4 correspond to the step numbers indicated on the operational phase side of Figure 2. Note that Figure 4 shows the operation procedure of the model operational system 10-2 portion of the information processing device 10.

[0066] First, the data acquisition unit 11 executes an operational process using a predictive model (step S21). In the operational process, the data acquisition unit 11 inputs the acquired operational data (IoT data) into the predictive model M12 to output future predicted values.

[0067] The first recording unit 13 acquires the predicted values ​​output by the prediction model M12 and stores them in the history information DB1 (step S22). The first recording unit 13 stores the outputted predicted values ​​in the history information DB1 each time the prediction model M12 outputs a predicted value. In addition, if the first recording unit 13 obtains actual values ​​corresponding to each predicted value, it may store those actual values ​​in the history information DB1 in association with the predicted values. Furthermore, the first recording unit 13 may input the predicted values ​​into the service system SV so that services corresponding to the predicted values ​​can be realized.

[0068] When sufficient predicted values ​​have been accumulated, the control unit 14 refers to the history information DB1 and determines whether the prediction performance of the prediction model M12 has deteriorated (step S23). For example, the control unit 14 determines whether the prediction performance of the prediction model M12 has deteriorated by comparing the predicted values ​​from the prediction model M12 with the measured values ​​corresponding to the predicted values.

[0069] If the prediction performance of the prediction model M12 has not deteriorated (step S23; No), the control unit 14 returns to step S21.

[0070] On the other hand, if the control unit 14 determines that the prediction performance of prediction model M12 has deteriorated (step S23; Yes), it creates a prompt based on the first evaluation information instructing the system to analyze the cause of the performance difference between prediction model M11 and prediction model M12, and inputs the generated prompt to the generating model M2 (step S24). Specifically, the control unit 14 creates a prompt that inquires about the cause of the deterioration in the prediction performance of prediction model M12, and inputs the created prompt to the generating model M2.

[0071] Furthermore, the control unit 14 may calculate the difference in performance between prediction model M11 and prediction model M12, and if the difference becomes larger than the baseline value (i.e., if the prediction performance of prediction model M12 is significantly worse than that of prediction model M11), it may output an alert to prompt the IT production team to create a prompt.

[0072] The generation model M2 accesses the model operation system 10-2 to perform analysis according to the prompt. In response to the generation model M2's access, the control unit 14 acquires historical information stored in the history information DB1 and actual prediction results accumulated in the history information DB1, and inputs them into the generation model M2 (step S25). Based on the input information, the generation model M2 analyzes information regarding the prediction performance defined by the prompt and outputs second evaluation information showing the analysis results.

[0073] The second recording unit 15 records the second evaluation information in the evaluation information DB2 (step S26). Specifically, the second recording unit 15 records the second evaluation information in the evaluation information DB2 together with the first evaluation information so that it can be used for future analysis.

[0074] [4. Other Embodiments] In the above embodiment, an example was shown in which the information processing device 10 is implemented as a system including a model development system 10-1 and a model operation system 10-2. However, the information processing device 10 may be implemented as a device independent of the model development system 10-1 and the model operation system 10-2. In this case, the information processing device 10 is connected to the model development system 10-1 and the model operation system 10-2 in a communicative manner.

[0075] Furthermore, the information processing device 10 does not necessarily have to have a history information DB1 and an evaluation information DB2. For example, the history information DB1 and the evaluation information DB2 may be implemented in an external device different from the information processing device 10.

[0076] Furthermore, in the above embodiment, an example was shown where the information processing device 10 is a control device that performs vibration management control of production equipment. However, the implementation of the information processing device 10 as a control device is not limited. For example, the information processing device 10 may be a control device that performs air conditioning management control in a room.

[0077] [5. Hardware Configuration] The information processing device 10 according to the embodiment may be implemented by a computer 1000 having a configuration such as that shown in Figure 5. Figure 5 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device 10 according to the embodiment. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0078] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, controlling various components. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0079] The HDD1400 stores programs executed by the CPU1100, as well as data used by such programs. The communication interface1500 receives data from other devices via a predetermined communication network and sends it to the CPU1100, and transmits data generated by the CPU1100 to other devices via the predetermined communication network.

[0080] The CPU 1100 controls output devices such as displays and input devices such as keyboards via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0081] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0082] For example, when computer 1000 functions as an information processing device 10 according to an embodiment, the CPU 1100 of computer 1000 implements each processing unit of the information processing device 10 by executing programs loaded on RAM 1200. The CPU 1100 of computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0083] [6. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0084] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0085] Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

[0086] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, including the embodiments described in the section on the present invention. [Explanation of Symbols]

[0087] 10 Information Processing Devices 10-1 Model Development System 10-2 Model Operating System 11 Data Acquisition Unit 12 Prediction Model Generation Unit 13. First Record Section 14 Control Unit 15. Second Records Section DB1 History Information DB2 Evaluation Information

Claims

1. A first recording unit records historical information regarding the development process of the predictive model, A control unit that outputs first evaluation information showing the analysis results by having the generative model analyze the status of the predictive model learned according to the development process based on the historical information, A second recording unit for recording the first evaluation information, Equipped with, The control unit, In the operational phase in which the developed prediction model, which is the prediction model developed through the aforementioned development process, is put into use, Based on information regarding the prediction results output when the developed prediction model is put into operation, the history information recorded during the development phase in which the development process is carried out, and the first evaluation information recorded during the development phase, the generative model is made to analyze information regarding the prediction performance of the developed prediction model, thereby outputting second evaluation information showing the analysis results. Information processing device.

2. The first recording unit records, as the history information, history information indicating the content of the learning process among the development process, The historical information indicating the contents of the learning process includes training data used to train the prediction model, validation data used to tune the prediction model, and information based on the prediction results obtained by the prediction model using the validation data. The information processing apparatus according to claim 1.

3. The first recording unit records, as historical information, historical information indicating the content of the evaluation process among the development process, The historical information indicating the content of the evaluation process includes test data used to evaluate the performance of the prediction model and information based on the prediction results obtained by the prediction model using the test data. The information processing apparatus according to claim 1.

4. The control unit provides the generation model with an instruction statement that instructs it to analyze information regarding the prediction performance of the prediction model based on the historical information, thereby causing the control unit to analyze the prediction performance of the prediction model. The information processing apparatus according to claim 1.

5. In the aforementioned operational stage, The control unit causes the generative model to analyze information regarding the prediction performance at predetermined timings associated with the operation of the developed prediction model. The information processing apparatus according to claim 1.

6. If the control unit detects that the prediction performance of the developed prediction model has deteriorated based on a comparison of the prediction results of the developed prediction model with the actual results, it will have the control unit analyze information regarding the prediction performance at the time the prediction performance deteriorated, based on the information regarding the prediction results, the historical information, and the first evaluation information. The information processing apparatus according to claim 5.

7. If the predictive performance of the developed predictive model has deteriorated, resulting in a performance difference between the predictive model at the development stage and the developed predictive model, the control unit will analyze the factors causing the difference based on the information regarding the prediction results, the historical information, and the first evaluation information, as information regarding the predictive performance. The information processing apparatus according to claim 6.

8. The control unit provides the generation model with an instruction statement that instructs it to analyze the factors causing the difference based on the historical information, the instruction statement which includes the first evaluation information as a condition for analysis, thereby causing the control unit to analyze the factors causing the difference. The information processing apparatus according to claim 7.

9. The second recording unit records the second evaluation information together with the first evaluation information so that it can be used for future analysis. The information processing apparatus according to claim 1.

10. An information processing method performed by an information processing device, A first recording step involves recording historical information related to the development process of the predictive model, A control step that causes the generative model to analyze the status of the predictive model learned according to the development step based on the historical information, thereby outputting first evaluation information that shows the analysis results, A second recording step for recording the first evaluation information, Includes, The control process described above is: In the operational phase in which the developed prediction model, which is the prediction model developed through the aforementioned development process, is put into use, Based on information regarding the prediction results output when the developed prediction model is put into operation, the history information recorded during the development phase in which the development process is carried out, and the first evaluation information recorded during the development phase, the generative model is made to analyze information regarding the prediction performance of the developed prediction model, thereby outputting second evaluation information showing the analysis results. Information processing methods.

11. A first recording procedure for recording historical information regarding the development process of a predictive model, A control procedure that causes the generative model to analyze the status of the predictive model learned according to the development process based on the historical information, thereby outputting first evaluation information that shows the analysis results, A second recording procedure for recording the first evaluation information, Have the computer run it, The control procedure described above is: In the operational phase in which the developed prediction model, which is the prediction model developed through the aforementioned development process, is put into use, Based on information regarding the prediction results output when the developed prediction model is put into operation, the history information recorded during the development phase in which the development process is carried out, and the first evaluation information recorded during the development phase, the generative model is made to analyze information regarding the prediction performance of the developed prediction model, thereby outputting second evaluation information showing the analysis results. Information processing program.

Images