Information processing device, information processing method, and program

Abstract

An information processing device (1) provided with: an accuracy degradation cause acquisition means (2) that acquires the cause of accuracy degradation of an analysis model; an improvement measure acquisition means (3) that acquires improvement measures for the analysis model that correspond to the cause of the accuracy degradation of the analysis model; and a display means (4) that displays the improvement measures for the analysis model. The information processing device (1) may be further provided with: a determination means that determines whether or not the analysis model satisfies evaluation criteria; an extraction means that, if it is determined that the analysis model does not satisfy the evaluation criteria, extracts, from data of the analysis model, information necessary to estimate the cause of the accuracy degradation; and an input means that inputs the information necessary to estimate the cause of the accuracy degradation into a degradation cause estimation device. The accuracy degradation cause acquisition means may acquire improvement measures for the analysis model that are output from the degradation cause estimation device and that correspond to the cause of the accuracy degradation of the analysis model.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a non-transitory computer-readable medium. [Background technology]

[0002] In the field of machine learning (AI), in order to improve the accuracy of analytical models, the person in charge considers specific measures to improve the analytical models based on experience and intuition (intuition based on experience). An analytical model is the learning result generated by machine learning (AI). Accuracy degradation refers to the large discrepancy between the predicted value returned by the analytical model and the actual value due to changes in the data input to the analytical model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-144504 ​​A Summary of the Invention [Problem to be solved by the invention]

[0004] However, since a considerable amount of experience is required to consider specific measures to improve analytical models, there is an issue that it is difficult for people who joined system development midway through or those who are new to AI to come up with appropriate improvement measures.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an information processing device, an information processing method, and a non-transitory computer-readable medium that are capable of presenting appropriate improvement measures. [Means for solving the problem]

[0006] The information processing device of the present disclosure includes: An accuracy degradation cause acquisition means for acquiring a cause of accuracy degradation of the analysis model; an improvement measure acquisition means for acquiring an improvement measure for the analytical model corresponding to a cause of deterioration in accuracy of the analytical model; and a display means for displaying improvement measures for the analysis model.

[0007] The information processing method of the present disclosure includes: An accuracy degradation cause acquisition step for acquiring a cause of accuracy degradation of the analytical model; an improvement measure acquisition step of acquiring an improvement measure for the analytical model corresponding to a cause of accuracy deterioration of the analytical model; and a display step means for displaying improvement measures for the analysis model.

[0008] The non-transitory computer readable medium of the present disclosure comprises: A non-transitory computer-readable medium storing a program for executing an information processing method, The information processing method includes: An accuracy degradation cause acquisition step for acquiring a cause of accuracy degradation of the analytical model; an improvement measure acquisition step of acquiring an improvement measure for the analytical model corresponding to a cause of accuracy deterioration of the analytical model; A display step means for displaying improvement measures for the analysis model; Equipped with. Effect of the Invention

[0009] The present disclosure makes it possible to provide an information processing device, an information processing method, and a non-transitory computer-readable medium that are capable of presenting appropriate improvement measures. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of a configuration of an information processing device according to a first embodiment. [Diagram 2] FIG. 11 is a diagram illustrating an example of a configuration of an information processing device according to a second embodiment. [Diagram 3] 1 is a diagram showing information held by a repository 10 (information holding unit 11). [Figure 4]2 is a diagram showing the relationship between an information input unit 21 and an information holding unit 11. FIG. [Diagram 5] 2 is a diagram showing the relationship between an information analysis unit 22 and an information holding unit 11. FIG. [Figure 6] 13 is a flowchart of a process for acquiring improvement measures for an analytical model from an information processing device 100 (AI model management device) and repeating the improvement measures. [Figure 7] 13 is a detailed flowchart of step S19. [Figure 8] These are specific examples of deterioration cause classification, search targets, display conditions, etc. [Figure 9] The following are specific examples of deterioration cause classification, display conditions, and improvement measures (recommendations). [Figure 10] 13 is a flowchart of a process for tuning a parameter (deterioration determination threshold) to be passed to an AI model deterioration cause estimation device 50. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Note that the following description and drawings are omitted and simplified as appropriate for clarity of explanation. In addition, in each of the following drawings, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0012] First, the terms used in this disclosure will be explained. In this disclosure, a design pattern indicating a pattern for designing a learning model is referred to as a "case". In addition, in this disclosure, a "case" is defined as a term that may also include design information for creating, verifying, and evaluating an analytical model. The design information may include designation of an AI engine, designation of learning data, verification data, and evaluation data, designation of hyperparameters and data division conditions, and designation of parameters used to run the AI ​​engine other than hyperparameters. Furthermore, the design information may include source code of an AI engine execution program, etc. For example, when a first learning model is created based on a first design pattern, the first design pattern is referred to as a first case, and information on the first design pattern (information used in the first design pattern) is referred to as first case information. In addition, in this disclosure, a learning model may be referred to as an analytical model.

[0013] (First embodiment) A configuration example of an information processing device 1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a configuration example of an information processing device according to the first embodiment. The information processing device 1 may be a personal computer or a server. The information processing device 1 includes an accuracy degradation cause acquisition means 2, an improvement measure acquisition means 3, and a display means 4.

[0014] The accuracy degradation cause acquisition means 2 acquires the cause of accuracy degradation of the analytical model output from a degradation cause estimation device (e.g., an AI model degradation cause estimation device 50 described later), for example. The accuracy degradation cause acquisition means 2 may also acquire the cause of accuracy degradation of the analytical model input by a user via an input device (e.g., an input device 30 described later). The improvement measure acquisition means 3 acquires improvement measures (for example, recommendations described later. See the improvement recommendations in FIG. 9) for the analytical model corresponding to the cause of accuracy deterioration of the analytical model. FIG. 9 shows a specific example of the deterioration cause classification, display conditions, and improvement measures (recommendations). The display means 4 (for example, the output device 40 described later) displays improvement measures for the analytical model (for example, recommendations described later; see the improvement recommendations in FIG. 9).

[0015] As described above, according to the first embodiment, the display means 4 (e.g., the output device 40 described below) displays improvement measures for the analytical model corresponding to the cause of the deterioration in accuracy of the analytical model (e.g., recommendations described below; see the improvement recommendations in Figure 9), so that appropriate improvement measures can be presented.

[0016] Second Embodiment Next, a second embodiment will be described. The second embodiment is a specific embodiment of the first embodiment. <Configuration example of information processing device> A configuration example of an information processing device 100 according to the second embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing a configuration example of an information processing device according to the second embodiment. The information processing device 100 corresponds to the information processing device 1 according to the first embodiment. The information processing device 100 is a device that analyzes an analytical model, which is a learning model obtained by machine learning. Hereinafter, it is also referred to as an AI model management device 100.

[0017] The information processing device 100 presents specific measures (recommendations) for improving the analytical model based on information on the cause of the deterioration in accuracy of the analytical model and data accumulated in the information processing device 100 (AI model management device).

[0018] An analytical model refers to a learning result generated by machine learning (AI). An analytical model outputs a classification result or a prediction result (for example, a prediction result by regression (linear regression)) based on the learning result for the input. In an AI engine, the input is learning data and the output is an analytical model, whereas in an analytical model, the input is data to be predicted and the output is a prediction result. Analytical models are included, for example, in analytical model information (see Figure 3) described later. Accuracy degradation refers to a large discrepancy between the predicted value returned by the analytical model and the actual value due to a change in the data input to the analytical model.

[0019] The information processing device 100 may be a personal computer or a server. The information processing device 100 includes a repository 10, a processing device 20, an input device 30, and an output device 40. In the following description, the learning model analyzed by the information processing device 100 is described as an analysis model.

[0020] The repository 10 is a storage device that stores (holds) case information to be analyzed by the information processing device 100 and various information related to the case information. The repository 10 may be, for example, an NEC Advanced Analytics Platform Modeler (AAPF Modeler). The repository 10 includes an information holding unit 11.

[0021] The information holding unit 11 inputs and holds various pieces of information received by an information input unit 21 included in the processing device 20 from the information input unit 21. The information holding unit 11 may be referred to as a storage unit.

[0022] Here, various types of information held (accumulated) by the information holding unit 11 will be described with reference to Fig. 3. Fig. 3 is a diagram showing information held by the repository 10 (information holding unit 11). As shown in Fig. 3, the information holding unit 11 holds analysis summary information, case information, analysis model information, evaluation record information, and issue information. In addition, the information holding unit 11 holds information on deterioration cause classification, deterioration determination threshold, recommendation template, incident, deterioration cause, and recommendation.

[0023] The analysis summary information is created for each analysis purpose to be performed by the analysis model, which is a learning model. For example, if a user (analyst) using the analysis model wants to perform a power demand forecast and sets the analysis purpose to power demand forecast, analysis summary information with "power demand forecast" as the analysis purpose is created. For example, if a user using the analysis model wants to perform a sales forecast different from the power demand forecast and sets the analysis purpose to sales forecast, analysis summary information with "sales forecast" as the analysis purpose is created. The analysis summary information includes an analysis summary name and an analysis purpose. In addition, the analysis summary information that bundles the cases includes the evaluation criteria of the analysis model used when verifying the hypothesis of the analysis model (at the time of hypothesis verification) and the evaluation criteria of the analysis model used when the AI ​​system is in production operation (at the time of production operation).

[0024] The name of the analysis outline is set in the analysis outline name. The purpose of creating the analytical model is set as the analysis purpose. Using the above example, for example, "electricity demand forecast" or "sales forecast" is set as the analysis purpose. In the evaluation criteria (at the time of hypothesis verification), information on the evaluation criteria of the analytical model used at the time of hypothesis verification of the analytical model is set. The evaluation criteria (for production use) is used to set information about the evaluation criteria for the analytical model to be used when the AI ​​system is in production.

[0025] The case information is information on a case (design information, design pattern) for creating an analytical model based on the analysis summary information. When one analysis summary information is created, an analytical model with high prediction accuracy is created according to the analysis purpose, etc. included in the analysis summary information. Since it is generally difficult to create an analytical model with high prediction accuracy by only one design, an analytical model with high prediction accuracy is created by creating multiple cases by trial and error through multiple designs and evaluating the analytical models. Therefore, multiple case information is created from one analysis summary information. In other words, the analysis summary information is information that bundles multiple case information, and the information holding unit 11 holds, for example, the analysis summary information and the case information in a hierarchical structure. In other words, the information holding unit 11 holds the case information so that it is stored in the hierarchy one level below the analysis summary information. Therefore, the analysis summary information and the case information are held by the information holding unit 11 so that corresponding information can be identified by tracing the held hierarchy. The case information includes a case name, learning candidate data, an AI engine algorithm, a target variable, an explanatory variable, and a corresponding issue.

[0026] In the case name, a name that identifies a case for designing an analysis model is set. The learning candidate data is set to a collection of data that may be used to create an analysis model. The learning candidate data includes data (variables) and column information. Specifically, the learning candidate data is set to a plurality of variable names that may be used as objective variables and explanatory variables, and data such as the numerical values ​​of each variable. The learning candidate data may include variables that are not used as objective variables and explanatory variables. The learning candidate data shown in FIG. 8 corresponds to the learning candidate data in FIG. 3. FIG. 8 shows specific examples of deterioration cause classifications, search targets, display conditions, etc.

[0027] The AI ​​engine and algorithm are set with the AI ​​engine name and the name of the algorithm used by the AI ​​engine. AI engine is a general term for AI that performs analysis based on a specific algorithm classification. An AI engine is a system that realizes analytical processing such as prediction and discrimination by generating an analytical model using machine learning technology in accordance with a specified data analysis method. An AI engine is, for example, a commercial software program or a software program provided as open source. Examples of AI engines include scikit-learn and PyTorch.

[0028] The objective variable is set with the variable name (objective variable name) of the information to be predicted by the analysis model (data to be predicted) and the data type. The data type of the objective variable indicates the type of the value of the objective variable and is a label used for categorization. Examples of data types include categorical and numeric types. For example, if the objective of the analysis is "power demand forecast," then the objective variable is set with "actual (10,000 kW)," which indicates the objective variable name of the actual power value related to power demand, and the data type of the objective variable.

[0029] The explanatory variables are multiple variables used by the analysis model when making predictions, and variable names (explanatory variable names) that are assumed to affect the objective variable are set. For example, all explanatory variable names are set in the form of a variable list as the explanatory variables. For example, if the analysis objective is "electricity demand forecasting," the explanatory variables are set in the form of a variable list with variable names such as "temperature," "precipitation," and "actual (10,000 kW)_2 days ago," which indicates the actual electric power value from two days ago, that are used to forecast the objective variable, electric power demand, set in a list format as a variable list.

[0030] The corresponding issue is information related to the issue information described below, and an issue to be resolved in each case is set in the corresponding issue. For example, when an analysis model created from a certain case is evaluated, if it is found that the learning candidate data contains insufficient data related to "temperature", the issue "There is insufficient data related to 'temperature'" is set in the issue information. If a newly considered case is based on the learning candidate data to which data related to "temperature" has been added, the corresponding issue included in the case information of that case is set to "There is insufficient data related to 'temperature'".

[0031] In the analytical model information, information on an analytical model created from one case information is set. Since multiple analytical models may be created from one case, at least one analytical model information is associated with one case information. The information holding unit 11 holds the analysis summary information, the case information, and the analytical model information in a hierarchical structure. Specifically, the information holding unit 11 holds the analysis summary information, the case information, and the analytical model information such that the case information is stored in a hierarchy one level below the analysis summary information, and the analytical model information is stored in a hierarchy one level below the case information. Therefore, the analysis summary information, the case information, and the analytical model information are held by the information holding unit 11 so that corresponding information can be identified by tracing the held hierarchy. The analytical model information includes an analytical model name, an accuracy index value (statistics), and data (variable values).

[0032] The name of the analysis model is set in the analysis model name. The accuracy index value (statistic) is set to the accuracy index value of the analytical model. The accuracy index value is, for example, the mean absolute error calculated from the data of the analytical model registered in the repository 10. For example, the "learning / verification / evaluation data" in FIG. 8 is set as the data (variable value).

[0033] The evaluation record information is information on the record when the case information and analytical model information of the evaluation target are evaluated. The evaluation record information includes the evaluation record name, the evaluation target, the evaluation result / opinion, the incident ID, and the recommendation ID. The name of the evaluation record is set in the evaluation record name. The evaluation target is set with information that identifies a case related to the analytical model to be evaluated.

[0034] The evaluation result / opinion indicates the opinion of the user who performs the evaluation regarding the analytical model and case to be evaluated. In the incident ID, information for identifying an incident related to the analytical model to be evaluated is set. In the recommendation ID, information for identifying a recommendation regarding the analytical model to be evaluated is set.

[0035] The task information is set with information about a task identified from the evaluation record information. For example, when an analysis model created from a certain case is evaluated and it is found that data related to "temperature" included in the learning candidate data is insufficient, the task information is set with information about the task that "data related to 'temperature' is insufficient." The task information includes the task name, task content, occurrence evaluation result name, occurrence case, problem handling case, case effectiveness, and recommendation ID.

[0036] The name of the assignment is set in the assignment name. If the assignment information is information about an assignment in which "data on 'temperature' is insufficient," then the assignment name is set to information such as "insufficient data on temperature."

[0037] The assignment content is set to the specific content of the assignment. If the assignment information is information about the assignment "there is insufficient data on 'temperature'", the assignment content is set to information such as "there is insufficient data on 'temperature' included in the learning candidate data".

[0038] The occurring evaluation result name is set to the evaluation record name included in the evaluation record information in which the problem was identified. The source case is set with information that identifies the case in which the problem was identified. The source case is set with information that identifies the case that was set as the evaluation target included in the evaluation record information in which the problem was identified.

[0039] Information for identifying a case corresponding to an issue is set in the issue response case. For example, when a new case is created for an issue, the case is set in the issue response case.

[0040] The case effectiveness is set to the judgment result of whether or not each new case corresponding to the issue has solved the issue. Suppose two new cases are created for an issue, and the first case does not solve the issue, while the second case does. In this case, as information regarding the case effectiveness, information indicating that the issue has not been solved is set for the first case, and information indicating that the issue has been solved is set for the second case. Information for identifying a recommendation is set in the recommendation ID.

[0041] For the deterioration cause classification, a name, parameters (deterioration judgment threshold) to be passed to the AI ​​model deterioration cause estimation device 50, and a template (recommendation template) for configuring improvement measures (recommendations) for the analysis model corresponding to the deterioration cause are set. When the deterioration judgment threshold is updated, information with a different update date and time is added, and a history of the update is kept. The deterioration cause classification includes an ID, a deterioration cause classification name, and a priority. In the ID, information for identifying the deterioration cause classification is set. The deterioration cause classification name is set in the deterioration cause classification name. Specific examples of the deterioration cause classification name are shown in Fig. 8 (see deterioration cause classification in Fig. 8). In the priority field, the priority of the deterioration cause classification name is set.

[0042] The deterioration determination threshold includes an ID, a deterioration cause classification ID, a value, and an update date and time. In the ID, information for identifying the deterioration determination threshold is set. In the deterioration cause classification ID, a deterioration cause classification ID associated with a deterioration cause determination threshold is set. A specific value of the deterioration determination threshold is set in the value. The deterioration determination threshold is also called a parameter. The date and time when the deterioration determination threshold was updated is set as the update date and time.

[0043] The recommendation template is composed of a search range and conditions for checking the analytical models registered in the AI ​​model management device 100 according to the cause of degradation, and a message indicating improvement measures for the cause of degradation. The recommendation template includes an ID, a degradation cause classification ID, a search target (candidate learning data or analytical model), a display condition, a message (Y), and a message (N). Information for identifying the recommendation template is set in the ID. The deterioration cause classification ID is set to the deterioration cause classification ID associated with the recommendation template. Information that identifies the search target (candidate learning data or analytical model) is set as the search target. A specific example of the search target is shown in FIG. 8. The display conditions are conditions for displaying a message. A specific example of the display conditions is shown in FIG. A message to be displayed when the display condition is satisfied is set in message (Y). Specific examples of message (Y) are shown in Figures 8 and 9 (see, for example, 1), 3), 5), 6), 7), 8), and 10) in Figures 8 and 9). A message to be displayed when the display conditions are not satisfied is set in the message (N). Specific examples of the message (N) are shown in Fig. 8 and Fig. 9 (see, for example, 2), 4), 9), and 11) in Fig. 8 and Fig. 9).

[0044] An incident is issued when an analytical model does not meet the evaluation criteria, and has information to be input to the analytical model degradation estimation device (analytical model data, degradation judgment threshold). An incident includes an ID, incident occurrence criteria, occurrence date and time, analytical model name, prediction result, prediction target data, and degradation judgment threshold ID. In the ID, information for identifying the incident is set. The incident occurrence origin criterion is set to an evaluation criterion (at the time of hypothesis verification) or an evaluation criterion (at the time of actual operation). The incident occurrence origin criterion is an item for managing whether the criterion is at the time of hypothesis verification or at the time of actual operation on the AI ​​model management device 100. The date and time of occurrence is set to the date and time when the processing is being performed. The analysis model name is set to the name of the uploaded analysis model (the analysis target). The uploaded prediction result is set as the prediction result. The prediction result is the prediction result output by the analysis model for the input, for example, a prediction result based on the learning result (for example, a prediction result by regression (linear regression)). A specific example of a prediction result is a weather forecast prediction result. The prediction result is included in the analysis model from before the incident is issued. The uploaded prediction target data is set as the prediction target data. The prediction target data is data that is input to the trained analytical model. A specific example of prediction target data is, for example, correct answer data for a weather forecast. The prediction target data is included in the analytical model before the occurrence of an incident. The ID of the "deterioration determination threshold" of the "deterioration cause classification" is set in the deterioration determination threshold ID. The deterioration determination threshold is used, for example, in the AI ​​model management device 100 to evaluate (determine) the distance between the prediction result and the correct answer data. Note that the prediction result, the prediction target data, and the degradation judgment threshold are essential for estimating the cause of model degradation, and the rest may be omitted. An incident (incident information) including at least the prediction result, the prediction target data, and the degradation judgment threshold is an example of information necessary for estimating the cause of accuracy degradation in the present disclosure.

[0045] The deterioration cause has information on the deterioration cause (name, location of the data on which it is based) returned from the analytical model deterioration estimation device and information on the input incident. The deterioration cause includes ID, incident ID, deterioration cause classification ID, deterioration cause name, target data, record, column, and data extraction condition. In the ID, information for identifying the cause of deterioration is set. The incident ID is set with information for identifying the incident associated with the degradation cause. The deterioration cause classification ID is set with information for identifying the deterioration cause classification corresponding to the deterioration cause. The deterioration cause name is set to a specific name of the deterioration cause. The target data includes record columns. Records and columns are information for identifying the location in the analysis data that is causing degradation. Since the data is in table (matrix) format, row numbers are set in records and column numbers (or names) are set in columns. By referencing records and columns, it is possible to identify a single piece of data that is causing degradation. The data extraction conditions include conditions for narrowing down the area and period, such as "specific area" and "specific period" in the "display conditions" in FIG.

[0046] When the cause of degradation is returned from the AI ​​model degradation cause estimation device 50, the recommendation is generated based on the information on the cause of degradation and the recommendation template. The recommendation has information on the analysis model improvement measures and the adoption results. The recommendation includes an ID, degradation cause ID, recommendation template ID, display condition judgment result (Y / N), response policy (adopted / not adopted), and registration date and time. Information for identifying the recommendation is set in the ID. The deterioration cause ID is set with information for identifying the deterioration cause. In the recommendation template ID, information for identifying the recommendation template is set. In the display condition judgment result (Y / N), the display condition judgment result (Y / N) is set. In the response policy (adopt / reject), adoption / rejection of the response policy is set. The registration date and time is set to the date and time when the recommendation was registered.

[0047] Returning to Fig. 2, an example of the configuration of the processing device 20 will be described. The processing device 20 functions as a control unit that performs various controls on data input from the input device 30. The processing device 20 also analyzes analysis summary information, case information, and analysis model information using various information held in the repository 10, and outputs the analysis results to the output device 40. The processing device 20 performs operations on external systems. The processing device 20 includes an information input unit 21 and an information analysis unit 22.

[0048] FIG. 4 is a diagram showing the relationship between the information input unit 21 and the information holding unit 11. As shown in FIG. 4, the information input unit 21 includes a design information input unit 21a, a model information input unit 21b, and an evaluation information input unit 21c. The information holding unit 11 (repository 10) includes a design information storage unit 11a, a model information storage unit 11b, and an evaluation information storage unit 11c. The design information input unit 21a registers (stores) the analysis outline, case, deterioration cause classification, and recommendation template input (uploaded) by the user from the input device 30 in the information holding unit 11 (design information storage unit 11a). The model information input unit 21b registers (stores) the analysis model (file) and learning candidate data input (uploaded) by the user from the input device 30 in the repository 10 (model information storage unit 11b) in association with pre-registered cases. The assessment information input unit 21c registers (stores) the assignments and assessment records input (uploaded) by the user from the input device 30 in the repository 10 (assessment information storage unit 11c).

[0049] FIG. 5 is a diagram showing the relationship between the information analysis unit 22 and the information holding unit 11. As shown in FIG. As shown in FIG. 5, the information analysis unit 22 includes an accuracy calculation unit 22a, a deterioration determination unit 22b, a deterioration cause estimation unit 22c, and a recommendation unit 22d. The accuracy calculation unit 22a calculates an accuracy index value such as a mean absolute error from the data of the analytical model registered in the repository 10 (model information storage unit 11b). The deterioration determination unit 22b compares the calculation result of the accuracy calculation unit 22a with the value of the evaluation criterion registered in advance by the design information input unit 21a to determine whether or not the evaluation criterion is satisfied. The deterioration determination unit 22b also generates information (incidents) to be input to the AI ​​model deterioration cause estimation device 50 from the data of the analysis model (prediction results, prediction target data) and parameters (deterioration determination thresholds). The deterioration cause estimation unit 22c inputs incident information to the AI ​​model deterioration cause estimation device 50 and acquires deterioration cause information from the output of the AI ​​model deterioration cause estimation device 50. The recommendation unit 22d issues a recommendation for the cause of the deterioration.

[0050] The input device 30 functions as an input unit. The input device 30 may be, for example, a keyboard, a mouse, a touch panel, or the like. When a user inputs various pieces of information held in the information holding unit 11 of the repository 10 to the input device 30, the input device 30 outputs the input information to the information input unit 21. When a user inputs an analytical model of an analysis target and an analytical model of a comparison target to be analyzed by the information analysis unit 22 to the input device 30, the input device 30 outputs the information to the information input unit 21.

[0051] The output device 40 functions as an output unit. The output device 40 is configured to include, for example, a display. The output device 40 displays the results of the calculations performed by the processing device 20 to the user.

[0052] As shown in FIG. 2, the AI ​​model degradation cause estimation device 50 is electrically connected to the information processing device 100 (processing device 20). The AI ​​model degradation cause estimation device 50 estimates (specifies) the degradation cause of the analysis model by executing a predetermined process based on the incident information input from the information processing device 100 (processing device 20), and outputs information on the estimated degradation cause. For example, the AI ​​model degradation cause estimation device 50 specifies the degradation cause (degradation cause classification ID, etc.) from a database (not shown) that stores the degradation cause (degradation cause classification ID, etc.) based on the prediction result, the prediction target data (correct answer data), and the degradation judgment threshold, and outputs the specified degradation cause (degradation cause classification ID, etc.). At that time, the AI ​​model degradation cause estimation device 50 outputs information (records, columns) for specifying the location causing the degradation by evaluating (determining) whether the distance between the prediction result and the correct answer data exceeds the degradation judgment threshold based on the prediction result, the prediction target data (correct answer data), and the degradation judgment threshold.

[0053] <Example of operation of information processing device> Next, an example of the operation of the information processing device 100 will be described with reference to FIGS.

[0054] The following describes a process of acquiring improvement measures for an analytical model from the information processing device 100 (AI model management device) and repeating the improvement measures. Fig. 6 is a flowchart of a process of acquiring improvement measures for an analytical model from the information processing device 100 (AI model management device) and repeating the improvement measures. Fig. 7 is a detailed flowchart of step S19. In the following, it is assumed that the analysis summary, deterioration cause classification, deterioration determination threshold, and recommendation template have been registered in the repository 10 in advance.

[0055] First, an analytical model is registered (step S10). For example, a user uploads an analytical model (file) to the AI ​​model management device 100 via the input device 30.

[0056] Next, the analytical model is registered in the repository 10 (step S11). This is performed by the model information input unit 21b. Specifically, the model information input unit 21b registers (stores) the analytical model (file) and learning candidate data input (uploaded) by the user from the input device 30 in the repository 10 (model information storage unit 11b) in association with a case registered in advance.

[0057] Next, it is determined whether the analytical model satisfies the evaluation criteria (step S12). This is an example of the determination means of the present disclosure, and is performed by the accuracy calculation unit 22a and the deterioration determination unit 22b. Specifically, first, the accuracy calculation unit 22a calculates an accuracy index value such as a mean absolute error from the data of the analytical model registered in the repository 10 (model information storage unit 11b). Next, the deterioration determination unit 22b compares the calculation result of the accuracy calculation unit 22a with the value of the evaluation criteria registered in advance from the design information input unit 21a to determine whether the evaluation criteria are satisfied. For example, the accuracy index value calculated by the accuracy calculation unit 22a is compared with a deterioration determination threshold (a deterioration determination threshold associated with the deterioration cause classification set in the evaluation criteria (at the time of hypothesis verification) or the evaluation criteria (at the time of actual operation) in the analysis overview), and if the display condition in the recommendation template is satisfied, it is determined that the analytical model does not satisfy the evaluation criteria. On the other hand, if the display condition in the recommendation template is not satisfied, it is determined that the analytical model satisfies the evaluation criteria.

[0058] Next, if it is determined in step S12 that the analytical model does not satisfy the evaluation criteria (step S12: NO), an incident is issued (step S13). This is an example of an extraction means of the present disclosure, and is performed by the deterioration determination unit 22b. Specifically, the deterioration determination unit 22b generates (extracts) information (incident) to be input to the AI ​​model deterioration cause estimation device 50 from the analytical model data (prediction result, prediction target data) and parameters (deterioration determination threshold). The processes of steps S12 and S13 are repeatedly executed the same number of times as the number of deterioration cause classifications. If it is determined in step S12 that the analytical model satisfies the evaluation criterion (step S12: YES), the processes in and after step S13 are executed in response to a user instruction (step S14).

[0059] Next, the cause of deterioration is estimated (step S15). This is performed by the deterioration cause estimation unit 22c. The deterioration cause estimation unit 22c inputs incident information (at least the prediction result, the prediction target data, and the deterioration judgment threshold) to the AI ​​model deterioration cause estimation device 50 (an example of an input means of the present disclosure) and acquires information on the deterioration cause from the output of the AI ​​model deterioration cause estimation device 50 (steps S16 to S18). This is an example of the accuracy deterioration cause acquisition means of the present disclosure. The AI ​​model deterioration cause estimation device 50 estimates (specifies) the deterioration cause of the analysis model by executing a predetermined process based on the incident information input from the information processing device 100 (processing device 20), and outputs information on the estimated deterioration cause. The information on the deterioration cause includes the deterioration cause classification (deterioration cause classification ID) and information on the part (record, column) that is the cause of the deterioration in the analysis data.

[0060] Next, a recommendation issuing process is executed (step S19), which is an example of the improvement measure acquiring means of the present disclosure.

[0061] FIG. 7 is a flowchart of the recommendation issuing process.

[0062] First, the deterioration cause classifications corresponding to all deterioration causes are acquired and rearranged in order of priority (step S191). This is an example of the deterioration cause classification acquisition means of the present disclosure. A specific example of the deterioration cause classification acquired here is shown in FIG.

[0063] Next, a recommendation template corresponding to the deterioration cause classification is obtained (step S192). This is an example of a recommendation template obtaining means of the present disclosure. Next, it is checked whether the data to be searched meets the display conditions (step S193). This is an example of the checking means of the present disclosure. A specific example of the search target and the display conditions is shown in FIG.

[0064] Next, a recommendation (message) having the check result (Y / N) of step S193 as the display condition determination result (Y / N) is issued (step S194). This is an example of the message acquisition means of the present disclosure. The processes of steps S192 to S194 are repeatedly executed the same number of times as the number of deterioration cause classifications acquired in step S191 (step S195: NO). If no deterioration cause is obtained in step S191 (step S195: YES), the process is interrupted.

[0065] Returning to FIG. 6, the explanation of the operation example will be continued. When the recommendation issuing process is completed, the incident (the incident ID of the incident issued in step S13) and the recommendation (the recommendation ID of the recommendation issued in step S19) are then registered in the evaluation result (step S20).

[0066] Next, a list of recommendations is displayed (step S21). This is an example of a display means of the present disclosure. For example, the recommendation information issued in S19 is displayed in a list format on the output device 40. A specific example of the recommendation information (list format) is shown in FIG. 9 (see improvement recommendation in FIG. 9).

[0067] Next, the user selects a recommendation to be adopted from the recommendations displayed in the list (step S22).

[0068] Next, if one or more recommendations have been adopted (step S23: YES), a task having recommendation information (for example, a task having the same recommendation ID as the recommendation selected in step S22) and a case for solving that task are issued (step S24). This is an example of a registration means of the present disclosure. For example, the evaluation information input unit 21c registers the improvement measures of the recommendation in the repository 10 as a task to be addressed in the next case. In addition, the design information input unit 21a registers the data of the above case in the repository 10. Note that if one or more recommendations have not been adopted (step S23: NO), the process ends.

[0069] Next, the user implements the improvement measures to create an analytical model, and registers the analytical model again in the information processing device 100 (AI model management device) (step S10). Thereafter, step S11 is repeatedly executed. As a result, multiple cases are issued. For example, when a series of processes from issuing an incident (step S13) to issuing a recommendation (step S19) is performed for case A and is completed up to S24, case B is issued to solve the problem derived from case A.

[0070] Next, a process (an example of the parameter update means of the present disclosure) for tuning the parameter (deterioration determination threshold) to be passed to the AI ​​model deterioration cause estimation device 50 and included in the incident issued in step S13 will be described. FIG. 10 is a flowchart of a process for tuning a parameter (deterioration determination threshold) to be passed to the AI ​​model deterioration cause estimation device 50.

[0071] The parameters (deterioration determination thresholds) passed to the AI ​​model deterioration cause estimation device 50 are set by the person in charge based on their experience and intuition (intuition based on experience) in past analysis work, making it difficult for a third party to tune them. 10, the parameters (deterioration determination thresholds) to be passed to the AI ​​model deterioration cause estimation device 50 are tuned based on data accumulated in the information processing device 100 (AI model management device). This increases the probability that the AI ​​model deterioration cause estimation device 50 will output the correct cause of accuracy deterioration.

[0072] First, an instruction to update the deterioration determination threshold is given (step S30). For example, the user performs an operation to update the value of the deterioration determination threshold registered in the repository 10 via the input device 30.

[0073] Next, the degradation determination threshold value with the latest update date and time is obtained from among the degradation cause classifications (step S31). This is performed by the design information input unit.

[0074] Next, a recommendation corresponding to the incident having the degradation judgment threshold acquired in step S31 is acquired (step S32). Incidents and recommendations correspond to each other via the degradation cause (see FIG. 3). Therefore, by narrowing down the combined table of recommendations and degradation causes by the incident ID, it is possible to acquire a recommendation corresponding to the incident.

[0075] Next, an assignment having the recommendation information acquired in step S32 is acquired (step S33). Specifically, an assignment having an ID (recommendation ID) in the recommendation acquired in step S32 is acquired.

[0076] Next, if the number of issues acquired in step S33 is five or more (step S36: YES), the processing from step S35 onwards is executed. On the other hand, if the number of issues acquired in step S33 is less than five (step S34: NO), no update is performed and the processing ends. Note that the number of issues acquired in step S33 becomes five or more when the same recommendation is adopted five or more times as a result of repeating the processing in FIG. 6. The reason for setting the number to five or more is to ensure a minimum population in the processing of step S36.

[0077] Next, the presence or absence of case effect for the issue is obtained (step S35). The result of whether or not the analytical model registered in the case that corresponds to the issue (= the improvement measures indicated in the recommendation were implemented) was improved is registered in the repository 10 as the presence or absence of case effect, so that information is obtained. Note that the presence or absence of case effect is registered at the timing when the model is evaluated by an AI model management device other than the information processing device 100 (AI model management device) shown in FIG. 2 (in JP 2020-38527 A, the timing when the evaluation record is created).

[0078] Next, the deterioration determination threshold is updated according to the ratio of the case effect to the absence or presence of the effect (steps S36 to S8). For example, if the rate of effectiveness is high (step S36: = 100%), the improvement measure is highly likely to be effective for other analysis models as well, so the value of the parameter (deterioration determination threshold) is updated so as to increase the probability that the deterioration cause corresponding to the recommendation will be output from the AI ​​model deterioration cause estimation device 50 (step S37).

[0079] On the other hand, if the proportion of effective results is low, there is a high possibility that there are many false positives, so the value of the parameter (deterioration determination threshold) is updated so as to reduce the probability that the cause of deterioration is output (step S38).

[0080] As described above, according to the second embodiment, the output device 40 displays improvement measures (recommendations; see the improvement recommendations in FIG. 9) for the analytical model corresponding to the cause of the deterioration in accuracy of the analytical model, so that appropriate improvement measures can be presented.

[0081] Furthermore, according to the second embodiment, the parameters (deterioration determination thresholds) passed to the AI ​​model deterioration cause estimation device 50 are tuned based on data accumulated in the information processing device 100 (AI model management device). This increases the probability that the AI ​​model deterioration cause estimation device 50 will output the correct cause of accuracy deterioration.

[0082] In addition, according to the second embodiment, by recommending improvement measures when the performance of an analytical model is insufficient or has deteriorated, improvement measures of a certain quality can be obtained regardless of the operator, thereby making the work of improving the analytical model more efficient.

[0083] Furthermore, according to the second embodiment, the parameter values ​​passed to the AI ​​model degradation cause estimation device 50 are automatically updated based on the results of the analysis model improvement work, making it possible to tune the parameters so as to increase the probability that the AI ​​model degradation cause estimation device 50 will output the correct cause of accuracy degradation regardless of the operator.

[0084] In the second embodiment, an example using step S12 has been described, but the automatic evaluation criterion determination process shown in step S12 may be omitted. This also makes it possible to achieve the above-mentioned effects.

[0085] (Other embodiments)

[0086] The information processing device 100 described in the above embodiment may have the following hardware configuration: Fig. 11 is a diagram showing an example of the hardware configuration of the information processing device according to the present disclosure.

[0087] 11, the information processing device 100 includes a processor 1201 and a memory 1202. The processor 1201 reads out software (computer programs) from the memory 1202 and executes them to perform the processing of the information processing device 100 described using the flowcharts in the above-mentioned embodiments. The processor 1201 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 1201 may include multiple processors.

[0088] The memory 1202 is configured by a combination of volatile memory and non-volatile memory. The memory 1202 may include storage located away from the processor 1201. In this case, the processor 1201 may access the memory 1202 via an I / O (Input / Output) interface (not shown).

[0089] 11, the memory 1202 is used to store a group of software modules. The processor 1201 reads out and executes the group of software modules from the memory 1202, thereby performing the processing of the information processing device 100 described in the above embodiment.

[0090] As described with reference to FIG. 11, each of the one or more processors included in the information processing device 100 executes one or more programs including a set of instructions for causing a computer to execute the algorithm described with reference to the drawings.

[0091] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0092] The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, the present disclosure may be implemented by appropriately combining the respective embodiments. [Explanation of symbols]

[0093] 1...Information processing device 2… Accuracy deterioration cause acquisition means 3…Means for acquiring improvement measures 4...Display means 10. Repository 11…Information holding unit 11a...Design information storage section 11b…Model information storage section 11c...Evaluation information storage unit 20... Processing equipment 21...Information input section 21a...Design information input section 21b…Model information input section 21c…Evaluation information input section 22…Information Analysis Department 22a...Accuracy calculation section 22b...Deterioration determination section 22c…Deterioration cause estimation section 22d…Recommendation section 30...Input device 40...Output device 50…AI model deterioration cause estimation device 100...Information processing device (AI model management device) 1201...processor 1202…Memory

Claims

1. An accuracy degradation cause acquisition means for acquiring a cause of accuracy degradation of the analysis model; an improvement measure acquisition means for acquiring an improvement measure for the analytical model corresponding to a cause of deterioration in accuracy of the analytical model; A display means for displaying an improvement measure for the analysis model, A determination means for determining whether the analytical model satisfies an evaluation criterion; an extraction means for extracting information necessary for estimating a cause of accuracy degradation from the data of the analytical model that is determined not to satisfy the evaluation criterion; and an input unit for inputting information required for estimating the cause of the accuracy degradation to the deterioration cause estimating device, The accuracy degradation cause acquisition means is an information processing device that acquires the cause of accuracy degradation of the analytical model output from the degradation cause estimation device.

2. The improvement measure acquisition means includes: a degradation cause classification acquisition means for acquiring a degradation cause classification corresponding to a cause of accuracy degradation of the analytical model; a recommendation template acquisition means for acquiring a recommendation template corresponding to the deterioration cause classification; A check means for checking whether or not the data to be searched in the recommendation template satisfies a display condition in the recommendation template; The information processing apparatus according to claim 1 , further comprising: a message acquisition means for acquiring a message corresponding to the check result in the recommendation template as an improvement measure for the analysis model.

3. A registration means for registering the results of the improvement measures taken by a user for the analysis model; The information processing apparatus according to claim 1 , further comprising: a parameter update means for updating parameters that are information necessary for estimating a cause of deterioration in accuracy of the analysis model, based on the result of the user's operation.

4. The registration means issues an issue when a user selects an improvement measure for the analysis model displayed by the display means; 4. The information processing apparatus according to claim 3, wherein the parameter update means updates the parameters in accordance with a ratio of the issued tasks having a case effect or not having a case effect when the issued tasks are equal to or greater than a predetermined number.

5. The information processing apparatus according to claim 1 , wherein the information necessary for estimating the cause of the accuracy deterioration is a prediction result, prediction target data, and a deterioration determination threshold value.

6. An accuracy degradation cause acquisition step for acquiring a cause of accuracy degradation of the analytical model; an improvement measure acquisition step of acquiring an improvement measure for the analytical model corresponding to a cause of accuracy deterioration of the analytical model; A display step of displaying an improvement measure for the analytical model, a determining step of determining whether the analytical model satisfies an evaluation criterion; an extraction step of extracting information necessary for estimating a cause of accuracy degradation from the data of the analytical model determined not to satisfy the evaluation criterion; and an input step of inputting information required for estimating the cause of the accuracy degradation into the deterioration cause estimating device, The accuracy degradation cause acquisition step is an information processing method for acquiring a cause of accuracy degradation of the analytical model output from the degradation cause estimation device.

7. A program for executing an information processing method, The information processing method includes: An accuracy degradation cause acquisition step for acquiring a cause of accuracy degradation of the analytical model; an improvement measure acquisition step of acquiring an improvement measure for the analytical model corresponding to a cause of accuracy deterioration of the analytical model; A display step of displaying an improvement measure for the analytical model, a determining step of determining whether the analytical model satisfies an evaluation criterion; an extraction step of extracting information necessary for estimating a cause of accuracy degradation from the data of the analytical model determined not to satisfy the evaluation criterion; and an input step of inputting information required for estimating the cause of the accuracy degradation into the deterioration cause estimation device, The accuracy degradation cause acquisition step is a program for acquiring a cause of accuracy degradation of the analytical model, the cause of accuracy degradation being output from the degradation cause estimation device.

Images