Model analysis device, model analysis method, and program

Abstract

In a model analysis device according to the present invention, a first embedding means embeds, in a space, a data set used by a prediction model. A second embedding means calculates an assurance region for the prediction accuracy of the prediction model and embeds the same in the space. An output means outputs a distribution graph indicating the space.

Description

Technical Field

[0001] This disclosure relates to the analysis of machine learning models.

Background Art

[0002] In recent years, prediction models obtained by machine learning have been used in various fields. Patent Document 1 describes a method for visualizing the error between the calculated value by the prediction model and the measured value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, it is difficult to grasp the relationship between the data set used for prediction and the prediction accuracy by the prediction model.

[0005] One object of this disclosure is to visualize the relationship between the data set used for prediction and the prediction accuracy by the prediction model.

Means for Solving the Problems

[0006] In one aspect of this disclosure, the model analysis device embeds the multiple data set used by the prediction model into , a 2D or 3D model with the features obtained from the aforementioned dataset as each axis a space by a first embedding means, calculates a guarantee region of the prediction accuracy of the prediction model, and embeds the The above in association with the dataset region into the space by a second embedding means, and outputs a distribution diagram showing the space by an output means. and includes 、 The first embedding means embeds the plurality of datasets into the space using the statistics of the dataset, or the distance between the dataset and a predetermined reference dataset, as the feature quantities. The second embedding means is Create a peripheral dataset located around the dataset in the space, measure the predicted accuracy of the prediction model for the peripheral dataset, and calculate the region in the space where the predicted accuracy exceeds a predetermined threshold as the guaranteed prediction accuracy region, or The distance from the dataset at which the prediction model has an expected accuracy of above a predetermined threshold is calculated, and the region within the distance from the dataset is calculated as the area where the prediction accuracy is guaranteed. 。

[0007] In another aspect of the present disclosure, a model analysis method executed by a computer is the dataset used by the prediction model multiple to , a 2D or 3D model with multiple features obtained from the aforementioned dataset as each axis perform a first embedding that embeds it into a space Perform , calculate a guarantee region of the prediction accuracy of the prediction model, and The above in association with the dataset perform a second embedding that embeds it into the Perform , output a distribution diagram showing the space death, The first embedding embeds the plurality of datasets into the space using the statistics of the dataset, or the distance between the dataset and a predetermined reference dataset, as the features. The second embedding is, Create a peripheral dataset located around the dataset in the space, measure the predicted accuracy of the prediction model for the peripheral dataset, and calculate the region in the space where the predicted accuracy exceeds a predetermined threshold as the guaranteed prediction accuracy region, or The distance from the dataset at which the prediction model has an expected accuracy of above a predetermined threshold is calculated, and the region within the distance from the dataset is calculated as the area where the prediction accuracy is guaranteed. 。

[0008] In still another aspect of the present invention, a program is the dataset used by the prediction model multiple to , a 2D or 3D model with multiple features obtained from the aforementioned dataset as each axis perform a first embedding that embeds it into a space Perform , calculate a guarantee region of the prediction accuracy of the prediction model, and The above in association with the dataset perform a second embedding that embeds it into the Perform , output a distribution diagram showing the space death, The first embedding embeds the plurality of datasets into the space using the statistics of the dataset, or the distance between the dataset and a predetermined reference dataset, as the features. The second embedding is, Create a peripheral dataset located around the dataset in the space, measure the predicted accuracy of the prediction model for the peripheral dataset, and calculate the region in the space where the predicted accuracy exceeds a predetermined threshold as the guaranteed prediction accuracy region, or The distance from the dataset at which the prediction model has an expected accuracy of above a predetermined threshold is calculated, and the region within the distance from the dataset is calculated as the area where the prediction accuracy is guaranteed. and cause a computer to execute the processing.

Advantages of the Invention

[0009] According to the present disclosure, it becomes possible to visualize the relationship between the dataset used for prediction and the prediction accuracy by the prediction model.

Brief Description of the Drawings

[0010] [Figure 1] This is a block diagram showing the overall configuration of the model generation system according to the first embodiment. [Figure 2] Hardware configuration block for the model generation device. [Figure 3] This block diagram shows the functional configuration of the model generation device according to the first embodiment. [Figure 4] An example of how evaluation information is displayed is shown. [Figure 5] Other examples of how evaluation information is displayed are shown below. [Figure 6] An example of a distribution map is shown. [Figure 7] Here are some other examples of distribution maps. [Figure 8] This is a flowchart of the model analysis process performed by the model generation device. [Figure 9] This is a diagram showing the schematic configuration of a model generation system using a server and terminal devices. [Figure 10] This diagram shows the functional configuration of the model analysis device according to the second embodiment. [Figure 11] This is a flowchart of the processing performed by the model analysis device of the second embodiment. [Modes for carrying out the invention]

[0011] Preferred embodiments of this disclosure will be described below with reference to the drawings. <First Embodiment> [Overall structure] Figure 1 is a block diagram showing the overall configuration of a model generation system according to the first embodiment. The model generation system 1 comprises a model generation device 100, a display device 2, and an input device 3. The model generation device 100 is an application of the model analysis device of this disclosure and is composed of a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device and displays the evaluation information generated by the model generation device 100. The input device 3 is, for example, a mouse or keyboard and is used by the user to give instructions and inputs necessary when modifying the model or viewing the evaluation information.

[0012] First, the operation of the model generation system 1 will be briefly explained. The model generation device 100 generates a machine learning model (hereinafter simply referred to as "the model") using pre-prepared training data. The model generation device 100 also analyzes and evaluates the generated model. Specifically, the model generation device 100 makes predictions using evaluation data and analyzes the prediction accuracy of the model based on the prediction results. The model generation device 100 then generates a distribution map showing the relationship between the dataset used by the model and the guaranteed area of ​​the model's prediction accuracy, and presents it to the user as evaluation information. The user can check the evaluation information and input correction information for modifying the model by operating the input device 3.

[0013] A machine learning model is information that represents the relationship between explanatory variables and a target variable. A machine learning model is a component for estimating the result of a target variable by calculating the target variable based on the explanatory variables. A machine learning model is generated by running a learning algorithm with training data in which the value of the target variable is already obtained and arbitrary parameters as input. A machine learning model may be represented, for example, by a function c that maps the input x to the correct answer y. A machine learning model may estimate the numerical value of the target, or it may estimate the label of the target. A machine learning model may output a variable that describes the probability distribution of the target variable. A machine learning model may also be described as a "learning model," "analysis model," "AI (Artificial Intelligence) model," "trained model," "inference model," or "prediction formula." Here, explanatory variables are variables used as input in a machine learning model. Explanatory variables may also be described as "features" or "features."

[0014] Furthermore, the learning algorithm used to generate the machine learning model is not particularly limited and any existing learning algorithm is acceptable. For example, the learning algorithm may be a random forest, support vector machine, naive Bayes, a piecewise linear model using FAB inference (Factorized Asymptotic Bayesian Inference), or a neural network.

[0015] Furthermore, the piecewise linear model method using FAB inference is disclosed, for example, in U.S. Patent Publication US2014 / 0222741A1.

[0016] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the model generation device 100. As shown in the figure, the model generation device 100 includes an interface (I / F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

[0017] I / F111 performs data input and output with external devices. Specifically, training data used for model generation, evaluation data, and instructions and inputs entered by the user using input device 3 are input to the model generation device 100 via I / F111. In addition, evaluation information of the model generated by the model generation device 100 is output to the display device 2 via I / F111.

[0018] The processor 112 is a computer such as a CPU (Central Processing Unit) and controls the entire model generation device 100 by executing a pre-prepared program. The processor 112 may also be a GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), quantum processor, or FPGA (Field-Programmable Gate Array). The processor 112 performs the model analysis processing described later.

[0019] Memory 113 consists of ROM (Read Only Memory), RAM (Random Access Memory), and other components. Memory 113 is also used as working memory while the processor 112 is executing various processes.

[0020] The recording medium 114 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the model generation device 100. The recording medium 114 stores various programs that the processor 112 executes. When the model generation device 100 performs various processes, the programs stored in the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0021] DB115 stores information about the model generated by the model generation device 100 (hereinafter referred to as the "existing model") and the model after retraining and modification (hereinafter referred to as the "modified model"). DB115 also stores training data, evaluation data, and modification information entered by the user, as needed, through I / F111.

[0022] (Functional Configuration) Figure 3 is a block diagram showing the functional configuration of the model generation device 100 according to the first embodiment. Functionally, the model generation device 100 comprises a training data DB 121, a model training unit 122, a model DB 123, an evaluation data DB 124, and an analysis unit 125.

[0023] The training data DB121 stores the training data used to generate the model. The training data D1 is input to the model training unit 122. The training data D1 consists of multiple combinations of input data and the correct labels (teacher labels) for that input data.

[0024] The model training unit 122 trains the model using the training data D1 and generates the model. The model training unit 122 outputs the model data M corresponding to the generated model to the model DB 123 and the analysis unit 125. The model data M includes information on multiple parameters that make up the model. The parameter information includes, for example, information on the explanatory variables used as input to the model, information on the weights for each explanatory variable, and information on the weights for each sample that makes up the input data.

[0025] Furthermore, the model training unit 122 retrains the existing model to generate a modified model. In this case, the model training unit 122 modifies the parameters that make up the model based on the modification information D3 input by the user using the input device 3, and retrains the model using training data for retraining as needed. The model training unit 122 stores the model data M of the modified model obtained through retraining in the model DB 123 and outputs it to the analysis unit 125.

[0026] The evaluation data DB124 stores evaluation data used to evaluate the generated model. Evaluation data includes, for example, some of the collected data that was not used as training data, newly collected data, and validation data prepared for verification. Training data may also be used as evaluation data. Evaluation data consists of multiple combinations of input data and the correct labels (training labels) for that input data.

[0027] Evaluation data is stored and used in units of datasets. Each dataset is associated with attribute information indicating its time series, such as when it was collected. For example, when using datasets collected in a real environment as training and evaluation data, each dataset is associated with information such as the time period (e.g., Spring 2019, Summer 2019, Autumn 2019) or version information indicating the time series of the dataset.

[0028] The analysis unit 125 analyzes the prediction accuracy of the current and past models using evaluation data. Specifically, the analysis unit 125 inputs the evaluation data into each model, performs predictions, and obtains the prediction results. Then, the analysis unit 125 estimates the prediction accuracy of each model based on the evaluation data used and the prediction results. Then, the analysis unit 125 creates evaluation information D2 for evaluating each model using the obtained prediction accuracy of each model. Specifically, the analysis unit 125 creates a distribution map showing multiple datasets used for a given model and the guaranteed region of the prediction accuracy of the current and past models of that model, and outputs it to the display device 2 as evaluation information D2. Here, the guaranteed region refers to the region in which the current model shows a prediction accuracy of above a predetermined threshold. The method for creating the distribution map will be explained in detail later. Note that the analysis unit 125 may use F1 score, precision, or recall instead of prediction accuracy. Also, the analysis unit 125 may use mean squared error or cross-entropy, where a lower value is considered better, instead of prediction accuracy. The analysis unit 125 is an example of a first embedding means, a second embedding means, and an output means.

[0029] The display device 2 displays the evaluation information D2 output by the analysis unit 125. By looking at the displayed distribution map, the user can easily understand the relationship between the multiple datasets used by a particular model and the guaranteed accuracy range of the current and past models of that model.

[0030] Furthermore, the user inputs correction information D3 into the input device 3 to modify the current model as needed. The correction information D3 includes information on modifications such as the explanatory variables used as input to the model, the weights for each explanatory variable, and the weights for each sample that makes up the input data. The model training unit 122 modifies the model by retraining it using the input correction information D3.

[0031] [Example of displaying evaluation information] Next, we will explain an example of how evaluation information is displayed on the display device 2. Figure 4 shows an example of the display of distribution map 40 as evaluation information. Distribution map 40 shows a two-dimensional space where the horizontal axis (X-axis) and vertical axis (Y-axis) are defined by specific features. Distribution map 40 shows the analysis results of past models up to a certain point in time, based on the current model.

[0032] In this embodiment, the distribution diagram will be described as a two-dimensional space, but the number of dimensions of the distribution diagram is not limited to two. For example, the distribution diagram may be represented in three dimensions. In other words, the distribution diagram only needs to be readable.

[0033] First, distribution diagram 40 shows multiple datasets 41. Each dataset 41 represents a collection of several hundred to several thousand data points. Each dataset 41 is embedded in a corresponding location in a two-dimensional space based on the features of the data it contains. Each dataset 41 is connected by a line segment that indicates a time series. That is, the line segments between datasets 41 connect datasets 41 that are consecutive in time series.

[0034] In the example in Figure 4, in the time series, dataset 41b is the next dataset after dataset 41a, dataset 41c is the next dataset after dataset 41b, and dataset 41x is the most recent dataset. That is, in the example in Figure 4, the datasets are newer in the order of dataset 41a, 41b, 41c, ..., 41x. Note that, for convenience, older datasets than dataset 41a are omitted from the illustration, as shown by the dashed line connected to dataset 41a. By embedding the datasets in a two-dimensional space in this way, users can see how the features of each dataset have changed. Note that "embedding" means converting high-dimensional vector data into low-dimensional vector data.

[0035] Distribution diagram 40 further shows the guaranteed prediction accuracy regions 42 and 43. The guaranteed region 42 of the current model, shown by a solid line, indicates the region where the current model exhibits prediction accuracy above a predetermined threshold. The guaranteed regions 43a and 43b of past models, shown by dashed lines, indicate the region where each past model exhibits prediction accuracy above a predetermined threshold. As can be seen from the changes in dataset 41, guaranteed region 43b represents the guaranteed region of the past model immediately preceding the current model, and guaranteed region 43a represents the guaranteed region of the past model two generations prior to the current model.

[0036] In this way, by displaying the guaranteed accuracy regions of the current and past models, users can see how the guaranteed regions of each model change with model updates. Furthermore, by displaying each dataset, the current model, and past models in the same two-dimensional space, users can see the correspondence between changes in the dataset and the resulting changes in the guaranteed regions of the models. In the example in Figure 4, the guaranteed regions of both the current and past models are displayed, but it is also possible to display only the guaranteed region of the current model. Additionally, users may be allowed to specify whether or not to display the guaranteed region of past models, and how far back in the past the guaranteed region of each model should be displayed.

[0037] Figure 5 shows another example of the distribution map display. In the distribution map 40a shown in Figure 5, the guaranteed region 42 of the current model is displayed using contour lines. The numerical values ​​on each contour line indicate the estimated value of the prediction accuracy (hereinafter referred to as "expected accuracy"), with the inner contour lines indicating higher expected accuracy. In this example, the threshold for expected accuracy is set to "80," and the region where the expected accuracy is 80 or higher is defined as the guaranteed region of prediction accuracy. Note that in the example in Figure 5, the contour line also displays contour lines outside the guaranteed region 42 of the current model (i.e., contour lines where the expected accuracy is less than "80"). By looking at the contour line, users can learn which directions the current model is strong and weak against changes in the trend of the dataset within the guaranteed region 42. For example, in the example in Figure 5, it can be seen that the current model is weak against changes in the trend of the dataset to the right in a two-dimensional space, and relatively strong against changes to the left or downward to the left. In this way, contour lines enable a detailed understanding of the model's characteristics.

[0038] One way to display the contour map is, for example, when the distribution map 40 shown in Figure 4 is displayed, the user can select the guaranteed area 42 using a mouse or the like, and the contour map shown in Figure 5 will be displayed. The guaranteed area 42 may be selected when the user hovers cursor C over the guaranteed area 42 (so-called mouse hover), or when the user clicks on the guaranteed area 42 with the mouse. Furthermore, the contour map may be displayed not only for the guaranteed area 42 of the current model, but also for the guaranteed area 43 of past models. For example, if the user moves cursor C and selects the guaranteed area 43b, the guaranteed area 43b may be displayed as a contour map.

[0039] [How to create a distribution map] Next, we will explain how to create a distribution map. A distribution map is created by embedding the dataset in a two-dimensional space, and by embedding the guaranteed region of the model's prediction accuracy in the same two-dimensional space.

[0040] (How to embed the dataset) First, let's explain how to embed a dataset into a two-dimensional space. (1) Method of embedding using statistics from the dataset The first method involves embedding each dataset into a two-dimensional space using the statistics of each dataset. Specifically, the analysis unit 125 retrieves a dataset from the evaluation data DB 124, calculates features such as the mean and variance of each feature in the dataset, or the covariance between features, and uses these as the features of the dataset. Then, the analysis unit 125 embeds the dataset into a two-dimensional space based on the obtained features. If the features of the dataset are three or more dimensions, the analysis unit 125 can compress the three or more dimensions into two-dimensional features using dimensionality reduction techniques such as PCA (Principal Component Analysis) or U-MAP (Uniform Manifold Approximation and Projection) and embed them into the two-dimensional space.

[0041] (2) Embedding method based on distance from reference dataset The second method involves embedding each dataset into a two-dimensional space based on its distance from a reference dataset. Specifically, the analysis unit 125 first determines a reference dataset for embedding from multiple (two or more) datasets. Next, the analysis unit 125 calculates the distance between the dataset to be embedded (hereinafter referred to as the "target dataset") and each reference dataset, and uses these distances as features of the target dataset. For example, KL divergence or Wasserstein distance can be used as the distance. Then, the analysis unit 125 embeds the dataset into a two-dimensional space based on the obtained features. If the features of a dataset are three or more dimensions, the analysis unit 125 can compress the three or more dimensions into two-dimensional features using a dimensionality reduction method, similar to the first method, and embed them into the two-dimensional space.

[0042] (Method for embedding the model's guaranteed area) Next, we will explain how to embed the guaranteed region of the model's prediction accuracy into a two-dimensional space. (1) A method of plotting regions by preparing multiple datasets and measuring their performance. In the first method, the analysis unit 125 first prepares multiple datasets surrounding the training data. The datasets can be prepared using one of the following methods. (A) The analysis unit 125 selects a random or specific portion of the training dataset. The portion of the training dataset becomes a dataset with properties that are somewhat different from the entire training dataset, and becomes a peripheral dataset to the training dataset in a two-dimensional space. (B) The analysis unit 125 perturbs the training dataset to create a slightly different dataset. (C) Analysis unit 125 uses the model trained on the training dataset to create a dataset similar to the training dataset.

[0043] Once surrounding datasets are created using the method described above, the analysis unit 125 measures the expected accuracy of the model for the surrounding datasets and plots the region in the two-dimensional space where the expected accuracy exceeds a certain threshold as the region where prediction accuracy is guaranteed.

[0044] (2) A method for plotting regions using a performance estimation method that uses the distance between datasets. Generally, the prediction accuracy of a model decreases for datasets located farther from the training dataset. Therefore, in the second method, the analysis unit 125 approximates and estimates the degree of decrease in prediction accuracy based on the distance from the training dataset. Then, using the above estimation method, the analysis unit 125 calculates the distance at which the model has a projected accuracy above a certain threshold, and plots the region within that distance from the training dataset as the region where prediction accuracy is guaranteed in a two-dimensional space.

[0045] The contour plot of the estimated accuracy shown in Figure 5 can be created by drawing the guaranteed region while changing the threshold for estimated accuracy using one of the methods described above.

[0046] [Example of analysis based on distribution maps] Next, we will explain an example of considering the performance of a model based on a distribution map. (Example 1) Figure 6(A) shows an example of a distribution map. In this example, the latest dataset 41x deviates from the guaranteed region 42 of the current model, so the user can determine that the current model needs to be retrained. It is preferable for the user to perform retraining when the latest dataset 41x moves outside the guaranteed region 42 of the current model.

[0047] (Example 2) Figure 6(B) shows an example of another distribution map. In this example, the current model's guaranteed region 42 covers a wider area than the locations of several recent datasets, so users can determine that the current model is a model with good long-term performance. Furthermore, the longer the period during which the dataset is included within the model's guaranteed region, the better the model can be evaluated in terms of operation.

[0048] (Example 3) Figure 6(C) shows an example of another distribution map. In this example, the recent dataset has deviated from the current model's guaranteed region 42 and returned to the historical model's guaranteed region 43. Therefore, the user can determine that they need to switch to a historical model that covers the latest dataset 41x.

[0049] (Example 4) Figure 7(A) shows an example of another distribution map. In this example, the user can see the need to select a model in response to changes in the dataset's trends. Specifically, in this example, since the dataset's trends change periodically, the user can see that they need to periodically change the model in response to these periodic changes in the dataset's trends. The user may also consider training a model specifically for datasets with periodicity, or removing the periodicity from the dataset through preprocessing before training the model.

[0050] (Example 5) Figure 7(B) shows an example of another distribution map. In this example as well, the user can see the need to select a model in response to changes in the dataset's trends. Specifically, in this example, since the dataset's trends are continuously changing in one direction, the user can build a model that is operational in the long term by predicting the next change in the dataset's trends and designing the model accordingly.

[0051] [Model analysis processing] Next, the model analysis process performed by the model generation device 100 will be described. Figure 8 is a flowchart of the model analysis process performed by the model generation device 100. The model analysis process analyzes the current model and past models, generates multiple datasets, and generates a distribution map in which the current model and past models are embedded in a two-dimensional space, and displays it on the display device 2. This process is realized by the processor 112 shown in Figure 2 executing a pre-prepared program and mainly operating as the analysis unit 125 shown in Figure 3.

[0052] First, the model generation device 100 selects the model to be analyzed (hereinafter referred to as the "target model") (step S10). This determines the current model and its corresponding past model. The user may also specify the target past model by specifying the analysis period, etc.

[0053] Next, the analysis unit 125 embeds multiple datasets corresponding to the target model into a two-dimensional space using the method described above (step S11). Next, the analysis unit 125 acquires the current model and past models of the target model (step S12), calculates the guaranteed prediction accuracy region of the current model and past models using the method described above, and embeds it into the two-dimensional space (step S13). Then, the analysis unit 125 outputs a distribution map of the two-dimensional space into which the datasets, current model, and past models are embedded to the display device 2 and displays it (step S14). The process then ends.

[0054] [Differentiation] In the above embodiment, the model generation device 100 is configured as an independent device such as a PC, but instead, the model generation device may be configured with a server and a terminal device. Figure 9 is a block diagram illustrating the schematic configuration of a model generation system 1x using a server and a terminal device. In Figure 9, the server 100x has the configuration of the model generation device 100 shown in Figure 3. In addition, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in Figure 3.

[0055] <Second Embodiment> Figure 10 is a block diagram showing the functional configuration of the model analysis device 70 of the second embodiment. The model analysis device 70 comprises a first embedding means 71, a second embedding means 72, and an output means 73.

[0056] Figure 11 is a flowchart of the processing performed by the model analysis device 70 of the second embodiment. The first embedding means 71 embeds the dataset used by the prediction model into space (step S71). The second embedding means 72 calculates the guaranteed region of the prediction accuracy of the prediction model and embeds it into space (step S72). The output means 73 outputs a distribution map showing the space (step S73).

[0057] According to the model analysis device 70 of the second embodiment, it is possible to visualize the relationship between the dataset used for prediction and the prediction accuracy of the prediction model.

[0058] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0059] (Note 1) A first embedding means for embedding the dataset used by the predictive model into space, A second embedding means calculates the guaranteed prediction accuracy region of the prediction model and embeds it in the space, Output means for outputting a distribution map showing the aforementioned space, A model analysis device equipped with the following features.

[0060] (Note 2) The first embedding means embeds a plurality of datasets into the space, In the distribution diagram, the multiple datasets are connected by line segments indicating time series, as described in Appendix 1 of the model analysis apparatus.

[0061] (Note 3) The aforementioned distribution diagram shows the guaranteed region of the current prediction model and the guaranteed region of past prediction models, as described in Appendix 1 or 2 of the model analysis apparatus.

[0062] (Note 4) The distribution map is a model analysis device according to any one of the appendices 1 to 3, which displays the guaranteed region of the prediction model selected by the user using contour lines indicating the prediction accuracy.

[0063] (Note 5) The first embedding means is a model analysis apparatus according to any one of the appendices 1 to 4, which embeds the dataset into the space using the statistics of the dataset.

[0064] (Note 6) The first embedding means is a model analysis apparatus according to any one of the appendices 1 to 4, which embeds the target dataset into the space using the distance between the target dataset and a predetermined reference dataset as a feature of the target dataset.

[0065] (Note 7) The second embedding means is a model analysis apparatus according to any one of the appendices 1 to 6, which calculates the guaranteed area of ​​the prediction model by measuring the accuracy of a plurality of datasets.

[0066] (Note 8) The model analysis apparatus according to any one of the appendices 1 to 6, wherein the second embedding means calculates the guaranteed region of the prediction model based on the distance from the training dataset used to generate the prediction model.

[0067] (Note 9) Embed the dataset used by the predictive model into space, The guaranteed accuracy range of the prediction model is calculated and embedded in the space. A model analysis method that outputs a distribution map showing the aforementioned space.

[0068] (Note 10) Embed the dataset used by the predictive model into space, The guaranteed accuracy range of the prediction model is calculated and embedded in the space. A recording medium containing a program that causes a computer to execute a process to output a distribution map showing the aforementioned space.

[0069] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of symbols]

[0070] 1. 1x Model Generation System 2, 2x display device 3. 3x Input Devices 7 Terminal devices 100 Model Generators 112 processors 121 Training Data DB 122 Model Training Department 123 Model DB 124 Evaluation Data Database 125 Analysis Department

Claims

1. A first embedding means for embedding multiple datasets used by a predictive model into a two-dimensional or three-dimensional space with features obtained from the datasets as each axis, A second embedding means calculates the guaranteed prediction accuracy range of the prediction model and embeds it in the space in correspondence with the dataset, Output means for outputting a distribution map showing the aforementioned space, Equipped with, The first embedding means embeds the plurality of datasets into the space using the statistical value of the dataset, or the distance between the dataset and a predetermined reference dataset, as the feature quantity. The second embedding means is Create a peripheral dataset located around the dataset in the space, measure the predicted accuracy of the prediction model for the peripheral dataset, and calculate the region in the space where the predicted accuracy exceeds a predetermined threshold as the guaranteed prediction accuracy region, or A model analysis device that calculates the distance from the dataset at which the prediction model will have an expected accuracy of a predetermined threshold or higher, and calculates the region within the distance from the dataset as the guaranteed region for the prediction accuracy.

2. The model analysis apparatus according to claim 1, wherein the plurality of datasets in the distribution diagram are connected by line segments indicating a time series.

3. The model analysis apparatus according to claim 1 or 2, wherein the distribution diagram shows the guaranteed region of the current prediction model and the guaranteed region of past prediction models.

4. The model analysis apparatus according to any one of claims 1 to 3, wherein the distribution map displays the guaranteed region of the prediction model selected by the user using contour lines indicating the prediction accuracy.

5. A model analysis method performed by a computer, A first embedding is performed in which multiple datasets used by the prediction model are embedded in a two-dimensional or three-dimensional space with multiple features obtained from the datasets as each axis. A second embedding is performed to calculate the guaranteed prediction accuracy range of the prediction model and embed it in the space in correspondence with the dataset. Output a distribution map showing the aforementioned space. The first embedding embeds the plurality of datasets into the space using the statistics of the dataset, or the distance between the dataset and a predetermined reference dataset, as the features. The second embedding is, Create a peripheral dataset located around the dataset in the space, measure the predicted accuracy of the prediction model for the peripheral dataset, and calculate the region in the space where the predicted accuracy exceeds a predetermined threshold as the guaranteed prediction accuracy region, or A model analysis method that calculates the distance from the dataset at which the prediction model has an expected accuracy of a predetermined threshold or higher, and calculates the region within the distance from the dataset as the guaranteed region for the prediction accuracy.

6. A first embedding is performed in which multiple datasets used by the prediction model are embedded in a two-dimensional or three-dimensional space with multiple features obtained from the datasets as each axis. A second embedding is performed to calculate the guaranteed prediction accuracy range of the prediction model and embed it in the space in correspondence with the dataset. Output a distribution map showing the aforementioned space. The first embedding embeds the plurality of datasets into the space using the statistics of the dataset, or the distance between the dataset and a predetermined reference dataset, as the features. The second embedding is, Create a peripheral dataset located around the dataset in the space, measure the predicted accuracy of the prediction model for the peripheral dataset, and calculate the region in the space where the predicted accuracy exceeds a predetermined threshold as the guaranteed prediction accuracy region, or A program that causes a computer to perform a process of calculating the distance from the dataset at which the prediction model will have an expected accuracy of a predetermined threshold or higher, and calculating the region within the distance from the dataset as the guaranteed region for the prediction accuracy.

Images