Model evaluation system, model evaluation method, and program

Abstract

The goal is to select a model from among several that offers high predictive accuracy while guaranteeing a certain level of stability and interpretability. [Solution] The model evaluation system (100) comprises a computing device (110) that executes a program and a storage device (130) that stores the program. The computing device selects a model from a plurality of machine learning models whose AUC value is equal to or greater than a standard value. The computing device calculates a first evaluation value using the contributions of the top features of the selected model, including at least one with the highest contribution. The computing device performs a first filtering to exclude the model with the smallest first evaluation value from the selected models. The computing device then selects the model with the highest AUC value from the models remaining after filtering.

Description

Technical Field

[0001] The present invention relates to a model evaluation system, a model evaluation method, and a program suitably applied to evaluation of machine learning models such as risk prediction models.

Background Art

[0002] In recent years, risk prediction using machine learning models has received widespread attention in the prevention and management of lifestyle-related diseases (diabetes, hypertension, dyslipidemia, etc.). Physicians provide guidance such as proposing solutions to individuals based on the results of risk prediction. These machine learning models are expected to perform highly accurate predictions based on large volumes of data, but several challenges exist in actual operation.

[0003] First, in the construction of machine learning models, a problem is that prediction accuracy varies significantly depending on the characteristics of the training dataset used and the settings of model parameters. In particular, AUC (Area Under the Curve), which is an indicator for evaluating model performance, and the extraction results of key factors in prediction often become unstable depending on the content of the training dataset and model parameters. A factor is an element or cause that can pose a risk to a prediction target (for example, a lifestyle-related disease). Such instability reduces the reliability of the model and makes it difficult to apply in actual medical settings. In a model with stable factors, the importance of key factors remains high and stable even when the content of the training dataset or model parameters change.

[0004] Furthermore, when evaluating machine learning models, it is required to select models with consistently high prediction accuracy even when the content of the training dataset and the settings of model parameters differ, but this has not been sufficiently achieved by conventional methods. For example, a model with a high AUC does not necessarily have stable factors. The key factors or their rankings may differ significantly between models, which makes interpretation of prediction results difficult.

[0005] Furthermore, when conducting parameter studies of machine learning models, it is necessary to generate many models using the same training dataset and compare their performance, but this process itself is very time-consuming and resource-intensive. Therefore, there is a need for methods to efficiently and effectively evaluate models and select the optimal model. On top of that, it is important to select a model that provides reliable prediction results from among multiple models.

[0006] Incidentally, Patent Document 1 discloses an information processing system that improves the efficiency and convenience of analyzing the likelihood of a mutation affecting the onset or progression of a disease. Patent Document 1 states that "each of multiple sequence mutations is classified, based on predetermined classification criteria, into either a high category, which has the highest probability of being selected as a target sequence mutation with harmful risk, or one or more low categories, which have a lower probability." It also states that "among the sequence mutations classified into low categories, those whose probability output by the model is above a certain level are reclassified into high categories." [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2023-102988 [Overview of the project] [Problems that the invention aims to solve]

[0008] The information processing system described in Patent Document 1 does not support evaluating multiple models and selecting the most appropriate model.

[0009] Given the above situation, there was a need for a method that could select a model from among several models that offered high predictive accuracy while guaranteeing a certain level of stability and interpretability. [Means for solving the problem]

[0010] To solve the above problems, a model evaluation system according to one aspect of the present invention comprises a computing device for executing a program and a storage device for storing the program, The aforementioned computing device selects a model from among multiple machine learning models whose AUC value is equal to or greater than a certain threshold. The computing device calculates a first evaluation value using the contributions of the top features, including at least one with the highest contribution among the selected model's features, and performs a first filtering to exclude the model with the smallest first evaluation value from the selected models. The computing device selects the model with the highest AUC value from among the models remaining after filtering.

[0011] Each of the model evaluation method and program according to one aspect of the present invention corresponds to a model evaluation system according to one aspect of the present invention. [Effects of the Invention]

[0012] According to at least one aspect of the present invention, it is possible to select a model from among several models that not only has high prediction accuracy but also guarantees a certain level of stability and interpretability. Other issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing an example configuration of a data processing system related to one embodiment of the present invention. [Figure 2] This figure shows an example of the AUC of a risk prediction model according to one embodiment of the present invention. [Figure 3] This figure shows an example of a Weight value for a risk prediction model relating to one embodiment of the present invention. [Figure 4] This flowchart shows an example of the procedure for evaluating a risk prediction model in a data processing system according to one embodiment of the present invention. [Figure 5]This figure shows an example of a GUI screen provided to a customer according to one embodiment of the present invention. [Modes for carrying out the invention]

[0014] Hereinafter, examples of embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the attached drawings. In this specification and the accompanying drawings, common or similar components are denoted by the same reference numeral, and redundant descriptions are omitted. Unless otherwise specified, the number of each component may be singular or plural. In the following examples of embodiments, various types of information are described in table format, but these types of information may be in data formats other than tables. Also, various names such as "XX Information," "XX Table," "XX List," and "XX List" are interchangeable.

[0015] [Data Processing System Configuration] First, the configuration of a data processing system according to one embodiment of the present invention will be described with reference to Figure 1. Figure 1 is a block diagram showing an example configuration of a data processing system according to one embodiment of the present invention. The data processing system 100 shown in Figure 1 is a system that applies a quantitative evaluation method for parameter studies in a lifestyle-related disease risk prediction model, and is an example of a model evaluation system. In this specification, a risk prediction model or machine learning model may be simply referred to as a "model".

[0016] The data processing system 100 comprises a central processing unit (CPU) 110, memory 120, storage unit 130, connection interface (IF) 141, and network interface (IF) 142. The data processing system 100 is an example of a computer. Each of the above devices is connected to each other via a system bus so that they can communicate with one another.

[0017] This data processing system 100 is particularly effective when using data with the knowledge that "features with higher contribution degrees have greater importance than features with lower contribution degrees", and aims to improve the stability and interpretability of the model. The contribution degree is the degree to which a certain factor (feature) contributes to the prediction result of the risk prediction model. A factor (feature) with a high contribution degree has a relatively greater impact on the prediction result of the model (than a factor with a low contribution degree). In general, a feature with a higher contribution degree is considered to have higher importance for the model.

[0018] The CPU 110 executes a computer program stored in the memory 120 to perform parameter study execution and model evaluation. The storage unit 130 stores medical data used for lifestyle-related disease prediction, health checkup data, and feature quantity data (explanatory variable data, objective variable data) extracted from these data. For example, the medical data is general medical data, such as data obtained by treating patients at medical institutions, medical open source data, medical paper data, and data obtained from the website of the Ministry of Health, Labour and Welfare. The health checkup data is data of examinees obtained in health checkups. The data processing system 100 performs data communication with an external database and the like via the network interface 142.

[0019] The connection interface 141 is connected to the display device 101 and the input device 102. The connection interface 141 is an interface used for operations and outputs of the data processing system 100, and a user can input data and check results through the display device 101 and the input device 102.

[0020] The display device 101 displays the model evaluation results and the selection status of main factors, and the user can set evaluation conditions and filtering criteria using the input device 102. A display device such as a liquid crystal display device can be used as the display device 101, and an input device such as a mouse or a keyboard can be used as the input device 102.

[0021] (Memory 120) Memory 120 stores a computer program that includes the following functional blocks: The statistical calculation function 121 calculates statistics (such as the mean, median, and difference in contributions) based on at least medical data and health checkup data used for predicting lifestyle-related diseases. The data extraction processing function 122 extracts data from the storage unit 130 based on specific conditions. For example, in this embodiment, the data extraction processing function 122 obtains the AUC value and Weight value (weight) of the risk prediction model to be evaluated by referring to the storage unit 130. The AUC value represents the area under the ROC (Receiver Operating Characteristic) curve. In this embodiment, the Weight value is an example of information corresponding to the contribution of the features of the risk prediction model. In machine learning models, the Weight value is the weight assigned to each feature during the model's learning process, and indicates the degree to which the feature has an impact on the prediction. The Weight value is adjusted during the model's learning process.

[0022] The priority calculation function 123 calculates the priority of the data (features) to be processed. The features correspond to the risk factors of the prediction target. Here, priority does not indicate the degree of priority in the narrow sense, but rather corresponds to an evaluation value used in the selection of models described later, or the level of said evaluation value.

[0023] The UI display processing function 124 performs the process of displaying information on the user interface (UI) of the display device 101. The UI display processing function 124 also performs the process of receiving input from the input device 102.

[0024] (Storage unit 130) The explanatory variable data storage unit 131 stores explanatory variable data used by the risk prediction model under evaluation for prediction and analysis. Explanatory variables are variables that are involved in an event and are responsible for the event. For example, the explanatory variable data in this embodiment includes numerical values ​​related to factors (features) that may be risk factors for lifestyle-related diseases.

[0025] The dependent variable data storage unit 132 stores the dependent variable data, which is the result that the risk prediction model under evaluation attempts to predict or analyze. A dependent variable is a variable that is involved in an event and is the result of that event. For example, the dependent variable data in this embodiment includes the risk of lifestyle-related diseases (such as diseases that one may contract) and their causes. In this embodiment, it is assumed that explanatory variable data and dependent variable data are prepared for each risk prediction model.

[0026] The system configuration file storage unit 133 stores files that define the configuration information and parameters of the data processing system 100. The evaluation result storage unit 134 stores the evaluation results of the risk prediction model performed by each functional block in the memory 120, such as the results of data analysis and statistical processing.

[0027] [AUC] Next, the AUC and Weight values ​​of the risk prediction model will be explained with reference to Figures 2 and 3. Figures 2 and 3 are specific examples of the AUC and Weight values ​​of the risk prediction model to be evaluated, which are stored in the memory unit 130 in this embodiment. For example, they show an example of the AUC and Weight values ​​for dyslipidemia.

[0028] Figure 2 shows an example of the AUC of the risk prediction model according to this embodiment. The AUC201 shown in Figure 2 has the items "seed", "ne", and "AUC for men under 60 years old". "Seed" is information that indicates the initial position of exploration for the risk prediction model, that is, the position where machine learning begins in the risk prediction model. "ne" indicates how many times the machine learning process will be iterated. As mentioned above, "AUC for men under 60" represents the ratio (≤1) of the area under the ROC (Receiver Operating Characteristic) curve. In this example, the AUC value for men under 60 is shown.

[0029] For example, in Figure 2, the AUC is "0.93131" when "seed" is "500" and "ne" is "10". Also, the AUC is "0.97258" when "seed" is "3500" and "ne" is "10", and the AUC is "0.91099" when "seed" is "5000" and "ne" is "10".

[0030] [Weight value] Figure 3 shows an example of the Weight values ​​for the risk prediction model according to this embodiment. The Weight value 202 shown in Figure 3 has the following items: "AUC", "Category", and "Features (factors)" ranked from 1st to 10th, along with their "Weight values". In this example, examples of "Weight values" for the "Features" ranked from 1st to 10th are displayed for each combination of "seed" and "ne".

[0031] For example, when "seed" is "500" and "ne" is "10", the AUC is "0.93131" and the "category" is "male under 60 years old". At this time, the weight values ​​were: 1st place "γ-GT" at "0.039581199", 2nd place "LDL" at "0.03023443", 3rd place "urinary protein" at "0.03007401", 4th place "urinary glucose" at "0.026328671", 5th place "triglycerides" at "0.021249108", 6th place "urinary occult blood" at "0.021067991", 7th place "BMI" at "0.020882722", 8th place "diastolic blood pressure" at "0.015237333", 9th place "GOT" at "0.014242183", and 10th place "systolic blood pressure" at "0.01315789".

[0032] In addition to the case where "seed" is "500" and "ne" is "10", examples of the "Weight values" of the top 10 "features" are shown for cases where "seed" is "3500" and "ne" is "10", and "seed" is "5000" and "ne" is "10". In the figure, the "features" shown in hatching are features that are generally considered to be major causative factors of dyslipidemia. Here, three factors are listed: LDL (Low-Density Lipoprotein), triglycerides, and HDL (High-Density Lipoprotein).

[0033] The combination of "features" and their "weight values" differ significantly depending on the combination of "seed" and "ne". For example, when "seed" is "500" and "ne" is "10", "LDL" is ranked 2nd and "triglycerides" is ranked 5th. The "weight values" are "0.03023443" for "LDL" and "0.021249108" for "triglycerides". Also, when "seed" is "5000" and "ne" is "10", "triglycerides" is ranked 1st, "LDL" is ranked 5th and "HDL" is ranked 8th. The "weight values" are "0.041609212" for "triglycerides", "0.028191383" for "LDL", and "0.011767983" for "HDL".

[0034] In this embodiment, the feature weight values ​​may be replaced with SHAP (Shapley Additive exPlanations) values. SHAP values ​​are values ​​that evaluate the contribution of each feature to the prediction. By using SHAP values, the influence of features can be evaluated locally, and the interpretability of the model's prediction results can be improved.

[0035] [Evaluation process for risk prediction models] Figure 4 is a flowchart showing an example of the procedure for evaluating a risk prediction model in a data processing system 100 (see Figure 1) according to one embodiment of the present invention. The CPU 110 executes a program stored in memory 120, thereby executing the evaluation process of the risk prediction model shown in Figure 4.

[0036] First, in step S1, the data extraction processing function 122, the explanatory variable data storage unit 131, and the target variable data storage unit 132 obtain AUC values ​​and Weight values ​​for the risk prediction model of the subject to evaluation (e.g., lifestyle-related diseases) and make them the subject of processing (input processing). In step S1, the data processing system 100 creates multiple risk prediction models with different model parameters, etc., using the explanatory variable data and dependent variable data stored in the storage unit 130. The data extraction processing function 122 calculates the AUC value and Weight value for each model and uses them for processing. Note that multiple risk prediction models may already be stored in the storage unit 130.

[0037] Next, in step S2, the statistical calculation function 121 selects only the models whose AUC value meets a certain criterion (e.g., 0.85 or higher) (evaluation process based on AUC value). In step S2, statistical filtering is applied to the evaluation process to exclude models whose AUC values ​​do not meet a certain standard. At this stage, low-accuracy models are eliminated, so that only better (higher accuracy) models are used in subsequent evaluation steps.

[0038] Next, in step S3, the priority calculation function 123 performs a first filtering evaluation on the risk prediction model selected in step S2. The priority calculation function 123 evaluates each model based on the absolute difference between the contribution of the top-ranking features (corresponding to factors) and the contribution of the lower-ranking features. Models with a small difference are excluded because it is determined that the contribution of the features, or in other words, their importance, cannot be clearly distinguished. In this way, the first filtering is performed on the models selected in step S2 to further select the models.

[0039] For example, for each model's main features, the absolute difference between the weight values ​​(corresponding to contribution) of the top 1st to 3rd most contributing features and the weight value of the 4th most contributing feature is calculated and used as the first evaluation value. In this case, the first evaluation value is obtained as the sum of, for example, "the difference between the 1st and 4th weight values", "the difference between the 2nd and 4th weight values", and "the difference between the 3rd and 4th weight values". Then, the model with the smallest first evaluation value, or models with the same order of first evaluation value in addition to that model, is excluded, and the remaining models are sent to the next step S3. Here, "same order" means that the number of digits of the number being discussed is the same as the number of digits of the number being compared (the difference is less than 10 times).

[0040] In this way, the priority calculation function 123 can select an appropriate model by performing a first filtering based on an index of the contribution of the model's features. Generally, features with a high contribution are considered to be important features for the model. In step S3, a first evaluation value is calculated using the contributions of the top features and the other features, and models that satisfy the predetermined conditions in the first evaluation value are selected. The top features of the models that remain without being excluded in the first filtering of step S3 have clearly distinguishable importance (contribution) compared to the lower-ranking features. Therefore, the interpretability of the model's prediction results is high. Also, because there is a difference in contribution between the top and lower-ranking features that satisfies the predetermined conditions, it can be said that the top features are consistently extracted relative to the lower-ranking features.

[0041] The difference between the weight values ​​of the top three features and the weight value of the fourth-ranked feature was calculated, but the method for calculating the first evaluation value can be changed depending on the data used (for example, the main features of the model being evaluated). Alternative filtering methods include, for example, keeping models where the difference between the contribution of the first-ranked feature and the contribution of the fourth-ranked feature is greater than a certain value, or keeping models where the difference between the contribution of the first-ranked feature and the contribution of the tenth-ranked feature is greater than a certain value. Filtering that excludes models where the contribution of the first-ranked feature is less than a predetermined value is also possible. Furthermore, filtering that excludes only the model with the lowest first evaluation value, or excludes a predetermined number of lower-ranking models with low first evaluation values ​​is also possible. In this way, the ranking of the features used to calculate the first evaluation value and the method for calculating the difference in contributions can be changed.

[0042] Next, in step S4, the priority calculation function 123 performs a second filtering evaluation on the risk prediction model selected in step S3. The priority calculation function 123 calculates the difference in feature contribution between consecutive ranks for each model, and compares the differences in contribution between consecutive ranks. Models with small differences in contribution between consecutive ranks are judged to have unstable feature contributions and are excluded. In this way, a second filtering is performed on the models selected in step S3 (first filtering) to further narrow down the selection of models.

[0043] For example, for each model's main features (e.g., ranked 1st to 10th), the absolute values ​​of the difference in contribution between the 1st and 2nd place features, the 2nd and 3rd place features, ..., the 9th and 10th place features are calculated, and the smallest absolute value among these differences is taken as the second evaluation value. Then, the model that obtained the smallest second evaluation value, or models with the same order of second evaluation value in addition to that model, is excluded, and the remaining models are sent to the next step S5.

[0044] After the first filtering, the priority calculation function 123 performs a more detailed second filtering based on an index of the contribution of the model's features to narrow down the selection to more appropriate models. In this embodiment, the model with the smallest second evaluation value, or models of the same order as that second evaluation value, are excluded, leaving only models with clearly defined and stable differences in contribution. In the models that were not excluded in the second filtering in step S4, each feature (contribution) is stable because the contributions of consecutively ranked features are relatively far apart.

[0045] Next, in step S5, the statistical calculation function 121 selects the model with the highest AUC among the risk prediction models selected in step S4 as the optimal model (optimal model determination process). The selected model is guaranteed to have not only high prediction accuracy but also excellent feature interpretability and stability. As mentioned above, the importance of the model's features is clear in step S3, and therefore the interpretability of the features is also improved. In model selection, it is important that the selection result (optimal model) can be explained in a way that is easy for humans to understand, and the "interpretability" of features refers to the property or ability of humans to interpret them. This interpretability can also be rephrased as "explainability," which refers to the property or ability of features to explain the selection result.

[0046] The statistical calculation function 121 records its optimal model as the evaluation result of the risk prediction model in the evaluation result storage unit 134.

[0047] Next, in step S6, the UI display processing function 124 outputs the selection result of the risk prediction model in step S5 and displays it to the user through the UI (output processing). In step S6, the selection results are output, and information on the selected optimal model is presented along with the reasons for the selection and the supporting data. The user makes a final decision based on the information presented, including the optimal model. After the processing in step S6 is completed, the series of evaluation processes is terminated.

[0048] In this embodiment, a series of processes from steps S1 to S6 have been described, but it is also possible to omit the second filtering process in step S4. If step S4 is not performed, the stability of the model's features will be slightly reduced compared to when step S4 is performed, but the series of processes will be simplified by omitting step S4.

[0049] Furthermore, in this embodiment, filtering was performed based on evaluation values ​​calculated from the difference in the contribution of feature quantities, and filtering was also performed by comparing evaluation values ​​corresponding to the contribution values ​​of feature quantities with a threshold. Therefore, since this embodiment filters models by performing simple calculations such as calculating and comparing differences, it is possible to effectively and efficiently evaluate multiple models and select the optimal model.

[0050] [GUI screen] Next, a graphical user interface (GUI) screen provided to a customer according to one embodiment of the present invention will be described with reference to Figure 5. Figure 5 shows an example of a GUI screen provided to a customer according to one embodiment of the present invention.

[0051] As shown in Figure 5, the GUI screen 500 has an input display area 510 and an output display area 520. The input display area 510 displays a file selection button 511 and a cutoff value specification field 512.

[0052] The file selection button 511 is an interface element that allows the user to select a file containing the AUC (see Figure 2) and factor contributions (see Figure 3) of each model from the memory unit 130 and store it in the memory 120. When the user clicks the file selection button 511, the user is taken to a screen where they can select a file from a folder on their computer. Here, a message is displayed prompting the user to select a file: "Please input a CSV file containing the AUC and factor contributions of each model."

[0053] The cutoff value specification field 512 is an interface element for the user to input a cutoff value using an input device 102 such as a keyboard. Here, the message "Please specify a cutoff value. The default value is 3." is displayed to prompt the user to specify a cutoff value. For example, if the cutoff value is "3rd place," in step S3 (first filtering) shown in Figure 4, the difference between the contribution of the top 1st to 3rd ranked features and the contribution of the 4th ranked feature is calculated. If the cutoff value is "4th place," in step S3, the difference between the contribution of the top 1st to 4th ranked features and the contribution of the 5th ranked feature is calculated. In this way, the cutoff value can be dynamically set by the user arbitrarily entering a cutoff value in the cutoff value specification field 512.

[0054] Furthermore, in the input display area 510, the input format of the cutoff value specification field 512 may be changed, or additional cutoff value specification fields may be added, to accommodate the setting of various other cutoff functions.

[0055] The output display area 520 displays the optimal model name in the optimal model name display field 521, the optimal model details display field 522, and the details display field 523 for other models, as a result of this model evaluation.

[0056] The optimal model name display field 521 displays the selection results stored in the evaluation result storage unit 134 (Figure 1), such as the name of the optimal model. The detailed display area 522 for the optimal model shows the Weight value 202 for the optimal model, as shown in Figure 3. The optimal model name display field 521 and the optimal model details display field 522 present the selection result (e.g., the optimal model name) and the reasons and supporting information for that selection on the display device 101. By presenting the user with the reasons and supporting data for the selection, the user can gain insight into the contribution of the features of the selected optimal model, thereby supporting the user's decision-making. Furthermore, by confirming the reasons and supporting information for the selection, the user can gain a sense of satisfaction with the selected optimal model.

[0057] In the details display area 523 for other models, information that identifies the model (e.g., model name) is added to the list of AUC201 as shown in Figure 2. When the user clicks on the model name in the GUI screen displaying the list of AUC201 and other information, the details display area 523 for other models displays the Weight value 202 for that model.

[0058] <The invention shown in the above-described embodiment> The embodiments described above illustrate the inventions of the following means. In the descriptions of the means below, the corresponding component names, expressions, and reference numerals used in the drawings in the embodiments described above are noted in parentheses for reference. However, the components of each invention are not limited to these notes.

[0059] [Means (1)] Invention relating to means (1): It has an arithmetic unit (e.g., CPU 110) that executes the program and a storage device (e.g., memory 120, storage unit 130) that stores the above program, The above computing device selects a model from among multiple machine learning models whose AUC (e.g., AUC201 in Figure 2) value is equal to or greater than a certain threshold (e.g., step S2 in Figure 4), The above-mentioned computing device calculates a first evaluation value using the contributions of the top features of the selected model, including at least one with the highest contribution (e.g., Weight value 202 in Figure 3), and performs a first filtering to exclude the model with the smallest first evaluation value from the selected models (e.g., step S3 in Figure 4). The above-mentioned computing device selects the model with the highest AUC value from among the models remaining after the above filtering (for example, step S5 in Figure 4). A model evaluation system (e.g., a data processing system 100).

[0060] According to this model evaluation system configuration, by filtering based on a combination of AUC and the contribution of top features, including at least the first-ranked feature, it is possible to select a model from among multiple models that has high predictive accuracy while guaranteeing a certain level of stability and interpretability. For example, this model evaluation system can reliably select a model even if the content of the training dataset or the settings of the model parameters differ slightly, thereby improving the reliability of the selected model. Furthermore, factor extraction based on the selected model is also stable, enabling the provision of highly interpretable (explainable) prediction results. In addition, this model evaluation system can achieve both the prediction accuracy and the validity of the interpretation of the model by filtering using a combination of AUC and the contribution of the features.

[0061] [Means (2)] Invention relating to means (2): The model evaluation system described in means (1), The above-mentioned computing device, after the first filtering, calculates the difference in feature contribution between consecutive ranks (e.g., 1st and 2nd, 2nd and 3rd, ...) for each model remaining after the first filtering, and takes the smallest of these differences as the second evaluation value, and performs a second filtering to exclude at least the model with the smallest second evaluation value (for example, step S4 in Figure 4). The above-mentioned computing device selects the model with the highest AUC value from among the models remaining after the second filtering (for example, step S5 in Figure 4).

[0062] According to the model evaluation system with this configuration, after the first filtering by means (1) described above, a more detailed second filtering can be performed to narrow down the selection to a more appropriate model. In other words, the model evaluation system with this configuration calculates the difference in the contribution of features between consecutive ranks for each rank of the feature contribution, takes the smallest value among these differences as the second evaluation value, and excludes at least the model with the smallest second evaluation value, thereby narrowing down the selection to models with features that have clear and stable differences in contribution.

[0063] [Means (3)] Invention relating to means (3): The model evaluation system described in means (2), In the first filtering, the above-mentioned computing device takes the sum of the differences between the contributions of the top-ranking features (e.g., only the 1st ranked feature, or features ranked 1st to 3rd) and the contributions of the lower-ranking features (e.g., only the 10th ranked feature, or features ranked 4th or lower) as the first evaluation value (for example, a variation of step S4 in Figure 4).

[0064] This model evaluation system, like method (1), allows for the selection of a model from among multiple models that possesses high prediction accuracy while guaranteeing a certain level of stability and interpretability. Furthermore, this model evaluation system calculates the difference between the contributions of top features (including at least the 1st place feature) and lower-ranking features and uses this difference for filtering. As a result, the importance (contribution) of top features is clearly distinguished from that of lower-ranking features. Therefore, the interpretability of the model's prediction results is improved.

[0065] [Means (4)] Invention relating to means (4): The model evaluation system described in means (3), In the first filtering, the above-mentioned computing device takes the sum of the differences between the contributions of the top multiple features (e.g., 1st to 3rd place, 1st to 4th place) among the selected model features, including the 1st place feature, and the contributions of the features lower in rank (e.g., 4th place or lower, 5th place or lower), as the first evaluation value (for example, a variation of step S4 in Figure 4).

[0066] This model evaluation system, like method (1), allows for the selection of a model from among multiple models that possesses high prediction accuracy while guaranteeing a certain level of stability and interpretability. Furthermore, this model evaluation system calculates the sum of the differences between the contributions of the top multiple features (including at least the 1st place feature) and the contributions of the lower-ranking features, and uses this for filtering. Therefore, the importance (contribution) of the top multiple features is clearly distinguished from that of the lower-ranking features. Consequently, the interpretability of the model prediction results is further improved in this model evaluation system (4) compared to method (3).

[0067] [Means (5)] Invention relating to means (5): The model evaluation system described in means (3), In the first filtering, the above-mentioned computing device takes the difference between the contribution of the feature with the highest contribution among the selected model features and the contributions of the features with lower contributions (e.g., 4th and 10th place) as the first evaluation value (for example, a variation of step S4 in Figure 4).

[0068] This model evaluation system, like method (1), allows for the selection of a model from among multiple models that possesses high predictive accuracy while guaranteeing a certain level of stability and interpretability. This model evaluation system calculates the difference between the contribution of the highest-ranking feature and the contribution of the lowest-ranking feature and uses this for filtering, thus achieving the same effect as method (3) with simple calculations.

[0069] [Means (6)] Invention relating to means (6): The model evaluation system described in means (1), In the first filtering, the above-mentioned computing device uses the contribution of the feature with the highest contribution among the selected model features as the first evaluation value, and excludes models whose first evaluation value is less than a predetermined value (for example, a variation of step S4 in Figure 4).

[0070] This model evaluation system, like method (1), allows for the selection of a model from among multiple models that possesses high predictive accuracy while guaranteeing a certain level of stability and interpretability. Since this model evaluation system utilizes the comparison result between the contribution of the highest-ranking feature and a predetermined value for filtering, it can achieve the same effect as method (3) with simple calculations.

[0071] [Means (7)] Invention relating to means (7): The model evaluation system described in means (1), The above-mentioned computing device outputs to a display device (e.g., GUI screen 500) the selection result of the model with the highest AUC value from among the models remaining after the above filtering (e.g., optimal model name display field 521) and information showing the details of the selected model (e.g., optimal model details display field 522) (e.g., step S6 in Figure 4).

[0072] According to this model evaluation system configuration, by presenting information detailing the selected model along with the model selection result, users can confirm the reasons and rationale for selecting that model.

[0073] The present invention is not limited to the embodiments described above, and various other modifications and applications are possible as long as they do not depart from the gist of the invention as described in the claims. For example, the embodiments described above are detailed and specific in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the components described. Furthermore, it is possible to add, replace, or delete other components in the configuration of each embodiment.

[0074] Furthermore, each of the above configurations, functions, and processing units may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Broadly defined processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may be used as hardware. Alternatively, each of the above configurations, functions, and processing units may be implemented in software by having the computer's processor interpret and execute programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, optical discs, and magneto-optical discs.

[0075] Furthermore, the model evaluation system and model evaluation method of the present invention are not limited to risk prediction models for lifestyle-related diseases, but can be applied to a wide range of fields, including risk prediction models for other diseases and quality control models in the manufacturing industry. [Explanation of Symbols]

[0076] 100…Data processing system, 101…Display device, 102…Input device, 110…Central processing unit (CPU), 120…Memory, 121…Statistical calculation function, 122…Data extraction processing function, 123…Priority calculation function, 124…UI display processing function, 130…Storage unit, 131…Explanatory variable data storage unit, 132…Target variable data storage unit, 133…System setting file storage unit, 134…Evaluation result storage unit, 141…Connection interface, 142…Network interface, 201…AUC, 202…Weight value, 500…GUI screen, 510…Input display area, 511…File selection button, 512…Cutoff value specification field, 520…Output display area, 521…Optimal model name display field, 522…Details display field, 523…Details display field

Claims

1. It comprises an arithmetic unit for executing a program and a storage device for storing the program, The aforementioned computing device selects a model from among multiple machine learning models in which the AUC value is equal to or greater than a certain threshold value. The computing device calculates a first evaluation value using the contributions of the top features, including at least one with the highest contribution among the selected model's features, and performs a first filtering to exclude the model with the smallest first evaluation value from the selected models. The arithmetic unit selects the model with the highest AUC value from among the models remaining after filtering. Model evaluation system.

2. The computing device, for the models remaining after the first filtering, calculates the difference in the contribution of features between consecutive ranks for each rank of the feature contribution, takes the smallest value among these differences as the second evaluation value, and performs a second filtering to exclude at least the model with the smallest second evaluation value. The calculation device selects the model with the highest AUC value from among the models remaining after the second filtering. The model evaluation system according to claim 1.

3. In the first filtering, the computing device takes the sum of the differences between the contributions of the top features, including at least one with a contribution of 1, and the contributions of the lower-ranking features as the first evaluation value. The model evaluation system according to claim 2.

4. In the first filtering, the computing device takes the sum of the differences between the contributions of the top multiple features, including the one with the highest contribution, and the contributions of the features with a lower contribution as the first evaluation value. The model evaluation system according to claim 3.

5. In the first filtering, the computing device takes the difference between the contribution of the feature with the highest contribution among the selected model features and the contribution of the features with a lower contribution as the first evaluation value. The model evaluation system according to claim 3.

6. In the first filtering, the computing device sets the contribution of the feature with the highest contribution among the selected model features as the first evaluation value, and excludes models whose first evaluation value is less than a predetermined value. The model evaluation system according to claim 1.

7. The calculation unit outputs to the display device the selection result of the model with the highest AUC value from among the models remaining after filtering, and information showing the details of the selected model. The model evaluation system according to claim 1.

8. A model evaluation method for a computer having an arithmetic unit for executing a program and a storage device for storing the program, wherein the computer evaluates and selects a model, The aforementioned computing device performs the process of selecting a model from among multiple machine learning models whose AUC value is equal to or greater than a certain threshold value, The calculation device performs a first filtering process in which it calculates a first evaluation value using the contributions of the top features, including at least one with the highest contribution among the selected model's features, and excludes the model with the smallest first evaluation value from the selected models. The calculation unit includes a process of selecting the model with the highest AUC value from among the models remaining after filtering. Model evaluation method.

9. The procedure for selecting a machine learning model from multiple models whose AUC value is above a certain threshold, A procedure for calculating a first evaluation value using the contributions of the top features, including at least one with the highest contribution among the selected model features, and performing a first filtering to exclude the model with the smallest first evaluation value from the selected models, A procedure for selecting the model with the highest AUC value from among the models remaining after the filtering, A program designed to be executed by a computer.

Images