Model creation method, program, and information processing device

Abstract

The objective of the present invention is to improve the prediction accuracy of a model. The objective of the present invention is to improve the prediction accuracy of a model. The present disclosure relates to a model creation method in which an information processing device creates a model that associates input data included in training data with output data included in the training data, the model creation method comprising: a first exclusion processing for excluding, from a training dataset, training data of which the input data is an outlier; a second exclusion processing for excluding, from the training dataset, training data of which the input data and the output data are outliers; a first model creation processing for creating a first model by using the training dataset from which the training data has been excluded in the first exclusion processing; a second model creation processing for creating a second model by using the training dataset from which the training data has been excluded in the second exclusion processing; and a model selection processing for selecting the model having the better prediction accuracy from the first model and the second model.

Description

Model creation method, program, and information processing device

[0001] The present invention relates to a model creation method, a program, and an information processing device.

[0002] Materials informatics is a well-known approach to improve the efficiency of materials development by using information science such as statistical analysis, machine learning, and data mining. One example of materials informatics is compounding research, which seeks to find optimal compounding of materials.

[0003] In blending search, a process of separating and excluding outliers from training data is performed to create a highly accurate model (see, for example, Patent Documents 1 to 3). Patent Document 1 discloses a technology for acquiring a data set configured in a predetermined format including a plurality of pairs of test conditions and substance information of a target substance generated based on the test conditions, performing statistical processing on the data set, and, based on the results, estimating recommended test conditions for optimizing substance information and the corresponding substance information of the target substance.

[0004] Patent Document 2 discloses a technology in which a data evaluation model constructed by machine learning on collected data evaluates input collected data and automatically selects training data for constructing a learning model from the evaluated data based on the evaluation results.

[0005] Patent Document 3 discloses a technology for selecting data from data stored in a data storage unit to update an existing learning model used for machine learning, generating a new learning model by machine learning based on the selected data, and updating the existing learning model to the new learning model.

[0006] Japanese Patent Publication No. 2021-193623 Japanese Patent Publication No. 2021-107970 Japanese Patent Publication No. 2021-157654

[0007] However, conventional techniques have the problem that they do not sufficiently remove outliers, hindering the improvement of the model's predictive accuracy. For example, when there are many explanatory variables and the dataset is sparse, the data in the search space is widely dispersed, making it difficult to properly identify outliers using conventional methods. Furthermore, since such datasets are not normally distributed, it is difficult to identify outliers using standard deviations, etc.

[0008] The present disclosure provides techniques for improving the predictive accuracy of a model.

[0009] The present disclosure relates to a model creation method in which an information processing device creates a model that associates input data included in training data with output data, the method comprising: a first exclusion process of excluding training data whose input data is an outlier from a training dataset; a second exclusion process of excluding training data whose input data and output data are outliers from the training dataset; a first model creation process of creating a first model using the training dataset from which the training data has been excluded in the first exclusion process; a second model creation process of creating a second model using the training dataset from which the training data has been excluded in the second exclusion process; and a model selection process of adopting the model with better prediction accuracy from the first model and the second model.

[0010] In a second disclosure, in the first disclosure, the first exclusion process excludes outliers from the input data using one of a plurality of different methods.

[0011] In a third disclosure, in the second disclosure, the first exclusion process excludes outliers from the input data using at least two different first or second methods.

[0012] In a fourth disclosure, in any of the first to third disclosures, the second exclusion process excludes outliers from the input data using one of a plurality of different methods, and if the output data is univariate, excludes outliers from the output data using a predetermined method.

[0013] In a fifth disclosure, in the fourth disclosure, the second elimination process removes outliers from the input data using at least two different third or fourth methods, and, if the output data is univariate, removes outliers from the output data using a fifth method; and the second elimination process removes outliers from the input data using the third method and removes outliers from the output data using the fifth method, or removes outliers from the input data using the fourth method and removes outliers from the output data using the fifth method.

[0014] In a sixth disclosure, in any one of the first to third disclosures, the second exclusion process excludes outliers from the input data using one of a plurality of different methods, and if the output data is multivariate, excludes outliers from the output data using one of a plurality of different methods.

[0015] A seventh disclosure is related to the sixth disclosure, wherein the second elimination process removes outliers from the input data using at least two different third methods or fourth methods, and, if the output data is multivariate, removes outliers from the output data using at least two different sixth methods or seventh methods, and the second elimination process removes outliers from the input data using the third method and removes outliers from the output data using the sixth method, or removes outliers from the input data using the fourth method and removes outliers from the output data using the sixth method, or removes outliers from the input data using the third method and removes outliers from the output data using the seventh method, or removes outliers from the input data using the fourth method and removes outliers from the output data using the seventh method.

[0016] In the eighth disclosure, in the third disclosure, the first method is a method for detecting outliers using quantiles based on the distance of each data item from a reference data item, and the second method is a method for detecting outliers using a local outlier factor method.

[0017] The ninth disclosure is related to the fifth disclosure, wherein the third method is a method for detecting outliers using quantiles based on the distance of each data item from a reference data item, the fourth method is a method for detecting outliers using a local outlier factor method, and the fifth method is a method for detecting outliers using quantiles.

[0018] In the tenth disclosure, in the seventh disclosure, the third method and the sixth method are methods for detecting outliers using quantiles based on the distance of each data item from a reference data item, and the fourth method and the seventh method are methods for detecting outliers using a local outlier factor method.

[0019] In an eleventh disclosure, in the eighth disclosure, in the first elimination process, the quantile, which is a hyperparameter of the first method, is determined by grid search, or the number of neighboring sites, which is a hyperparameter of the second method, is determined by grid search.

[0020] In a twelfth disclosure, in the ninth disclosure, in the second exclusion process, at least one of the quantile, which is a hyperparameter of the third method, or the quantile, which is a hyperparameter of the fifth method, is determined by grid search, or at least one of the number of nearby sites, which is a hyperparameter of the fourth method, or the quantile, which is a hyperparameter of the fifth method, is determined by grid search.

[0021] The thirteenth disclosure is the tenth disclosure, wherein in the second exclusion process, at least one of the quantile, which is a hyperparameter of the third method, or the quantile, which is a hyperparameter of the sixth method, is determined by grid search; or at least one of the quantile, which is a hyperparameter of the third method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search; or at least one of the number of neighboring sites, which is a hyperparameter of the fourth method, or the quantile, which is a hyperparameter of the sixth method, is determined by grid search; or at least one of the number of neighboring sites, which is a hyperparameter of the fourth method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search.

[0022] The fourteenth disclosure provides a program that causes an information processing device that creates a model that associates input data included in training data with output data to execute: a first exclusion process that excludes training data whose input data is an outlier from the training data set; a second exclusion process that excludes training data whose input data and output data are outliers from the training data set; a first model creation process that creates a first model using the training data set from which the training data has been excluded in the first exclusion process; a second model creation process that creates a second model using the training data set from which the training data has been excluded in the second exclusion process; and a model selection process that adopts the model with better prediction accuracy from the first model and the second model.

[0023] The fifteenth disclosure is an information processing device that creates a model that associates input data included in training data with output data, the information processing device having: an exclusion unit that excludes training data whose input data is an outlier from a training data set, and excludes training data whose input data and output data are outliers from the training data set; a model creation unit that creates a first model using the training data set from which training data whose input data is an outlier has been excluded, and creates a second model using the training data set from which training data whose input data and output data are outliers has been excluded; and a model selection unit that adopts the model with better prediction accuracy from the first model and the second model.

[0024] The present disclosure can improve the predictive accuracy of the model.

[0025] 8A , 8B, 8C, 8D, 8E, 8F, 8G, 8G, 8H, 8H, 8I, 8J, 8K ... FIG. 13 is an example of a flowchart illustrating the process of performing a grid search for an optimal quantile in step S215 of FIG. 12 . FIG. 14 is an example of a flowchart illustrating the process of performing a grid search for an optimal number of neighboring sites in step S218 of FIG. 12 . FIG. 15 is an example of a flowchart illustrating the process of creating a model by excluding training data in which both input data and output data are outliers in step S4 of FIG. 9 . FIG. 16 is an example of a flowchart illustrating the process of performing a grid search for an optimal quantile in step S416 of FIG. 15 . FIG. 17 is an example of a flowchart illustrating the process of performing a grid search for an optimal number of neighboring sites in step S419 of FIG. 15 . FIG. 18 is an example of a flowchart illustrating the process of creating a model by excluding training data in which both input data and output data are outliers in step S421 of FIG. 15 .FIG. 19 is an example of a flowchart illustrating the process of grid searching the optimal quantile for the input data and the output data in step S4215A of FIG. 18. FIG. 19 is an example of a flowchart illustrating the process of grid searching the optimal quantile for the input data and the optimal number of neighboring sites for the output data in step S4215B of FIG. 18. FIG. 19 is an example of a flowchart illustrating the process of grid searching the optimal number of neighboring sites for the input data and the optimal quantile for the output data in step S4215C of FIG. 18. FIG. 19 is an example of a flowchart illustrating the process of grid searching the optimal number of neighboring sites for the input data and the output data in step S4215D of FIG. 18. FIG. 20 is an example of a diagram illustrating a technique for excluding outliers in input data in the first exclusion process. FIG. 21 is an example of a diagram illustrating a combination of techniques for excluding outliers in input data and output data in the second exclusion process.

[0026] An information processing device and a model creation method performed by the information processing device will be described below as an example of an embodiment of the present invention.

[0027] <Supplementary information on the comparison technology> In blending search technology, which searches for optimal blends of materials, a model is created using a training dataset to search for blends of materials with desirable properties. Material developers sometimes manufacture prototypes with blends of materials that are significantly different from conventional ones, and these prototypes are also measured to confirm whether they have desirable properties. However, it is not uncommon for no improvement in properties to be observed, and further search around that blend may not be carried out. In such cases, the blend of the prototype may be an outlier.

[0028] However, unlike measurement errors, data on prototype characteristics cannot be used to determine whether or not to exclude it at the time the data is acquired; rather, the decision is often made after the development process has progressed, which can lead to oversights or forgetting to exclude something.

[0029] 1A and 1B are diagrams illustrating a comparison between an outlier 101 due to a measurement error and an outlier 102 measured on a prototype. The outlier 101 due to a measurement error shown in Fig. 1A can be flagged as a "measurement error" by the developer when measuring the characteristics, or can be removed from the accumulated data on the spot.

[0030] In contrast, the outlier 102 measured for the prototype shown in FIG. 1B corresponds to the coordinates of a data point for the formulation (ratio of materials 1 and 2). For example, a formulation significantly different from the conventional formulation may be tested, but no improvement in properties may be observed. If no improvement in properties is observed, further searches around that formulation are not performed. Furthermore, such prototype measurement data may reduce the accuracy of the model or may be disadvantageous for searching for a formulation with good properties, so it is desirable to exclude it. As such, the outlier 102 is not due to a measurement error, nor is it something that is immediately removed from the accumulated data, making it difficult to remove during measurement. For this reason, the outlier 102 may be overlooked or forgotten to be removed later in the development process.

[0031] Furthermore, when detecting outliers when the type and amount of ingredients used are used as explanatory variables, there will be a large number of explanatory variables. However, experiments are rarely conducted by uniformly changing the type and amount of ingredients, and the training dataset tends to be sparse. This makes it difficult to apply common outlier detection methods such as outlier detection using standard deviation or quartiles. Furthermore, an efficient method is needed that can detect outliers even when the training dataset contains a large amount of data.

[0032] Therefore, when creating a model for blending search using machine learning or the like, the information processing device of the present disclosure excludes outliers from the learning dataset as follows.

[0033] FIG. 2 is a diagram illustrating the flow of processing in which an information processing device creates a model by removing outliers from a training data set.

[0034] S100: The information processing device measures the distance from the reference data for the multivariate explanatory variables (input data) and excludes training data including input data that is an outlier in the distance distribution. Quartiles or the like may be used as a method for detecting outliers. The information processing device may also detect outliers using other methods, such as the local outlier factor method.

[0035] At step S200, the information processing device excludes learning data in which the input data is an outlier in the distribution of distances and the dependent variable (output data) is also an outlier.

[0036] S300 The information processing device creates a model using the training data set from which the training data was excluded in step S100, creates a model using the training data set from which the training data was excluded in step S200, and adopts the model with the better accuracy.

[0037] The processes of steps S100 and S200 may be performed in either order, or may be performed in parallel.

[0038] As described above, the information processing device of this embodiment easily removes outliers even when the training dataset (explanatory variables) is sparse, making it difficult to apply general outlier detection techniques. Therefore, it is easy to remove data measured on prototypes, etc. Since training data in which both the dependent variable and the explanatory variables are outliers can be removed, it is possible to detect cases where measurements were stopped due to deterioration of characteristics after experimental blending (prototypes result in outliers in the explanatory variables, and poor characteristics result in the dependent variable being an outlier). Since the information processing device creates a model and verifies its accuracy by removing outliers, it can remove outliers in the explanatory variables if they are detrimental to the model's predictive accuracy. A model is created by removing outliers in only the explanatory variables, and another model is created by removing outliers in both the dependent variables and the dependent variables, and the model with the better accuracy is adopted. Therefore, a model created by removing outliers can be adopted from the perspective of accuracy. Furthermore, since the number of models created is not as large as when the leave-one-out method is comprehensively applied to a training dataset with a large amount of data, outliers can be detected even when the training dataset contains a large amount of data.

[0039] <Terminology> A learning dataset is a collection of data required for machine learning, and is a set of learning data consisting of a combination of input data and output data. A learning dataset is also called a training dataset.

[0040] An outlier is a value that is significantly different from other values ​​among the obtained observed values. In this embodiment, the outlier may reduce the accuracy of the model or may include data that is unfavorable for searching for a blend of good properties.

[0041] In machine learning, a model is a mechanism (program, file, etc.) that derives output data from input data. A model associates input data with output data. A model is sometimes called AI or artificial intelligence.

[0042] <System Configuration Example> Fig. 3A is an example of a system configuration diagram of a data processing system 100. The data processing system 100 in Fig. 3A is a client-server system, and includes a server device 10 and an information processing device 30. However, the information processing device 30 may be a general-purpose computer, and may not be included in the data processing system 100.

[0043] The server device 10 and the information processing device 30 are communicatively connected via a wide-area network N1 such as the Internet. The server device 10 may be installed in a cloud or a data center, or may be installed on-premise. The server device 10 may be a web server that returns processing results in response to requests from the information processing device 30. A server is a computer or software that provides information and processing results in response to requests from clients.

[0044] The server device 10 is one or more information processing devices. The server device 10 creates a model for performing a blending search from training data specified by the developer 9 via the information processing device 30. One of the features of the server device 10 is that it creates a model after appropriately excluding outliers from the training data.

[0045] The server device 10 may have a data storage device in which the training data is stored in advance. Alternatively, the server device 10 may acquire the training data from a data server or a network-attached storage (NAS). Alternatively, the developer 9 may operate the information processing device 30 to transmit the training data to the server device 10.

[0046] The server device 10 may be compatible with cloud computing. Cloud computing refers to a form of utilization in which resources on a network are utilized without regard to specific hardware resources. Therefore, the server device 10 does not need to be housed in a single housing or provided as a single integrated device. The functions of the server device 10 may be distributed among multiple information processing devices, or multiple information processing devices may each have all of the functions, and the information processing device that processes the data may be switched by load balancing or the like.

[0047] The information processing device 30 is located in a facility such as a company, a research institute, a development department, or a factory, and is connected to a network N2. The network N2 may be a LAN, Wi-Fi (registered trademark), wide area Ethernet (registered trademark), or a mobile phone network such as 4G, 5G, or 6G.

[0048] The information processing device 30 is a machine learning or general-purpose computer used by a developer 9. Here, the developer 9 is a person who creates a model from training data using the server device 10. The person who uses the information processing device 30 may include not only the person who creates the model, but also a person who contracts to create the model. The person who uses the information processing device 30 may also be a person who performs a breeding search.

[0049] A web browser and a native application dedicated to the server device 10 run on the information processing device 30. When the information processing device 30 runs a web browser, the information processing device 30 and the server device 10 run a web application. A web application is an application that runs through cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server (server device 10) side. When a web application is executed, a model may be created by the server device 10, or by the information processing device 30 that has received the web application.

[0050] An application that cannot be executed unless it is installed on the information processing device 30 is called a native application. In this embodiment, the application executed on the information processing device 30 may be a web application or a native application. In this case, too, the process of creating a model by removing outliers from the training data may be performed by the server device 10, or may be performed by the information processing device 30 using a native application.

[0051] Furthermore, in this embodiment, instead of the client-server system shown in FIG. 3A , the information processing device 30 may create a model by itself, excluding outliers, as shown in FIG. 3B . In this case, a native application that creates a model by excluding the model from training data runs on the information processing device 30. However, even in this case, the information processing device 30 may acquire training data from a data server on a network. Therefore, even in the configuration shown in FIG. 3B , it is preferable that the information processing device 30 be connectable to a network.

[0052] The information processing device 30 is, for example, a desktop PC, a notebook PC, a smartphone, a personal digital assistant (PDA), a tablet terminal, or the like used by the developer 9. Alternatively, the information processing device 30 may be any device on which a web browser or a native application runs.

[0053] Furthermore, the developer 9 may appropriately use either the form in Fig. 3A or the form in Fig. 3B. The information processing device 30 removes outliers, and the server device 10 creates a model. The server device 10 removes outliers, and the information processing device 30 creates a model. Note that, in this embodiment, unless otherwise specified, the description will be based on the form in Fig. 3B (in which the information processing device 30 performs processing independently).

[0054] <Hardware Configuration Example> With reference to FIG. 4, the hardware configuration of the server device 10 and the information processing device 30 included in the data processing system 100 according to this embodiment will be described.

[0055] <<Information Processing System and Information Processing Device 30>> Fig. 4 is a diagram showing an example of the hardware configuration of the server device 10 and the information processing device 30 according to this embodiment. As shown in Fig. 4, the server device 10 and the information processing device 30 are constructed by a computer 500, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0056] Of these components, the CPU 501 controls the overall operation of the server device 10 and the information processing device 30. The ROM 502 stores programs, such as IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HDD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HDD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 4 .

[0057] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, DVD, Blu-Ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0058] <Regarding Functions> Next, the functional configuration of the information processing device 30 according to this embodiment will be explained by dividing it into blocks using Fig. 5. Fig. 5 is a diagram showing an example of the functional configuration of the information processing device 30 according to this embodiment. Note that the lines between the functional blocks in Fig. 5 indicate the relationship between them, but this does not mean that the processing is limited to proceeding in the order connected by these lines.

[0059] The information processing device 30 includes a display control unit 31, an operation reception unit 32, a measurement error data detection unit 33, a distance calculation unit 34, a grid search unit 35, a quantile outlier detection unit 36, a local outlier detection unit 37, an exclusion unit 38, a model creation unit 39, an accuracy calculation unit 40, a verification unit 41, and a model selection unit 42. Each of these units is a function or a means that is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded in the RAM 503.

[0060] The information processing device 30 also has a training data storage unit 49. The training data storage unit 49 is constructed in the HD 504, RAM 503, etc. shown in FIG. 4. A training data set is stored in the training data storage unit 49. The training data set is assumed to be divided into input data and output data. In the case of a multiple regression model, the input data is referred to as an explanatory variable and the output data as a target variable. In this embodiment, the terms explanatory variable and target variable are sometimes used, but the machine learning technique is not limited to a multiple regression model. The training data storage unit 49 may be located on a network accessible by the information processing device 30.

[0061] The display control unit 31 interprets screen information held by the native application and displays it on the display 506. The operation receiving unit 32 receives various operations by the developer 9 on various screens displayed on the display 506.

[0062] The measurement error data detection unit 33 acquires the learning data set stored in the learning data storage unit 49 and detects the measurement error data included in the learning data set. It is assumed that the measurement error data is assigned a predetermined flag.

[0063] The distance calculation unit 34 calculates the distance of the training data (which may be either input data or output data) relative to reference data set as standard data, for example, to remove outliers using quartiles.

[0064] The grid search unit 35 prepares all combinations of the specified hyperparameters. In this embodiment, the hyperparameters are the quantile number Q or the number of neighboring sites N. Note that the grid search is not essential.

[0065] The quantile outlier detector 36 detects outliers (which may be input data or output data) from the training data based on, for example, quartiles. When a grid search is performed, the quantile outlier detector 36 detects outliers from the training data based on the quantile Q determined by the grid search unit 35.

[0066] The local outlier detection unit 37 detects outliers (which may be input data or output data) from the training dataset, for example, with the number of neighboring sites N set to 3. Furthermore, when a grid search is performed, the local outlier detection unit 37 detects outliers from the training dataset based on the number of neighboring sites N determined by the grid search unit 35. A local outlier is an outlier detected based on the density of data in space. The local density is the reciprocal of the average distance between the point of interest and k neighboring points. Given an outlier point A, its local density is low because it is far from the k neighboring points. The k non-outlier points have high local densities because they are close to their respective neighboring points. Therefore, in the case of point A, which is an outlier, its local density is low but the local densities of its k neighboring points are high, while non-outlier points have similar values ​​for their local density and the local densities of their neighboring points. Therefore, a point is determined to be an outlier when the ratio of its local density to the local densities of its neighboring points differs by more than a threshold value.

[0067] The exclusion unit 38 excludes from the training data set any training data that includes input data that is an outlier detected by the quantile outlier detection unit 36 ​​or the local outlier detection unit 37. The exclusion unit 38 also excludes from the training data set any training data that includes input data and output data that are outliers detected by the quantile outlier detection unit 36 ​​or the local outlier detection unit 37. Note that exclusion does not necessarily involve erasure of the training data itself; it is sufficient that the training data is not used to create a model.

[0068] The model creation unit 39 creates a model for blending search using a training data set from which outliers have been removed. The model may be created using various machine learning techniques. Machine learning is a technology that allows a computer to acquire human-like learning capabilities. The computer autonomously generates algorithms necessary for judgments such as data identification from training data previously acquired, and applies these algorithms to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, and reinforcement learning (supervised learning is mainly used in this embodiment), or a combination of these learning methods. Machine learning techniques include perceptrons, deep learning, support vector machines, logistic regression, naive Bayes, decision trees, random forests, and the like, and are not limited to the techniques described in this embodiment.

[0069] For example, deep learning is an algorithm that predicts output data based on input data and then adjusts the weights of a neural network using error backpropagation to reduce errors with training data. In the case of blending search, the developer prepares a training data set containing multiple training data sets, with the input data being the blending amounts of each material and the output data being the material properties (strength properties, physical properties, electrical properties, corrosion resistance, etc.), and the model creation unit 39 learns the correspondence between the input data and the output data.

[0070] The accuracy calculation unit 40 calculates the accuracy of the model created by the model creation unit 39. The learning dataset is divided into L parts in advance, and a model is created using training data excluding one of the parts (referred to as test data). The accuracy calculation unit 40 inputs the excluded test data into the created model and compares the output with the output data in the test data to calculate the accuracy of the created model.

[0071] When the grid search unit 35 removes outliers with different quantiles or neighbor counts, a model is created for each training data set from which different outliers have been removed. The accuracy calculation unit 40 calculates the accuracy for each model. Furthermore, the accuracy calculation unit 40 also calculates the accuracy for each model when cross-validation is performed.

[0072] The verification unit 41 creates L models from the training data set using a method such as cross-validation, and averages the accuracy of the output of each model. By averaging, differences in model accuracy due to differences in training data are reduced.

[0073] The model selection unit 42 selects the model with the higher accuracy from among a model created by excluding learning data in which only the input data is an outlier, and a model created by excluding learning data in which both the input data and the output data are outliers.

[0074] <Detection of Outliers Using Quartiles> With reference to Fig. 6, outlier detection using quartiles, which is a comparative technique to this embodiment and is used in some cases, will be explained in more detail. Fig. 6 is a diagram illustrating a method for detecting outliers 110 using quartiles. Quartiles are a detection method in which, when data is sorted in ascending order, quartiles (Q1, Q2, Q3) that divide the data equally into four are found, and data that is smaller than 1.5 times the IQR (Q3 - Q1) for Q1 or larger than 1.5 times the IQR (Q3 - Q1) for Q3 is considered to be an outlier 110.

[0075] Such a method can be said to be a method for detecting outliers 110 based on the distribution of data. However, even if training data is removed using this method, it is not necessarily possible to remove outliers that reduce the prediction accuracy of the model.

[0076] <Detection of Outliers by the Leave One Out Method> With reference to FIG. 7 , outlier detection by the Leave One Out method, which is a comparative technique to this embodiment, will be described in more detail. FIG. 7 is a diagram illustrating a method for detecting outliers using the Leave One Out method. The Leave One Out method is a method in which one piece of training data is excluded as unknown data 112, and a model is constructed using the remaining training data. In FIG. 7 , there are eight training data sets. Therefore, eight models are created, and the accuracy of each model when the unknown data 112 is input is compared. The unknown data 112 with poor accuracy is excluded as an outlier. The difference from cross-validation is that the Leave One Out method divides the data into a much larger number of parts.

[0077] This method can be said to utilize the predictive accuracy of machine learning models. However, when the leave-one-out method is comprehensively applied to a training dataset with a large amount of data, the number of models created increases, which can lead to the inconvenience of high computational costs, and outliers in the composition (explanatory variables) are not necessarily eliminated.

[0078] <Outline of Outlier Removal Method of This Embodiment> An outline of the outlier removal method of this embodiment will be described with reference to FIGS. 8A and 8B. FIGS. 8A and 8B are diagrams illustrating outliers of explanatory variables, outliers of dependent variables, and a method for removing outliers. First, FIG. 8A is a diagram illustrating a method for detecting outliers of explanatory variables and dependent variables. The quantile outlier detection unit 36 ​​and the local outlier detection unit 37 detect outliers of explanatory variables from the distribution of input data, which is the explanatory variable. In FIG. 8A, two outliers 121 and 122 are detected from the explanatory variable (mixture).

[0079] As mentioned above, it can be difficult to detect the "prototype (challenging formulation, formulation for comparison)" as an outlier in a formulation system (explanatory variables). This is because there are many explanatory variables and the training dataset is sparse, so the data in the search space is widely spread and not normally distributed. In such cases, the following process is effective. 1. Set a reference formulation and calculate the distance from that formulation (the definition of distance is arbitrary). 2. The larger outlier in the distance distribution is considered to be the outlier of the explanatory variables.

[0080] The quantile used to determine an outlier and the number N of neighboring sites in a local outlier can also be automatically determined as hyperparameters by grid search.

[0081] Next, the quantile outlier detector 36 and the local outlier detector 37 detect outliers of the objective variable from the distribution of the output data, which is the objective variable. This is to pick up cases where further search has been stopped due to deterioration of characteristics. In this embodiment, even if only the objective variable is an outlier, it is not excluded. Therefore, if the explanatory variables are also outliers, it is possible to pick up cases where further search has been stopped due to deterioration of characteristics of the explanatory variables.

[0082] Two outliers 123 and 124 were detected from the dependent variable. Removal of outliers based on the dependent variable is expected to improve the accuracy of the model.

[0083] 8B is a diagram illustrating a method for removing outliers detected in FIG. 8A. The removal unit 38 removes training data including outliers 121 and 122 in the explanatory variables, and the model creation unit 39 creates a model. Similarly, the removal unit 38 removes training data including outliers 121 and 123 that are outliers in both the explanatory variables and the objective variable, and the model creation unit 39 creates a model. The accuracy calculation unit 40 calculates the accuracy of these models, and the model selection unit 42 adopts the model with the better accuracy.

[0084] 8B, there is a possibility that the search region of the model will narrow (the accuracy will increase but the prediction of the extrapolation region will be weak). Therefore, it is preferable to exclude only outliers whose characteristics (objective variables) are farther away from the target value on the side of inferior characteristics.

[0085] <Operation Procedure> Hereinafter, the flow of processing for creating a model for blending search while excluding outliers will be described with reference to FIGS.

[0086] <Processing without using grid search> First, a flowchart illustrating the overall flow of model creation is shown in Fig. 9. Hereinafter, explanatory variables will be referred to as "input data" and objective variables will be referred to as "output data".

[0087] First, the measurement error data detection unit 33 excludes learning data due to measurement errors to which a flag or the like is attached from the learning data set (S1).

[0088] The information processing device 30 performs a first exclusion process to exclude outliers from the input data. The information processing device 30 also performs a first model creation process to create a model (an example of a first model) from a training data set from which training data including input data that are outliers have been excluded (S2). Details will be described with reference to FIG. 10.

[0089] Next, the verification unit 41 performs the process of step S2 multiple times to perform cross-validation (S3). That is, the verification unit 41 creates multiple models (examples of multiple first models) from the training data set after excluding training data including outlier input data, by changing the test data not used in model creation. The verification unit 41 calculates the accuracy of each model calculated by the accuracy calculation unit 40.

[0090] Next, the information processing device 30 performs a second exclusion process to exclude training data whose input data is an outlier and whose output data is an outlier. The information processing device 30 then performs a second model creation process to create a model (an example of a second model) from the training data set from which training data whose input data is an outlier and whose output data is an outlier has been excluded (S4). Details will be described with reference to FIG. 11.

[0091] The verification unit 41 performs the process of step S4 multiple times to perform cross-validation (S5). That is, the verification unit 41 creates multiple models (examples of multiple second models) by changing the test data not used for model creation from the training data set after excluding training data whose input data is an outlier and whose output data is an outlier, and calculates the accuracy of each model.

[0092] The model selection unit 42 performs a model selection process (S6) to select the model with the better prediction accuracy from the models in step S2 and step S4. Prediction accuracy refers to the accuracy of the model for data other than the training data. Known indicators of the accuracy of a regression model include MAE (Mean Absolute Error), MAPE (Mean Absolute Percentage Error), WAPE (Weighted Absolute Percentage Error), MSE (Mean Squared Error), and RMSE (Square Root of Mean Squared Error).

[0093] FIG. 10 is a flowchart illustrating the process of creating a model by excluding outliers from input data in step S2 of FIG.

[0094] First, input data is converted by normalization into values ​​in a fixed range such as 0 to 1, or by standardization into values ​​with a mean of "0" and a variance of "1" (S21).

[0095] Next, the distance calculation unit 34 acquires reference input data (S22). The reference input data can be set in advance by the developer. Alternatively, input data whose output data (target value) indicates the median value may be automatically selected. Since the output data indicates the median value, the input data can be expected to be distributed near the center of the other multiple input data.

[0096] Next, the distance calculation unit 34 calculates the distance from the reference input data for each input data (S23). The distance may be any distance that allows detection of outliers, such as Euclidean distance or Mahalanobis distance.

[0097] The quantile outlier detector 36 detects outliers, for example, by quartiles, based on the distances in step S23 (an example of a first method), and the excluding unit 38 excludes training data including input data that is an outlier from the training data set (S24). Quintiles, etc., may be used instead of quartiles.

[0098] Instead of the processes in steps S23 and S24 described above, the local outlier detector 37 may detect outliers using a local outlier factor method (an example of a second technique).

[0099] Thereafter, the model creation unit 39 focuses on the input data and creates a model using the training data set from which outliers have been removed (S25).

[0100] FIG. 11 is a flowchart illustrating the process of creating a model by excluding outliers from the input data and output data in step S4 of FIG.

[0101] First, it is determined whether the output data is univariate (S41). Univariate means that the output data has one element, and multivariate means that the output data has multiple elements.

[0102] In the case of a univariate (Yes in S41), the quantile outlier detector 36 detects outliers in the output data using, for example, quartiles (an example of a fifth method) (S42). Quintiles, etc., may be used instead of quartiles.

[0103] In the case of multivariate data (No in S41), the local outlier detection unit 37 detects outliers in the output data using the local outlier factor method (an example of a seventh method) with, for example, three neighbors (S43). The number of neighbors may be four, instead of three. In the case of multivariate data (No in S41), the quantile outlier detection unit 36 ​​may calculate the distance from the reference data for each data item, and detect outliers using, for example, quartiles based on this distance (an example of a sixth method).

[0104] The exclusion unit 38 excludes from the training data set any training data whose output data and input data are outliers (S44).Then, the model creation unit 39 focuses on the input data and the output data and creates a model using the training data set from which the outliers have been excluded.

[0105] <Processing Using Grid Search on Input Data> Next, a process for determining optimal quantiles for input data using grid search and removing outliers will be described with reference to Figures 12 to 14. The overall flow of model creation is the same as that shown in Figure 9.

[0106] FIG. 12 is a flowchart illustrating the process of creating a model by excluding input data that is an outlier in step S2 of FIG.

[0107] First, input data is converted by normalization into values ​​in a predetermined range such as 0 to 1, or by standardization into values ​​with a mean of "0" and a variance of "1" (S211).

[0108] Next, the grid search unit 35 selects one of the outlier detection methods (distance measurement from the reference or local outlier factor method) (S212). The selection method may be preset by the developer, or both may be performed and the one with the better model accuracy may be selected.

[0109] When distance measurement from a reference (an example of a first method) is selected, the distance calculation unit 34 acquires reference input data (S213). The reference input data can be set in advance by the developer. Alternatively, input data whose output data (target value) indicates the median may be automatically selected.

[0110] Next, the grid search unit 35 determines quantile candidates q[1], q[2], q[3]...q[f] (S214). The quantile candidates q[1], q[2], q[3]...q[f] are natural numbers such as 2, 3, 4, 5, etc.

[0111] The grid search unit 35 performs a grid search for the optimal quantile using the quantile candidates (S215), as will be described in detail with reference to FIG.

[0112] The model creation unit 39 determines the most accurate model from among the models created using different quantiles (S216). The index of accuracy may be the same as in step S6.

[0113] When the local outlier factor method (an example of the second method) is selected, the grid search unit 35 determines candidates n[1], n[2], n[3]...n[f] for the number of neighboring sites (S217). The candidates n[1], n[2], n[3]...n[f] for the number of neighboring sites are natural numbers such as 1, 2, 3, 4, and 5.

[0114] The grid search unit 35 performs a grid search using the candidates for the number of nearby sites to find the optimum number of nearby sites (S218), as will be described in detail with reference to FIG.

[0115] The model creation unit 39 determines the most accurate model from among the models created using different numbers of neighboring sites (S219). The index of accuracy may be the same as in step S6.

[0116] In the process of FIG. 12, only either the quantile number or the number of neighboring sites may be grid searched.

[0117] FIG. 13 is a flowchart illustrating the process of performing a grid search for the optimal quantile in step S215 of FIG.

[0118] The grid search unit 35 sets the quantile candidates q[1], q[2], q[3], . . . q[f] as the quantile Q, and repeats the following process for all the candidates (S2151).

[0119] The distance calculation unit 34 calculates the distance from the reference input data for each input data (S2152). The distance may be any distance that allows detection of outliers, such as Euclidean distance or Mahalanobis distance.

[0120] The quantile outlier detector 36 detects outliers by the quantile Q based on the distance in step S2152 (S2153).

[0121] The exclusion unit 38 excludes the training data containing this outlier (input data), and the model creation unit 39 creates a model (S2154).

[0122] The accuracy calculation unit 40 calculates the accuracy of the created model (S2155). The accuracy index may be the same as that in step S6.

[0123] Thus, a model is created for each of the quantile candidates q[1], q[2], q[3], ..., q[f], and the accuracy of each model is calculated. Processing returns to step S216 in Fig. 12, where the model creation unit 39 determines the model with the highest accuracy (optimal quantile).

[0124] FIG. 14 is a flowchart illustrating the process of performing a grid search to find the optimum number of nearby sites in step S218 of FIG.

[0125] The grid search unit 35 sets the candidates n[1], n[2], n[3] . . . n[f] for the optimum number of neighboring sites as the number N of neighboring sites, and repeats the following process for all the candidates (S2181).

[0126] The local outlier detector 37 detects outliers based on the number N of nearby sites (S2182).

[0127] The exclusion unit 38 excludes the training data containing this outlier (input data), and the model creation unit 39 creates a model (S2183).

[0128] The accuracy calculation unit 40 calculates the accuracy of the created model (S2184). The accuracy index may be the same as that in step S6.

[0129] Thus, a model is created for each of the candidate numbers of neighboring sites n[1], n[2], n[3]...n[f], and the accuracy of each model is calculated. Processing returns to step S219 in Fig. 12, where model creation unit 39 determines the model with the highest accuracy (optimal number of neighboring sites).

[0130] 13 or 14, an optimal model using the optimal quantile or the optimal number of neighboring sites is created, and the process proceeds to step S3 in Fig. 9, where cross-validation is performed. Note that the process in step S4 may be the same as the process described in Fig. 11, but it is also possible to perform a grid search on the output data as described below.

[0131] <Processing Using Grid Search on Input and Output Data> Next, the process of determining the optimal quantile or number of neighboring sites for the input and output data using grid search and excluding outliers will be described with reference to Figures 15 to 17. The overall flow of model creation is the same as that shown in Figure 9.

[0132] FIG. 15 is a flowchart illustrating the process of creating a model by excluding training data in which both the input data and the output data are outliers in step S4 of FIG.

[0133] First, it is determined whether the output data is univariate (S411).

[0134] If the output data is univariate (Yes in S411), the input data is converted to a value in a fixed range such as 0 to 1 by normalization, or to a value with a mean of "0" and a variance of "1" by standardization (S412).

[0135] Next, the grid search unit 35 selects one of the outlier detection methods (distance measurement from the reference or local outlier factor method) (S413). The selection method may be preset by the developer, or both may be performed and the one with the better model accuracy may be selected.

[0136] If distance measurement from a reference (an example of a third method) is selected, the distance calculation unit 34 acquires reference input data (S414). The reference input data can be set in advance by the developer. Alternatively, input data whose output data (target value) indicates the median may be automatically selected.

[0137] Next, the grid search unit 35 determines quantile candidates q[1], q[2], q[3]...q[f] (S415). The quantile candidates q[1], q[2], q[3]...q[f] are natural numbers such as 2, 3, 4, 5, etc.

[0138] The grid search unit 35 performs a grid search for the optimal quantile using the quantile candidates (S416), as will be described in detail with reference to FIG.

[0139] The model creation unit 39 determines the most accurate model from among the models created using different quantiles (S417). The index of accuracy may be the same as in step S6.

[0140] When the local outlier factor method (an example of the fourth method) is selected, the grid search unit 35 determines candidates n[1], n[2], n[3]...n[f] for the number of neighboring sites (S418). The candidates n[1], n[2], n[3]...n[f] for the number of neighboring sites are natural numbers such as 1, 2, 3, 4, and 5.

[0141] The grid search unit 35 performs a grid search for the optimum number of neighboring sites using the candidates for the number of neighboring sites (S419), the details of which will be described with reference to FIG.

[0142] The model creation unit 39 determines the most accurate model from among the models created using different numbers of neighboring sites (S420). The index of accuracy may be the same as in step S6.

[0143] If the output data is determined to be multivariate in step S411 (No in S411), the processing flow is the same as in the univariate case (S421), except that distance measurement from the reference and the local outlier factor method can be selected as the method for detecting outliers in the output data.

[0144] In the process of FIG. 15, only either the quantile number or the number of neighboring sites may be grid searched.

[0145] FIG. 16 is a flowchart illustrating the process of performing a grid search for the optimal quantile in step S416 of FIG.

[0146] The grid search unit 35 sets the quantile candidates q[1], q[2], q[3], ... q[f] as the quantile Q, and repeats the following process for all the candidates (S4161).

[0147] The distance calculation unit 34 calculates the distance from the reference input data for each input data (S4162). The distance may be any distance that allows detection of outliers, such as Euclidean distance or Mahalanobis distance.

[0148] The quantile outlier detector 36 detects outliers by the quantile Q based on the distance in step S4162 (S4163).

[0149] Similarly, the quantile outlier detector 36 detects outliers based on the quantile Q for the output data (univariate) (S4164).

[0150] The exclusion unit 38 excludes the learning data in which the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S4165).

[0151] The accuracy calculation unit 40 calculates the accuracy of the created model (S4166). The accuracy index may be the same as that in step S6.

[0152] Thus, a model is created for each of the quantile candidates q[1], q[2], q[3], ..., q[f], and the accuracy of each model is calculated. Processing returns to step S417 in Fig. 15, where the model creation unit 39 determines the model with the highest accuracy (optimal quantile).

[0153] FIG. 17 is a flowchart illustrating the process of performing a grid search to find the optimum number of nearby sites in step S419 of FIG.

[0154] The grid search unit 35 sets the quantile candidates q[1], q[2], q[3] ... q[f] as the quantile Q, and sets the neighboring site number candidates n[1], n[2], n[3] ... n[f] as the neighboring site number N (S4191). The grid search unit 35 repeats the following process for all combinations of the quantile candidates and the neighboring site number candidates (S4192, S4193).

[0155] The local outlier detector 37 detects outliers for the input data when the number of neighboring sites is N (S4194).

[0156] The quantile outlier detector 36 detects outliers based on the quantile Q for the output data (S4195).

[0157] The exclusion unit 38 excludes the training data in which the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S4196).

[0158] The accuracy calculation unit 40 calculates the accuracy of the model created when the number of neighboring sites is N and the quantile is Q (S4197). The index of accuracy may be the same as in step S6.

[0159] A model is created for each combination of the candidate numbers of neighboring sites and candidate quantiles, and the accuracy of each model is calculated. Processing returns to step S420 in Fig. 15, where the model creation unit 39 determines the model with the highest accuracy (optimum quantile, optimum number of neighboring sites).

[0160] 16 or 17, outliers were removed using the optimal quantiles and the optimal number of neighboring sites, and an optimal model was created. The process then proceeds to step S5 in FIG. 9, where cross-validation is performed.

[0161] <<When Output Data is Multivariate>> Fig. 18 is a flowchart illustrating the process of creating a model by excluding training data in which both the input data and the output data are outliers in step S421 (when the output data is multivariate) in Fig. 15. Fig. 18 may mainly explain the differences from steps S412 to S420 in Fig. 15.

[0162] 18 , in step S4212, the grid search unit 35 selects an outlier detection method (distance measurement from reference, local outlier factor method) for each of the input data and the output data (S4212). Therefore, the process from step S4212 onward branches into the following four cases: A. "Input data: distance from reference (an example of the third method), output data: distance from reference (an example of the sixth method)" B. "Input data: distance from reference (an example of the third method), output data: local outlier factor method (an example of the seventh method)" C. "Input data: local outlier factor method (an example of the fourth method), output data: distance from reference (an example of the sixth method)" D. "Input data: local outlier factor method (an example of the fourth method), output data: local outlier factor method (an example of the seventh method)" A. When "input data: distance from reference, output data: distance from reference" is selected, the distance calculation unit 34 acquires the input data and output data used as references (S4213A).

[0163] Next, the grid search unit 35 determines the quantile candidates q1[1], q1[2], q1[3] ... q1[f] on the input data side, and the quantile candidates q2[1], q2[2], q2[3] ... q2[f] on the output data side (S4214A).

[0164] The grid search unit 35 performs a grid search to find the optimal combination from all combinations of quantile candidates on the input data side and quantile candidates on the output data side (S4215A). Details will be described with reference to FIG.

[0165] The model creation unit 39 determines the most accurate model among the models created by excluding training data that has input data determined to be an outlier based on the quantiles of the input data and output data determined to be an outlier based on the quantiles of the output data (S4216A). The accuracy index may be the same as in step S6.

[0166] B. When "input data: distance from reference, output data: local outlier factor method" is selected, the distance calculation unit 34 acquires the input data to be used as the reference (S4213B).

[0167] Next, the grid search unit 35 determines the quantile candidates q[1], q[2], q[3] ... q[f] on the input data side, and the candidate numbers of neighboring sites n[1], n[2], n[3] ... n[f] on the output data side (S4214B).

[0168] The grid search unit 35 performs a grid search to find the optimum combination from all combinations of the quantile candidates on the input data side and the number of neighboring sites candidates on the output data side (S4215B). Details will be explained with reference to FIG.

[0169] The model creation unit 39 determines the most accurate model from among the models created by excluding the training data having input data determined to be outliers based on the quantiles on the input data side and having output data determined to be outliers based on the number of neighboring sites on the output data side (S4216B). The accuracy index may be the same as in step S6.

[0170] C. If "input data: local outlier factor method, output data: distance from reference" is selected, the distance calculation unit 34 acquires the output data to be used as the reference (S4213C).

[0171] Next, the grid search unit 35 determines candidates n[1], n[2], n[3] ... n[f] for the number of neighboring sites on the input data side, and candidates q[1], q[2], q[3] ... q[f] for the quantiles on the output data side (S4214C).

[0172] The grid search unit 35 performs a grid search to find the optimum combination among all combinations of candidates for the number of neighboring sites on the input data side and candidates for the quantiles on the output data side (S4215C). Details will be explained with reference to FIG.

[0173] The model creation unit 39 determines the most accurate model from among the models created by excluding the training data having input data determined to be an outlier based on the number of neighboring sites on the input data side and having output data determined to be an outlier based on the quantile on the output data side (S4216C). The accuracy index may be the same as in step S6.

[0174] D. When "input data: local outlier factor method, output data: local outlier factor method" is selected, the grid search unit 35 determines candidates n1[1], n1[2], n1[3] ... n1[f] for the number of neighboring sites on the input data side, and candidates n2[1], n2[2], n2[3] ... n2[f] for the number of neighboring sites on the output data side (S4214D).

[0175] Next, the grid search unit 35 performs a grid search to find the optimum combination among all combinations of candidates for the number of neighboring sites on the input data side and candidates for the number of neighboring sites on the output data side (S4215D). Details will be described with reference to FIG.

[0176] The model creation unit 39 determines the most accurate model from among the models created by excluding the training data having input data determined to be an outlier based on the number of neighboring sites on the input data side and having output data determined to be an outlier based on the number of neighboring sites on the output data side (S4216D). The accuracy index may be the same as in step S6.

[0177] Note that grid search may not be performed in steps S4215A, S4215B, S4215C, and S4215D in FIG. 18, or may be performed in only one or more of these steps.

[0178] FIG. 19 is a flowchart illustrating the process of performing a grid search to find optimal quantiles for input data and output data in step S4215A of FIG.

[0179] The grid search unit 35 sets the quantile candidates q1[1], q1[2], q1[3] ... q1[f] on the input data side as the quantile Q1, and sets the quantile candidates q2[1], q2[2], q2[3] ... q2[f] on the output data side as the quantile Q2 (S42151A). The grid search unit 35 repeats the following process for all combinations of Q1 and Q2 (S42152A, S42153A).

[0180] The distance calculation unit 34 calculates the distance from the reference data for each input data (S42154A). The distance may be any distance that allows detection of outliers, such as Euclidean distance or Mahalanobis distance.

[0181] The quantile outlier detector 36 detects an outlier by the quantile Q1 based on the distance in step S42154A (S42155A).

[0182] The same applies to steps S42156A and S42157A.

[0183] The exclusion unit 38 excludes the learning data in which the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158A).

[0184] The accuracy calculation unit 40 calculates the accuracy of the model created for the quantiles Q1 and Q2 (S42159A). The accuracy index may be the same as in step S6.

[0185] In this way, a model is created for each combination of the quantile candidates on the input data side and the quantile candidates on the output data side, and the accuracy of each model is calculated. Processing returns to step S4216A in Fig. 18, where the model creation unit 39 determines the model with the highest accuracy (the optimal quantile for the input data and the output data).

[0186] FIG. 20 is a flowchart illustrating the process of grid searching the optimum quantile number on the input data side and the optimum number of neighboring sites on the output data side in step S4215B of FIG.

[0187] The grid search unit 35 sets the quantile candidates q[1], q[2], q[3] ... q[f] on the input data side as the quantile Q, and sets the neighboring site number candidates n[1], n[2], n[3] ... n[f] on the output data side as the neighboring site number N (S42151B). The grid search unit 35 repeats the following process for all combinations of Q and N (S42152B, S42153B).

[0188] The distance calculation unit 34 calculates the distance from the reference data for each input data (S42154B). The distance may be any distance that allows detection of outliers, such as Euclidean distance or Mahalanobis distance.

[0189] The quantile outlier detector 36 detects outliers by the quantile Q based on the distance in step S42154B (S42155B).

[0190] Similarly, the local outlier detection unit 37 detects outliers from the output data (multivariate) using the local outlier factor method based on the number N of neighboring sites (S42156B).

[0191] The exclusion unit 38 excludes the learning data in which the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158B).

[0192] The accuracy calculation unit 40 calculates the accuracy of the model created when the quantile number Q and the number of neighboring sites N are used (S42159B). The accuracy index may be the same as that used in step S6.

[0193] In this way, a model is created for each combination of candidate quantiles and candidate numbers of neighboring sites, and the accuracy of each model is calculated. Processing returns to step S4216B in Fig. 18, where model creation unit 39 determines the model with the highest accuracy (the optimal quantile for the input data and the optimal number of neighboring sites for the output data).

[0194] FIG. 21 is a flowchart illustrating the process of grid searching the optimum number of neighboring sites on the input data side and the optimum quantile number on the output data side in step S4215C of FIG.

[0195] Next, the grid search unit 35 sets the candidates n[1], n[2], n[3] ... n[f] for the number of neighboring sites on the input data side as the number N of neighboring sites, and sets the candidates q[1], q[2], q[3] ... q[f] for the quantile on the output data side as the quantile Q (S42151C). The grid search unit 35 repeats the following process for all combinations of N and Q (S42152C, S42153C).

[0196] The local outlier detector 37 detects outliers from the input data using the local outlier factor method based on the number N of neighboring sites (S42154C).

[0197] The distance calculation unit 34 calculates the distance from the reference data for each output data (S42155C). The distance may be any distance that allows detection of outliers, such as Euclidean distance or Mahalanobis distance.

[0198] The quantile outlier detector 36 detects an outlier by the quantile Q based on the distance in step S42155C (S42156C).

[0199] The exclusion unit 38 excludes the training data in which the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158C).

[0200] The accuracy calculation unit 40 calculates the accuracy of the model created when the number of neighboring sites is N and the quantile is Q (S42159C). The accuracy index may be the same as in step S6.

[0201] In this way, a model is created for each combination of the candidate numbers of neighboring sites and the candidate quantiles, and the accuracy of each model is calculated. Processing returns to step S4216C in Fig. 18, where model creation unit 39 determines the model with the highest accuracy (the optimal number of neighboring sites for the input data and the optimal quantile for the output data).

[0202] FIG. 22 is a flowchart illustrating the process of performing grid search to determine the optimum number of neighboring sites for the input data and output data in step S4215D of FIG.

[0203] The grid search unit 35 sets the candidates for the number of neighboring sites n1[1], n1[2], n1[3] ... n1[f] as the number of neighboring sites N1, and sets the candidates for the number of neighboring sites n2[1], n2[2], n2[3] ... n2[f] as the number of neighboring sites N2 (S42151D). The grid search unit 35 repeats the following process for all combinations of N1 and N2 (S42152D, S42153D).

[0204] The local outlier detector 37 detects outliers from the input data using the local outlier factor method based on the number of neighboring sites N1 (S42154D).

[0205] The local outlier detector 37 detects outliers in the output data using the local outlier factor method based on the number N2 of nearby sites (S42155D).

[0206] The exclusion unit 38 excludes the training data in which the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158D).

[0207] The accuracy calculation unit 40 calculates the accuracy of the model created when the number of neighboring sites is N1 and N2 (S42159D). The accuracy index may be the same as in step S6.

[0208] In this way, models are created for all combinations of candidate numbers of neighboring sites on the input data side and the output data side, and the accuracy of each model is calculated. Processing returns to step S4216D in Fig. 18, where model creation unit 39 determines the model with the highest accuracy (the optimal number of neighboring sites for the input data and the output data).

[0209] 19 to 22, outliers were removed using the optimal quantiles or number of neighboring sites on the input data side and the optimal quantiles or number of neighboring sites on the output data side, and an optimal model was created. The process proceeds to step S5 in FIG. 9, where cross-validation is performed.

[0210] <Supplementary Information Regarding Combinations of Methods for Removing Outliers> Figures 23A and 23B show examples of combinations of various methods in the first elimination process and the second elimination process, respectively. Figure 23A shows a method for removing outliers from input data in the first elimination process. Multiple methods are available for removing outliers from input data, and any one of them may be adopted. Note that outlier removal using quantiles (first method) and outlier removal using the local outlier factor method (second method) are merely examples.

[0211] 23B shows a combination of methods for excluding outliers in the input data and the output data in the second excluding process. First, a number of methods are available for excluding outliers in the input data, and any one of these methods may be adopted.

[0212] In the present embodiment, when the output data is univariate, outlier removal using quantiles is exemplified as an example of a method for removing outliers from the output data. Note that outlier removal using quantiles (the third method) is merely one example. Therefore, outliers in the input data and output data can be removed by combining either the third or fourth method with the fifth method. Furthermore, when the output data is univariate, multiple methods for removing outliers may be provided.

[0213] In the present embodiment, when the output data is multivariate, outlier removal using quantiles and outlier removal using the local outlier factor method are exemplified as examples of methods for removing outliers in the output data. Note that outlier removal using quantiles (sixth method) and outlier removal using the local outlier factor method (seventh method) are merely examples. Therefore, outliers in the input data and output data can be removed by combining either the third or fourth method with either the sixth or seventh method.

[0214] <Major Effects> The information processing device of this embodiment easily eliminates outliers even when the training dataset (explanatory variables) is sparse, making it difficult to apply general outlier detection techniques. Therefore, it is easy to eliminate data measured on prototypes, etc. Because training data in which both the objective variable and the explanatory variables are outliers can be excluded, it can detect cases where measurements were stopped due to deterioration of characteristics after experimental blending (prototypes result in outliers in the explanatory variables, and poor characteristics result in the objective variable being an outlier). The information processing device creates models by excluding outliers and verifies their accuracy, so it can exclude outliers in explanatory variables if they are detrimental to the model's predictive accuracy. Models are created by excluding outliers in training data for only the explanatory variables, and models are created by excluding outliers in both the explanatory variables and the objective variable, and the model with the better accuracy is adopted. Therefore, from the perspective of accuracy, it is possible to adopt models created by excluding outliers. Furthermore, because the number of models created is not very large, outliers can be detected even when the training dataset contains a large amount of data.

[0215] <Other Application Examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0216] For example, in this embodiment, an example of detecting outliers from a training dataset of a model mainly used for compounding exploration has been described, but the present invention is not limited to such material informatics, and the training dataset from which outliers are removed may be any dataset that includes input data and output data used in machine learning.

[0217] Furthermore, the removal of outliers using quartiles or the local outlier factor method is just one example, and outliers may be detected using standard deviation, Smirnoff-Grubbs test, or Hotelling's theory in the case of a single variable, or standard deviation, cluster analysis, or Hotelling's T2 method in the case of a multivariate variable. In other words, the number of methods for detecting outliers is not limited to two (it is preferable to have at least two or more), and any of a number of different methods may be used.

[0218] In the case of a client-server system, a user may connect the information processing device 30 to the server device 10 and request the creation of a model, whereby the server device 10 creates the model and transmits it to the information processing device 30. The server device 10 may retain the model internally, and in this case, when explanatory variables are transmitted from the information processing device 30, the server device 10 returns objective variables to the information processing device 30. The information processing device 30 may execute a web application provided by the server device 10 as appropriate.

[0219] In addition, the configuration examples in Figure 5 and the like are divided according to main functions to facilitate understanding of the processing by the information processing device 30 and the server device 10. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the information processing device 30 and the server device 10 can be divided into even more processing units depending on the processing content. Furthermore, the processing can be divided so that one processing unit includes even more processes.

[0220] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0221] This application claims priority based on Japanese Patent Application No. 2023-134755, filed with the Japan Patent Office on August 22, 2023, the entire contents of which are incorporated herein by reference.

[0222] 10 Server device 30 Information processing device 100 Data processing system

Claims

1. A model creation method in which an information processing device creates a model that matches input data included in training data with output data, the model creation method performing the following steps: a first exclusion process of excluding from the training data set training data whose input data is an outlier; a second exclusion process of excluding from the training data set training data whose input data and output data are outliers; a first model creation process of creating a first model using the training data set from which the training data has been excluded by the first exclusion process; a second model creation process of creating a second model using the training data set from which the training data has been excluded by the second exclusion process; and a model selection process of adopting the model with better predictive accuracy from the first model and the second model.

2. The model creation method according to claim 1, wherein the first elimination process excludes outliers from the input data using one of a number of different techniques.

3. The model creation method according to claim 2, wherein the first exclusion process excludes outliers from the input data using at least two different first or second methods.

4. A model creation method according to any one of claims 1 to 3, wherein the second exclusion process excludes outliers in the input data using one of a plurality of different methods, and, if the output data is univariate, excludes outliers in the output data using a predetermined method.

5. The model creation method of claim 4, wherein the second elimination process removes outliers in the input data using at least two different third or fourth methods, and, if the output data is univariate, removes outliers in the output data using a fifth method, and wherein the second elimination process either removes outliers in the input data using the third method and removes outliers in the output data using the fifth method, or removes outliers in the input data using the fourth method and removes outliers in the output data using the fifth method.

6. A model creation method according to any one of claims 1 to 3, wherein the second exclusion process excludes outliers in the input data using any one of a plurality of different methods, and when the output data is multivariate, excludes outliers in the output data using any one of a plurality of different methods.

7. The model creation method of claim 6, wherein the second elimination process removes outliers in the input data by at least two different third methods or fourth methods, and, if the output data is multivariate, removes outliers in the output data by at least two different sixth methods or seventh methods, and wherein the second elimination process removes outliers in the input data by the third method and removes outliers in the output data by the sixth method, or removes outliers in the input data by the fourth method and removes outliers in the output data by the sixth method, or removes outliers in the input data by the third method and removes outliers in the output data by the seventh method, or removes outliers in the input data by the fourth method and removes outliers in the output data by the seventh method.

8. The model creation method according to claim 3, wherein the first method is a method for detecting outliers using quantiles based on the distance of each data item from a reference data item, and the second method is a method for detecting outliers using a local outlier factor method.

9. The model creation method of claim 5, wherein the third method is a method for detecting outliers using quantiles based on the distance of each data item from a reference data item, the fourth method is a method for detecting outliers using a local outlier factor method, and the fifth method is a method for detecting outliers using quantiles.

10. The model creation method described in claim 7, wherein the third method and the sixth method are methods for detecting outliers using quantiles based on the distance of each data item from a reference data item, and the fourth method and the seventh method are methods for detecting outliers using a local outlier factor method.

11. The model creation method described in claim 8, wherein in the first elimination process, the quantile, which is a hyperparameter of the first method, is determined by grid search, or the number of nearby sites, which is a hyperparameter of the second method, is determined by grid search.

12. The model creation method described in claim 9, wherein in the second exclusion process, at least one of the quantile, which is a hyperparameter of the third method, or the quantile, which is a hyperparameter of the fifth method, is determined by grid search, or at least one of the number of nearby sites, which is a hyperparameter of the fourth method, or the quantile, which is a hyperparameter of the fifth method, is determined by grid search.

13. The model creation method described in claim 10, wherein in the second exclusion process, at least one of the quantile, which is a hyperparameter of the third method, or the quantile, which is a hyperparameter of the sixth method, is determined by grid search, or at least one of the quantile, which is a hyperparameter of the third method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search, or at least one of the number of neighboring sites, which is a hyperparameter of the fourth method, or the quantile, which is a hyperparameter of the sixth method, is determined by grid search, or at least one of the number of neighboring sites, which is a hyperparameter of the fourth method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search.

14. A program that causes an information processing device that creates a model that matches input data included in training data with output data to execute the following: a first exclusion process that excludes training data whose input data is an outlier from the training data set; a second exclusion process that excludes training data whose input data and output data are outliers from the training data set; a first model creation process that creates a first model using the training data set from which the training data has been excluded by the first exclusion process; a second model creation process that creates a second model using the training data set from which the training data has been excluded by the second exclusion process; and a model selection process that adopts the model with better predictive accuracy from the first model and the second model.

15. An information processing device that creates a model that matches input data included in training data with output data, the information processing device having: an exclusion unit that excludes training data whose input data is an outlier from a training data set, and excludes training data whose input data and output data are outliers from the training data set; a model creation unit that creates a first model using the training data set from which the training data whose input data is an outlier has been excluded, and creates a second model using the training data set from which the training data whose input data and output data are outliers has been excluded; and a model selection unit that adopts the model with better predictive accuracy from the first model and the second model.

Images