Model creation methods, programs, and information processing devices.

Abstract

The objective of the present invention is to improve the prediction accuracy of a model. The objective of the present invention is to improve the prediction accuracy of a model. The present disclosure relates to a model creation method in which an information processing device creates a model that associates input data included in training data with output data included in the training data, the model creation method comprising: a first exclusion processing for excluding, from a training dataset, training data of which the input data is an outlier; a second exclusion processing for excluding, from the training dataset, training data of which the input data and the output data are outliers; a first model creation processing for creating a first model by using the training dataset from which the training data has been excluded in the first exclusion processing; a second model creation processing for creating a second model by using the training dataset from which the training data has been excluded in the second exclusion processing; and a model selection processing for selecting the model having the better prediction accuracy from the first model and the second model.

Description

Technical Field

[0001] The present invention relates to a model creation method, a program, and an information processing apparatus.

Background Art

[0002] Materials informatics that aims to improve the efficiency of material development using information sciences such as statistical analysis, machine learning, and data mining is known. One of the materials informatics is formulation search for searching the optimal formulation of materials.

[0003] In formulation search, a process of separating or excluding outliers from learning data is performed to create an accurate model (see, for example, Patent Documents 1 to 3). Patent Document 1 discloses acquiring a data set composed of a plurality of sets of test conditions and substance information of a target substance generated based on the test conditions, performing statistical processing on the data set, and based on the result, recommended test conditions for optimizing the substance information and a technique for estimating the substance information of the corresponding target substance.

[0004] Patent Document 2 discloses a technique in which a data evaluation model constructed by performing machine learning on collected data evaluates the input collected data, and automatically selects training data for constructing a learning model from the evaluated data based on the evaluation result.

[0005] Patent Document 3 discloses a technique of selecting data for updating an existing learning model used for machine learning from the data stored in a data storage unit, generating a new learning model by machine learning based on the selected data, and updating the existing learning model with the new learning model.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] However, conventional techniques have a problem in that they do not adequately exclude outliers, which hinders improvements in the predictive accuracy of the model. For example, when there are many explanatory variables and the dataset is sparse, the spread of data in the search space is large, making it difficult to appropriately identify outliers with conventional methods. Also, since such datasets are not normally distributed, it is difficult to identify outliers using standard deviation, etc.

[0008] This disclosure provides a technique for improving the prediction accuracy of a model. [Means for solving the problem]

[0009] This disclosure provides a model creation method for which an information processing device creates a model that maps input data included in training data to output data, comprising: a first exclusion process that excludes training data in which the input data is an outlier from the training dataset; a second exclusion process that excludes training data in which both the input data and the output data are outliers from the training dataset; a first model creation process that creates a first model using the training dataset from which the training data excluded in the first exclusion process; a second model creation process that creates a second model using the training dataset from which the training data excluded in the second exclusion process; and a model selection process that adopts the model with better prediction accuracy among the first model and the second model.

[0010] In the second disclosure, in the first disclosure, In the first exclusion process, outliers in the input data are excluded using one of several different methods.

[0011] In the third disclosure, in the second disclosure, In the first exclusion process, outliers in the input data are excluded using at least two different first or second methods.

[0012] In the fourth disclosure, in any of the first to third disclosures, In the second exclusion process, outliers in the input data are excluded using one of several different methods. If the output data is univariate, outliers in the output data are removed using a predetermined method.

[0013] In the fifth disclosure, in the fourth disclosure, The second exclusion process excludes outliers from the input data using at least two different third or fourth methods, and if the output data is univariate, excludes outliers from the output data using a fifth method. In the second exclusion process, outliers in the input data are excluded using the third method, and outliers in the output data are excluded using the fifth method, or, The fourth method removes outliers from the input data, and the fifth method removes outliers from the output data.

[0014] In the sixth disclosure, in any of the first to third disclosures, In the second exclusion process, outliers in the input data are excluded using one of several different methods, and if the output data is multivariate, outliers in the output data are excluded using one of several different methods.

[0015] In the seventh disclosure, in the sixth disclosure, The second exclusion process excludes outliers from the input data using at least two different third or fourth methods, and if the output data is multivariate, excludes outliers from the output data using at least two different sixth or seventh methods. In the second exclusion process, outliers in the input data are excluded using the third method, and outliers in the output data are excluded using the sixth method, or, Exclude the outliers of the input data by the fourth method and exclude the outliers of the output data by the sixth method, or Exclude the outliers of the input data by the third method and exclude the outliers of the output data by the seventh method, or Exclude the outliers of the input data by the fourth method and exclude the outliers of the output data by the seventh method.

[0016] In the eighth disclosure, in the third disclosure, The first method is a method for detecting outliers by quantiles based on the distance of each data from the reference data. The second method is a method for detecting outliers by the local outlier factor method.

[0017] In the ninth disclosure, in the fifth disclosure, The third method is a method for detecting outliers by quantiles based on the distance of each data from the reference data. The fourth method is a method for detecting outliers by the local outlier factor method. The fifth method is a method for detecting outliers by quantiles.

[0018] In the tenth disclosure, in the seventh disclosure, The third method and the sixth method are methods for detecting outliers by quantiles based on the distance of each data from the reference data. The fourth method and the seventh method are methods for detecting outliers by the local outlier factor method.

[0019] In the eleventh disclosure, in the eighth disclosure, In the first exclusion process, determine the quantile, which is the hyperparameter of the first method, by grid search, or Determine the number of neighboring sites, which is the hyperparameter of the second method, by grid search.

[0020] In the twelfth disclosure, in the ninth disclosure, In the second exclusion process, at least one of the quantiles, which are hyperparameters of the third method, or the quantiles, which are hyperparameters of the fifth method, is determined by grid search, or At least one of the hyperparameters of the fourth method, namely the number of neighboring sites, or the hyperparameter of the fifth method, namely the number of quantiles, is determined by grid search.

[0021] In the 13th disclosure, in the 10th disclosure, In the second exclusion process, at least one of the quantiles, which are hyperparameters of the third method, or the quantiles, which are hyperparameters of the sixth method, is determined by grid search, or The quantile, which is a hyperparameter of the third method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search, or The number of neighboring sites, which is a hyperparameter of the fourth method, or the quantile, which is a hyperparameter of the sixth method, is determined by grid search, or At least one of the hyperparameters of the fourth method, namely the number of neighboring sites, or the hyperparameter of the seventh method, namely the number of neighboring sites, is determined by grid search.

[0022] The 14th disclosure concerns an information processing device that creates a model that maps input data included in training data to output data, A first exclusion process removes training data where the input data is an outlier from the training dataset, A second exclusion process that excludes training data from the training dataset in which the input data and output data are outliers, A first model creation process, which creates a first model using the training dataset from which the training data has been excluded in the first exclusion process, A second model creation process, which creates a second model using the training dataset from which the training data has been excluded in the second exclusion process, The present invention provides a program that performs a model selection process to select the model with better prediction accuracy from among the first and second models.

[0023] Disclosure No. 15 is an information processing device that creates a model that maps input data included in training data to output data, An exclusion unit that excludes training data in which the input data is an outlier from the training dataset, and excludes training data in which both the input data and the output data are outliers from the training dataset, A model creation unit creates a first model using the training dataset from which training data with outlier input data have been excluded, and creates a second model using the training dataset from which training data with outlier input data and output data have been excluded. The system includes a model selection unit that selects the model with better prediction accuracy from among the first model and the second model. [Effects of the Invention]

[0024] This disclosure can improve the prediction accuracy of the model. [Brief explanation of the drawing]

[0025] [Figure 1A] This diagram illustrates the comparison between outliers due to measurement errors and outliers measured for prototypes. [Figure 1B] This diagram illustrates the comparison between outliers due to measurement errors and outliers measured for prototypes. [Figure 2] This diagram illustrates the process by which an information processing device removes outliers from a training dataset to create a model. [Figure 3A] This is an example of a system configuration diagram for a data processing system. [Figure 3B] This figure shows an example of an information processing device that creates a model by excluding outliers. [Figure 4] This figure shows the hardware configuration of an example of a server device and information processing device. [Figure 5]This figure shows an example of the functional configuration of an information processing device. [Figure 6] This diagram illustrates a method for detecting outliers using quartiles. [Figure 7] This diagram illustrates the outlier detection method using the D-One-Out method. [Figure 8A] This diagram illustrates methods for detecting outliers in the explanatory and dependent variables. [Figure 8B] This figure illustrates the method for removing outliers detected in Figure 8A. [Figure 9] This is an example of a flowchart illustrating the overall process of model creation. [Figure 10] Figure 9 shows an example of a flowchart illustrating the process of creating a model by excluding outliers from the input data in step S2. [Figure 11] Figure 9 shows an example of a flowchart illustrating the process of creating a model by excluding outliers from the input and output data in step S4. [Figure 12] Figure 9 shows an example of a flowchart illustrating the process of creating a model by excluding outlier input data in step S2. [Figure 13] This is an example of a flowchart illustrating the process of performing a grid search for the optimal quantile in step S215 of Figure 12. [Figure 14] This is an example of a flowchart illustrating the process of performing a grid search to find the optimal number of neighboring sites in step S218 of Figure 12. [Figure 15] Figure 9 shows an example of a flowchart illustrating the process of creating a model by excluding training data where both the input and output data are outliers in step S4. [Figure 16] This is an example of a flowchart illustrating the process of performing a grid search for the optimal quantile in step S416 of Figure 15. [Figure 17] This is an example of a flowchart illustrating the process of performing a grid search to find the optimal number of neighboring sites in step S419 of Figure 15. [Figure 18] Figure 15 shows an example of a flowchart illustrating the process of creating a model by excluding training data where both the input and output data are outliers, as shown in step S421. [Figure 19] Figure 18 shows an example of a flowchart illustrating the process of performing a grid search for the optimal quantiles for the input and output data in step S4215A. [Figure 20] Figure 18 shows an example flowchart illustrating the process of performing a grid search for the optimal quantile on the input data side and the optimal number of neighboring sites on the output data side in step S4215B. [Figure 21] Figure 18 shows an example of a flowchart illustrating the process of performing a grid search to find the optimal number of neighboring sites on the input data side and the optimal quantiles on the output data side in step S4215C. [Figure 22] Figure 18 shows an example of a flowchart illustrating the process of performing a grid search for the optimal number of neighboring sites for the input and output data in step S4215D. [Figure 23A] This is an example diagram illustrating a method for excluding outliers from input data during the first exclusion process. [Figure 23B] This is an example of a diagram showing a combination of methods for excluding outliers from input and output data in the second exclusion process. [Modes for carrying out the invention]

[0026] The following describes an example of an embodiment for carrying out the present invention: an information processing device and a model creation method performed by the information processing device.

[0027] <Supplementary information on comparative technology> In formulation search techniques, which explore the optimal blending of materials, a model is created using a training dataset to search for material blends with desirable properties. Material developers sometimes manufacture prototypes with material blends that differ significantly from conventional blends, and these prototypes are also measured to confirm whether or not they possess desirable properties. However, it is not uncommon for no improvement in properties to be observed, and further exploration around that blend may be discontinued. In such cases, the blend of the prototype may become an outlier.

[0028] However, unlike measurement errors, data on the characteristics of prototypes cannot be judged at the time of data acquisition, and decisions are often made later in the development process, which can lead to oversights or omissions in exclusion.

[0029] Figures 1A and 1B illustrate the comparison between outlier 101 due to measurement error and outlier 102 measured for the prototype. Outlier 101 due to measurement error, shown in Figure 1A, can be flagged as a "measurement error" by the developer during characteristic measurement, or excluded from the accumulated data on the spot.

[0030] In contrast, the outlier 102 measured for the prototype shown in Figure 1B corresponds to the coordinates of the data points relative to the formulation (ratio of materials 1 and 2). However, even if a formulation significantly different from the conventional one is experimentally performed, there may be cases where no improvement in properties is observed. If no improvement in properties is observed, further exploration around that formulation is not carried out. Furthermore, measurement data from such prototypes should be excluded if it degrades the accuracy of the model or is unfavorable for searching for a formulation with good properties. Thus, since outlier 102 is not due to a measurement error and is not excluded from the accumulated data on the spot, it is difficult to exclude it during measurement. For this reason, outlier 102 may be overlooked or forgotten to be excluded later in the development process.

[0031] Furthermore, when detecting outliers using the types and amounts of ingredients as explanatory variables, a large number of explanatory variables will exist. However, experiments are rarely conducted uniformly by varying the types and amounts of ingredients evenly, so training datasets tend to be sparse. For this reason, it can be difficult to apply general outlier detection methods such as standard deviation or quartile detection. Moreover, there is a need for a method (an efficient method) that can detect outliers even when the number of data points in the training dataset is large.

[0032] Therefore, when creating a model for formulation search using machine learning or the like, the information processing device disclosed here excludes outliers from the training dataset as follows.

[0033] Figure 2 illustrates the process flow in which an information processing device removes outliers from a training dataset to create a model.

[0034] The S100 information processing device measures the distance of multivariate explanatory variables (input data) from reference data and excludes training data containing input data that are outliers in the distance distribution. Quartiles or other methods may be used to detect outliers. The information processing device may also detect outliers using methods such as the local outlier factorization method or other techniques.

[0035] The S200 information processing device excludes training data where the input data is an outlier in the distance distribution and also an outlier in the target variable (output data).

[0036] In step S300, the information processing device creates a model using the training dataset from which the training data has been removed in step S100, and in step S200, it creates a model using the training dataset from which the training data has been removed, and adopts the model with better accuracy.

[0037] Note that the processes in steps S100 and S200 may be performed in either order, or they may be performed in parallel.

[0038] Thus, because the training dataset (explanatory variables) of this embodiment is sparse, the information processing device makes it easier to exclude outliers even when it is difficult to apply general outlier detection methods. Therefore, it is easy to exclude data measured on prototypes, etc. • Since training data where both the dependent and independent variables are outliers can be excluded, cases where measurements were stopped because the characteristics deteriorated due to experimental formulations can be detected (because it is a prototype, the independent variable is an outlier, and because the characteristics are poor, the dependent variable is also an outlier). • The information processing device creates a model and verifies its accuracy by excluding outliers, so it can exclude outliers in the explanatory variables if they are detrimental to the model's prediction accuracy. • By creating a model that excludes training data where only the explanatory variables are outliers, and by creating a model that excludes training data where both the explanatory and dependent variables are outliers, the model with better accuracy can be adopted. Therefore, from an accuracy standpoint, the model created by excluding outliers can be adopted. Furthermore, compared to applying the Leave One Out method comprehensively to a training dataset with a large amount of data, the number of models created is not as large, so outliers can be detected even when the training dataset has a large amount of data.

[0039] <About Terminology> A training dataset is a collection of data necessary for machine learning, consisting of sets of input and output data. It is also called a training dataset.

[0040] An outlier is a value among the obtained observations that deviates significantly from other values. In this embodiment, outliers may include data that reduces the accuracy of the model or is unfavorable to the search for good trait combinations.

[0041] In machine learning, a model refers to a mechanism (such as a program or file) that derives output data from input data. The model establishes a correspondence between input and output data. Models are sometimes also referred to as AI or artificial intelligence.

[0042] <Example System Configuration> Figure 3A is an example of a system configuration diagram of the data processing system 100. The data processing system 100 in Figure 3A is a client-server system and includes a server device 10 and an information processing device 30. However, the information processing device 30 may be a general-purpose computer and may not be included in the data processing system 100.

[0043] The server device 10 and the information processing device 30 are connected via a wide-area network N1 such as the Internet. The server device 10 may be installed in a cloud or data center, or it may be installed on-premises. The server device 10 may be a web server that returns processing results in response to requests from the information processing device 30. A server is a computer or software that performs the function of providing information and processing results in response to requests from a client.

[0044] The server device 10 is one or more information processing devices. The server device 10 creates a model that performs compound search from training data specified by the developer 9 via the information processing device 30. One of the features of the server device 10 is that it appropriately removes outliers from the training data before creating the model.

[0045] The server device 10 may have a data storage device that has training data stored in advance. Alternatively, the server device 10 may acquire training data from a data server or NAS (Network Attached Storage). Alternatively, the developer 9 may operate the information processing device 30 to send training data to the server device 10.

[0046] The server device 10 may be compatible with cloud computing. Cloud computing refers to a usage model in which network resources are utilized without being aware of specific hardware resources. Therefore, the server device 10 does not need to be housed in a single enclosure or provided as a single unit. The functions of the server device 10 may be distributed among multiple information processing units, or multiple information processing units may each have all the functions, and the information processing unit being used may be switched through load balancing or other means.

[0047] The information processing device 30 is located in facilities such as companies, research institutes, development departments, and factories, and is connected to network N2. Network N2 may be a LAN, Wi-Fi (registered trademark), Wide Area Ethernet (registered trademark), 4G, 5G, 6G, or other mobile phone network.

[0048] The information processing device 30 is a machine learning or general-purpose computer used by the developer 9. Here, the developer 9 is the person who uses the server device 10 to create a model from training data. The person using the information processing device 30 may include not only the person who creates the model, but also the person who has been contracted to create the model. Furthermore, the person using the information processing device 30 may also be the person performing the formulation search.

[0049] The information processing device 30 runs a dedicated native application on the web browser and the server device 10. When the information processing device 30 runs a web browser, the information processing device 30 and the server device 10 run a web application. A web application is an application that operates through the cooperation of a program written in a programming language (e.g., JavaScript®) that runs on a web browser and a program on the web server (server device 10). When a web application is executed, the model may be created by the server device 10, or by the information processing device 30 that receives the web application.

[0050] An application that cannot be executed unless it is installed on the information processing device 30 is called a native application. In this embodiment as well, the application executed on the information processing device 30 may be either a web application or a native application. In this case as well, the process of removing outliers from the training data and creating a model may be performed by the server device 10, or by the information processing device 30 using a native application.

[0051] Furthermore, in this embodiment, instead of the client-server system shown in Figure 3A, the information processing device 30 may create a model by excluding outliers on its own, as shown in Figure 3B. In this case, the information processing device 30 runs a native application that creates a model by excluding the model from the training data. However, even in this case, the information processing device 30 may acquire training data from a data server on the network. Therefore, even in the configuration shown in Figure 3B, it is preferable that the information processing device 30 can connect to the network.

[0052] The information processing device 30 can be, for example, a desktop PC, notebook PC, smartphone, PDA (Personal Digital Assistant), or tablet device used by the developer 9. Alternatively, the information processing device 30 can be any device capable of running a web browser or native applications.

[0053] Furthermore, developer 9 may use the configurations shown in Figure 3A and Figure 3B as appropriate. The information processing device 30 removes outliers, and the server device 10 creates the model. The server device 10 removes outliers, and the information processing device 30 creates the model. In this embodiment, unless otherwise specified, the description will be based on the configuration shown in Figure 3B (where the information processing device 30 processes the data independently).

[0054] <Example Hardware Configuration> Referring to Figure 4, the hardware configuration of the server device 10 and information processing device 30 included in the data processing system 100 according to this embodiment will be described.

[0055] <<Information Processing System and Information Processing Device 30>> Figure 4 shows the hardware configuration of an example of a server device 10 and information processing device 30 according to this embodiment. As shown in Figure 4, the server device 10 and information processing device 30 are built by a computer 500 and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0056] Of these, the CPU 501 controls the operation of the entire server device 10 and information processing device 30. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0057] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving a cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0058] <About the features> Next, the functional configuration of the information processing device 30 according to this embodiment will be explained in blocks using Figure 5. Figure 5 is a diagram showing an example of the functional configuration of the information processing device 30 according to this embodiment. Note that the lines between the functional blocks in Figure 5 indicate the relationships between them, but this does not mean that processing is limited to the order in which these lines connect.

[0059] The information processing device 30 includes a display control unit 31, an operation reception unit 32, a measurement error data detection unit 33, a distance calculation unit 34, a grid search unit 35, a quantile outlier detection unit 36, a local outlier detection unit 37, an exclusion unit 38, a model creation unit 39, an accuracy calculation unit 40, a verification unit 41, and a model selection unit 42. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program deployed in the RAM 503.

[0060] Furthermore, the information processing device 30 has a learning data storage unit 49. The learning data storage unit 49 is constructed using the HD504, RAM503, etc., shown in Figure 4. The learning data storage unit 49 stores a learning dataset. The learning dataset is assumed to be divided into input data and output data. In the case of a multiple regression model, the input data is called the explanatory variable and the output data is called the dependent variable. In this embodiment as well, the terms explanatory variable and dependent variable may be used, but the machine learning method is not limited to the multiple regression model. The learning data storage unit 49 may be located on a network accessible by the information processing device 30.

[0061] The display control unit 31 interprets the screen information held by the native application and displays it on the display 506. The operation reception unit 32 receives various operations from the developer 9 on the various screens displayed on the display 506.

[0062] The measurement error data detection unit 33 retrieves the training dataset stored in the training data storage unit 49 and detects measurement error data included in the training dataset. Measurement error data is assumed to be assigned a predetermined flag.

[0063] The distance calculation unit 34 calculates the distance between the training data (which may be either input data or output data) and reference data set as reference data, for example, in order to exclude outliers using quartiles.

[0064] The grid search unit 35 prepares all combinations of the specified hyperparameters. In this embodiment, the hyperparameters are the quantile Q or the number of neighboring sites N. Note that grid search is not mandatory.

[0065] The quantile outlier detection unit 36 ​​detects outliers (which may be in either the input or output data) from the training data, for example, based on quartiles. Furthermore, when a grid search is performed, the quantile outlier detection unit 36 ​​detects outliers from the training data based on the quantile Q determined by the grid search unit 35.

[0066] The local outlier detection unit 37 detects outliers (which may be in either the input or output data) from the training dataset, for example, by setting the number of neighboring sites N to 3. Furthermore, when a grid search is performed, the local outlier detection unit 37 detects outliers from the training dataset based on the number of neighboring sites N determined by the grid search unit 35. A local outlier is an outlier detected based on the data density in space. Local density is the reciprocal of the average distance between the point of interest and its k neighboring points. If there is an outlier point A, its local density will be low because the distance to its k neighboring points is large. The k points that are not outliers will have high local densities because their distances to their neighboring points are close. Therefore, in the case of point A, which is an outlier, its own local density is low, but the local densities of its k neighboring points are high, while points that are not outliers will have similar local densities to those of their neighboring points. Thus, a point is judged to be an outlier if the ratio of its own local density to the local densities of its neighboring points differs by more than a threshold.

[0067] The exclusion unit 38 excludes training data containing input data that are outliers detected by the quantile outlier detection unit 36 ​​or the local outlier detection unit 37 from the training dataset. The exclusion unit 38 also excludes training data containing input and output data that are outliers detected by the quantile outlier detection unit 36 ​​or the local outlier detection unit 37 from the training dataset. Note that exclusion does not require the complete deletion of the training data itself; it is sufficient that the data is not used in the creation of the model.

[0068] The model creation unit 39 creates a model for combination search using a training dataset from which outliers have been removed. The model may be created using various machine learning techniques. Machine learning is a technique that enables computers to acquire human-like learning abilities, in which a computer autonomously generates algorithms necessary for judgments such as data identification from pre-incorporated training data, and applies these algorithms to new data to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, or reinforcement learning (in this embodiment, supervised learning is mainly used), or a combination of these learning methods may be used, and the learning method for machine learning is not limited. Examples of machine learning techniques include perceptron, deep learning, support vector machine, logistic regression, naive Bayes, decision tree, and random forest, and are not limited to the techniques described in this embodiment.

[0069] For example, deep learning is an algorithm that predicts output data based on input data, and then adjusts the weights of a neural network using backpropagation to reduce the error with the training data. In the case of blending search, the developer prepares a training dataset containing multiple training datasets, where the blending amount of each material is the input data and the material properties (strength properties, physical properties, electrical properties, corrosion resistance, etc.) are the output data. The model creation unit 39 then learns the correspondence between the input data and the output data.

[0070] The accuracy calculation unit 40 calculates the accuracy of the model created by the model creation unit 39. The training dataset has been pre-divided into L parts, and a model is created using the training data excluding one of these parts (called the test data). The accuracy calculation unit 40 inputs the excluded test data into the created model and calculates the accuracy of the created model by comparing its output with the output data of the test data.

[0071] When the outlier removal unit 35 excludes outliers with different quantiles or neighborhood numbers, models are created for each learning dataset with different outliers removed. The accuracy calculation unit 40 calculates the accuracy for each of these models. Also, when cross-validation is performed, the accuracy calculation unit 40 calculates the accuracy for each model.

[0072] The verification unit 41 creates L models from the learning dataset by a method such as cross-validation and averages the output accuracy of each model. By averaging, the difference in model accuracy due to differences in the learning data is reduced.

[0073] The model selection unit 42 adopts the model with better accuracy among the model created by excluding the learning data where only the input data is an outlier and the model created by excluding the learning data where both the input data and the output data are outliers.

[0074] <Detection of Outliers by Quartiles> Referring to FIG. 6, supplementary explanation is given on the detection of outliers by quartiles, which is a comparative technique for this embodiment and is partly used. FIG. 6 is a diagram for explaining the detection method of the outlier 110 by quartiles. The quartiles are the quartiles (Q1, Q2, Q3) that evenly divide the data into four parts when the data is arranged in ascending order, and the data smaller than 1.5 times of IQR (Q3 - Q1) with respect to Q1 or larger than 1.5 times of IQR (Q3 - Q1) with respect to Q3 is regarded as the outlier 110.

[0075] Such a method can be said to be a method for detecting outliers 110 based on the data distribution. However, even if the learning data is excluded by this method, it is not always possible to exclude outliers that necessarily reduce the prediction accuracy of the model.

[0076] <Detection of Outliers by Leave One Out Method> Referring to Figure 7, we will supplement this explanation with information on outlier detection using the Leave One Out method, which is a comparative technique to this embodiment. Figure 7 is a diagram illustrating the outlier detection method using the Leave One Out method. The Leave One Out method excludes one of the training data as unknown data 112, and builds a model using the remaining training data. In Figure 7, there are eight training datasets. Therefore, eight models are created, and the accuracy of each model when unknown data 112 is input is compared. Unknown data 112 with poor accuracy is excluded as an outlier. The difference from cross-validation is that the Leave One Out method uses a much larger number of data divisions.

[0077] This method can be described as a technique that relies on the predictive accuracy of machine learning models. However, when the Leave One Out method is applied exhaustively to training datasets with a large amount of data, the number of models created tends to increase, leading to computational costs. Furthermore, outliers in the combination (explanatory variables) are not always excluded.

[0078] <Outline of the method for excluding outliers in this embodiment> The outline of the outlier exclusion method in this embodiment will be explained with reference to Figures 8A and 8B. Figures 8A and 8B illustrate outliers of the explanatory variables, outliers of the dependent variable, and the method for excluding outliers. First, Figure 8A illustrates the detection method for outliers of the explanatory variables and the dependent variable. The quantile outlier detection unit 36 ​​and the local outlier detection unit 37 detect outliers of the explanatory variables from the distribution of the input data, which are the explanatory variables. In Figure 8A, two outliers 121 and 122 were detected from the explanatory variable (combination).

[0079] As described above, it can be difficult to detect "prototypes (experimental formulations, formulations for comparison)" as outliers in the formulation system (explanatory variables). This is because there are many explanatory variables and the training dataset is sparse, resulting in a large spread of data in the search space and a non-normal distribution. In such cases, the following processing is effective. 1. Set a standard formula and calculate the distance from that formula (the definition of distance is arbitrary). 2. The larger outlier in the distance distribution is considered the outlier of the explanatory variable.

[0080] The quantiles used to identify outliers, and the number of neighboring sites N for local outliers, can also be automatically determined as hyperparameters using grid search.

[0081] Next, the quantile outlier detection unit 36 ​​and the local outlier detection unit 37 detect outliers of the target variable from the distribution of the output data, which is the target variable. This is to pick up cases where further exploration was stopped because the characteristics deteriorated. In this embodiment, outliers in the target variable alone are not excluded, so if the explanatory variables are also outliers, it is possible to pick up cases where further exploration was stopped because both the explanatory variables are outliers and the characteristics deteriorated.

[0082] Two outliers, 123 and 124, were detected from the dependent variable. Removing these outliers based on the dependent variable is expected to improve the model's accuracy.

[0083] Figure 8B illustrates the method for removing outliers detected in Figure 8A. The exclusion unit 38 removes the training data containing the explanatory variable outliers 121 and 122, and the model creation unit 39 creates a model. Similarly, the exclusion unit 38 removes the training data containing the outliers 121 and 123, which are outliers in both the explanatory and dependent variables, and the model creation unit 39 creates a model. The accuracy calculation unit 40 calculates the accuracy of these models, and the model selection unit 42 adopts the model with better accuracy.

[0084] Note that if all outliers in Figure 8B are excluded, the model's search domain may narrow (accuracy may improve, but predictions in the extrapolated domain may be weak). Therefore, it is preferable to exclude only outliers whose characteristics (dependent variable) are farther from the target value on the side of the characteristic's inferiority.

[0085] <Operating Procedure> The following describes the process of creating a model for blending exploration by excluding outliers, referring to Figures 9 to 11.

[0086] <Processing without using grid search> First, Figure 9 is a flowchart illustrating the overall model creation process. Hereafter, explanatory variables will be referred to as "input data," and the dependent variable as "output data."

[0087] First, the measurement error data detection unit 33 excludes training data from the training dataset that has been flagged or otherwise identified as measurement errors (S1).

[0088] The information processing device 30 performs a first exclusion process to remove outliers from the input data. The information processing device 30 also performs a first model creation process (S2) to create a model (an example of a first model) from the training dataset, which excludes the training data containing outlier input data. The details are explained in Figure 10.

[0089] Next, the validation unit 41 performs the process in step S2 multiple times to perform cross-validation (S3). That is, the validation unit 41 creates multiple models (an example of multiple first models) from the training dataset after excluding the training data containing outlier input data, by changing the test data that is not used to create the model. The validation unit 41 calculates the accuracy of each model calculated by the accuracy calculation unit 40.

[0090] Next, the information processing device 30 performs a second exclusion process to exclude training data where the input data is an outlier and the output data is also an outlier. The information processing device 30 then performs a second model creation process (S4) to create a model (an example of a second model) from the training dataset from which the training data where the input data is an outlier and the output data is also an outlier has been excluded. Details are explained in Figure 11.

[0091] The validation unit 41 performs the process in step S4 multiple times to perform cross-validation (S5). That is, the validation unit 41 creates multiple models (an example of multiple second models) from the training dataset after excluding training data where both the input data and output data are outliers, by changing the test data that is not used to create the model, and calculates the accuracy of each model.

[0092] The model selection unit 42 performs a model selection process (S6) to select the model with better prediction accuracy from among the models in step S2 and step S4. Prediction accuracy refers to the accuracy of the model on data other than the training data. Known indicators of the accuracy of regression models include MAE (mean absolute error), MAPE (mean absolute percentage error), WAPE (weighted absolute percentage error), MSE (mean squared error), or RMSE (root mean squared error).

[0093] Figure 10 is a flowchart illustrating the process of creating a model by excluding outliers from the input data in step S2 of Figure 9.

[0094] First, the input data is normalized to a fixed range of values ​​such as 0 to 1, or standardized to a value with a mean of "0" and a variance of "1" (S21).

[0095] Next, the distance calculation unit 34 acquires reference input data (S22). The reference input data can be set in advance by the developer. Alternatively, input data in which the output data (target value) represents the median may be automatically selected. Since the output data represents the median, it can be expected that its input data will also be distributed around the middle of several other input data.

[0096] Next, the distance calculation unit 34 calculates the distance from the reference input data for each input data (S23). The distance can be any distance capable of detecting outliers, such as the Euclidean distance or the Mahalanobis distance.

[0097] The quantile outlier detection unit 36 ​​detects outliers based on the distance in step S23, for example, by quartiles (an example of the first method), and the exclusion unit 38 excludes the training data containing the input data that is an outlier from the training dataset (S24). Quintiles, quintiles, or other numbers may be used instead of quartiles.

[0098] Instead of the processes described in steps S23 and S24 above, the local outlier detection unit 37 may detect outliers using the local outlier factor method (an example of a second method).

[0099] Next, the model creation unit 39 creates a model using a training dataset from which outliers have been removed by focusing on the input data (S25).

[0100] Figure 11 is a flowchart illustrating the process of creating a model by excluding outliers from the input and output data in step S4 of Figure 9.

[0101] First, it is determined whether the output data is univariate or not (S41). Univariate means that the output data has one element, while multivariate means that the output data has multiple elements.

[0102] In the case of a univariate (Yes in S41), the quantile outlier detection unit 36 ​​detects outliers in the output data, for example, using quartiles (an example of the fifth method) (S42). Quintiles or other numbers may be used instead of quartiles.

[0103] In the case of multivariate data (No. S41), the local outlier detection unit 37 detects outliers in the output data using the local outlier factor method (an example of the seventh method), for example, by setting the number of neighbors to 3 (S43). The number of neighbors may be 4 or other values ​​instead of 3. In the case of multivariate data (No. S41), the quantile outlier detection unit 36 ​​calculates the distance from the reference data for each data point and, based on this distance, may detect outliers, for example, by quartiles (an example of the sixth method).

[0104] The exclusion unit 38 excludes training data from the training dataset where the output data is an outlier and the input data is an outlier (S44). Then, the model creation unit 39 focuses on the input data and output data and creates a model using the training dataset from which the outliers have been excluded.

[0105] <Processing that uses grid search on input data> Next, referring to Figures 12 to 14, we will explain the process of determining the optimal quantiles for the input data using grid search and removing outliers. The overall flow of model creation is the same as in Figure 9.

[0106] Figure 12 is a flowchart illustrating the process of creating a model by excluding outlier input data in step S2 of Figure 9.

[0107] First, the input data is normalized to a fixed range of values ​​such as 0 to 1, or standardized to a value with a mean of "0" and a variance of "1" (S211).

[0108] Next, the grid search unit 35 selects one of two outlier detection methods (distance measurement from the reference, or local outlier factor method) (S212). The selection method may be predetermined by the developer, or both methods may be performed and the one that yields better model accuracy may be selected.

[0109] If distance measurement from a reference point (an example of the first method) is selected, the distance calculation unit 34 acquires the reference input data (S213). The reference input data can be set in advance by the developer. Alternatively, input data in which the output data (target value) represents the median may be automatically selected.

[0110] Next, the grid search unit 35 determines candidate quantiles q[1], q[2], q[3]...q[f] (S214). Candidate quantiles q[1], q[2], q[3]...q[f] are, for example, natural numbers such as 2, 3, 4, 5, etc.

[0111] The grid search unit 35 performs a grid search for the optimal quantile using candidate quantiles (S215). The details are explained in Figure 13.

[0112] The model creation unit 39 determines the model with the best accuracy among those created using different quantiles (S216). The accuracy metric can be the same as in step S6.

[0113] If the local outlier factor method (an example of the second method) is selected, the grid search unit 35 determines candidate neighbor site numbers n[1], n[2], n[3]...n[f] (S217). Candidate neighbor site numbers n[1], n[2], n[3]...n[f] are, for example, natural numbers such as 1, 2, 3, 4, 5, etc.

[0114] The grid search unit 35 uses candidate numbers of neighboring sites to perform a grid search to find the optimal number of neighboring sites (S218). The details are explained in Figure 14.

[0115] The model creation unit 39 determines the model with the best accuracy among those created using different numbers of neighboring sites (S219). The accuracy metric can be the same as in step S6.

[0116] Note that in the process shown in Figure 12, only the quantiles or the number of neighboring sites may be used for the grid search.

[0117] Figure 13 is a flowchart illustrating the process of grid-searching for the optimal quantile in step S215 of Figure 12.

[0118] The grid search unit 35 sets the candidate quantiles q[1], q[2], q[3]...q[f] as the quantile Q and repeats the following process for all candidates (S2151).

[0119] The distance calculation unit 34 calculates the distance from the reference input data for each input data (S2152). The distance can be any distance capable of detecting outliers, such as the Euclidean distance or the Mahalanobis distance.

[0120] The quantile outlier detection unit 36 ​​detects an outlier by quantile Q based on the distance in step S2152 (S2153).

[0121] The exclusion unit 38 excludes the training data containing this outlier (input data), and the model creation unit 39 creates the model (S2154).

[0122] The precision calculation unit 40 calculates the precision of the created model (S2155). The precision index can be the same as in step S6.

[0123] With the above steps, models are created for each of the candidate quantiles q[1], q[2], q[3]...q[f], and the accuracy of each model is calculated. The process returns to step S216 in Figure 12, where the model creation unit 39 determines the model with the best accuracy (optimal quantile).

[0124] Figure 14 is a flowchart illustrating the process of performing a grid search to find the optimal number of neighboring sites in step S218 of Figure 12.

[0125] The grid search unit 35 sets the number of neighboring sites N to candidates n[1], n[2], n[3]...n[f] for the optimal number of neighboring sites, and repeats the following process for all candidates (S2181).

[0126] The local outlier detection unit 37 detects outliers based on the number of neighboring sites N (S2182).

[0127] The exclusion unit 38 excludes the training data containing this outlier (input data), and the model creation unit 39 creates the model (S2183).

[0128] The precision calculation unit 40 calculates the precision of the created model (S2184). The precision index can be the same as in step S6.

[0129] With the above steps, models are created for each candidate number of neighboring sites n[1], n[2], n[3]...n[f], and the accuracy of each model is calculated. The process returns to step S219 in Figure 12, where the model creation unit 39 determines the model with the best accuracy (optimal number of neighboring sites).

[0130] The process shown in Figure 13 or Figure 14 creates an optimal model using the optimal quantiles or the optimal number of neighboring sites, so the process proceeds to step S3 in Figure 9, where cross-validation is performed. The process in step S4 can be the same as the process described in Figure 11, but as described below, it is also possible to perform a grid search on the output data.

[0131] <Processing that uses grid search for input and output data> Next, referring to Figures 15 to 17, we will explain the process of determining the optimal quantiles or number of neighboring sites for the input and output data using grid search and excluding outliers. The overall flow of model creation is the same as in Figure 9.

[0132] Figure 15 is a flowchart illustrating the process of creating a model by excluding training data where both the input and output data are outliers, as shown in step S4 of Figure 9.

[0133] First, it is determined whether the output data is univariate or not (S411).

[0134] If the output data is univariate (Yes in S411), the input data is normalized to a fixed range of values ​​such as 0 to 1, or standardized to a value with a mean of "0" and a variance of "1" (S412).

[0135] Next, the grid search unit 35 selects one of two outlier detection methods (distance measurement from the reference, or local outlier factorization) (S413). The selection method may be predetermined by the developer, or both methods may be performed and the one that yields better model accuracy may be selected.

[0136] If distance measurement from a reference point (an example of a third method) is selected, the distance calculation unit 34 acquires the reference input data (S414). The reference input data can be set in advance by the developer. Alternatively, input data in which the output data (target value) represents the median may be automatically selected.

[0137] Next, the grid search unit 35 determines candidate quantiles q[1], q[2], q[3]...q[f] (S415). Candidate quantiles q[1], q[2], q[3]...q[f] are, for example, natural numbers such as 2, 3, 4, 5, etc.

[0138] The grid search unit 35 performs a grid search to find the optimal quantile using the candidate quantiles (S416). The details are explained in Figure 16.

[0139] The model creation unit 39 determines the model with the best accuracy among those created using different quantiles (S417). The accuracy metric can be the same as in step S6.

[0140] If the local outlier factor method (an example of the fourth method) is selected, the grid search unit 35 determines candidate neighbor site numbers n[1], n[2], n[3]...n[f] (S418). Candidate neighbor site numbers n[1], n[2], n[3]...n[f] are, for example, natural numbers such as 1, 2, 3, 4, 5, etc.

[0141] The grid search unit 35 uses candidate numbers of neighboring sites to perform a grid search to find the optimal number of neighboring sites (S419). The details are explained in Figure 17.

[0142] The model creation unit 39 determines the model with the best accuracy among those created using different numbers of neighboring sites (S420). The accuracy metric can be the same as in step S6.

[0143] If the output data is determined to be multivariate in step S411 (No. in S411), the processing flow is the same as in the case of univariate data (S421). However, as methods for detecting outliers in the output data, distance measurement from the reference and local outlier factorization become selectable.

[0144] Note that in the process shown in Figure 15, only the quantiles or the number of neighboring sites may be used for the grid search.

[0145] Figure 16 is a flowchart illustrating the process of performing a grid search for the optimal quantile in step S416 of Figure 15.

[0146] The grid search unit 35 sets the candidate quantiles q[1], q[2], q[3]...q[f] as quantile Q and repeats the following process for all candidates (S4161).

[0147] The distance calculation unit 34 calculates the distance from the reference input data for each input data (S4162). The distance can be any distance capable of detecting outliers, such as the Euclidean distance or the Mahalanobis distance.

[0148] The quantile outlier detection unit 36 ​​detects an outlier by quantile Q based on the distance in step S4162 (S4163).

[0149] Similarly, the quantile outlier detection unit 36 ​​detects outliers based on quantile Q in the output data (univariate) (S4164).

[0150] The exclusion unit 38 excludes training data where the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S4165).

[0151] The precision calculation unit 40 calculates the precision of the created model (S4166). The precision index can be the same as in step S6.

[0152] With the above steps, models are created for each of the candidate quantiles q[1], q[2], q[3]...q[f], and the accuracy of each model is calculated. The process returns to step S417 in Figure 15, where the model creation unit 39 determines the model with the best accuracy (optimal quantile).

[0153] Figure 17 is a flowchart illustrating the process of performing a grid search to find the optimal number of neighboring sites in step S419 of Figure 15.

[0154] The grid search unit 35 sets candidate quantiles q[1], q[2], q[3]...q[f] as quantile Q and candidate neighbor site numbers n[1], n[2], n[3]...n[f] as neighbor site numbers N (S4191). The grid search unit 35 repeats the following process for all combinations of candidate quantiles and candidate neighbor site numbers (S4192, S4193).

[0155] The local outlier detection unit 37 detects outliers in the input data for a number of neighboring sites N (S4194).

[0156] The quantile outlier detection unit 36 ​​detects outliers in the output data based on quantile Q (S4195).

[0157] The exclusion unit 38 excludes training data where the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S4196).

[0158] The accuracy calculation unit 40 calculates the accuracy of the model created for the number of neighboring sites N and the number of quantiles Q (S4197). The accuracy index may be the same as in step S6.

[0159] A model is created for every possible combination of candidate numbers of neighboring sites and candidate numbers of quantiles, and the accuracy of each model is calculated. The process returns to step S420 in Figure 15, where the model creation unit 39 determines the model with the best accuracy (optimal quantiles, optimal number of neighboring sites).

[0160] The processing shown in Figure 16 or Figure 17 removes outliers with the optimal quantiles and number of neighboring sites, creating an optimal model. The process then proceeds to step S5 in Figure 9, where cross-validation is performed.

[0161] <<When the output data is multivariate>> Figure 18 is a flowchart illustrating the process of creating a model by excluding training data where both the input and output data are outliers, as shown in step S421 (multivariate output data) of Figure 15. Figure 18 is mainly used to explain the differences from steps S412 to S420 of Figure 15.

[0162] In the process shown in Figure 18, in step S4212, the grid search unit 35 selects an outlier detection method (distance measurement from a reference, local outlier factor method) for both the input data and the output data (S4212). Therefore, the processing from step S4212 onward branches into the following four cases. A. "Input data: distance from the reference point (example of the third method), Output data: distance from the reference point (example of the sixth method)" B. "Input data: Distance from a reference point (an example of the third method), Output data: Local outlier factorization (an example of the seventh method)" C. "Input data: Local outlier factorization (an example of the fourth method), Output data: Distance from the reference (an example of the sixth method)" D. "Input data: Local outlier factor analysis (an example of the fourth method), Output data: Local outlier factor analysis (an example of the seventh method)" A. If "Input data: distance from reference, Output data: distance from reference" is selected, the distance calculation unit 34 acquires the reference input data and output data (S4213A).

[0163] Next, the grid search unit 35 determines candidate quantiles q1[1], q1[2], q1[3]...q1[f] on the input data side, and candidate quantiles q2[1], q2[2], q2[3]...q2[f] on the output data side (S4214A).

[0164] The grid search unit 35 performs a grid search to find the optimal combination from all combinations of candidate quantiles on the input data side and candidate quantiles on the output data side (S4215A). The details are explained in Figure 19.

[0165] The model creation unit 39 determines the model with the best accuracy among the models created by excluding training data that have input data judged to be outliers based on the quantiles of the input data and output data that have output data judged to be outliers based on the quantiles of the output data (S4216A). The accuracy metric may be the same as in step S6.

[0166] B. If "Input data: distance from reference, output data: local outlier factor method" is selected, the distance calculation unit 34 acquires the reference input data (S4213B).

[0167] Next, the grid search unit 35 determines candidate quantiles q[1], q[2], q[3]...q[f] on the input data side, and candidate neighbor site numbers n[1], n[2], n[3]...n[f] on the output data side (S4214B).

[0168] The grid search unit 35 performs a grid search to find the optimal combination from all combinations of candidate quantiles on the input data side and candidate neighbor site counts on the output data side (S4215B). The details are explained in Figure 20.

[0169] The model creation unit 39 determines the model with the best accuracy among the models created by excluding training data that have input data judged to be outliers based on the quantiles of the input data and output data that have output data judged to be outliers based on the number of neighboring sites of the output data (S4216B). The accuracy metric may be the same as in step S6.

[0170] C. If "Input data: Local outlier factor method, Output data: Distance from reference" is selected, the distance calculation unit 34 acquires the reference output data (S4213C).

[0171] Next, the grid search unit 35 determines candidate neighbor site numbers n[1], n[2], n[3]...n[f] on the input data side, and candidate quantile numbers q[1], q[2], q[3]...q[f] on the output data side (S4214C).

[0172] The grid search unit 35 performs a grid search to find the optimal combination for all combinations of candidate neighbor site counts on the input data side and candidate quantile counts on the output data side (S4215C). The details are explained in Figure 21.

[0173] The model creation unit 39 determines the model with the best accuracy among the models created by excluding training data that have input data judged to be outliers based on the number of neighboring sites on the input data side, and output data that have output data judged to be outliers based on the quantiles on the output data side (S4216C). The accuracy metric may be the same as in step S6.

[0174] D. If "Input data: Local outlier factor method, Output data: Local outlier factor method" is selected, the grid search unit 35 determines candidate neighbor site numbers n1[1], n1[2], n1[3]...n1[f] on the input data side, and candidate neighbor site numbers n2[1], n2[2], n2[3]...n2[f] on the output data side (S4214D).

[0175] Next, the grid search unit 35 performs a grid search to find the optimal combination for all combinations of candidate neighbor site counts on the input data side and candidate neighbor site counts on the output data side (S4215D). The details are explained in Figure 22.

[0176] The model creation unit 39 determines the model with the best accuracy among the models created by excluding training data that have input data judged to be outliers based on the number of neighboring sites on the input data side, and output data that have output data judged to be outliers based on the number of neighboring sites on the output data side (S4216D). The accuracy metric may be the same as in step S6.

[0177] Note that in Figure 18, grid searches do not necessarily have to be performed in steps S4215A, S4215B, S4215C, and S4215D, or grid searches may be performed in only one or more of these steps.

[0178] Figure 19 is a flowchart illustrating the process of performing a grid search for the optimal quantiles for the input and output data in step S4215A of Figure 18.

[0179] The grid search unit 35 sets candidate quantiles q1[1], q1[2], q1[3]...q1[f] from the input data as quantile Q1, and sets candidate quantiles q2[1], q2[2], q2[3]...q2[f] from the output data as quantile Q2 (S42151A). The grid search unit 35 repeats the following processing for all combinations of Q1 and Q2 (S42152A, S42153A).

[0180] The distance calculation unit 34 calculates the distance from the reference data for each input data (S42154A). The distance can be any distance capable of detecting outliers, such as the Euclidean distance or the Mahalanobis distance.

[0181] The quantile outlier detection unit 36 ​​detects an outlier based on quantile Q1 based on the distance in step S42154A (S42155A).

[0182] The same applies to steps S42156A and S42157A.

[0183] The exclusion unit 38 excludes training data where the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158A).

[0184] The precision calculation unit 40 calculates the precision of the model created for quantiles Q1 and Q2 (S42159A). The precision index can be the same as in step S6.

[0185] With the above steps, a model is created for every possible combination of candidate quantiles from the input data and candidate quantiles from the output data, and the accuracy of each model is calculated. The process returns to step S4216A in Figure 18, where the model creation unit 39 determines the model with the best accuracy (the optimal quantiles for the input and output data).

[0186] Figure 20 is a flowchart illustrating the process of performing a grid search to find the optimal quantile on the input data side and the optimal number of neighboring sites on the output data side in step S4215B of Figure 18.

[0187] The grid search unit 35 sets candidate quantiles q[1], q[2], q[3]...q[f] on the input data side as quantile Q, and candidate neighbor site numbers n[1], n[2], n[3]...n[f] on the output data side as neighbor site number N (S42151B). The grid search unit 35 repeats the following processing for all combinations of Q and N (S42152B, S42153B).

[0188] The distance calculation unit 34 calculates the distance from the reference data for each input data (S42154B). The distance can be any distance capable of detecting outliers, such as the Euclidean distance or the Mahalanobis distance.

[0189] The quantile outlier detection unit 36 ​​detects an outlier by quantile Q based on the distance in step S42154B (S42155B).

[0190] Similarly, the local outlier detection unit 37 detects outliers in the output data (multivariate) using the local outlier factor method based on the number of neighboring sites N (S42156B).

[0191] The exclusion unit 38 excludes training data where the input data is an outlier and the output data is also an outlier, and the model creation unit 39 creates a model (S42158B).

[0192] The accuracy calculation unit 40 calculates the accuracy of the model created for a given quantile Q and number of neighboring sites N (S42159B). The accuracy index can be the same as in step S6.

[0193] With the above steps, a model is created for every possible combination of candidate quantiles and candidate neighbor site numbers, and the accuracy of each model is calculated. The process returns to step S4216B in Figure 18, where the model creation unit 39 determines the model with the best accuracy (optimal quantiles for the input data and optimal neighbor site numbers for the output data).

[0194] Figure 21 is a flowchart illustrating the process of performing a grid search to find the optimal number of neighboring sites on the input data side and the optimal quantiles on the output data side in step S4215C of Figure 18.

[0195] Next, the grid search unit 35 sets the candidate neighbor site numbers n[1], n[2], n[3]...n[f] on the input data side to the number of neighbor sites N, and sets the candidate quantile numbers q[1], q[2], q[3]...q[f] on the output data side to the quantile number Q (S42151C). The grid search unit 35 repeats the following process for all combinations of N and Q (S42152C, S42153C).

[0196] The local outlier detection unit 37 detects outliers in the input data using a local outlier factor method based on the number of neighboring sites N (S42154C).

[0197] The distance calculation unit 34 calculates the distance from the reference data for each output data (S42155C). The distance can be any distance capable of detecting outliers, such as the Euclidean distance or the Mahalanobis distance.

[0198] The quantile outlier detection unit 36 ​​detects an outlier by quantile Q based on the distance in step S42155C (S42156C).

[0199] The exclusion unit 38 excludes training data where the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158C).

[0200] The accuracy calculation unit 40 calculates the accuracy of the model created for the number of neighboring sites N and the number of quantiles Q (S42159C). The accuracy index can be the same as in step S6.

[0201] With the above steps, a model is created for every possible combination of candidate neighbor site numbers and candidate quantiles, and the accuracy of each model is calculated. The process returns to step S4216C in Figure 18, where the model creation unit 39 determines the model with the best accuracy (optimal neighbor site numbers for the input data and optimal quantiles for the output data).

[0202] Figure 22 is a flowchart illustrating the process of performing a grid search for the optimal number of neighboring sites for the input and output data in step S4215D of Figure 18.

[0203] The grid search unit 35 sets candidate neighbor site counts n1[1], n1[2], n1[3]...n1[f] as neighbor site count N1, and candidate neighbor site counts n2[1], n2[2], n2[3]...n2[f] as neighbor site count N2 (S42151D). The grid search unit 35 repeats the following process for all combinations of N1 and N2 (S42152D, S42153D).

[0204] The local outlier detection unit 37 detects outliers in the input data using a local outlier factor method based on the number of neighboring sites N1 (S42154D).

[0205] The local outlier detection unit 37 detects outliers in the output data using a local outlier factor method based on the number of neighboring sites N2 (S42155D).

[0206] The exclusion unit 38 excludes training data where the input data is an outlier and the output data is an outlier, and the model creation unit 39 creates a model (S42158D).

[0207] The accuracy calculation unit 40 calculates the accuracy of the model created for the number of neighboring sites N1 and N2 (S42159D). The accuracy index can be the same as in step S6.

[0208] With the above steps, a model is created for every possible combination of candidate neighbor site counts for the input data and the output data, and the accuracy of each model is calculated. The process returns to step S4216D in Figure 18, where the model creation unit 39 determines the model with the best accuracy (the optimal number of neighbor sites for the input and output data).

[0209] Through one of the processes shown in Figures 19 to 22, outliers are removed using the optimal quantiles or number of neighboring sites on both the input and output data sides, resulting in the creation of an optimal model. The process then proceeds to step S5 in Figure 9, where cross-validation is performed.

[0210] <Supplementary information regarding combinations of methods for excluding outliers> Figures 23A and 23B show examples of various method combinations in the first and second exclusion processes, respectively. Figure 23A shows a method for excluding outliers from the input data in the first exclusion process. Multiple methods are available for excluding outliers from the input data, and any one of them may be adopted. Note that outlier exclusion by quantiles (first method) and outlier exclusion by local outlier factorization (second method) are just examples.

[0211] Figure 23B shows combinations of methods for excluding outliers from input and output data in the second exclusion process. First, there are several methods for excluding outliers from input data, and any one of them can be arbitrarily adopted.

[0212] In this embodiment, when the output data is univariate, outlier exclusion using quantiles is given as an example of a method for excluding outliers from the output data. Note that outlier exclusion using quantiles (third method) is only one example. Therefore, outliers in both input and output data can be excluded by combining either the third or fourth method with the fifth method. Furthermore, when the output data is univariate, multiple methods for excluding outliers may be provided.

[0213] When the output data is multivariate, this embodiment illustrates outlier exclusion using quantiles and outlier exclusion using the local outlier factor method as examples of methods for removing outliers from the output data. Note that outlier exclusion using quantiles (sixth method) and outlier exclusion using the local outlier factor method (seventh method) are merely examples. Therefore, outliers in both input and output data can be removed by combining either the third or fourth method with either the sixth or seventh method.

[0214] <Main effects> • Because the training dataset (explanatory variables) of this embodiment is sparse, the information processing device makes it easier to exclude outliers even when it is difficult to apply general outlier detection methods. Therefore, it is easy to exclude data measured on prototypes, etc. • Since training data where both the dependent and independent variables are outliers can be excluded, cases where measurements were stopped because the characteristics deteriorated due to experimental formulations can be detected (because it is a prototype, the independent variable is an outlier, and because the characteristics are poor, the dependent variable is also an outlier). • The information processing device creates a model and verifies its accuracy by excluding outliers, so it can exclude outliers in the explanatory variables if they are detrimental to the model's prediction accuracy. • By creating a model that excludes training data where only the explanatory variables are outliers, and by creating a model that excludes training data where both the explanatory and dependent variables are outliers, the model with better accuracy can be adopted. Therefore, from an accuracy standpoint, the model created by excluding outliers can be adopted. Furthermore, because the number of models created is not very large, outliers can be detected even when the training dataset has a large amount of data.

[0215] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0216] For example, in this embodiment, we have described an example of detecting outliers from a training dataset of a model mainly used for formulation search, but this is not limited to such material informatics; the training dataset from which outliers are excluded can include input and output data used for machine learning.

[0217] Furthermore, the exclusion of outliers using quartiles or local outlier factorization is just one example; outliers may also be detected using standard deviation, Smirnov-Grubbs test, or Hotelling's theory for univariate data, or standard deviation, cluster analysis, or Hotelling's T2 method for multivariate data. In other words, the methods for detecting outliers are not limited to two (it is preferable to have at least two or more), and any of several different methods may be used.

[0218] In the case of a client-server architecture, the user may connect the information processing device 30 to the server device 10 and request the creation of a model, at which point the server device 10 may create the model and send it to the information processing device 30. The server device 10 may also keep the model within itself; in this case, when explanatory variables are sent from the information processing device 30, the server device 10 returns the target variable to the information processing device 30. The information processing device 30 may, as appropriate, execute a web application provided by the server device 10.

[0219] Furthermore, the configuration examples shown in Figure 5 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the information processing device 30 and the server device 10. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the information processing device 30 and the server device 10 can be further divided into many more processing units depending on the processing content. Also, one processing unit can be divided to include even more processing.

[0220] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above.

[0221] This application claims priority based on Japanese Patent Application No. 2023-134755, filed with the Japan Patent Office on August 22, 2023, and the entire contents of Japanese Patent Application No. 2023-134755 are incorporated herein by reference. [Explanation of Symbols]

[0222] 10 Server devices 30 Information Processing Devices 100 Data Processing Systems

Claims

1. A model creation method in which an information processing device creates a model that maps input data included in training data to output data, A first exclusion process removes training data where the input data is an outlier from the training dataset, A second exclusion process that excludes training data from the training dataset in which the input data and output data are outliers, A first model creation process, which creates a first model using the training dataset from which the training data has been excluded in the first exclusion process, A second model creation process, which creates a second model using the training dataset from which the training data has been excluded in the second exclusion process, A model selection process that adopts the model with better prediction accuracy from among the first model and the second model, A method for creating a model that performs this task.

2. The model creation method according to claim 1, wherein the first exclusion process excludes outliers from the input data using one of several different methods.

3. The model creation method according to claim 2, wherein the first exclusion process excludes outliers from the input data using at least two different first or second methods.

4. In the second exclusion process, outliers in the input data are excluded using one of several different methods. The model creation method according to any one of claims 1 to 3, wherein if the output data is univariate, outliers in the output data are removed by a predetermined method.

5. The second exclusion process excludes outliers from the input data using at least two different third or fourth methods, and if the output data is univariate, excludes outliers from the output data using a fifth method. In the second exclusion process, outliers in the input data are excluded using the third method, and outliers in the output data are excluded using the fifth method, or, The model creation method according to claim 4, wherein outliers in the input data are removed by the fourth method and outliers in the output data are removed by the fifth method.

6. In the second exclusion process, outliers in the input data are excluded using one of several different methods. The model creation method according to any one of claims 1 to 3, wherein, if the output data is multivariate, outliers in the output data are excluded using one of several different methods.

7. The second exclusion process excludes outliers from the input data using at least two different third or fourth methods, and if the output data is multivariate, excludes outliers from the output data using at least two different sixth or seventh methods. In the second exclusion process, outliers in the input data are excluded using the third method, and outliers in the output data are excluded using the sixth method, or, The outliers in the input data are removed using the fourth method, and the outliers in the output data are removed using the sixth method, or The third method described above removes outliers from the input data, and the seventh method described above removes outliers from the output data, or, The model creation method according to claim 6, wherein outliers in the input data are removed by the fourth method and outliers in the output data are removed by the seventh method.

8. The first method described above is a method for detecting outliers based on the quantile, using the distance of each data point from a reference data point. The model creation method according to claim 3, wherein the second method is a method for detecting outliers by local outlier factorization.

9. The third method described above is a method for detecting outliers based on the distance of each data point from a reference data point, using quantiles. The fourth method is a method for detecting outliers using the local outlier factorization method, The model creation method according to claim 5, wherein the fifth method is a method for detecting outliers using quantiles.

10. The third and sixth methods described above are methods for detecting outliers based on the distance of each data point from a reference data point, using quantiles. The model creation method according to claim 7, wherein the fourth and seventh methods are methods for detecting outliers using the local outlier factor method.

11. In the first exclusion process, the quantile, which is a hyperparameter of the first method, is determined by grid search, or The model creation method according to claim 8, wherein the number of neighboring sites, which is a hyperparameter of the second method, is determined by grid search.

12. In the second exclusion process, at least one of the quantiles, which are hyperparameters of the third method, or the quantiles, which are hyperparameters of the fifth method, is determined by grid search, or The model creation method according to claim 9, wherein at least one of the number of neighboring sites, which is a hyperparameter of the fourth method, or the number of quantiles, which is a hyperparameter of the fifth method, is determined by grid search.

13. In the second exclusion process, at least one of the quantiles, which are hyperparameters of the third method, or the quantiles, which are hyperparameters of the sixth method, is determined by grid search, or The quantile, which is a hyperparameter of the third method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search, or The number of neighboring sites, which is a hyperparameter of the fourth method, or the quantile, which is a hyperparameter of the sixth method, is determined by grid search, or The model creation method according to claim 10, wherein at least one of the number of neighboring sites, which is a hyperparameter of the fourth method, or the number of neighboring sites, which is a hyperparameter of the seventh method, is determined by grid search.

14. An information processing device that creates a model that maps input data included in training data to output data, A first exclusion process removes training data where the input data is an outlier from the training dataset, A second exclusion process that excludes training data from the training dataset in which the input data and output data are outliers, A first model creation process, which creates a first model using the training dataset from which the training data has been excluded in the first exclusion process, A second model creation process, which creates a second model using the training dataset from which the training data has been excluded in the second exclusion process, A model selection process that adopts the model with better prediction accuracy from among the first model and the second model, A program that executes the command.

15. An information processing device that creates a model that maps input data included in training data to output data, An exclusion unit that excludes training data in which the input data is an outlier from the training dataset, and excludes training data in which both the input data and the output data are outliers from the training dataset, A model creation unit creates a first model using the training dataset from which training data where the input data are outliers have been excluded, and creates a second model using the training dataset from which training data where the input data and output data are outliers have been excluded. A model selection unit that adopts the model with better prediction accuracy from among the first model and the second model, An information processing device having

Images