Information processing method, information processing device, and program

Abstract

The invention provides an information processing method, an information processing device, and a program for extracting a subset having domain generalization. An information processing method executed by one or more processors, the information processing method including the following processes performed by the one or more processors: extracting a subset of a data set of an analysis object under a specified condition; and evaluating the field generalization of the extracted subsets. In a classification model or a prediction model learned by using the subset, the domain generalization representation of the subset shows that the reduction of classification performance or prediction performance is relatively small due to the change of joint probability distribution of an explanatory variable and a target variable caused by the change of various conditions in a data generation process; or the classification performance or prediction precision at the moment is high.

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing device and a program, and in particular to a technology for constructing a model with domain generalization. Background Art

[0002] Classification models or prediction models based on data are widely used. For example, disease classification based on a medical image is known.

[0003] However, a model learned using data from a certain hospital facility may not achieve the expected accuracy when imported into another facility. This is because there are often cases of domain shifts where data distribution differs between the learning facility and the importing facility.

[0004] Research on improving robustness to domain shift is called domain generalization, and has been actively conducted in recent years. There are known methods for learning invariant feature quantity expressions regardless of domains and meta-learning methods for learning evaluation performance after simulating domain shifts.

[0005] For example, a method for selecting domain-wide feature quantities is proposed in non-patent document 1. Specifically, for learning data in multiple fields, the correlation coefficient between each feature quantity (explanatory variable) and the target variable is calculated for each field, and the feature quantity whose absolute value of the correlation coefficient is above a threshold in all fields in the learning data is selected.

[0006] Previous technical literature

[0007] Non-patent literature

[0008] Non-patent literature 1: Vikas Garg, Adam Tauman Kalai, Katrina Ligett, Steven Wu "Learn to Expect the Unexpected: Probably Approximately Correct DomainGeneralization" (2021AISTATS)

[0009] Non-patent document 2: Ivan Cantador, Ignacio Fenandez-Tobias, Shlomo Bwrkovsky, Paolo Cremonesi, Chapter 27: “Cross-domain Recommender System” (2015 Springer)

[0010] Non-patent document 3: G Chandrashekar, F Sahin, Computers & Electrical Engineering, “A survey on feature selection methods” (2014 Elsevier)

[0011] Non-patent document 4: Y Saeys, I Inza, P Larranaga "A review of feature selection techniques in bioinformatics" (bioinformatics, 2007) Summary of the invention

[0012] Technical issues to be solved by the invention

[0013] The method of Non-Patent Document 1 is based on the premise that a classification model or prediction model with domain generalization can be constructed for the entire data. However, in reality, this is often impossible and there is a problem of performance degradation.

[0014] The present invention has been made in view of such circumstances, and an object of the present invention is to provide an information processing method, an information processing device, and a program for extracting a subset having domain generalization.

[0015] Means for solving technical problems

[0016] In order to achieve the above-mentioned purpose, the information processing method involved in the first embodiment of the present invention is executed by one or more processors, and the information processing method includes the following processing performed by one or more processors: extracting a subset from the data set of the analysis object under specified conditions; and evaluating the domain generalization of the extracted subset. The domain generalization of a subset means that in a classification model or prediction model (classification / prediction model) learned using a subset, the change (domain shift) of the joint probability distribution (joint distribution) of the explanatory variable and the target variable caused by the change of various conditions in the data generation process makes the decrease in classification performance or prediction performance relatively small, or the classification performance or prediction accuracy during the domain shift is relatively high. According to this embodiment, a subset with domain generalization can be extracted, so a classification / prediction model with domain generalization can be constructed.

[0017] The information processing method involved in the second embodiment of the present invention is preferably the information processing method involved in the first embodiment, wherein the data set includes data sets of multiple fields, and the evaluation includes the following processing performed by one or more processors: extracting a learning subset from a data set of one or more learning fields among the multiple fields under specified conditions, and using the learning subset to learn the learning model; extracting an evaluation subset from a data set of one or more evaluation fields different from the learning fields among the multiple fields under specified conditions, and evaluating the learning model using the evaluation subset; and evaluating the domain generalization using the evaluation result of the learning model.

[0018] The information processing method involved in the third embodiment of the present invention is preferably an information processing method involved in the first embodiment or the second embodiment, wherein the data set includes data sets of multiple fields, and the evaluation includes the following processing performed by one or more processors: extracting a learning subset from a data set of one or more learning fields among the multiple fields under specified conditions, and using the learning subset to learn the learning model; extracting a first evaluation subset different from the learning subset from the data set of the learning field under specified conditions, and evaluating the learning model using the first evaluation subset; extracting a second evaluation subset from a data set of one or more evaluation fields different from the learning field among the multiple fields under specified conditions, and evaluating the learning model using the second evaluation subset; and evaluating the domain generalization using the difference between the evaluation result based on the first evaluation subset and the evaluation result based on the second evaluation subset.

[0019] The information processing method involved in the 4th aspect of the present invention is preferably an information processing method involved in any one of the 1st to 3rd aspects, wherein the data set includes data sets of multiple fields, and the evaluation includes the following processing performed by one or more processors: evaluating the correlation between the feature quantity of the subset and the target variable according to each field; and in multiple fields, the higher the correlation is relatively, the higher the domain universality of the feature quantity is relatively, and the more feature quantities of the subset with relatively high domain universality have a relatively higher evaluation of the domain generalization.

[0020] The information processing method involved in the 5th aspect of the present invention is preferably an information processing method involved in any one of the 1st to 4th aspects, wherein the data set includes data sets of multiple fields, and the evaluation includes the following processing performed by one or more processors: evaluating the correlation between the feature quantity of the subset and the target variable according to each field; and setting the feature quantity having a correlation above a specified threshold in more than a specified number of fields as a feature quantity with relatively high field universality, and using the number of feature quantities with relatively high field universality to evaluate the field generalization of the subset.

[0021] The information processing method involved in the 6th aspect of the present invention is preferably an information processing method involved in any one of the 1st to 5th aspects, wherein the data set includes data sets of multiple fields, and the evaluation includes the following processing performed by one or more processors: extracting a learning subset from a data set of one or more learning fields among the multiple fields under specified conditions, and extracting a feature value set from the learning subset; extracting an evaluation subset from a data set of one or more evaluation fields different from the learning field among the multiple fields under specified conditions, and evaluating the extracted feature value set using the evaluation subset; and evaluating the domain generalization of the learning subset using the proportion of feature values ​​in the extracted feature value set that are also valid in the evaluation subset.

[0022] The information processing method involved in the 7th embodiment of the present invention is preferably the information processing method involved in the 6th embodiment, which includes the following processing performed by one or more processors: evaluating the correlation between the feature quantity of the learning subset and the target variable according to each field; and in multiple fields, setting the higher the correlation is, the higher the field universality of the feature quantity is, and the extracted feature quantity set includes feature quantities with relatively high field universality.

[0023] The information processing method involved in the 8th embodiment of the present invention is preferably the information processing method involved in the 6th embodiment, which includes the following processing performed by one or more processors: evaluating the correlation between the feature quantity of the learning subset and the target variable according to each field; and setting the feature quantity having a correlation above a specified threshold in more than a specified number of fields as a feature quantity with relatively high field universality, and the extracted feature quantity set includes feature quantities with relatively high field universality.

[0024] The information processing method involved in the 9th aspect of the present invention is preferably an information processing method involved in any one of the 1st to 8th aspects, wherein the evaluation includes the following processing performed by one or more processors: evaluating the usefulness of the subset based on known usefulness information for each sample in the data set and the samples included in the subset; and combining the usefulness and domain generalization as an evaluation of the subset.

[0025] The information processing method involved in the tenth aspect of the present invention is preferably an information processing method involved in any one of the first to ninth aspects, wherein the data set includes data sets in multiple fields, and the evaluation includes the following processing performed by one or more processors: evaluating the domain uniformity representing the degree of distribution proximity between the number of data in the data set in each field and the number of data in the subset; and combining the domain uniformity and the domain generalization as an evaluation of the subset.

[0026] The information processing method involved in the 11th mode of the present invention is preferably an information processing method involved in any one of the 1st to 10th modes, wherein one or more processors learn a subset classification model that classifies whether data in a data set is a subset.

[0027] The information processing method involved in the 12th embodiment of the present invention is preferably the information processing method involved in the 11th embodiment, wherein the evaluation includes the following processing performed by one or more processors: evaluating the subset classification performance of the subset classification model; and combining the subset classification performance with the domain generalization as an evaluation of the subset.

[0028] The information processing method according to the 13th aspect of the present invention is preferably the information processing method according to any one of the 1st to 12th aspects, wherein the extraction and evaluation include the following processing performed by one or more processors: searching for a subset with higher domain generalization by repeatedly adding or deleting samples from the starting subset. The extraction may also include the processing of repeatedly adding or deleting samples from the starting subset, and the evaluation may also include the processing of searching for a subset with higher domain generalization.

[0029] The information processing method involved in the 14th embodiment of the present invention is preferably an information processing method involved in the 13th embodiment, wherein the data set includes data sets in multiple fields, and the search includes a search performed by one or more processors using the following evaluation method: the usefulness of the subset evaluated based on known usefulness information for each sample in the data set and the samples included in the subset, the field uniformity indicating the degree of distribution proximity between the number of data in the data set in each field and the number of data in the subset, and the subset classification performance of a subset classification model for classifying whether the data in the data set is a subset are also evaluated.

[0030] The information processing method involved in the 15th mode of the present invention is preferably an information processing method involved in any one of the 1st to 14th modes, which includes the following processing performed by one or more processors: prompting multiple different subset conditions; evaluating subsets extracted respectively with multiple different subset conditions; and extracting subsets with the subset condition with the best evaluation result among the multiple different subset conditions.

[0031] In order to achieve the above-mentioned object, the information processing device involved in the 16th aspect of the present invention comprises: one or more processors; and one or more memories storing commands for the one or more processors to execute, wherein the one or more processors extract subsets of the data set of the analysis object under specified conditions and evaluate the domain generalization of the extracted subsets. According to this aspect, a subset with domain generalization can be extracted, so a classification / prediction model with domain generalization can be constructed.

[0032] In order to achieve the above-mentioned object, the program involved in the 17th mode of the present invention is a program that enables a computer to realize the following functions: a function of extracting a subset of a data set of an analysis object under specified conditions; and a function of evaluating the domain generalization of the extracted subset. According to this mode, a subset with domain generalization can be extracted, so a classification / prediction model with domain generalization can be constructed.

[0033] Effects of the Invention

[0034] According to the present invention, a subset having domain generalization can be extracted. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an explanatory diagram showing the introduction process of the classification / prediction system.

[0036] Figure 2 This is an explanatory diagram for performing learning of a model based on domain adaptation.

[0037] Figure 3 This is an explanatory diagram showing examples of learning data and evaluation data used in machine learning.

[0038] Figure 4 This is a diagram schematically showing how the performance of the model varies depending on the data set.

[0039] Figure 5 This is a block diagram schematically showing an example of the hardware configuration of the information processing device according to the embodiment.

[0040] Figure 6 This is a functional block diagram showing the functional structure of the information processing device.

[0041] Figure 7 This is an explanatory diagram showing the classification or prediction involved in Implementation Example 1.

[0042] Figure 8 This is an explanatory diagram showing the processing involved in Implementation Example 2.

[0043] Fig. 9 This is an explanatory diagram showing the processing involved in Implementation Example 4.

[0044] Fig.10 This is an explanatory diagram showing the processing involved in Implementation Example 5.

[0045] Fig.11 This is an explanatory diagram showing the processing involved in Implementation Example 6.

[0046] Fig.12 is a diagram used to illustrate the usefulness of subsets.

[0047] Fig.13 This is an explanatory diagram showing the processing involved in Implementation Example 7.

[0048] Fig.14 This is an explanatory diagram showing the processing involved in Implementation Example 8.

[0049] Fig.15 This is an explanatory diagram showing an algorithm for extracting a subset according to the ninth embodiment.

[0050] Fig.16 It is a table showing an example of a data set.

[0051] Fig.17 Yes means Fig.16 The data set shown is a table of an example of a data set that is abstracted.

[0052] Fig.18 This is a table showing other examples of abstracted data sets. DETAILED DESCRIPTION

[0053] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0054] 〔Construction and application of classification / prediction models〕

[0055] Figure 1 This is an explanatory diagram showing the introduction process of the classification system or prediction system (classification / prediction system).

[0056] Therein, a typical process for importing a classification / prediction system into a facility is shown. Regarding the import of the classification / prediction system, first, a model 14 is constructed, and the model 14 performs a target classification task or prediction task (classification / prediction task) (step 1), and then the constructed model 14 is imported and applied (step 2). In the case of a machine learning model, "constructing" the model 14 includes learning the model 14 using learning (training) data, and creating a classification / prediction model that meets a practical level of classification performance or prediction performance. "Applying" the model 14 means, for example, obtaining an output of the employee's turnover risk from the learned model 14 when a data set related to an employee is input.

[0057] The construction of model 14 requires data for learning. Model 14 of a general classification / prediction system is learned based on data collected at the facility at the import destination. By learning using data collected from the facility at the import destination, model 14 learns the behavior of employees at the facility at the import destination, and can make a high-precision prediction of the turnover risk of employees at the facility at the import destination.

[0058] However, due to various circumstances, the data of the facility at the import destination may not be available. For example, in the case of a document information classification system in a company's internal system or a hospital's in-house system, the company that develops the classification model often cannot access the data of the facility at the import destination. In the case where the data of the facility at the import destination cannot be obtained, it is necessary to learn from the data collected at a different facility instead.

[0059] Figure 1 Indicates the classification process or prediction process when the data of the facility at the import destination cannot be obtained. If the model 14 learned using data collected at a facility different from the facility at the import destination is used in the facility at the import destination, there is a problem that the prediction accuracy of the model 14 decreases due to differences in user behavior between facilities.

[0060] The problem that the machine learning model cannot function well in other unknown facilities that are different from the facility where learning is completed can be broadly understood as the following technical topic: improving robustness against the problem of domain offset where the source domain where the model 14 is learned is different from the target domain to which the model 14 is applied. Domain adaptation is a setting for problems related to domain generalization. This is a method of learning using data from two domains, the source domain and the target domain. Although there is data in the target domain, the purpose of using data from a different domain is to compensate for the part where the target domain has a small amount of data and is insufficient for learning.

[0061] Figure 2 This is an explanatory diagram of the learning of the model 14 based on domain adaptation. Although the amount of data collected at the facility of the import destination in the target domain is relatively smaller than the amount of data collected at different facilities, by using the data of both for learning, the model 14 can also predict the behavior of users of the facility of the import destination with a certain degree of accuracy.

[0062] Since data from the target domain cannot be accessed during learning, domain generalization is a more difficult problem than domain adaptation.

[0063] 〔Explanation of the field〕

[0064] In Non-Patent Document 2, which is a document related to the study of domain adaptation in information recommendation, differences in domains are classified into the following four types.

[0065] [1] Item attribute level: For example, comedy movies and horror movies are different categories.

[0066] [2] Item type level: For example, movies and TV series are different fields.

[0067] [3] Item level: For example, movies and books are different fields.

[0068] [4] System level: For example, movies shown in cinemas and movies shown on television are in different fields.

[0069] Figure 1 and Figure 2 The differences in “facilities” shown correspond to the areas of the [4] system level within the 4 types described above.

[0070] The domain is defined by the joint probability distribution P(X, Y) of the target variable Y and the explanatory variable X. When Pd1(X, Y)≠Pd2(X, Y), d1 and d2 are different domains. That is, domain shift occurs.

[0071] The joint probability distribution P(X,Y) can be represented by the product of the distribution of the explanatory variable P(X) and the conditional probability distribution P(Y|X), or the product of the distribution of the target variable P(Y) and the conditional probability distribution P(Y|X).

[0072] P(X,Y)=P(Y|X)P(X)=P(X|Y)P(Y)

[0073] Therefore, if one or more of P(X), P(Y), P(Y|X), and P(X|Y) changes, different domains are formed.

[0074] [Reasons for the influence of domain shift]

[0075] The classification / prediction model for the task of prediction or classification makes inferences based on the correlation between the explanatory variable X and the target variable Y, so as long as P(Y|X) changes, the classification performance or prediction performance will naturally decrease. In addition, when the classification / prediction model is machine-learned, the classification error or prediction error in the learning data is minimized. For example, when the frequency of the explanatory variable becoming X=X_1 is greater than the frequency of becoming X=X_2, that is, when P(X=X_1)>P(X=X_2), there are more data of X=X_1 than data of X=X_2, so the error reduction of X=X_1 is learned with priority over the error reduction of X=X_2. Therefore, when P(X) changes between facilities, the classification performance or prediction performance will also decrease.

[0076] [Example of Domain Shift]

[0077] Domain shift can be a problem for models of various tasks. For example, in the case of a model that predicts employee turnover risk, domain shift can be a problem when a prediction model learned using data from a certain company is used in another company.

[0078] Furthermore, in the case of a model that predicts the amount of antibody produced by a cell, domain shift may become a problem when a model learned using data on a certain antibody is applied to another antibody. Furthermore, in the case of a model that classifies customer voices (VOC), for example, a model that classifies VOCs into "product function," "support response," and "other," domain shift may become a problem when a classification model learned using data related to a certain product is applied to another product.

[0079] [Typical pattern of domain shift]

[0080] [Covariate shift] When the distribution of the explanatory variable P(X) is different, it is called covariate shift. For example, when the distribution of user attributes is different between data sets, more specifically, when the ratio of men to women is different, it is a covariate shift.

[0081] [Prior probability shift] When the distribution P(Y) of the target variable is different, it is called prior probability shift. For example, the average browsing rate and average purchase rate between data sets are different, which is a prior probability shift.

[0082] [Concept shift] When the conditional probability distributions P(Y|X) and P(X|Y) are different, it is called concept shift. For example, the probability of a company's research and development department reading data analysis materials is P(Y|X), but it is different between data sets, which is a concept shift.

[0083] There are two types of research on domain adaptation or domain generalization: research that assumes any of the above modes as the main cause and research that considers coping with changes in P(X,Y) without considering which mode is the main cause. In the former case, covariate shift is often assumed.

[0084] 〔Subset extraction〕

[0085] When a classification / prediction model with domain generalization cannot be constructed for the entire data, a classification / prediction model with domain generalization can be constructed within the subset by properly extracting a partial set (subset) from the data. The present invention proposes an appropriate subset extraction method.

[0086] A subset is a partial set of data sets to be analyzed. As an example of a subset extraction condition, for example, a subset extracted from data of only employees in the "sales department" of a certain company's data can be used to learn a model for predicting employee turnover risk. When a prediction model learned using the subset of the "sales department" is used in another company, the subset is universal in the sales department.

[0087] Furthermore, for a model predicting the amount of antibody produced by a cell, a subset extracted from only the data of an antibody whose "cell size is greater than a predetermined value" can be used. When a model learned using the subset whose "cell size is greater than a predetermined value" is applied to another antibody, the subset is universal in the field as long as the "cell size is greater than a predetermined value".

[0088] Furthermore, for a model that classifies customer voices into "product function," "support measures," and "others," a subset extracted from only the data of "cat owners" among data related to a certain product can be used. When a classification model learned using the subset of "cat owners" is applied to another product, the subset is universal for the entire field as long as it is "cat owners."

[0089] [About generalizability]

[0090] Figure 3This is an explanatory diagram showing an example of learning data and evaluation data used in machine learning. The data set obtained from the joint probability distribution Pd1(X,Y) of a certain field d1 is divided into learning data and evaluation data. The learning data and the evaluation data in the same field are referred to as "first evaluation data". Figure 3 In the example, the evaluation data is marked as "Evaluation Data 1". In addition, a data set obtained from the joint probability distribution Pd2(X,Y) of the domain d2 different from the domain d1 is prepared and used as the evaluation data. The evaluation data in the domain different from the learning data is called "2nd evaluation data". Figure 3 It is marked as "Evaluation Data 2".

[0091] The model 14 is learned using the learning data of the domain d1 , and the performance of the learned model 14 is evaluated using the first evaluation data of the domain d1 and the second evaluation data of the domain d2 .

[0092] Figure 4 This is a diagram schematically showing the difference in model performance due to different data sets. If the performance of model 14 in the learning data is set to performance A, the performance of model 14 in the first evaluation data is set to performance B, and the performance of model 14 in the second evaluation data is set to performance C, then generally Figure 4 As shown, the relationship is as follows: performance A>performance B>performance C.

[0093] The high generalization performance of the model 14 generally means that the performance B is high or the difference between the performance A and B is small. That is, the purpose is to avoid overfitting to the learning data and to have high prediction performance for the unlearned data.

[0094] In the context of domain generalization in this specification, it means that performance C is high or the difference between performance B and performance C is small. That is, the purpose is to always demonstrate high performance even in a field different from the field used in learning.

[0095] Furthermore, in this specification, the domain generalization of learning data means that in a classification / prediction model learned using the learning data, the change in the joint probability distribution of the explanatory variable and the target variable (domain shift) caused by the change in various conditions of the data generation process causes a relatively small decrease in classification performance or prediction performance, or the classification performance or prediction accuracy during the domain shift is relatively high. That is, when the domain generalization of the model 14 is relatively high, the domain generalization of the learning data used in the learning of the model 14 is relatively high.

[0096] Strictly speaking, the domain generalization of learning data means that there is a classification / prediction model with relatively little performance degradation. In other words, the performance of the classification / prediction model with the smallest performance degradation for domain shift among the classification / prediction model candidates is the domain generalization. For example, when one gene is selected from gene A, gene B, and gene C and its expression level is greater than the average value of all learning data, it is determined to be cancer. The classification / prediction model is uniquely determined by the selected gene (= feature value), so there are three model candidates: model A using gene A, model B using gene B, and model C using gene C.

[0097] Here, it is assumed that when there is no domain shift, the classification accuracy of Model A, Model B, and Model C is 90%, and when there is a domain shift, the classification accuracy is 70%, 80%, and 50%, respectively. In this case, the performance of Model A, Model B, and Model C decreases by -20%, -10%, and -40%, respectively, and the -10% of the best model, Model B, is the domain generalization.

[0098] Among them, when the classification accuracy of Model A, Model B, and Model C without domain shift is 75%, 90%, and 80% respectively, the performance degradation caused by domain shift is -5%, -10%, and -30% respectively. In this case, the -5% of Model A with the smallest performance degradation can be defined as the domain generalization performance, and the 80% of Model B with the largest accuracy when domain shift occurs can also be defined as the domain generalization performance.

[0099] In addition, if it is a universal characteristic, it is considered to be universally applicable, so if the subset is universal in the field, it has a relationship with field generalization. For example, if the characteristic that a gene is cancerous when expressed is universal regardless of race (field), then the method of determining cancer by the expression of this gene can be universally used regardless of race.

[0100] [Overview of information processing device]

[0101] Figure 5 1 is a block diagram schematically showing an example of the hardware configuration of the information processing device 100 according to the embodiment. The information processing device 100 has the following functions: a function of extracting a subset of a data set to be analyzed under a specified condition; and a function of evaluating domain generalization.

[0102] The information processing device 100 can be implemented using computer hardware and software. The physical form of the information processing device 100 is not particularly limited, and it can be a server computer, a workstation, a personal computer, or a tablet terminal, etc. Here, an example of using a single computer to implement the processing function of the information processing device 100 is described, but the processing function of the information processing device 100 can also be implemented by using a computer system composed of multiple computers.

[0103] The information processing device 100 includes a processor 102 , a non-transitory tangible computer-readable medium 104 , a communication interface 106 , an input / output interface 108 , and a bus 110 .

[0104] The processor 102 includes a CPU (Central Processing Unit). The processor 102 may include a GPU (Graphics Processing Unit). The processor 102 is connected to the computer-readable medium 104, the communication interface 106, and the input / output interface 108 via a bus 110. The processor 102 reads various programs and data stored in the computer-readable medium 104, and performs various processes. The term "program" includes the concept of a program module, and includes commands corresponding to the program.

[0105] The computer readable medium 104 is, for example, a storage device including a memory 112 as a main storage device and a storage device 114 as an auxiliary storage device. The storage device 114 is configured using, for example, a hard disk drive (HDD) device, a solid state drive (SSD) device, an optical disk, a magneto-optical disk, or a semiconductor memory, or a suitable combination thereof. The storage device 114 stores various programs and data.

[0106] The memory 112 is used as a work area for the processor 102 and as a storage unit for temporarily storing programs and various data read from the storage device 114. The program stored in the storage device 114 is loaded into the memory 112, and the processor 102 executes the instructions of the program, thereby the processor 102 functions as a mechanism for performing various processes specified in the program.

[0107] The memory 112 stores an extraction program 130 , a learning program 132 , an evaluation program 134 , various programs, various data, and the like that are executed by the processor 102 .

[0108] The extraction program 130 is a program that executes a process of acquiring a data set and extracting a subset of the data set that is a partial set of analysis targets from the acquired data set.

[0109] The learning program 132 is a program that performs processing for learning a classification / prediction model for classification or prediction according to each subset extracted by the extraction program 130 .

[0110] The evaluation program 134 is a program that executes a process of evaluating the domain generalization of the subset extracted by the extraction program 130. The evaluation program 134 can execute a process of evaluating the domain generalization of the classification / prediction model learned by the learning program 132.

[0111] The memory 112 includes a dataset storage unit 140 and a learning model storage unit 142. The dataset storage unit 140 is a storage area for storing datasets collected in the learning facility. The learning model storage unit 142 is a storage area for storing classification / prediction models learned by the learning program 132.

[0112] The communication interface 106 performs communication processing with an external device via wired or wireless communication, and exchanges information with the external device. The information processing device 100 is connected to a communication line (not shown) via the communication interface 106. The communication line can be a local area network, a wide area network, or a combination thereof. The communication interface 106 can function as a data acquisition unit that receives input of various data such as an original data set.

[0113] The information processing device 100 may include an input device 152 and a display device 154. The input device 152 and the display device 154 are connected to the bus 110 via the input / output interface 108. The input device 152 may be, for example, a keyboard, a mouse, a multi-touch panel or other pointing device, or a sound input device, or a suitable combination thereof. The display device 154 may be, for example, a liquid crystal display, an organic EL (organic electro-luminescence: OEL) display or a projector, or a suitable combination thereof. In addition, the input device 152 and the display device 154 may be integrally formed, such as a touch panel, and the information processing device 100, the input device 152 and the display device 154 may also be integrally formed, such as a touch panel type tablet terminal.

[0114] Figure 6 1 is a functional block diagram showing the functional structure of the information processing device 100. The information processing device 100 includes a data set acquisition unit 160, a subset extraction unit 162, a learning unit 164, and a domain generalization evaluation unit 166.

[0115] The dataset acquisition unit 160 acquires the dataset collected in the learning facility from the dataset storage unit 140. The dataset is a collection of data in each field, including explanatory variables and target variables.

[0116] The subset extraction unit 162 extracts a subset from the data set acquired by the data set acquisition unit 160 according to a specified condition. The subset includes all feature quantities (explanatory variables) and the target variable of the data set. That is, the subset extraction unit 162 does not narrow the feature quantities. The extraction condition can be specified by the user.

[0117] The learning unit 164 generates a classification / prediction model for performing classification or prediction for each subset.

[0118] The domain generalization evaluation unit 166 is a program that executes processing for evaluating the domain generalization of the subset extracted by the subset extraction unit 162. The domain generalization evaluation unit 166 can evaluate the domain generalization of the classification / prediction model learned by the learning unit 164.

[0119] [Implementation method 1]

[0120] The information processing apparatus 100 implements an information processing method in which a subset is extracted from a data set to be analyzed under a specified condition, and domain generalization of the extracted subset is evaluated.

[0121] Figure 7 1 is an explanatory diagram showing classification or prediction according to Embodiment 1. The data set of the analysis object in Embodiment 1, i.e., the data set for learning or evaluation (learning / evaluation data set), includes a data set DS1 of domain 1, a data set DS2 of domain 2, and a data set DS3 of domain 3. Domain 1, domain 2, and domain 3 are different domains.

[0122] The subset extraction unit 162 extracts subsets from the data set according to a specified condition. That is, the subset extraction unit 162 extracts the subset SS1 from the data set DS1, extracts the subset SS2 from the data set DS2, and extracts the subset SS3 from the data set DS3.

[0123] The domain generalization evaluation unit 166 evaluates the domain generalization of the subsets SS1 , SS2 , and SS3 .

[0124] When the subsets SS1, SS2, and SS3 have domain generalization, the learning unit 164 generates the model 14 for classification or prediction using the subsets SS1, SS2, and SS3. Since the subsets SS1, SS2, and SS3 have domain generalization, the model 14 having domain generalization can be constructed.

[0125] Furthermore, in this embodiment, there are data sets DS4 and DS5 in the field to which the learning model is applied. This field is an unknown field when the model 14 is learned.

[0126] The subset extraction unit 162 extracts subsets from the data set under the specified conditions. That is, the subset extraction unit 162 extracts subset SS4 from the data set DS4 and extracts subset SS5 from the data set DS5. The specified conditions here are the same as the conditions for extracting subsets SS1 to SS3 from the data sets DS1 to DS3.

[0127] By applying model 14 to subsets SS4 and SS5, high-precision classification or prediction can be performed.

[0128] [Implementation method 2]

[0129] The information processing device 100 can divide the data sets of multiple fields according to each field, extract a learning subset from the data sets of one or more learning fields among the multiple fields according to specified conditions, and use the learning subset to learn the learning model, extract an evaluation subset from the data sets of one or more evaluation fields different from the learning fields among the multiple fields according to specified conditions, and use the evaluation subset to evaluate the learning model, and use the evaluation result of the learning model to evaluate the domain generalization.

[0130] Figure 8 This is an explanatory diagram showing the processing according to Embodiment 2. Among them, Domain 1 and Domain 2 are learning domains, and Domain 3 is an evaluation domain.

[0131] The subset extraction unit 162 extracts a subset SS1 from the data set DS1 of the domain 1 which is the learning domain, and extracts a subset SS2 from the data set DS2 under a designated condition.

[0132] Furthermore, the subset extraction unit 162 divides the subsets SS1 and SS2 into learning data TD and first evaluation data ED, respectively. That is, the subset extraction unit 162 divides the subset SS1 into learning data TD1 and evaluation data ED1, and divides the subset SS2 into learning data TD2 and evaluation data ED2. The learning data TD includes the learning data TD1 and the learning data TD2, and the first evaluation data ED includes the evaluation data ED1 and the evaluation data ED2.

[0133] Furthermore, the subset extracting unit 162 extracts a subset SS3 from the data set DS3 of the field 3 which is the evaluation field under a specified condition. The subset SS3 corresponds to the second evaluation data.

[0134] The learning unit 164 performs learning of the model 14 using the learning data TD (an example of a “learning subset”).

[0135] The domain generalization evaluation unit 166 Figure 4The performance C shown evaluates the domain generalization property. That is, the domain generalization property evaluation unit 166 evaluates the domain generalization property of the learned model 14 using the subset SS3 (an example of the “evaluation subset”) as the second evaluation data.

[0136] The information processing device 100 can divide the data sets of multiple fields according to each field, extract a learning subset from the data sets of one or more learning fields among the multiple fields according to specified conditions, and use the learning subset to learn the learning model, extract a first evaluation subset different from the learning subset from the data set of the learning field according to specified conditions, and evaluate the learning model using the first evaluation subset, extract a second evaluation subset from the data sets of one or more evaluation fields different from the learning field among the multiple fields according to specified conditions, evaluate the learning model using the second evaluation subset, and evaluate the domain generalization using the difference between the evaluation results based on the first evaluation subset and the evaluation results based on the second evaluation subset.

[0137] That is, the domain generalization evaluation unit 166 evaluates Figure 4 The difference between the performance B and the performance C shown in FIG. 1 is used to evaluate the domain generalization. In this case, the domain generalization evaluation unit 166 can evaluate the domain generalization by using the difference between the performance of the model 14 learned using the first evaluation data ED (an example of the "first evaluation subset") and the performance of the model 14 learned using the second evaluation data, that is, the subset SS3 (an example of the "second evaluation subset"). The smaller the difference, the higher the domain generalization.

[0138] The domain generalization evaluation unit 166 can comprehensively judge Figure 4 The performance shown is C and Figure 4 The domain generalization ability was evaluated based on the difference between the performance B and the performance C shown.

[0139] [Implementation method 3]

[0140] The information processing device 100 can divide the data sets of multiple fields into each field, and evaluate the correlation between the feature quantities (explanatory variables) of the subsets of the data sets divided into each field and the target variable, set the feature quantities with a correlation above a specified threshold in more than a specified number of fields as feature quantities with relatively high field universality, and use the number of feature quantities with relatively high field universality to evaluate the field generalization of the subset.

[0141] In this embodiment, S is a set of data samples, which is expressed as follows:

[0142] S={1,2,...,i,...,|S|}.

[0143] F is a set of feature quantities, expressed as follows:

[0144] F={1,2,...,k,...,|F|}.

[0145] D is the set of fields, which is expressed as follows:

[0146] D={1,2,...,d,...,|D|}.

[0147] Each sample i includes a feature vector x included in a real number vector of |F| dimensions i (x i ∈R |F| ) and target variable yi. The value of the kth feature value of sample i is expressed as follows:

[0148] [Formula 1]

[0149] Each sample is included in any domain. The domain of sample i is denoted as di, and di∈D. The subset G is extracted from the dataset S, and is

[0150] [Formula 2]

[0151]

[0152] The above formula represents the correlation between the kth feature quantity of domain d and the target variable. For example, in G∩[i|d i =d], are the target variable yi and the value of the kth feature value of sample i.

[0153] [Formula 3]

[0154]

[0155] There is a Pearson correlation coefficient between the above equations.

[0156] A subset of domain-universal features It is defined as the following formula 1.

[0157] [Formula 4]

[0158]

[0159] in,

[0160] [Formula 5]

[0161]

[0162] In the above formula, the correlation between the feature quantity and the target variable is 1 when it is greater than θ or less than -θ, and 0 in other cases. θ is a specified threshold, for example, θ = 0.8. m is a specified number, for example, the total number of domains is 5, and m = 4. The domain generalization of the subset G is calculated by F DG The number of feature quantities of (G) is |F DG (G)|Evaluate.

[0163] The information processing device 100 can divide the data sets of multiple fields according to each field, and evaluate the correlation between the feature values ​​of the subsets of the data sets divided according to each field and the target variable. In multiple fields, the higher the correlation is, the higher the field universality of the feature values ​​is. The more feature values ​​of the subsets with relatively high field universality are, the higher the evaluation of the field generalization is.

[0164] That is, more generally, for larger θ, m, |F DG The larger the (G)|, the better. This evaluation method can be expressed as the following formula (2).

[0165] [Formula 6]

[0166]

[0167] Here, α and β are scaling factors used as follows: the larger the values ​​of m and θ, the greater the weights given.

[0168] There are various feature quantity selection methods, but as described in Non-Patent Documents 3 and 4, they are classified into filter methods, wrapper methods, and embedded methods.

[0169] The filtering method evaluates the feature quantity separately from the classification model based on the correlation between the feature quantity and the target variable. The packaging method evaluates the feature quantity based on the performance of a specific classification model. The embedding method has feature quantity selection built into the algorithm of the classification model. For example, decision trees, Lasso, etc.

[0170] Filtering methods are classified into univariate methods that evaluate feature quantities one by one and multivariate methods that evaluate feature quantity sets. The method using Pearson correlation is one of the univariate methods of filtering methods. Of course, other univariate methods can also be used. It is also easy to expand to multivariate methods and packaging methods. In this case, the correlation v is defined for the feature quantity set, and the feature quantity set F' that satisfies the conditions of the following formula 3 is a feature quantity set with field universality.

[0171] [Formula 7]

[0172]

[0173] [Implementation method 4]

[0174] The information processing device 100 can divide the data sets of multiple fields according to each field, extract a learning subset from the data sets of one or more learning fields among the multiple fields according to specified conditions, extract a feature value set from the learning subset, extract an evaluation subset from the data sets of one or more evaluation fields different from the learning fields among the multiple fields according to specified conditions, evaluate the extracted feature value set using the evaluation subset, and evaluate the domain generalization of the learning subset using the proportion of feature values ​​in the extracted feature value set that are also valid in the evaluation subset.

[0175] The information processing device 100 evaluates the correlation between the feature quantity of the learning subset and the target variable according to each field. In multiple fields, the higher the correlation is, the higher the field universality of the feature quantity is. The extracted feature quantity set can include feature quantities with relatively high field universality.

[0176] The information processing device 100 can evaluate the correlation between the feature quantities of the learning subset and the target variable according to each field, and set the feature quantities having a correlation above a specified threshold in more than a specified number of fields as feature quantities with relatively high field universality. The extracted feature quantity set may include feature quantities with relatively high field universality.

[0177] Fig. 9 This is an explanatory diagram showing the processing according to Embodiment 4. Among them, domain 1 and domain 2 are learning domains, and domain 3 is an evaluation domain.

[0178] The subset extraction unit 162 extracts subset SS1 from the dataset DS1 of the domain 1 as the learning domain and subset SS2 from the dataset DS2 under the specified conditions. Furthermore, the subset extraction unit 162 extracts subset SS3 from the dataset DS3 of the domain 3 as the evaluation domain under the specified conditions.

[0179] The domain generalization evaluation unit 166 extracts a feature set VS from all feature values ​​included in the subset SS1 and the subset SS2 (an example of a “learning subset”). The feature set VS is a partial set of explanatory variables of the subset SS1 and the subset SS2, or a feature value composed of the partial set.

[0180] The domain generalization evaluation unit 166 basically extracts the feature set VS based on the correlation between the target variable and the explanatory variable. For example, when height is the target variable and the expression level of each of 10,000 genes is the explanatory variable, the Pearson correlation coefficient between the expression level of each gene and the height is calculated, and the genes with a correlation coefficient greater than a specified value are extracted. In the case where the explanatory variable is not a continuous value such as height but a binary variable (binary variable) of whether it is cancer, the selection is made based on the p-value of the t-test or the AUC (Area Under the Curve). These correspond to the filtering method described above.

[0181] In the case of the packaging method, the performance of the classification model is evaluated, for example, the classification accuracy is compared when each gene is added to the explanatory variable in the logistic regression model and when it is not added to the explanatory variable, and genes with large differences (genes whose accuracy is greatly improved by adding to the explanatory variable) are selected.

[0182] However, when extracting the feature quantity set VS in consideration of the domain universality in the learning domain, for example, in both domain 1 and domain 2, a feature quantity set having a Pearson correlation coefficient equal to or greater than a predetermined value is extracted.

[0183] Then, the domain generalization evaluation unit 166 verifies the validity of the feature value set VS using the subset SS3 (an example of the “evaluation subset”).

[0184] As a method of verifying the validity, any of the following methods can be considered.

[0185] · In the evaluation domain, the same feature quantity as that in the learning domain is extracted and overlapped.

[0186] The positive correlation and negative correlation between the learning domain and the feature quantity are the same. That is, if there is a positive correlation in the learning domain, there is at least a positive correlation in the evaluation domain.

[0187] Fig. 9In F9A shown, the feature set VS is expressed by the proportion of features that are also valid in the evaluation field. For example, when s extracts 15 features and 9 features that are valid, the proportion of valid features is higher than when s extracts 20 features and 10 features that are valid, so the latter subset has a relatively higher field generalization.

[0188] [Implementation method 5]

[0189] The information processing device 100 may present a plurality of different subset conditions, evaluate the subsets extracted respectively by the plurality of different subset conditions, and extract the subset by the subset condition with the best evaluation result among the plurality of different subset conditions.

[0190] Fig.10 1 is an explanatory diagram showing the processing involved in Embodiment 5. The information processing device 100 includes a subset condition presenting unit 168. The subset condition presenting unit 168 presents a plurality of different subset conditions (an example of "specifying"). Among them, as the learning / evaluation datasets, there are a dataset DS1 of domain 1, a dataset DS2 of domain 2, and a dataset DS3 of domain 3.

[0191] exist Fig.10 In the example shown, the subset condition presenting unit 168 presents different subset conditions A and subset conditions B. As long as it is a subset of a model for predicting the antibody production amount of a cell, for example, the subset condition A is "cell size is 10 μm or more", and the subset condition B is "cell size is 15 μm or more". That is, the different subset conditions A and subset conditions B define the same feature quantity item such as "cell size" by different values ​​(attributes) of "10 μm or more" and "15 μm or more".

[0192] The subset extracting unit 162 extracts subsets SS1A, SS2A, and SS3A from the data sets DS1, DS2, and DS3, respectively, using the subset condition A. Furthermore, the subset extracting unit 162 extracts subsets SS1B, SS2B, and SS3B from the data sets DS1, DS2, and DS3, respectively, using the subset condition B.

[0193] The domain generalization evaluation unit 166 adopts the subset condition with the best evaluation result among the subset conditions A and B. The subset extraction unit 162 extracts a subset using the adopted subset condition.

[0194] The learning unit 164 generates the model 14 using the subsets SS1A and SS2A of the subsets of the subset condition A as learning data and the subsets SS1B and SS2B of the subset condition B as learning data.

[0195] The domain generalization evaluation unit 166 evaluates the model 14 of subset condition A using subset SS3A of the subset of subset condition A as evaluation data, and evaluates the model 14 of subset condition B using subset SS3B of the subset of subset condition B as evaluation data. Thus, subset conditions A and B can be evaluated.

[0196] Subset conditions A and B may also be evaluated using feature sets as in Embodiment 4. In this case, the domain generalization evaluation unit 166 extracts feature sets from subsets SS1A and SS2A, and verifies the validity of the feature sets using subset SS3A. Furthermore, the domain generalization evaluation unit 166 extracts feature sets from subsets SS1B and SS2B, and verifies the validity of the feature sets using subset SS3B.

[0197] [Implementation method 6]

[0198] The information processing device 100 may evaluate the usefulness of the subset based on known usefulness information for each sample in the data set and the samples included in the subset, and combine the usefulness and domain generalization as the evaluation of the subset.

[0199] Fig.11 This is an explanatory diagram showing the processing according to Embodiment 6. The domain generalization evaluation unit 166 includes a subset usefulness evaluation unit 166A. The subset usefulness evaluation unit 166A evaluates the usefulness of a subset based on known usefulness information for each sample in the data set and the samples included in the subset.

[0200] like Fig.11 As shown, the datasets for learning / evaluation include a dataset DS1 for domain 1, a dataset DS2 for domain 2, and a dataset DS3 for domain 3. The subset extraction unit 162 extracts subsets SS1, SS2, and SS3 from the datasets DS1, DS2, and DS3, respectively.

[0201] The domain generalization evaluation unit 166 evaluates the domain generalization of the subsets SS1 , SS2 , and SS3 .

[0202] Then, the subset usefulness evaluation unit 166A evaluates the usefulness of the subsets SS1 , SS2 , and SS3 . Fig.12This is a graph used to illustrate the usefulness of a subset. It is assumed that in the VOC classification of cameras, the more photos a user prints in a year, the better the user is, i.e., a sample with relatively high usefulness. The subset usefulness evaluation unit 166A calculates the average number of photos printed in a year for each user of the subsets SS1, SS2, and SS3, for "having a cat", "having a dog", "other pets", and "having no pets". The larger the average, the higher the evaluation of the usefulness of the subset. Fig.12 In the example shown, "having a cat", "having a dog", "having other pets", and "having no pets" have higher usefulness in this order.

[0203] Furthermore, in the case of VOC classification, the subset usefulness evaluation unit 166A may evaluate that the more loyal customers there are, the higher the usefulness of the subset is. In the case of antibody production prediction, the subset usefulness evaluation unit 166A may evaluate that the faster the cell proliferation rate is, the higher the usefulness of the subset is.

[0204] The subset usefulness evaluation unit 166A averages the usefulness of the samples belonging to the subsets SS1 , SS2 , and SS3 , and obtains the average of the usefulness of the samples as the usefulness of the subsets SS1 , SS2 , and SS3 .

[0205] The domain generalization evaluation unit 166 combines the usefulness evaluated by the subset usefulness evaluation unit 166A with the domain generalization to obtain the evaluation of the subsets SS1 , SS2 , and SS3 .

[0206] [Implementation method 7]

[0207] The data set includes data sets in multiple fields. The information processing device 100 can evaluate the domain uniformity representing the degree of distribution proximity between the number of data in each field data set and the number of data in the subset, and combine the domain uniformity and the domain generalization as the evaluation of the subset.

[0208] Fig.13 : is an explanatory diagram showing the processing according to Embodiment 7. The domain generalization evaluation unit 166 includes a domain uniformity evaluation unit 166B. The domain uniformity evaluation unit 166B evaluates domain uniformity indicating how close the distribution of the number of data in a data set and the number of data in a subset are for each domain.

[0209] like Fig.13 As shown, there are datasets DS1 for domain 1, DS2 for domain 2, and DS3 for domain 3 as datasets for learning / evaluation. The numbers of data in datasets DS1, DS2, and DS3 are n1, n2, and n3, respectively.

[0210] The subset extracting unit 162 extracts subsets SS1, SS2, and SS3 from the data sets DS1, DS2, and DS3 according to a specified condition. The number of data in the subsets SS1, SS2, and SS3 is m1, m2, and m3, respectively.

[0211] The domain generalization evaluation unit 166 evaluates the domain generalization of the subsets SS1 , SS2 , and SS3 .

[0212] Then, the domain uniformity evaluation unit 166B evaluates the domain uniformity. The domain uniformity evaluation unit 166B evaluates the data set as uniform if the existence ratio of the subset in the data set is 40% in the domains 1, 2, and 3. For example, if the existence ratio of the subset in the data set is 90% in the domain 1, 50% in the domain 2, and 10% in the domain 3, the domain uniformity evaluation unit 166B evaluates the data set as non-uniform.

[0213] The domain uniformity evaluation unit 166B evaluates the distribution closeness in the data numbers n1, n2, and n3 of the data sets DS1, DS2, and DS3 and the data numbers m1, m2, and m3 of the subsets SS1, SS2, and SS3. For example, the domain uniformity evaluation unit 166B evaluates that the smaller the KL divergence (Kullback-Leibler divergence) is, the better the domain uniformity is. The KL divergence can be expressed as follows.

[0214] Kullback-Leibler divergence=Σ k P1(d=k)(log(P1(d=k)-log(P2(d=k)))

[0215] k is the area ID, which is 1, 2, and 3. P1 is the distribution of n1, n2, and n3, and P1(d=k)=n1 / (n1+n2+n3). Also, P2 is the distribution of m1, m2, and m3, and P2(d=k)=m1 / (m1+m2+m3).

[0216] [Implementation method 8]

[0217] The information processing apparatus 100 may learn a subset classification model. The information processing apparatus 100 may evaluate the subset classification performance of the subset classification model, and combine the subset classification performance with the domain generalization as the evaluation of the subset.

[0218] Fig.141 is an explanatory diagram showing the processing involved in Embodiment 8. Here, as the datasets for learning / evaluation, there are datasets DS1 for domain 1, datasets DS2 for domain 2, and datasets DS3 for domain 3. The subset extraction unit 162 extracts subsets SS1, SS2, and SS3 from the datasets DS1, DS2, and DS3, respectively, based on the subset classification model 162B.

[0219] Here, the subset extraction unit 162 according to Embodiment 8 includes a subset classification learning unit 162A. The subset classification learning unit 162A learns a subset classification model 162B for classifying whether data in a data set is a subset or not.

[0220] For example, in the prediction of antibody production, prediction is performed using gene expression as a feature quantity, and subset extraction is performed based on DNA mutations. In practice, since subset extraction is also more convenient through gene expression, the subset classification model from gene expression is learned separately. Of course, the higher the performance of the classification model, the better, so the ease of classification is also included in the evaluation of the generalization of the subset domain.

[0221] Fig.14 F14A is a diagram showing a subset extracted from a certain data set. In F14A, the vertical axis and the horizontal axis are feature quantities or indicators calculated based on the feature quantities, and the samples of the data set are plotted according to the values ​​of the vertical axis and the horizontal axis. In this example, an example is shown in which the samples of the data set are classified into samples belonging to the subset represented by the black circular mark and samples not belonging to the subset represented by the white circular mark. For example, in the VOC classification, the samples belonging to the subset are users who "keep cats", and the samples not belonging to the subset are users other than "keep cats".

[0222] Whether it is a subset is defined by the subset extraction unit 162. F14A shows an example when "keeping cats" is a subset extraction condition. In the example of stability prediction described later, the antibody production amount is above the average value as a subset extraction condition, and as long as the antibody production amount is above the average value, it is defined as a subset.

[0223] The subset classification learning unit 162A learns the subset classification model 162B using the data shown in F14A.

[0224] The domain generalization evaluation unit 166 evaluates the domain generalization of the subsets SS1, SS2, and SS3. In addition, the domain generalization evaluation unit 166 evaluates the subset classification performance of the subset classification model 162B, and combines the subset classification performance and the domain generalization as the subset evaluation.

[0225] Subset classification performance is similar to the evaluation of general machine learning models. The data is divided into "learning" and "evaluation" data, and the model learned with the learning data is evaluated with the evaluation data. For example, 80% of "keeping cats" and other data except "keeping cats" are divided into learning data, and 20% are divided into evaluation data. The model learned with the former is evaluated as follows: whether the latter data can be correctly classified into "keeping cats" and other data except "keeping cats". Learning and evaluation of subset classification can be done by using data that combines the data of domain 1, domain 2, and domain 3.

[0226] [Implementation method 9]

[0227] The information processing device 100 can search for a subset with higher domain generalization by repeatedly adding or deleting samples (an example of the "specified condition") in the subset from the starting subset. The information processing device 100 can search by the following evaluation method: the usefulness of the subset evaluated based on the known usefulness information of each sample in the data set and the samples included in the subset, the domain uniformity indicating the distribution closeness of the number of data in the data set and the number of data in the subset in each field, and the subset classification performance of the subset classification model for classifying whether the data in the data set is a subset are also evaluated.

[0228] Fig.15 : is an explanatory diagram showing an algorithm for extracting a subset according to Embodiment 9. In this algorithm, inputs are a threshold value θ of the correlation between a feature quantity and a target variable, a maximum size n of a subset, and a convergence condition e for terminating when the improvement in domain generalization becomes less than a specified value, and an output is a subset G with domain generalization. The domain generalization of a subset is evaluated by the number of feature quantities with domain generalization.

[0229] exist Fig.15 In the first line of , G as a subset is initialized to the starting subset. The method for determining the starting subset is arbitrary, and it can be the entire data set or an empty set.

[0230] exist Fig.15 In the second row of , the range of |G|, which represents the number of samples of the subset G used for processing, and the increase amount Δ of domain generalization is defined. That is, the processing is continued when |G|<n and Δ>e are satisfied, and the processing is terminated when they are not satisfied.

[0231] exist Fig.15 In line 3 of F ,exist Fig.15 In line 4 of BHere, the samples are increased or decreased one by one, but multiple samples can also be increased or decreased at once. The object of argmax can be not only |F DG |, it can also be the weighted addition of usefulness evaluation, domain uniformity, and subset classification performance.

[0232] exist Fig.15 In the 5th row of F Is the domain universality of G greater than that of the subset G B The domain universality of the domain is relatively higher (whether the number of feature quantities with domain generalization is large). F In the case of a relatively higher domain prevalence, Fig.15 In line 6 of DG (G F )|-|F DG (G)|, and update G to G F .

[0233] In the subset G F In the case of relatively low universality in the field, Fig.15 In line 9 of DG (G B )|-|F DG (G)|, and update G to G B .

[0234] The above treatment is carried out until |G| or Δ is separated from Fig.15 Thus, it is possible to search for subsets with high domain generalization.

[0235] [Implementation method 10]

[0236] Fig.16 It is a table showing an example of a data set. Fig.16 The data set shown is the data set of "Company 1" of a certain company. Fig.16 As shown, the data set has the items of "occupation type", "age", "years of work experience", "years since last promotion", "annual income", "final education", "commuting time", "average overtime hours" and "age difference with supervisor" as explanatory variables for each sample (employee), and the item of "whether to quit" as the target variable. In addition, in "whether to quit", "yes" is "1" and "no" is "0".

[0237] For example, Fig.16Among the employees in the top row, “Occupation type”, “Age”, “Years of experience”, “Years since last promotion”, “Annual income”, “Ultimate education level”, “Commuting time”, “Average overtime hours”, “Age difference with supervisor”, and “Whether to leave” are “sales”, “32”, “10”, “5”, “600”, “college”, “40”, “20”, “3”, and “1” respectively.

[0238] Fig.17 Yes Fig.16 The table shows an example of a data set that is abstracted from the data set shown in FIG. Among them, "occupation type" is a "variable for subset extraction", "age", "years of work experience", "years since last promotion", "annual income", "final education", "commuting time", "average overtime hours" and "age difference with supervisor" are respectively "feature quantity A", "feature quantity B", "feature quantity C", "feature quantity D", "feature quantity E", "feature quantity F", "feature quantity G" and "feature quantity H". And "whether to resign" is a "target variable".

[0239] The turnover risk prediction model is explained. Here, each company is a field. There are five data sets, "Company 1", "Company 2", "Company 3", "Company 4", and "Company 5", and it is desired to build a robust prediction model for companies other than "Company 1" to "Company 5" (other fields). Fig.17 The eight feature quantities AH shown are used to construct a prediction model, which predicts whether the employee will resign (1 / 0). The feature quantities are converted into numerical values ​​and are all considered continuous variables. Among them, subsets are extracted based on "occupation type". In the extraction of subsets, the feature quantities are not reduced.

[0240] In the five companies (fields), the correlation between each feature quantity and the target variable is evaluated separately, and the feature quantity with high correlation in more than four of the five fields is regarded as a feature quantity with high field universality. The correlation is evaluated based on AUC, and as long as AUC is above 0.8, it is evaluated as having high correlation. More precisely, AUC is symmetrical around 0.5, so as long as |AUC-0.5| is above 0.3, it is evaluated as having high correlation. The field generalization evaluation of the extracted subset is performed based on the number of feature quantities with high field universality.

[0241] First, without extracting a subset, the domain universality was evaluated using all the data. As a result, there was only one feature quantity with high domain universality. Next, subsets were extracted for each occupation type and the domain universality was evaluated. Subsets were extracted in "sales", "development", "R&D", and "production" respectively. As a result, the number of feature quantities with high domain universality was 5, 2, 1, and 2, respectively. It can be seen that if "sales" is extracted as a subset, it has high universality.

[0242] Therefore, we prepare a subset of data extracted from each company's data for only sales positions, and use this data to learn a turnover risk prediction model. For the "sales" subset, the model that has been learned can be expected to have high domain generalization, and can be used with confidence even for other unknown companies.

[0243] [Other examples of datasets]

[0244] Fig.18 This is a table showing other examples of abstracted data sets. Fig.18 The dataset shown is a dataset of a certain company, and it is envisioned that the VOC classification model constructed based on the dataset will be applied to other companies.

[0245] Fig.18 The data set shown has the following items in addition to "variable for subset extraction": "word A", "word B", "word C", "word D", "word E", "word F", "word G", "word H" and "target variable". "Word A", "word B", "word C", "word D", "word E", "word F", "word G" and "word H" correspond to feature quantities, respectively.

[0246] [Implementation method 11]

[0247] The production of pharmaceutically useful antibodies using CHO (Chinese hamster ovary) cells and the like is widely performed. In such production methods, it is desired that the production amount per unit time of the cells is large and that the production is stable over a long period of time (the production amount does not vary greatly).

[0248] Therefore, a model for predicting the stability label (1 or 0) is constructed based on the gene expression value. It is desirable to construct a model that can predict stability regardless of the type of antibody produced by the cell. That is, it is important to predict performance for unknown antibody types, and a prediction model with domain generalization is required that takes the antibody type as a domain.

[0249] Among them, there are data sets for five fields, namely antibody types a, b, c, d and e. The value of gene expression is a count value that takes a positive integer, and is logarithmically transformed and used as a feature. The correlation between each feature and the target variable (stability label) is evaluated using five antibody types (fields), and the feature with high correlation in more than four of the five fields (m=4) is used as a feature with high field universality. The correlation is evaluated based on the absolute value of the difference caused by the target variable of the feature value average value. As long as the difference is greater than 0.2 (θ=0.2), it is evaluated as having a high correlation. The field generalization evaluation of the extracted subset is performed based on the number of feature quantities with high field universality |F(DG)|.

[0250] First, without extracting a subset, the domain universality was evaluated using all the data. As a result, there was only one feature quantity with high domain universality. Next, the data with an antibody production amount above the average value was extracted as a subset, and the feature quantity with high domain universality was extracted for the same. As a result, there were 224 feature quantities with high domain universality. In addition, if the data with a viable cell density above the average value was extracted as a subset, there were 33 feature quantities with high domain universality. Therefore, the subset was extracted according to the antibody production amount.

[0251] Using the extracted subset (antibody production is above the average value) data, the stability prediction model is machine-learned. Here, a logistic regression model is used. The feature quantity is selected from 224 with high field universality, and 50 are selected in turn and learned in a way that the prediction accuracy of logistic regression is the largest. Learning is limited to a subset with high field generalization, so the learned model can robustly predict stability for the subset even for other antibody types.

[0252] When using a model that has been trained to predict stability (predicting unlearned antibody types), first, a subset is extracted from the samples that are to be predicted under the same conditions (antibody production is above the average value), and a stability label is predicted for the subset based on gene expression. During operation, samples predicted to be stable are transferred to the next step.

[0253] In the above example, the domain generalization of the subset is evaluated based on the number of feature quantities with high domain generalization, but it may be evaluated based on the ratio of the extracted feature quantities that are also effective in other domains.

[0254] Specifically, for example, using data from 4 of the 5 fields, the top 100 feature quantities with high AUC are extracted, and the feature quantity is also evaluated in the same manner in the remaining 1 field to see whether it is effective in prediction (here, the condition of AUC being 0.6 or above). In the case where the subset is extracted based on the live cell density, there are 52 (ratio 0.52), and in the case where the antibody production is superior, there are 83 (ratio 0.83), and the subset evaluated as superior in antibody production has higher field generalization.

[0255] Furthermore, as another evaluation method, as described in Embodiment 2, the prediction performance of the model learned in four fields can be evaluated in the remaining one field, thereby evaluating the field generalization of the subset.

[0256] [About the programs that make computers run]

[0257] A program that enables a computer to implement part or all of the processing functions in the information processing device 100 can be recorded on a computer-readable medium such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible non-transitory information storage medium, and the program can be provided through the information storage medium.

[0258] Furthermore, instead of providing the program by storing it in such a tangible non-transitory computer-readable medium, it is also possible to provide the program signal as a download service using telecommunication lines such as the Internet.

[0259] Furthermore, part or all of the processing functions in the information processing device 100 may be realized by cloud computing, and may also be provided as SaaS (Software as a Service).

[0260] [About the hardware structure of each processing unit]

[0261] The hardware configuration of processing units that execute various processes such as the data set acquisition unit 160 , the subset extraction unit 162 , the learning unit 164 , and the domain generalization evaluation unit 166 in the information processing device 100 is, for example, various processors as shown below.

[0262] Various processors include general-purpose processors such as CPUs, GPUs, and FPGAs (Field Programmable Gate Arrays) that execute programs and function as various processing units; processors whose circuit structures can be changed after manufacturing, such as programmable logic devices (PLDs) and ASICs (Application Specific Integrated Circuits); and processors with circuit structures specifically designed to perform specific processing, such as dedicated circuits.

[0263] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same type or different types. For example, a processing unit can be composed of multiple FPGAs or a combination of CPU and FPGA or a combination of CPU and GPU. In addition, multiple processing units can also be composed of one processor. As an example of a plurality of processing units composed of one processor, first, as represented by a computer such as a client or server, there is the following method: a processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. Secondly, as represented by a system on chip (System On Chip; SoC), there is a method of using a processor, and the processor is composed of an IC (Integrated Circuit) chip to realize the function of the entire system including multiple processing units. In this way, various processing units are configured as a hardware structure using one or more of the above-mentioned various processors.

[0264] Furthermore, more specifically, the hardware structure of these various processors is a circuit (circuitry) formed by combining circuit elements such as semiconductor elements.

[0265] [Advantages based on implementation method]

[0266] According to the present embodiment, a subset having domain generalization can be extracted, and thus a classification / prediction model having domain generalization can be constructed.

[0267] [Other application examples]

[0268] In the above embodiments, a model for predicting employee turnover risk, a model for predicting antibody production in cells, and a model for classifying VOCs are described as examples, but the technology of the present invention can be applied to the construction of various classification / prediction models.

[0269] The domain generalization evaluation unit 166 may evaluate the domain generalization of a subset by combining the subset evaluation methods according to the various embodiments.

[0270] 〔other〕

[0271] The present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the technical concept of the present invention.

[0272] Explanation of symbols

[0273] 14-model, 100-information processing device, 102-processor, 104-computer readable medium, 106-communication interface, 108-input / output interface, 110-bus, 112-memory, 114-storage device, 130-extraction program, 132-learning program, 134-evaluation program, 140-dataset storage unit, 142-learning model storage unit, 152-input device, 154-display device, 160-dataset acquisition unit, 162-subset extraction unit, 162A-subset classification learning unit, 162B-subset classification model, 164-learning unit, 166-domain generalization evaluation unit, 166 A-subset usefulness evaluation unit, 166B-domain uniformity evaluation unit, 168-subset condition prompt unit, DS1-dataset, DS2-dataset, DS3-dataset, DS4-dataset, DS5-dataset, ED-first evaluation data, ED1-evaluation data, ED2-evaluation data, SS1-subset, SS1A-subset, SS1B-subset, SS2-subset, SS2A-subset, SS2B-subset, SS3-subset, SS3A-subset, SS3B-subset, SS4-subset, SS5-subset, TD-learning data, TD1-learning data, TD2-learning data.

Claims

1. An information processing method, which is performed by one or more processors, The information processing method includes the following processing performed by the one or more processors: Extracting a subset from the data set of the analysis object according to the specified conditions; and The domain generalization of the extracted subset is evaluated.

2. The information processing method according to claim 1, wherein: The datasets include datasets from multiple fields. The evaluation includes the following processing performed by the one or more processors: Extracting a learning subset from a data set of one or more learning fields among the plurality of fields according to the specified condition, and using the learning subset to make a learning model learn; extracting an evaluation subset from data sets of one or more evaluation fields different from the learning field among the plurality of fields according to the specified condition, and evaluating the learning model using the evaluation subset; and The domain generalization is evaluated using the evaluation results of the learning model.

3. The information processing method according to claim 1, wherein: The datasets include datasets from multiple fields. The evaluation includes the following processing performed by the one or more processors: Extracting a learning subset from a data set of one or more learning fields among the plurality of fields according to the specified condition, and using the learning subset to make a learning model learn; Extracting a first evaluation subset different from the learning subset from the data set of the learning field according to the specified condition, and evaluating the learning model using the first evaluation subset; extracting a second evaluation subset from a data set of one or more evaluation fields different from the learning field among the plurality of fields according to the specified condition, and evaluating the learning model using the second evaluation subset; and The domain generalization is evaluated using a difference between an evaluation result based on the first evaluation subset and an evaluation result based on the second evaluation subset.

4. The information processing method according to claim 1, wherein: The datasets include datasets from multiple fields. The evaluation includes the following processing performed by the one or more processors: Evaluating the correlation between the feature quantity of the subset and the target variable according to each of the fields; and In a plurality of fields, the higher the correlation is, the higher the field universality of the feature quantity is. The subset having more feature quantities with relatively higher field universality has a relatively higher evaluation of the field generalization.

5. The information processing method according to claim 1, wherein: The datasets include datasets from multiple fields. The evaluation includes the following processing performed by the one or more processors: Evaluating the correlation between the feature quantity of the subset and the target variable according to each of the fields; and Feature quantities having correlations greater than or equal to a predetermined threshold in a predetermined number of fields or more are set as feature quantities with relatively high field universality, and the field generalization of the subset is evaluated using the number of feature quantities with relatively high field universality.

6. The information processing method according to claim 1, wherein: The datasets include datasets from multiple fields. The evaluation includes the following processing performed by the one or more processors: Extracting a learning subset from a data set of one or more learning fields among the plurality of fields according to the specified condition, and extracting a feature quantity set from the learning subset; extracting an evaluation subset from data sets of one or more evaluation fields different from the learning field among the plurality of fields according to the specified condition, and evaluating the extracted feature quantity set using the evaluation subset; and The domain generalization of the learning subset is evaluated using the ratio of feature amounts that are also effective in the evaluation subset among the feature amounts in the extracted feature amount set.

7. The information processing method according to claim 6, comprising the following processing performed by the one or more processors: Evaluating the correlation between the feature quantity of the learning subset and the target variable according to each of the fields; and In a plurality of fields, it is assumed that the higher the correlation is, the higher the field universality of the feature quantity is. The extracted feature quantity set includes feature quantities with relatively high universality in the field.

8. The information processing method according to claim 6, comprising the following processing performed by the one or more processors: Evaluating the correlation between the feature quantity of the learning subset and the target variable according to each of the fields; and A feature quantity having a correlation degree of at least a predetermined threshold value in at least a predetermined number of fields is regarded as a feature quantity having relatively high field universality. The extracted feature quantity set includes feature quantities with relatively high universality in the field.

9. The information processing method according to claim 1, wherein: The evaluation includes the following processing performed by the one or more processors: Evaluating the usefulness of the subset based on known usefulness information for each sample in the data set and the samples included in the subset; and The usefulness and the domain generalization are combined as the evaluation of the subset.

10. The information processing method according to claim 1, wherein: The datasets include datasets from multiple fields. The evaluation includes the following processing performed by the one or more processors: evaluating the domain uniformity indicating how closely the number of data in the data set of each of the domains is distributed to the number of data in the subset; and The domain uniformity and the domain generalization are combined together as the evaluation of the subset.

11. The information processing method according to claim 1, comprising the following processing performed by the one or more processors: A subset classification model is learned, wherein the subset classification model classifies whether the data in the data set is a subset.

12. The information processing method according to claim 11, wherein: The evaluation includes the following processing performed by the one or more processors: evaluating the subset classification performance of the subset classification model; and The subset classification performance and the domain generalization are combined as the evaluation of the subset.

13. The information processing method according to claim 1, wherein: The extracting and the evaluating include the following processing performed by the one or more processors: The subset with higher generalization of the domain is searched by repeatedly adding or deleting samples from the initial subset.

14. The information processing method according to claim 13, wherein: The datasets include datasets from multiple fields. The search includes a search performed by the one or more processors in the following evaluation manner: The usefulness of the subset evaluated based on the known usefulness information of each sample in the data set and the samples included in the subset, the field uniformity indicating the degree of closeness of the distribution of the number of data in the data set and the number of data in the subset in each field, and the subset classification performance of the subset classification model for classifying whether the data in the data set belongs to a subset will also be evaluated.

15. The information processing method according to any one of claims 1 to 14, comprising the following processing performed by the one or more processors: Prompt for multiple different subset conditions; evaluating the subsets respectively extracted according to the plurality of different subset conditions; and The subset is extracted according to the subset condition with the best evaluation result among the multiple different subset conditions.

16. An information processing device comprising: More than one processor; and One or more memories storing instructions for execution by the one or more processors, wherein: The one or more processors perform the following processing: Extracting a subset of the data set of the analysis object according to the specified conditions; and The domain generalization of the extracted subset is evaluated.

17. A program that enables a computer to implement the following functions: The function of extracting a subset of the data set of the analysis object according to the specified conditions; and Function to evaluate the domain generalization of the extracted subset. 18 . A recording medium which is non-transitory and computer-readable and has the program according to claim 17 recorded thereon.

Images