Information processing program, information processing method and information processing device

Abstract

To properly tune a model.SOLUTION: An information processing program according to an embodiment causes a computer to execute the processing of: acquiring an evaluation result that indicates evaluation for each of a plurality of indexes in a machine learning model; clustering the evaluation result for each combination pattern of the plurality of indexes; calculating a variance of the evaluation results in a cluster for each combination pattern; determining a combination pattern that satisfies a predetermined condition, from among the plurality of combination patterns, based on the calculated variance of the evaluation results; aggregating the evaluations for each of the plurality of indexes for each cluster, based on the evaluation result included in each cluster obtained by performing clustering on the determined combination pattern; and determining a solution for each of the plurality of indexes in the machine learning model based on the aggregated evaluation for each of the plurality of indexes.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Conventionally, there are information processing systems that use machine learning models (hereafter, models) to support the decision-making of loan examiners and recruiters in people's decision-making processes such as loan screening and personnel recruitment. In these information processing systems, the results of past cases (loan, hiring or not) are used as training data to train the model, and the model inputs the cases to be judged, and outputs the judgment results (loan, hiring or not).

[0003] The output from this information processing system has the problem of bias based on race, gender, etc. Such bias can be reduced by tuning the model using performance indicators and fairness indicators, but it is important to tune the system in a way that is acceptable to all stakeholders who are affected by it.

[0004] Regarding tuning of such models, there is a conventional technique that considers the diverse preferences of stakeholders affected by the system by aggregating preference information through majority voting and using the aggregated results for tuning. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-87966 A [Patent Document 2] JP 2013-101700 A [Patent Document 3] JP 2007-172427 A [Patent Document 4] US Patent Application Publication No. 2023 / 0024361 [Patent Document 5] US Patent Application Publication No. 2016 / 0180451 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the above conventional technology, majority voting results in wasted votes, which means that the preferences of stakeholders with minority opinions (for example, women who have few loan screening cases, people with disabilities who are few job seekers, etc.) are eliminated. For this reason, the conventional technology has the problem that it is difficult to tune the model in a way that satisfies all stakeholders, including those with minority opinions.

[0007] In one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that are capable of appropriately tuning a model. [Means for solving the problem]

[0008] In one proposal, the information processing program causes a computer to execute an acquisition process, a clustering process, a calculation process, a combination pattern determination process, a tabulation process, and a solution determination process. The acquisition process acquires evaluation results indicating evaluations for each of a plurality of indicators in a machine learning model. The clustering process clusters the evaluation results for each combination pattern of the plurality of indicators. The calculation process calculates the variance of the evaluation results in a cluster for each combination pattern. The determination process determines a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results. The tabulation process tabulates evaluations for each of the plurality of indicators for each cluster based on evaluation results included in each cluster by clustering performed on the determined combination pattern. The solution determination process determines a solution for each of the plurality of indicators in the machine learning model based on the tabulated evaluations for each of the plurality of indicators. Effect of the Invention

[0009] According to one embodiment, the model can be properly tuned. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a flowchart illustrating an operation example of the information processing system according to the embodiment. [Diagram 3] FIG. 3 is an explanatory diagram for explaining an overview of model learning. [Figure 4] FIG. 4 is an explanatory diagram for explaining an outline of the questionnaire. [Diagram 5] FIG. 5 is an explanatory diagram for explaining an outline of the clustering calculation. [Figure 6] FIG. 6 is an explanatory diagram for explaining an outline of the Euclidean distance calculation. [Figure 7] FIG. 7 is an explanatory diagram for explaining an overview of intra-cluster distributed calculation. [Figure 8] FIG. 8 is an explanatory diagram for explaining an outline of the calculation of the minimum value of the average variance within a cluster. [Figure 9] FIG. 9 is an explanatory diagram for explaining an outline of intra-cluster preference value calculation. [Figure 10] FIG. 10 is an explanatory diagram for explaining an outline of the cluster-wide preference value calculation. [Figure 11] FIG. 11 is an explanatory diagram for explaining an overview of the index parameter set. [Figure 12] FIG. 12 is an explanatory diagram illustrating an example of a computer configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an information processing program, an information processing method, and an information processing device according to the embodiments will be described with reference to the drawings. Configurations having the same functions in the embodiments are given the same reference numerals, and duplicated descriptions will be omitted. Note that the information processing program, the information processing method, and the information processing device described in the following embodiments are merely examples, and do not limit the embodiments. In addition, the following embodiments may be appropriately combined within a range that does not cause contradiction.

[0012] Fig. 1 is a block diagram showing a configuration example of an information processing system according to an embodiment, and Fig. 2 is a flowchart showing an operation example of the information processing system according to the embodiment.

[0013] As shown in Fig. 1, an information processing system 100 includes a model learning device 1, a questionnaire device 2, and a model adjustment device 3. In the information processing system 100, the model learning device 1, the questionnaire device 2, and the model adjustment device 3 operate, for example, as shown in the flowchart (S1 to S9) in Fig. 2, thereby tuning the model 10b in consideration of the diverse preferences of the interested party 20a.

[0014] In this embodiment, the model 10b is a machine learning model trained using past case data 10a as training data to support the decision-making of the hiring manager. Specifically, the model 10b receives data on a job seeker and outputs a judgment result on whether or not the job seeker can be hired based on whether or not multiple indicators (e.g., age, career, desired annual income, etc.) meet the hiring criteria. The model 10b is not limited to a model for supporting the decision-making of the hiring manager, and may be applied to a machine learning model trained to support the decision-making of the examiner (whether or not to grant a loan) in loan screening, for example.

[0015] Stakeholders 20a involved in the decision-making of the recruiter using the past case data 10a include not only the recruiter but also job seekers.

[0016] As an example, a recruiter has a preference to minimize the possibility of mistakenly hiring a job seeker who does not meet the hiring criteria (i.e., to minimize the false positive probability and maximize the specificity index). A job seeker has a preference to avoid being mistakenly determined not to be hired despite meeting the hiring criteria (i.e., to minimize the false positive probability and maximize the sensitivity index). Thus, each of the interested parties 20a has a variety of preferences.

[0017] Therefore, in the information processing system 100, the model 10b is tuned in consideration of the preferences of the recruiters and job seekers included in the interested parties 20a.

[0018] In this embodiment, an information processing system 100 including multiple information processing devices, namely, the model learning device 1, the questionnaire device 2, and the model adjustment device 3, is exemplified, but the information processing system 100 may be configured such that the processing of the model learning device 1, the questionnaire device 2, and the model adjustment device 3 is executed by a single information processing device.

[0019] The model learning device 1 has a model learning unit 10, and learns a model 10b (S1). Figure 3 is an explanatory diagram for explaining an overview of model learning.

[0020] As shown in Figure 3, the model learning unit 10 is a processing unit that performs machine learning of a model 10b based on past case data 10a in which job seeker data (age, career history, desired annual income, etc.) is associated with the results of whether or not the job seeker will be hired.

[0021] Specifically, the model learning unit 10 performs machine learning of the model 10b based on past case data 10a while repeatedly adjusting the trade-off relationships between multiple indicators M (M=1, 2, ..., m) such as age, career history, and desired annual income, and forms an m-dimensional Pareto surface (multi-objective optimization).

[0022] The multi-objective optimization problem in machine learning of Model 10b can be defined as follows:

[0023]

number

[0024] where f k (x) is an objective function, k (k≧2) is the number of objective functions, and X is a matrix of options (a matrix of a plurality of indexes in repeated learning using past case data 10a).

[0025] In a multi-objective optimization problem, since it is not possible to simultaneously optimize all objective functions, attention is focused on the solution set that is Pareto optimal.

[0026] A feasible solution x1∈X Pareto dominates x2∈X if the following equations (2) and (3) are satisfied.

[0027]

number

[0028]

number

[0029] Argument x * If ∈X is not dominated by any other solution (Pareto optimal), then the solution set X * is called the Pareto surface.

[0030] The model learning unit 10 acquires table data (Pareto surface data 11) of an m-dimensional Pareto surface (solution set) for a plurality of indexes through machine learning of a model 10b using past case data 10a.

[0031] 1 and 2, the questionnaire device 2 has a questionnaire unit 20, and obtains an evaluation result indicating an evaluation (preference) for each of a plurality of indicators M (M=1, 2, ..., m) in the machine learning model (model 10b) by conducting a questionnaire to the interested parties 20a (S2). Note that the interested parties 20a who conduct the questionnaire are people randomly selected from people (recruiters, job seekers) who have been registered in advance as users of the model 10b.

[0032] Fig. 4 is an explanatory diagram for explaining an outline of the questionnaire. As shown in Fig. 4, the questionnaire unit 20 receives evaluations (preferences) for each of a plurality of indexes M (M = 1, 2, 3, 4) by operation input (such as a slider operation) from the interested party 20a via the UI (User Interface) 20b. Specifically, the questionnaire unit 20 receives preference values ​​for each index between 0.0 and 1.0 from each of the interested parties 20a (i = 1, ..., 10).

[0033] Here, the preference value (evaluation value) of individual (i) for each index M is V iM The questionnaire section 20 obtains the preference value V iM The input data is converted into a table to generate stakeholder preference data 21.

[0034] The stakeholder preference data 21 includes, for each individual (i) of the stakeholder 20a, a preference value V for each indicator M (M=1, 2, 3, 4) along with the stakeholder group of recruiters / job seekers at the time of user registration. iM is shown.

[0035] In the illustrated stakeholder preference data 21, the preferences of individuals i=4, 5 and the preferences of individuals i=9, 10 have different selection patterns from other individuals among job seekers. For example, the preference patterns of individuals i=9, 10 show large individual differences in the index M=1, 2, but show small individual differences in the index M=3, 4.

[0036] Returning to FIGS. 1 and 2, the model adjustment device 3 has a clustering unit 30 , an intra-cluster preference value calculation unit 31 , an entire-cluster preference value calculation unit 32 , and a Pareto solution calculation unit 33 .

[0037] The clustering unit 30 uses K-means clustering to derive an evaluation result (preference value V iM ) are clustered. The clustering unit 30 calculates the variance of the evaluation results in each cluster by this clustering, and determines the pattern that minimizes the variance. In this way, by determining the pattern that minimizes the variance of the evaluation results in each cluster, the clustering unit 30 selects and discards multiple indicators, and uses the clustering result with the highest consistency within the cluster (the lowest variance).

[0038] In this way, by using the clustering result with the highest intra-cluster consistency (lowest variance), individuals with similar preference patterns can be accurately grouped (clustered).

[0039] For example, assuming that individuals cannot express their preferences by taking into account the balance relationship of all indicators, one can assume that there are indicators that are not taken into consideration (are not important) for many people.

[0040] Furthermore, if all indicators are used in the clustering aggregation, unimportant indicators will act as noise, resulting in individuals with different preference patterns being mixed within a cluster, which reduces the consistency of preferences within a cluster.

[0041] Therefore, by creating multiple patterns with different indices used for clustering and selecting the combination pattern of indices that maximizes consistency within a cluster, the similarity of the preference patterns of individuals within the cluster can be maximized, enabling accurate modeling of stakeholder subgroups. In other words, individuals (minority) with similar preference patterns can be accurately clustered, and minority opinions can be appropriately reflected.

[0042] Specifically, the clustering section 30 has a clustering calculation section 30a, a Euclidean distance calculation section 30b, an intra-cluster variance calculation section 30c, and a minimum variance calculation section 30d.

[0043] The clustering calculation unit 30a uses K-means clustering to calculate an evaluation result (preference value V iM ) is a processing unit that performs clustering calculation (S3).

[0044] Fig. 5 is an explanatory diagram for explaining an outline of the clustering calculation. As shown in Fig. 5, the clustering calculation unit 30a performs K-means clustering for each pattern (p) obtained by changing the combination of a plurality of indexes M (M = 1, 2, 3, 4).

[0045] For example, in K-means clustering, the clustering calculation unit 30a selects k points as initial center points in the feature space. In the feature space, preference values ​​of multiple machine learning indices are used (i.e., preference values ​​V for M=1, 2, 3, 4). iM ). Next, the clustering calculation unit 30a forms clusters by assigning the preference value of each person (i) to the closest center point. Next, the clustering calculation unit 30a replaces the center point of each cluster with the center of the points that belong to each center point. Then, the clustering calculation unit 30a repeats the cluster formation and center point calculation until the position of the center point stops moving.

[0046] The total number of combination patterns p is p = Σ m r=1m C m-r+1 Individual (i) is assigned to one of the clusters for all patterns. The number of clusters k is determined from multiple candidates by the well-known elbow method.

[0047] Clustering calculation unit 30a generates k clusters (cluster numbers K=1, ..., k) for each pattern number P (P=1, ..., p). p Let us assume that.

[0048] In the illustrated example of the calculation result 30e, the number of indices m is 4, so the total number of patterns is p=15. p As an example of the output, when P=1 (M=1,2,3,4), individual (i) is classified into four clusters, K1=0,1,2,3.

[0049] Returning to FIGS. 1 and 2, the Euclidean distance calculation unit 30b is a processing unit that calculates (S4) the Euclidean distance between the average value of the preference values ​​of all individuals included in the stakeholder preference data 21 and the preference value of each individual.

[0050] 6 is an explanatory diagram for explaining an outline of the Euclidean distance calculation. As shown in FIG. 6, the Euclidean distance calculation unit 30b calculates the preference values ​​(V iM ) is obtained. Next, the Euclidean distance calculation unit 30b calculates the average value of all the individual preferences (V iM (upper bar)) and compare the preference values ​​of all individuals with the average (V iM ) and the Euclidean distance (d pi ) and obtain the result 30f.

[0051] Here, the Euclidean distance is calculated as in the following formula (4), and the average value is calculated as in the following formula (5).

[0052]

number

[0053]

number

[0054] In addition, cluster Kp The Euclidean distance (d pi ) is d PK Let us assume that this d PK takes a common value for all patterns.

[0055] Returning to FIGS. 1 and 2, the intra-cluster variance calculation unit 30c is a processing unit that calculates (S5) the variance of the evaluation results within a cluster for each combination pattern.

[0056] 7 is an explanatory diagram for explaining an outline of intra-cluster variance calculation. As shown in FIG. 7, the intra-cluster variance calculation unit 30c calculates the d PKi The variance value 30h is calculated using the following formula (6).

[0057]

number

[0058] d PK (Upper bar) is the d of all individuals in the cluster PK The average value, 30g, can be calculated using the following formula (7).

[0059]

number

[0060] 1 and 2, the minimum variance calculation unit 30d determines a combination pattern that satisfies a predetermined condition from among a plurality of combination patterns based on the variance of the evaluation results calculated by the intra-cluster variance calculation unit 30c. Specifically, the minimum variance calculation unit 30d performs an intra-cluster variance average minimum calculation (S6) and determines a pattern with the minimum variance from among the plurality of combination patterns. Note that the minimum variance calculation unit 30d may select and determine from a plurality of patterns whose variance is equal to or less than a predetermined threshold.

[0061] When the minimum variance calculation unit 30d determines a pattern from among a plurality of patterns with the minimum variance or a plurality of patterns with a variance equal to or less than a predetermined threshold, the predetermined pattern may be selected based on the number of indexes that the pattern has.

[0062] For example, a pattern with a large number of indicators is highly likely to have a positive or negative proportional relationship or an inverse proportional relationship between the indicators. In this case, a pattern that holds more information about the relationship between the indicators (a pattern with a large number of indicators) has more elements that explain the preference pattern of the stakeholder group. Therefore, when determining a pattern from among a plurality of patterns, the minimum variance calculation unit 30d selects a pattern with a large number of indicators.

[0063] 8 is an explanatory diagram for explaining an outline of the calculation of the minimum intra-cluster variance average value. As shown in FIG. 8, the minimum variance value calculation unit 30d calculates the variance value Mean of each pattern by performing calculation according to the following formula (8). p The calculation result is 30i.

[0064]

number

[0065] Next, the minimum variance calculation unit 30d calculates the argmin Mean p The minimum variance calculation unit 30d determines a pattern p such that p=6, 12, 13, and obtains a calculation result 30i. In the illustrated example, three patterns (P=6, 12, 13) are obtained with the same probability as the calculation result 30i. From among these three patterns, the minimum variance calculation unit 30d determines the pattern (P=6) with the largest number of indexes, as described above.

[0066] 1 and 2, the intra-cluster preference value calculation unit 31 is a processing unit that calculates preferences (evaluations) for each of a plurality of indices for each cluster based on the preference results included in each cluster by clustering performed on the determined combination pattern. Specifically, the intra-cluster preference value calculation unit 31 performs intra-cluster preference value calculation (S7), calculates the average value of the preference values ​​of individuals (i) in each cluster for all indices, and calculates the average value for each cluster.

[0067] FIG. 9 is an explanatory diagram for explaining an outline of intra-cluster preference value calculation. As shown in FIG. 9, the intra-cluster preference value calculation unit 31 obtains a calculation result 30k by merging the stakeholder preference data 21 and the clustering result (P=6(M=1,2)) by the clustering unit 30. Next, the intra-cluster preference value calculation unit 31 calculates the average value of the preference values ​​of the individuals (i) in each cluster for all indexes, and represents the average value of each cluster as V PKM The calculation result is 30l.

[0068] Returning to FIGS. 1 and 2, the cluster-wide preference value calculation unit 32 is a processing unit that performs cluster-wide preference value calculation (S8) for aggregating the preference values ​​of all clusters.

[0069] FIG. 10 is an explanatory diagram for explaining an outline of the cluster whole preference value calculation. As shown in FIG. 10, the cluster whole preference value calculation unit 32 calculates V PKM Calculate the geometric mean of V ave PKM The calculation result is 30m, i.e., the aggregated evaluation result.

[0070] Returning to FIGS. 1 and 2, the Pareto solution calculation unit 33 is a processing unit that performs Pareto solution calculation (S9) to determine solutions for each of the multiple indexes in the model 10b based on the calculation result 30m.

[0071] Specifically, as shown in FIG. 10, the Pareto solution calculation unit 33 selects a V ave PKMThe Pareto solution that has the smallest Euclidean distance to the selected Pareto solution is selected. This Pareto solution is a parameter set for tuning the model 10b that takes into account the preferences of each stakeholder 20a. The Pareto solution calculation unit 33 outputs an index parameter set 34 including each index value of the selected Pareto solution as tuning data for the model 10b.

[0072] Fig. 11 is an explanatory diagram for explaining an outline of the index parameter set. Case C1 in Fig. 11 is a case in which preference values ​​are simply collected for a stakeholder group of recruiters and job seekers, and tuning data for model 10b is calculated. Case C1 is a case in which tuning data for model 10b is calculated in the above-mentioned embodiment.

[0073] As shown in Fig. 11, in case C1, calculation result 35b for tuning model 10b is obtained based on calculation result 35a, which is a simple aggregation of preference values ​​for the stakeholder group of recruiters and job seekers. Therefore, in case C1, the preferences of individuals i=4,5 and i=9,10 are affected by the difference in numbers due to other stakeholders and are under-reflected in calculation result 35b.

[0074] In contrast, in case C2, the preferences of individuals i=4,5 and i=9,10 are reflected in the indicator parameter set 34 (with the same weights as the other clusters) without being affected by the differences in numbers of other stakeholders.

[0075] As described above, the information processing system 100 obtains evaluation results indicating evaluations for each of the multiple indicators in the machine learning model (model 10b). The information processing system 100 clusters the evaluation results for each combination pattern of the multiple indicators. The information processing system 100 calculates the variance of the evaluation results in the cluster for each combination pattern. The information processing system 100 determines a combination pattern that satisfies a predetermined condition from among the multiple combination patterns based on the calculated variance of the evaluation results. The information processing system 100 aggregates the evaluations for each of the multiple indicators for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern. The information processing system 100 determines a solution for each of the multiple indicators in the model 10b based on the aggregated evaluations for each of the multiple indicators.

[0076] As a result, the information processing system 100 can obtain a solution that properly reflects even a small number of evaluation results, and can properly tune the model.

[0077] Furthermore, the information processing system 100 determines a pattern with the smallest variance of the evaluation results from among the multiple combination patterns. This allows the information processing system 100 to determine solutions for each of the multiple indicators in the model 10b based on the clustering result with the highest intra-cluster consistency.

[0078] Furthermore, the information processing system 100 determines a solution that has a close Euclidean distance to the evaluation for each of the multiple indicators from among the solution sets (Pareto surface data 11) for each of the multiple indicators in the model 10b. This allows the information processing system 100 to obtain a solution that is closer to the evaluation for each of the multiple indicators.

[0079] Moreover, the evaluation result in the information processing system 100 is the result of a questionnaire survey of people (interested parties 20a) related to the judgment using the model 10b. This enables the information processing system 100 to tune the model to reflect the opinions of people related to the judgment using the model 10b.

[0080] Note that each component of each device shown in the figure does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0081] In addition, the various processing functions of the model learning unit 10, the questionnaire unit 20, the clustering unit 30, the intra-cluster preference value calculation unit 31, the entire-cluster preference value calculation unit 32, and the Pareto solution calculation unit 33 performed in the model learning device 1, the questionnaire device 2, and the model adjustment device 3 may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or an MCU (Micro Controller Unit)). Needless to say, the various processing functions may be executed in whole or in part on a program analyzed and executed by a CPU (or a microcomputer such as an MPU or an MCU), or on hardware using wired logic. In addition, the various processing functions performed in the model learning device 1, the questionnaire device 2, and the model adjustment device 3 may be executed in an information processing device (computer) such as a single server device, or may be executed by multiple computers working together using cloud computing.

[0082] The various processes described in the above embodiment can be realized by executing a program prepared in advance on a computer. Therefore, an example of a computer configuration (hardware) that executes a program having the same functions as those in the above embodiment will be described below. Figure 12 is an explanatory diagram for explaining an example of a computer configuration.

[0083] 12, computer 200 has CPU 201 for executing various arithmetic processes, input device 202 for receiving data input, monitor 203, and speaker 204. Computer 200 also has medium reading device 205 for reading programs and the like from a storage medium, interface device 206 for connecting with various devices, and communication device 207 for connecting to external devices by wire or wirelessly. Computer 200 also has RAM 208 for temporarily storing various information, and hard disk device 209. Each unit (201 to 209) in computer 200 is connected to bus 210.

[0084] The hard disk drive 209 stores a program 211 for executing various processes in the functional configurations (e.g., the model learning unit 10, the questionnaire unit 20, the clustering unit 30, the intra-cluster preference value calculation unit 31, the entire-cluster preference value calculation unit 32, and the Pareto solution calculation unit 33) described in the above embodiment. The hard disk drive 209 also stores various data 212 referenced by the program 211. The input device 202, for example, accepts input of operation information from an operator. The monitor 203, for example, displays various screens operated by the operator. The interface device 206 is connected to, for example, a printing device. The communication device 207 is connected to a communication network such as a LAN (Local Area Network), and exchanges various information with external devices via the communication network.

[0085] The CPU 201 reads out the program 211 stored in the hard disk device 209, expands it in the RAM 208, and executes it to perform various processes related to the above-mentioned functional configuration (for example, the model learning unit 10, the questionnaire unit 20, the clustering unit 30, the intra-cluster preference value calculation unit 31, the entire-cluster preference value calculation unit 32, and the Pareto solution calculation unit 33). The program 211 does not have to be stored in the hard disk device 209. For example, the program 211 stored in a storage medium readable by the computer 200 may be read out and executed. The storage medium readable by the computer 200 corresponds to, for example, a portable storage medium such as a CD-ROM or a DVD disk, a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, a hard disk drive, etc. Also, the program 211 may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 200 may read out and execute the program 211 from the device.

[0086] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0087] (Appendix 1) Obtain an evaluation result showing an evaluation for each of a plurality of indicators in a machine learning model, clustering the evaluation results for each combination pattern of the plurality of indexes; Calculating the variance of the evaluation results within a cluster for each of the combination patterns; determining a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results; aggregating evaluations for each of the plurality of indexes for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern; determining a solution for each of the plurality of indicators in the machine learning model based on an evaluation for each of the plurality of indicators that have been collected; An information processing program that causes a computer to execute a process.

[0088] (Additional Note 2) The process of determining the combination pattern determines a pattern having a minimum variance of the evaluation results from among the plurality of combination patterns. 2. The information processing program according to claim 1,

[0089] (Appendix 3) The process of determining the solution determines a solution having a close Euclidean distance to an evaluation for each of the multiple indicators from among a solution set for each of the multiple indicators in the machine learning model. 2. The information processing program according to claim 1,

[0090] (Appendix 4) The evaluation results are the results of a survey of people related to the judgment using the machine learning model. 2. The information processing program according to claim 1,

[0091] (Appendix 5) Obtain an evaluation result showing an evaluation for each of a plurality of indicators in the machine learning model, clustering the evaluation results for each combination pattern of the plurality of indexes; Calculating the variance of the evaluation results within a cluster for each of the combination patterns; determining a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results; aggregating evaluations for each of the plurality of indexes for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern; determining a solution for each of the plurality of indicators in the machine learning model based on an evaluation for each of the plurality of indicators that have been collected; An information processing method characterized in that the processing is executed by a computer.

[0092] (Additional Note 6) The process of determining the combination pattern determines a pattern having a minimum variance of the evaluation result from among the plurality of combination patterns. 6. The information processing method according to claim 5,

[0093] (Appendix 7) The process of determining the solution determines a solution having a close Euclidean distance to an evaluation for each of the multiple indicators from among a solution set for each of the multiple indicators in the machine learning model. 6. The information processing method according to claim 5,

[0094] (Appendix 8) The evaluation results are the results of a survey of people related to the judgment using the machine learning model. 6. The information processing method according to claim 5,

[0095] (Appendix 9) Obtain an evaluation result showing an evaluation for each of a plurality of indicators in the machine learning model, clustering the evaluation results for each combination pattern of the plurality of indexes; Calculating the variance of the evaluation results within a cluster for each of the combination patterns; determining a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results; aggregating evaluations for each of the plurality of indexes for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern; determining a solution for each of the plurality of indicators in the machine learning model based on an evaluation for each of the plurality of indicators that have been collected; 1. An information processing device comprising: a control unit that executes processing.

[0096] (Additional Note 10) The process of determining the combination pattern determines a pattern having a minimum variance of the evaluation result from among the plurality of combination patterns. 10. The information processing device according to claim 9,

[0097] (Appendix 11) The process of determining the solution determines a solution having a close Euclidean distance to an evaluation for each of the multiple indicators from among a solution set for each of the multiple indicators in the machine learning model. 10. The information processing device according to claim 9,

[0098] (Appendix 12) The evaluation results are the results of a survey of people related to the judgment using the machine learning model. 10. The information processing device according to claim 9, [Explanation of symbols]

[0099] 1...Model learning device 2. Questionnaire device 3. Model adjustment device 10…Model Learning Section 10a…Past case data 10b…Model 11...Pareto surface data 20…Survey Section 20a…Interested parties 20b…UI 21...Stakeholder preference data 30…Clustering Department 30a…Clustering calculation section 30b…Euclidean distance calculation section 30c…Intra-cluster distributed computing section 30d…Minimum variance calculation section 30e~30m, 35a, 35b...calculation results 31…Intra-cluster preference value calculation section 32…Cluster-wide preference value calculation section 33...Pareto solution calculation section 34…Indicator parameter set 100...Information processing systems 200…Computer 201...CPU 202...Input device 203…Monitor 204…Speaker 205... Media reader 206...Interface device 207...Communication equipment 208…RAM 209...Hard disk drive 210…Bus 211…Program 212...Various data C1, C2…Case

Claims

1. Obtaining an evaluation result indicating an evaluation for each of a plurality of indicators in the machine learning model; clustering the evaluation results for each combination pattern of the plurality of indexes; Calculating the variance of the evaluation results within a cluster for each of the combination patterns; determining a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results; aggregating evaluations for each of the plurality of indexes for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern; determining a solution for each of the plurality of indicators in the machine learning model based on an evaluation for each of the plurality of indicators that have been collected; An information processing program that causes a computer to execute a process.

2. The process of determining the combination pattern determines a pattern having a minimum variance of the evaluation results from among the plurality of combination patterns.

2. The information processing program according to claim 1,

3. The process of determining the solution includes determining a solution having a short Euclidean distance to an evaluation for each of the plurality of indicators from among a solution set for each of the plurality of indicators in the machine learning model.

2. The information processing program according to claim 1,

4. The evaluation result is a survey result for people related to the judgment using the machine learning model.

2. The information processing program according to claim 1,

5. Obtaining an evaluation result indicating an evaluation for each of a plurality of indicators in the machine learning model; clustering the evaluation results for each combination pattern of the plurality of indexes; Calculating the variance of the evaluation results within a cluster for each of the combination patterns; determining a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results; aggregating evaluations for each of the plurality of indexes for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern; determining a solution for each of the plurality of indicators in the machine learning model based on an evaluation for each of the plurality of indicators that have been collected; An information processing method characterized in that the processing is executed by a computer.

6. Obtaining an evaluation result indicating an evaluation for each of a plurality of indicators in the machine learning model; clustering the evaluation results for each combination pattern of the plurality of indexes; Calculating the variance of the evaluation results within a cluster for each of the combination patterns; determining a combination pattern that satisfies a predetermined condition from among the plurality of combination patterns based on the calculated variance of the evaluation results; aggregating evaluations for each of the plurality of indexes for each cluster based on the evaluation results included in each cluster obtained by clustering the determined combination pattern; determining a solution for each of the plurality of indicators in the machine learning model based on an evaluation for each of the plurality of indicators that have been collected; 1. An information processing device comprising: a control unit that executes processing.

Images