Information processing device, information processing method, and program

Abstract

An information processing device according to one embodiment of the present technology comprises a first learning unit, a second learning unit, an evaluation unit, and an adjustment unit. The first learning unit is caused to learn a prescribed learning model. The second learning unit is caused to learn a conversion model for converting the output of the prescribed learning model to a rule group that is notated in a format which can be interpreted by a user. The evaluation unit acquires evaluation information in which the rule group has been evaluated according to a prescribed standard. The adjustment unit adjusts the learning of the prescribed learning model on the basis of the evaluation information.

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a program that can be applied to constructing a learning model using machine learning. [Background technology]

[0002] Conventionally, there are known techniques for checking whether a processing result using a computer violates laws and regulations. For example, Patent Document 1 describes an ethical review support system that supports the review of whether the contents of video data violate ethical regulations. In this system, for example, the levels of various parameters related to the video expression (blood color, frequency, etc.) are analyzed to determine whether the video data violates ethical regulations. In this way, the system automatically analyzes the video expression, making it possible to obtain objective review results (paragraphs

[0046] ,

[0051] ,

[0056] , etc. of the specification of Patent Document 1).

[0003] In recent years, technologies that use machine learning to predict various objects and to realize image recognition, etc., have been developed. By using a learning model, it is possible to predict, for example, whether or not a person who comes to an interview at a company should be hired, or to predict the expected length of employment of the person who comes to an interview. When making such predictions, it is important to check whether the processing by the model meets the necessary standards such as laws and regulations. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2002-149884 A Summary of the Invention [Problem to be solved by the invention]

[0005] Learning models using machine learning are expected to be applied in a variety of situations, and there is a demand for technology that makes it easy to build learning models that meet the necessary standards.

[0006] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program that are capable of easily constructing a learning model that meets required standards. [Means for solving the problem]

[0007] In order to achieve the above object, an information processing device according to an embodiment of the present technology includes a first learning unit, a second learning unit, an evaluation unit, and an adjustment unit. The first learning unit learns a predetermined learning model. The second learning unit trains a conversion model that converts an output of the predetermined learning model into a set of rules described in a format that can be interpreted by a user. The evaluation unit acquires evaluation information that evaluates the rule group according to a predetermined criterion. The adjustment unit adjusts the learning process of the predetermined learning model based on the evaluation information.

[0008] In this information processing device, a learning process is performed between a predetermined learning model and a conversion model that converts the output of the learning model. The conversion model converts the output of the learning model into a set of rules described in a format that can be interpreted by a user. The learning process of the learning model is adjusted using evaluation information that evaluates the set of rules according to a predetermined standard. This makes it possible to easily build a learning model that meets the required standards.

[0009] The predetermined criteria may include at least one of criteria defined by laws and regulations and criteria defined by the user.

[0010] The learning model may be a predictive model that predicts a target item.

[0011] The set of rules may include at least one output rule that explains an output of the predictive model, in which case the evaluator may generate at least one of an explanatory text and a diagram for each of the output rules.

[0012] The evaluation section may generate a check item for each of the output rules, which causes the user to check whether or not the predetermined criterion is satisfied.

[0013] The evaluation unit may read, as the evaluation information, a check result of the user for each check item.

[0014] The evaluation unit may generate the check item targeting a data item specified by the user among a plurality of data items included in learning data of the prediction model.

[0015] The information processing device may further include a storage unit in which a database relating to the predetermined criteria is stored. In this case, the evaluation unit may determine whether or not the output rule satisfies the predetermined criteria based on the database.

[0016] The evaluation section may generate, for the output rule that has been determined not to satisfy the predetermined criterion, a check item that causes the user to check whether or not the output rule satisfies the predetermined criterion.

[0017] The evaluation section may generate, as the evaluation information, information regarding the output rules that are determined not to satisfy the predetermined criterion.

[0018] The evaluation information may include information on a violation rule that is the output rule that does not satisfy the predetermined criterion. In this case, the adjustment unit may adjust at least one of learning data of the prediction model or learning parameters of the prediction model based on a data range specified by the violation rule.

[0019] The adjustment unit may perform at least one of a process of reducing the number of learning data included in the data range specified by the violation rule that causes the violation rule to not satisfy the specified criterion, and a process of adding dummy data adjusted to satisfy the specified criterion to the data range specified by the violation rule as the learning data.

[0020] The learning parameters may include at least one of a parameter for adjusting an output of the prediction model regarding the training data, or a parameter for adjusting a loss function of the prediction model.

[0021] The prediction model may be a classification model in which a classification for the target item is used as a predicted value. In this case, the adjustment unit may adjust a learning process of the prediction model so that a predicted value of the prediction model in a data range specified by the violation rule substantially matches a predicted value of the prediction model in a data range specified by the output rule that satisfies the predetermined criterion.

[0022] The prediction model may be a regression model in which a value of the target item is used as a predicted value. In this case, the adjustment unit may adjust a learning process of the prediction model so that a distribution of predicted values ​​of the prediction model in a data range specified by the violation rule substantially matches a distribution of predicted values ​​of the prediction model in a data range specified by the output rule that satisfies the predetermined criterion.

[0023] The evaluation unit may present a plurality of adjustment methods for an output of the prediction model in a selectable manner, and in this case, the adjustment unit may adjust the learning process of the prediction model based on a method selected by the user from the plurality of adjustment methods.

[0024] The second learning unit may learn the conversion model according to the predetermined criterion.

[0025] The conversion model may be a learning model using at least one of a decision tree and a rule fitting algorithm.

[0026] An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes training a predetermined learning model. A conversion model is trained to convert the output of the predetermined learning model into a set of rules described in a format that can be interpreted by a user. Evaluation information is obtained by evaluating the rule group according to a predetermined criterion. A learning process of the predetermined learning model is adjusted based on the evaluation information.

[0027] A program according to an embodiment of the present technology causes a computer system to execute the following steps. A step of training a predetermined learning model. A step of training a conversion model that converts the output of the predetermined learning model into a set of rules written in a format that can be interpreted by a user. A step of acquiring evaluation information that evaluates the group of rules according to a predetermined criterion. Adjusting the learning process of the predetermined learning model based on the evaluation information. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a schematic diagram for explaining an outline of an operation of a data analysis device according to an embodiment of the present technology. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a data analysis device. [Diagram 3] 1 is a table illustrating an example of a learning database. [Figure 4] 11 is a table illustrating an example of a rule database. [Diagram 5] FIG. 1 is a schematic diagram for explaining a summary model. [Figure 6] 10 is a flowchart illustrating an example of a basic operation of the data analysis apparatus. [Figure 7]13 is an example of a UI screen for specifying learning data to be used in a predictive model. [Figure 8] 13 is an example of a UI screen for inputting various settings related to a prediction model. [Figure 9] 13 is an example of an evaluation screen that displays characteristics of a prediction model. [Figure 10] 13 is another example of an evaluation screen displaying characteristics of a prediction model. [Figure 11] 13 is an example of a check screen for an output rule. [Figure 12] 13 is another example of a check screen for an output rule. [Figure 13] 13 is an example of a check screen displayed when a regression model is used. [Figure 14] 1 is a map of the output of a comparative predictive model. [Figure 15] 13 is an example of a check screen displayed when an authentication model is used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0030] [Data analysis equipment configuration] 1 is a schematic diagram for explaining an outline of an operation of a data analysis device 100 according to an embodiment of the present technology. The data analysis device 100 is configured using a computer or the like, and is a device that provides a predictive analysis tool for performing predictive analysis or the like. Here, predictive analysis is a technique for predicting future events based on past cases, for example, by machine learning. In predictive analysis, a predictive model 10 constructed according to the target of prediction is used. By using predictive analysis (predictive model 10), it is possible to predict, for example, whether a new interviewee should be hired using the results of a past personnel interview. Of course, the target of predictive analysis is not limited and can be set arbitrarily. In this embodiment, the predictive model 10 is an example of a predetermined learning model. A user of the data analysis device 100 can construct a predictive model 10 that predicts a desired item by specifying training data 1, input items, model parameters, etc., via a GUI (Graphical User Interface) displayed on a display (not shown), for example.

[0031] FIG. 1 is a schematic diagram showing the flow of basic operations performed by a data analysis device 100 when constructing a prediction model 10. As shown in FIG. 1, a data analysis device 100 executes a step of constructing a prediction model 10, a step of constructing a summary model 11, a step of evaluating the model, and a step of adjusting the prediction model 10.

[0032] The summary model 11 is a learning model that converts the output of the prediction model 10 into a format that can be interpreted by a user. In this embodiment, the summary model 11 corresponds to a conversion model. In general, since the processing contents of the prediction model 10 using machine learning are complex, it is often difficult to extract, for example, rules (conditions) expressing the output from the output of the prediction model 10. For example, it is difficult to interpret the conditions of the data that apply to the prediction result just by looking at the distribution of the prediction results indicated by the output of the prediction model 10. The summary model 11 generates a set of rules that summarize (approximate) such a complex output. Here, the set of rules means a collection of rules consisting of at least one rule that explains the output of the prediction model 10. These rules are described using conditions that can be interpreted by a user, for example. This makes it possible to easily interpret the contents of even complex output. Hereinafter, the rules that explain the output of the prediction model 10 are referred to as output rules.

[0033] The step of evaluating the model is a step of checking whether the prediction process by the prediction model 10 satisfies a predetermined standard. In the data analysis device 100, the output of the prediction model 10 is evaluated based on the set of rules generated by the summary model 11, i.e., the set of rules summarizing the output of the prediction model 10. The predetermined criteria include criteria stipulated by laws and regulations. For example, in a situation where a company predicts whether or not to hire a person who has come for an interview, the output (rule set) of the prediction model 10 is evaluated according to the criteria stipulated by the Equal Employment Opportunity Law and other laws. The predetermined criteria may also include criteria set by the user. For example, even when predicting the hiring of interviewees, depending on the company's situation (e.g., the gender and age composition of employees), it may be preferable not to use the criteria set by laws and regulations as they are. In such cases, the rule group may be evaluated according to criteria set by the user according to the situation (e.g., the user's judgment).

[0034] In the data analysis apparatus 100, information on evaluation of a rule group (output rule) according to a predetermined criterion is generated. The information on evaluation of the rule group includes, for example, a determination result on whether or not each output rule satisfies the predetermined criterion, information indicating the degree to which the output rule deviates from the predetermined criterion, etc. Furthermore, the determination as to whether each output rule satisfies a predetermined criterion may be made by a user, or may be made automatically by data analysis apparatus 100 (explanation generating unit 24 described later). Hereinafter, information obtained by evaluating a rule group according to a predetermined criterion will be referred to as “evaluation information.” The evaluation information can be said to be information indicating whether the output of the prediction model 10 satisfies the predetermined criterion.

[0035] In the step of adjusting the prediction model 10, the learning process of the prediction model 10 is adjusted based on the evaluation information. Specifically, the learning process of the prediction model 10 is adjusted so that the output of the prediction model 10 satisfies a predetermined standard, that is, so that an output rule that does not satisfy the predetermined standard is not extracted from the output of the prediction model 10. By repeating the adjustment of the learning process in this manner, it is possible to build a prediction model 10 that satisfies the predetermined standard. The method of adjusting the learning process will be described in detail later.

[0036] For example, when the output of the predictive model 10 does not meet a predetermined standard, the prediction by the predictive model 10 may be ethically questionable or may lack fairness. For example, a predictive model 10 that biases predicted values ​​such as hiring rates by gender or age may be considered an ethical violation. In the data analysis device 100, two learning models are prepared: a "prediction model 10" and a "summary model 11" that summarizes the prediction model 10, and the "summary model 11" is used to check whether there are any ethical issues in the output of the "prediction model 10." The "prediction model 10" is then adjusted using the check results. Therefore, by using the data analysis device 100, it is possible to check whether there are any ethical issues in the prediction model 10 and correct any issues that may exist.

[0037] This makes it possible for even a user who does not understand the algorithm of the prediction model 10 to easily check whether the output of the prediction model 10 poses any ethical issues. In addition, for any part of the output of the prediction model 10 that may be problematic in terms of ethical issues, the output of the prediction model 10 can be corrected through the summary model 11. Furthermore, it is possible to automatically check whether the predictive model 10 contains any ethical issues, etc., and it is also possible to correct problems that the user is not aware of. As a result, it becomes possible to easily construct a prediction model 10 that meets the required standards.

[0038] 2 is a block diagram showing an example of the configuration of data analysis apparatus 100. Data analysis apparatus 100 has a display unit 12, an operation unit 13, a communication unit 14, a storage unit 15, and a control unit 16.

[0039] The display unit 12 is a display that displays various pieces of information, and displays, for example, the above-mentioned GUI screen. For example, a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display is used as the display unit 12. The specific configuration of the display unit 12 is not limited, and for example, a display equipped with a touch panel or the like that functions as the operation unit 13 may be used. Furthermore, a head mounted display (HMD) may be used as the display unit 12.

[0040] The operation unit 13 includes an operation device for the user to input various information. As the operation unit 13, for example, a device capable of inputting information, such as a mouse, a keyboard, or a trackpad, is used. The specific configuration of the operation unit 13 is not limited to this. For example, a touch panel or the like may be used as the operation unit 13. Furthermore, as the operation unit 13, a camera or the like that captures an image of the user may be used, and input may be possible by gaze or gesture.

[0041] The communication unit 14 is a module that performs communication processing between other devices and the data analysis device 100. The communication unit 14 is configured, for example, with a wireless LAN (Local Area Network) module such as Wi-Fi, or a wired LAN module. In addition, a communication module capable of short-range wireless communication such as Bluetooth (registered trademark), optical communication, or the like may be used.

[0042] The storage unit 15 is a non-volatile storage device, such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 15 stores a learning database 17 and a rule database 18. The storage unit 15 also functions as a storage unit of the data analysis device 100, and stores programs executed by the control unit 16, etc.

[0043] 3 is a table showing an example of the learning database 17. The learning database 17 is a database in which learning data 1 for learning (training) the prediction model 10 is stored. In the example shown in FIG. 3, personnel data on job applicants who have been selected for employment in the past is used as learning data 1. Each learning data set 1 includes the following items: the job applicant's ID, application period, age, gender, rank, employment type, desired annual salary, number of job changes, whether or not the applicant has registered email address, whether or not they have qualifications, the hiring manager, and the result of the hiring decision. For example, when training a prediction model 10 that judges whether to adopt or not, the result of the adoption judgment becomes the target item to be trained. Also, items other than the target items (the result of the adoption judgment) of the prediction model 10 become parameters used in training the prediction model 10. The learning database 17 is a collection of such learning data 1, and is recorded as data in, for example, CSV format. The format of the learning data 1 is not limited to this.

[0044] 4 is a table showing an example of the rule database 18. The rule database 18 is a database that stores information on laws and regulations and other rules such as general ethics (hereinafter referred to as evaluation rules). In this embodiment, the laws and regulations and the evaluation rules are examples of the predetermined criteria, and the rule database corresponds to a database related to the predetermined criteria.

[0045] 4, the rule database 18 includes items such as rule name, rule field, problem setting, items to note, comparison method, comparison area, area details (area A, area B, ...), etc. The rule database 18 is configured as table data in, for example, CSV format. In the rule name field, the name of the law, evaluation rule, etc., and its description, etc. are input. The rule field field is entered with the field to which the rule applies (for example, human resources field, sales field, insurance field, etc.). The problem setting item records the type of problem that the predictive model 10 solves, such as classification, regression, or object recognition. For example, when "all" is entered in the problem setting item, this means that the applicable laws and regulations are rules that must be followed regardless of the type of predictive model 10. Also, when "classification" is entered in the problem setting item, this means that the applicable laws and regulations are rules that must be followed when the predictive model 10 solves a classification problem. The items to be noted are inputted as the items that are the subject of the laws and regulations or the evaluation rules (for example, age, gender, etc.). In other words, the items to be noted are items that specify what the laws and regulations prescribe. The comparison method field is where you enter the method for comparing the items of interest. For example, if the comparison method is entered as "equally," it means that the items of interest (gender and age) should be treated equally. In this case, you will compare whether there is any bias in the predicted values ​​for each gender and age. In the comparison area field, the area in which the comparison should be performed is entered. For example, if the item to be noted is age, information specifying a comparison for all ages or a comparison for some ages is entered. In the area details field, when areas of interest are to be divided into smaller areas for comparison, information for dividing the areas is entered. The rule database 18 is a collection of such laws and evaluation rules, and is recorded as data in, for example, CSV format. The format of the rule database 18 is not limited to this.

[0046] Returning to Fig. 2, the control unit 16 controls the operation of each block of the data analysis device 100. The control unit 16 has a hardware configuration required for a computer, such as a CPU and memory (RAM, ROM), and functions as an information processing device according to this embodiment. The CPU loads a program stored in the storage unit 15 into the RAM and executes it, thereby executing various processes. As the control unit 16, a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or other device such as an ASIC (Application Specific Integrated Circuit) may be used.

[0047] In this embodiment, the CPU of the control unit 16 executes the program according to this embodiment to realize functional blocks including a UI generation unit 20, a prediction model learning unit 21, a characteristic evaluation unit 22, a summary model learning unit 23, an explanation generation unit 24, a query generation unit 25, and an adjustment processing unit 26. These functional blocks execute the information processing method according to this embodiment. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0048] The UI generation unit 20 generates a UI screen (display interface) to be displayed on the display unit 12. Typically, the UI generation unit 20 generates a GUI such as a UI screen to be displayed when performing a predictive analysis using the predictive model 10. The UI screen displays, for example, information to be presented to the user, input fields for the user to input information, etc. The user can specify various settings, values, etc. by operating the operation unit 13 (keyboard, etc.) while viewing the UI screen. The UI generating unit 20 also accepts information input or specified by the user via the UI screen. Information input via the UI screen includes, for example, information specifying the learning data to be used, the field to which the prediction model 10 is applied (human resources, sales, insurance industry, etc.), the type of problem setting (classification, regression, etc.), items to be noted (age, gender, etc.), and the like. Check items, etc., which will be described later, are displayed on the UI screen that is displayed after the prediction model 10 is generated. Check results for these check items, etc. are accepted via the UI screen. In this way, the UI generation unit 20 realizes a display function on a UI screen that processes screen input, and a data input / output function via the UI screen.

[0049] The prediction model learning unit 21 trains a prediction model 10 that predicts a target item. The learning process of the prediction model 10 is a process of training a learning model designed to predict a target item designated by a user, for example, using a plurality of learning data 1. Typically, the prediction model 10 is configured using a machine learning algorithm (decision tree learning, neural network, etc.) that performs learning using training data 1 in which feature items (weather, day of the week, gender, age, etc.) and target items (values ​​to be predicted) are paired. Therefore, it can be said that the prediction model learning unit 21 performs supervised learning that trains the model using the predicted values ​​of the training data 1 as teacher data. The target item, the algorithm used in the prediction model 10, the learning data 1 to be used, etc. are input via the above-mentioned UI screen, etc. In this way, the prediction model learning unit 21 realizes a prediction model creating function. In this embodiment, the prediction model learning unit 21 corresponds to a first learning unit.

[0050] In this embodiment, a classification model in which a classification regarding a target item is used as a predicted value is used as the prediction model 10. For example, if the target item is an item indicating whether or not a subject will be employed, whether or not the subject will be employed is predicted. Also, for example, if the target item is an item indicating whether or not a product will be purchased, whether or not a customer will purchase the product is predicted. In addition, a prediction model 10 capable of solving any classification problem including multiple options may be used. A regression model in which the value of the target item is the predicted value may be used as the prediction model 10. For example, if the target item is the subject's length of service, the expected length of service of the subject is predicted. Also, if the target item is the date of withdrawal from a service, the expected date of withdrawal of a customer from the service is predicted. The machine learning algorithm used in the prediction model 10 is not limited, and any algorithm may be used as appropriate according to the processing content. The present technology is applicable regardless of the type of algorithm.

[0051] When the predictive model 10 is generated for the first time, or when the predictive model 10 is evaluated as having no ethical issues, the predictive model learning unit 21 performs normal supervised learning to train the predictive model 10 to hit the correct answer recorded in the learning data 1. Furthermore, if there is an ethical problem with the prediction model 10 (for example, if a user points out a rule violation or an ethical violation), the learning process of the prediction model 10 is adjusted. Specifically, the prediction model learning unit 21 changes the learning data 1 used in the learning process and learning parameters set for each algorithm, etc., in accordance with an adjustment instruction generated by an adjustment processing unit 26 described later, and causes the prediction model 10 to learn again. The prediction model learning unit 21 receives learning data 1 to be passed to the prediction model 10, setting data, and adjustment instructions generated by the adjustment processing unit 26. In addition, the prediction model learning unit 21 outputs the created prediction model 10.

[0052] The characteristic evaluation unit 22 evaluates the characteristics of the prediction model 10. Specifically, an evaluation index indicating the performance of the prediction model 10, the prediction tendency, and the like is calculated. For example, when the prediction model 10 is a classification model, the accuracy rate, precision, etc. of the predicted value are calculated as the evaluation index. When the prediction model 10 is a regression model, the error mean, square error, etc. of the predicted value are calculated as the evaluation index. The type of evaluation index is not limited, and for example, an evaluation index according to the type of algorithm of the prediction model 10 may be calculated as appropriate. The created prediction model 10 is input to the characteristic evaluation unit 22. In addition, the characteristic evaluation unit 22 outputs various evaluation indexes and the like. The evaluation index of the prediction model 10 is displayed, for example, on a UI screen and presented to the user (see FIGS. 9 and 10). This enables the user to understand the performance of the prediction model 10.

[0053] The summary model learning unit 23 trains the summary model 11 that converts the output of the prediction model 10 into a set of rules written in a format that can be interpreted by the user. As described with reference to FIG. 1, the summary model 11 is a learning model that outputs, as a set of rules, at least one output rule that explains the output of the prediction model 10. The created prediction model 10 is input to the summary model learning section 23. Also, the created summary model 11 is output from the summary model learning section 23. In this embodiment, the summary model learning section 23 corresponds to a second learning section.

[0054] 5 is a schematic diagram for explaining the summarization model 11. Here, it is assumed that a classification model is used as the prediction model 10, and the summarization model 11 that summarizes the output of the classification model will be explained. 5A is a map showing the distribution of true correct answers of data (learning data 1, etc.) input to the prediction model 10. In this map, items to be classified are classified into category α (the dark gray area on the lower side of the map) and category β (the light gray area on the upper side of the map). The horizontal and vertical axes of the map are the first and second parameters related to the items to be classified. Note that three or more parameters may be set.

[0055] As an example, a case will be described in which a prediction model 10 (classification model) is used to predict whether or not a subject will be hired. In this case, the categories α and β represent the cases where the subject will be hired and where the subject will not be hired. In addition, the horizontal axis of the map (first parameter) represents the age of the subject, and the vertical axis of the map (second parameter) represents the annual income desired by the subject. In this case, in the map shown in Figure 5A, for example, if the desired annual income is sufficiently high, the subject will not be hired regardless of age, and if the subject is sufficiently old, he or she will be hired even if the desired annual income is relatively high. The prediction model 10 is trained so as to be able to predict, for example, the distribution represented by such a true correct answer based on the first parameter, the second parameter, and the like.

[0056] 5B is a map showing the distribution of prediction results (output) of the prediction model 10. The rectangular regions on the map are condition regions 2 described using conditions related to the first and second parameters. For example, one condition region 2 is specified by specifying the ranges of the first and second parameters. As shown in FIG. 5B, in the actual prediction model 10, a prediction process is performed in which a large number of condition regions 2 are classified into categories α and β. That is, each condition region 2 is classified into either category α or category β. Therefore, the boundary between condition regions 2 of different categories can be said to be the decision boundary when the prediction model 10 performs classification. Here, the decision boundary is a surface that serves as a judgment criterion for changing the prediction. For example, in the case of hiring prediction, the dividing line between hiring and not hiring is the decision boundary. In this way, in the actual prediction model 10, the condition region 2 is sufficiently detailed, so it is possible to reproduce the distribution of true correct answers shown in Fig. 5A with high accuracy. On the other hand, the conditions for each parameter become too complicated, making it difficult to interpret information such as which data in which region is classified into which category.

[0057] 5C is a map showing the distribution of the output of summary model 11. Summary model 11 is trained to provide an output as close as possible to the predictive model. As a result, the output of summary model 11 becomes a condition region 2 that approximates condition region 2 of predictive model 10 and covers a wider region (data range). Each of these approximated condition regions 2 corresponds to an output rule 4 that explains the output of predictive model 10 in an interpretable manner. For example, in FIG. 5C, the area classified into category α is represented by four condition areas 2 (output rules 4). This makes it possible to describe, for example, the output of the prediction model 10 as four types of conditions described within the ranges of the first and second parameters. In the following, the condition region 2 specified by the output rule 4 may be simply referred to as a region specified by the output rule. In this embodiment, the condition region corresponds to a data range.

[0058] Methods of constructing such a summary model 11 include methods using algorithms such as a decision tree or rule fit. In other words, the summary model 11 is a learning model that uses at least one of the decision tree and rule fit algorithms. By using decision trees and rule fitting, it is possible to describe the output of a complex prediction model 10 in simple conditions (output rules 4).

[0059] The summary model learning unit 23 may also train the summary model 11 according to a predetermined criterion for evaluating the output of the prediction model 10. For example, if the predetermined criterion is a law or the like that sets a criterion related to age, the summary model 11 is configured to generate an output rule 4 described with age as a parameter. In this case, for example, an output rule 4 is generated that combines a condition specifying an age range with a condition described with other parameters (gender, desired annual income, etc.). This makes it possible to easily determine whether the output of the prediction model 10 satisfies a predetermined standard. Furthermore, the specific configuration of the summary model 11 is not limited, and any algorithm capable of summarizing the output of the algorithm used in the prediction model 10 may be used, for example.

[0060] Returning to FIG. 2, the explanation generating unit 24 generates information that explains each output rule 4 (rule group) generated by the summary model 11. The explanation generating unit 24 also acquires evaluation information that evaluates the rule group according to a predetermined criterion. In this embodiment, the explanation generating unit 24 corresponds to an evaluator.

[0061] Specifically, the explanation generating unit 24 generates an explanation for each of the output rules 4. The explanation is displayed by the UI generating unit 20 on the UI screen. When, for example, a decision tree algorithm is used as the summary model 11, the output rules 4 of the prediction model 10 can be described as a set of multiple simple rules. The explanation generation unit 24 generates an explanation that represents such a set of rules. For example, assume that a rule is generated as output rule 4 such that a target person will be hired if the following conditions are met: the gender item is "male", the age item is "45 years or older", and the qualification item is "qualified". In this case, the explanation generating unit 24 generates an explanation such as "hire a target person who is male, 45 years or older, and has qualifications." Such an explanatory text is generated for each output rule 4, for example.

[0062] Also, a diagram relating to each of the output rules 4 may be generated. For example, the contents of the conditions relating to gender, age, qualifications, etc. as described above may be illustrated as an image on the map described with reference to FIG. 3, etc. This allows for an intuitively easy-to-understand explanation of the output rules 4. Alternatively, a table showing the conditions relating to each item may be generated. This makes it possible to easily grasp the differences between multiple output rules 4, for example. In addition, the explanation generating unit 24 can generate information explaining the output rule 4 in various formats according to the algorithm used in the summary model 11 that summarizes the prediction model 10.

[0063] In this embodiment, the explanation generating unit 24 generates check items for each of the output rules 4 to allow the user to check whether or not a predetermined criterion is met. For example, an explanation or the like generated to enable the user to determine whether or not the corresponding output rule 4 meets the predetermined criterion is used for the check items. The generated check items are output to the UI generating unit 20 and displayed on the UI screen.

[0064] The check items are generated, for example, for items that require attention designated by the user. In other words, the check items are generated using explanatory text that allows the user to determine whether the items designated by the user meet a predetermined standard. Also, check items may be generated after determining whether each output rule 4 satisfies a predetermined criterion based on the rule database. In this case, for example, a check item is generated to request the user to confirm an output rule 4 that the explanation generating unit 24 has determined not to satisfy the predetermined criterion. The method of generating check items will be described in detail later.

[0065] The explanation generating unit 24 also reads the results of the checks made by the user for the check items. The check results are an example of evaluation information obtained by evaluating a group of rules according to a predetermined criterion. For example, suppose that the user judges that a certain output rule 4 does not satisfy a predetermined criterion and checks the check item 5. In this case, the checked output rule 4 becomes a violating rule that does not satisfy the predetermined criterion. Therefore, the evaluation information includes information specifying the output rule 4 that the user judges to be a violating rule. In addition, the explanation generating unit 24 generates, as the evaluation information, information specifying the laws and regulations that the violating rule does not satisfy and information indicating the degree to which the violating rule deviates from the standard, in association with the violating rule. In this way, the evaluation information includes information about the violating rule that is the output rule 4 that does not satisfy the predetermined standard.

[0066] In this way, the explanation generating unit 24 receives the created summary model 11, information on items of caution specified by the user, or information on laws and regulations obtained from the rule database. In addition, the explanation generating unit 24 outputs explanations (check items) of the output rules 4 to be presented to the user, and evaluation information on each output rule.

[0067] The query generation unit 25 generates a query to inquire of the rule database 18 about information on laws and regulations, etc., that are necessary to evaluate the prediction model 10. The query can also be said to be a command for collecting laws and regulations, etc., that the prediction model 10 must comply with. For example, a query is generated based on information on the field to which the prediction model 10 is applied, the type of problem setting, and items to be noted via a UI screen by the user. Alternatively, a query inquiring about laws and regulations to be referred to may be generated according to the output rules 4 generated by the summary model 11. The query generation unit 25 receives information such as the created summary model 11 and items of caution designated by the user, and also outputs a query inquiring about laws and regulations.

[0068] For example, in a prediction model 10 for predicting hiring, when investigating laws and regulations related to age and gender, the following query is generated. Query: (area = human resources, problem setting = classification, items to pay attention to = (age, gender)) Of course, the format of the query is not limited. When such a query is input to the rule database 18, a rule is obtained in which the rule field item is "personnel", the problem setting item is "classification" or "all", and the item to be noted is "age" or "gender". For example, in FIG. 4, information on the Equal Employment Opportunity Law and the Act on Promotion of Women's Participation and Advancement in the Workplace is obtained. In this manner, a query inquiring about laws and regulations is input to the rule database 18. Furthermore, the rule database 18 outputs information about laws and regulations that correspond to the specification of the query.

[0069] The adjustment processing unit 26 adjusts the learning process of the prediction model 10 based on the evaluation information. More specifically, the adjustment processing unit 26 adjusts the learning process of the prediction model 10 by generating an adjustment instruction for adjusting the prediction model 10 based on the evaluation information. In this embodiment, the adjustment processing unit 26 corresponds to an adjustment unit. Examples of methods for adjusting the prediction model 10 include methods for adjusting the learning data 1 or learning parameters. For example, when adjusting the learning data 1, information specifying the target learning data 1, information indicating how to handle the learning data 1, and the like are generated as adjustment instructions. Also, for example, when adjusting a learning parameter, information specifying the target learning parameter, information indicating the amount of adjustment of the learning parameter, and the like are generated as adjustment instructions. These adjustment instructions are set according to the content of the violation, for example, so that the violating rule specified by the evaluation information is eliminated. In this way, the evaluation information output from the explanation generating unit 24 is input to the adjustment processing unit 26. In addition, the adjustment processing unit 26 outputs an adjustment instruction for adjusting the prediction model 10. The adjustment processing unit 26 may be configured, for example, as a part of the prediction model learning unit 21. That is, a configuration is also possible in which the prediction model learning unit 21 adjusts the learning process of the prediction model 10 based on the evaluation information.

[0070] 6 is a flowchart showing an example of a basic operation of data analysis apparatus 100. The process shown in FIG. In the following, a basic operation of the data analysis device 100 will be described using a classification model that predicts whether or not to hire a subject as an example of the prediction model 10. Of course, the following description is applicable regardless of the type, subject, etc. of the prediction model 10.

[0071] The learning data 1 and information on the setting items input via the UI screen are read (step 101). 7 is an example of a UI screen for specifying the learning data 1 to be used for the prediction model 10. The UI screen shown in FIG. The start screen has a “Model Name” input field for inputting the name of the prediction model 10 to be created, and a “Description” input field for inputting an explanation of the prediction model 10. The start screen also has an input field for "learning data setting" for specifying the learning data 1 (learning database 17) to be used for learning the prediction model 10. Here, the user is instructed to specify a file in CSV format or TSV format as the learning data 1. For example, the learning data 1 to be used is specified by dragging the file to be used as the learning data 1 onto the start screen or by selecting it from a folder tree. After entering the necessary information in each input field, press the decision button and the model name and training data 1 will be loaded.

[0072] FIG. 8 is an example of a UI screen for inputting various settings related to the prediction model 10. 8 is displayed after an input operation via the start screen, and is a setting screen for setting the prediction model 10. The setting screen is generated based on, for example, items of the learning data 1 specified via the start screen. As shown in FIG. 8, the setting screen has a plurality of setting fields. In the setting field for "Prediction target," it is possible to specify the prediction target (target item) of the prediction model 10. In this case, "recruitment decision," which targets the item of "recruitment result" (see FIG. 3) in the learning data 1, is specified as the prediction target. In the "Prediction type" setting field, the type of prediction model 10 can be specified. Here, the items "binary classification," "multiple-value classification," and "numerical prediction" (regression prediction) are displayed as selectable items. In the adoption decision, binary classification is selected as the type of prediction model 10. In the setting field for "Scenes in which the predictive model is to be used," it is possible to specify the scenes in which the predictive model 10 is to be applied. Here, items such as "Demand forecast," "Personal and recruitment," "Economy and finance," "Price and sales price," and "Contains personal information" are displayed as selectable items. For hiring decisions, the items "Personal and recruitment" and "Contains personal information" are selected.

[0073] In the "Input Items" setting field, it is possible to specify the items to be used in training the prediction model 10 from among the items contained in the training data 1. Here, each data item of the training data 1 shown in FIG. 3 is displayed selectably. Also displayed are the data type, unique number, etc. of each data item. The setting field for the "input items" also includes a field for specifying "items to be careful about." "Items to be careful about" are, for example, items that are subject to regulations such as laws and regulations, or sensitive items that may cause ethical problems. For example, the data items specified as "items to pay attention to" are referred to as appropriate when generating a query to inquire of the rule database 18, or when generating check items for the output rule 4. In the adoption decision, the items "age" and "gender" are selected as "items to pay attention to". It is not necessary to input "situations in which the predictive model should be used" or "items to be careful of." After inputting the necessary information into each setting field, pressing the execute button to execute learning and evaluation starts the learning process of the prediction model 10 and the like.

[0074] When the user completes the input operation via the UI screen, a learning process is executed to train the prediction model 10 (step 102). For example, an appropriate machine learning algorithm is selected according to the type of prediction model 10 specified in the "Prediction Type" setting field, and a learning model is constructed using the selected algorithm. Then, training of the learning model is executed using the learning data 1 specified by the user so as to learn the target item specified in the "Prediction Target" setting field. This trained learning model becomes the created prediction model 10.

[0075] In this embodiment, the characteristic evaluation unit 22 evaluates the characteristics of the created prediction model 10. This characteristic evaluation may be performed when the prediction model 10 is generated, or when an evaluation screen or the like, which will be described later, is displayed. FIG. 9 is an example of an evaluation screen that displays the characteristics of the prediction model 10. The evaluation screen shown in FIG. 9 displays a "prediction accuracy level" indicating the level of prediction accuracy of the created prediction model 10, and a "degree of contribution of items." In the "Prediction Accuracy Level" field, for example, the AUC (Area Under the Curve) of a Receiver Operating Characteristic (ROC) curve is displayed as an evaluation index showing the performance of the prediction model 10. The AUC is an index showing the classification accuracy of a classification model. In addition, a bar graph showing the contribution of each item to the classification is displayed in the "Contribution of Items" section. This makes it possible to compare items, for example, those that influenced the classification of "adopt" and those that influenced the classification of "not adopt."

[0076] FIG. 10 shows another example of the evaluation screen that displays the characteristics of the prediction model 10. FIG. 10 shows an evaluation screen showing details of the contribution degrees shown in FIG. On the left side of this evaluation screen, a bar graph similar to that in Figure 9 is displayed for each item. When an item is selected, the composition of the item's contribution rate is displayed on the right side of the evaluation screen. For example, suppose the "Age" item is selected. In this case, the contribution of each age range to the classification of "hiring" (or "not hiring") is displayed. For example, in Figure 10, learning data 1, which specifies ages between 45 and 50, has the highest contribution to the classification of "hiring." In this way, details of the contribution level may be analyzed and displayed to the user.

[0077] Returning to FIG. 6, when the learning process of the prediction model 10 is completed, a learning process is executed to train the summary model 11 (step 103). The summary model 11 is trained so as to obtain an output close to that of the prediction model 10, for example, using the predicted value by the prediction model 10 as correct answer data. For example, a learning model is constructed using an algorithm such as a decision tree or rule fitting. Then, the learning model is trained using the predicted values ​​by the created prediction model 10 as correct answer data. This trained learning model becomes the created summary model 11. As a result, as described with reference to FIG. 5C, a set of rules (output rules 4) is generated in which the complex output of the prediction model 10 is described in a format that can be interpreted by the user.

[0078] When the learning process of the summary model 11 is completed, a check item 5 regarding the output of the prediction model 10 is displayed (step 104). First, an explanatory text is generated for each output rule 4 generated by the summary model 11. Then, using the explanatory text, check items for checking each output rule 4 are generated.

[0079] As an example of a method for creating an explanatory text, a rule is created by omitting the items specified by the user as "items to note" from the set of rules (conditions) that make up the output rule 4. Then, an explanatory text is created that explains what kind of output will be produced by the rule omitting the "items to note". For example, let's say that output rule 4, "If they are male, over 45 years old, and qualified, hire them," is extracted. If gender is specified as an "item to pay attention to," the explanation for the output rule will be something like, "This model predicts that, among people over 45 years old who are qualified, X% of men and Y% of women should be hired."

[0080] This is an explanatory text that explains output rule 4, focusing on the item (gender) that the user should pay attention to. Using such an explanatory text, a check item of output rule 4 is generated. In this manner, in this embodiment, check items are generated that target data items designated by the user ("items to be careful of") among a plurality of data items included in the learning data 1 of the prediction model 10. As described with reference to Fig. 5C, each output rule 4 corresponds to a condition region 2 that divides the distribution of the output of the prediction model 10. Therefore, creating an explanation in this manner can be said to be a process of outputting an explanation about the distribution of items that require attention in each condition region 2 (such as the ratio by gender). This makes it possible, for example, in cases where attention should be paid to gender, to explain the adoption rate for each gender and then have the user check whether the rule violates laws and regulations.

[0081] FIG. 11 shows an example of a check screen for output rule 4. Fig. 11A is a display example of a map in which a condition region 2 corresponding to output rule 4 is drawn. Fig. 11B is a display example of a check item 5 related to output rule 4. Fig. 11C is a display example when check item 5 shown in Fig. 11B is checked. On the check screen for output rule 4, for example, the map shown in Fig. 11A and check item 5 shown in Fig. 11B are displayed simultaneously. Note that the map on which condition area 2 is drawn does not necessarily have to be displayed.

[0082] The map shown in FIG. 11A is a heat map showing the hiring decision rate output by the prediction model 10 that performs hiring decisions. The horizontal axis of the map is the number of job changes, and the vertical axis is age. In this map, the darker the gray, the lower the hiring decision rate. 11A, output rules 4 (conditions 1 to 3 in the figure) that explain the output of the prediction model 10 are illustrated as condition regions 2. Each output rule 4 is described as a condition indicated by a numerical range of the number of job changes and a numerical range of age. In this way, by illustrating each output rule 4 (conditions 1 to 3) as a condition region, the user can easily grasp the correspondence between each output rule 4 and the adoption determination rate.

[0083] In FIG. 11B, an explanatory text for output rule 4, which describes conditions 1 to 3, is displayed as check item 5. Condition 1 is a condition that 80% of applicants aged 45 or older should be hired, condition 2 is a condition that applicants aged 30 to 45 who have changed jobs three or more times should not be hired, and condition 3 is a condition that applicants aged 30 or younger should be hired. This screen displays instructions to check whether these conditions (output rule 4) are ethically problematic. Also provided below the check items 5 are a confirmation button 30 for confirming that there are no problems, and a start adjustment button 31 for starting adjustment of the prediction model 10. For example, when no conditions are checked, the confirmation button 30 is highlighted.

[0084] For example, suppose that the user judges that the hiring rate of the subject specified in condition 2 is excessively lowered. In this case, as shown in FIG. 11C, the check box for condition 2 is checked, and output rule 4 indicated in condition 2 is designated as a rule that does not meet the predetermined criteria, that is, is ethically questionable. When the check box is checked, the adjustment start button 31 is highlighted. Pressing the adjustment start button 31 starts a process of adjusting the prediction model 10 so that, for example, the adoption decision rate is uniform. In this way, by displaying the output rules 4 in a manner that is interpretable by the user, it is possible to easily identify areas in which the output of the prediction model 10 violates ethics. The contents displayed on the check screen are not limited. For example, an adjustment value for adjusting a learning parameter of the prediction model 10 may be input.

[0085] 11, a case has been described in which it is determined whether or not each output rule 4 satisfies a predetermined criterion. The explanation generating unit 24 can also determine whether or not an output rule 4 satisfies a predetermined criterion based on the rule database 18. A method for creating an explanatory message (check item 5) that prompts the user to check the laws and regulations in the rule database 18 will be described below. For example, assume that information on laws and regulations, etc., necessary for evaluating the prediction model 10 is read from the rule database 18 by a query generated by the query generation unit 25. In this case, the explanation generation unit 24 compares the output of the prediction model 10 (or the output of the summary model 11) according to a comparison method and a comparison area related to the rules of laws and regulations, etc., read from the rule database 18.

[0086] For example, it is assumed that rules stipulated in the "Equal Employment Opportunity Law" are output from the rule database 18 as laws and regulations for evaluating the prediction model 10 that performs hiring decisions. It is also assumed that gender is set as an "item to be careful of." The rules regarding gender in the "Equal Employment Opportunity Law" (the rules written in the first column of the table in FIG. 4) state that the comparison method is "equal" and the comparison area is "all areas" for gender. In the explanation generation unit 24, for example, the hiring decision rates for men and women are compared for the learning data 1 specified by the output rule 4 to be evaluated. In other words, when the output of the prediction model 10 is divided into the output of the learning data 1 for men and the output of the learning data 1 for women, it is confirmed whether the probability of hiring men and women is "equal."

[0087] If the probabilities are not equal, the output rule 4 in question is determined to not satisfy the rules (standards) set forth in the Equal Employment Opportunity Law, and an explanatory message is generated to prompt the user to confirm. Specifically, a check item 5 is generated that includes an explanatory text such as, "Under these conditions, we will hire X% of men and Y% of women. Is this correct from the perspective of the [Equal Employment Opportunity Law]?" In this way, the explanation generating unit 24 generates a check item 5 for the output rule 4 that is determined not to satisfy the predetermined criterion, which allows the user to check whether the output rule 4 satisfies the predetermined criterion.

[0088] Fig. 12 is another example of the check screen for output rule 4. Fig. 12 shows the check screen when a possible legal or ethical problem is automatically detected. Fig. 12A is a display example of a map on which a boundary line 6 corresponding to an output rule 4 determined by the explanation generating unit 24 to not satisfy a predetermined criterion is drawn. Fig. 12B is a display example of a check item 5 related to the output rule 4. Fig. 12C is a display example of a selection screen for selecting a method for adjusting the prediction model 10. On the check screen for output rule 4, for example, the map shown in Fig. 12A and check item 5 shown in Fig. 12B are simultaneously displayed. In addition, the selection screen shown in Fig. 12C is displayed when the user selects adjustment of the prediction model 10.

[0089] For example, since the predictive model 10 that performs hiring decisions is used in personnel-related tasks, the personnel / hiring item is checked as a "situation in which the predictive model is used" on the setting screen shown in Fig. 8. As a result, the laws and regulations in the field of personnel / hiring in the rule database 18 are checked, and it is determined whether the output of the predictive model 10 meets the standards set forth by the laws and regulations. The check screen shown in Fig. 12 is a screen for prompting the user to check the fact that an unnatural distribution bias was found in the hiring decision rate at the age of 35. Fig. 12A shows a boundary line 6 where the hiring decision rate changes. This boundary line 6 can be detected as the boundary of, for example, the condition area 2 indicating the output rule 4.

[0090] In FIG. 12B, an explanatory text about this boundary line 6 is displayed as a check item 5. This explanatory text describes the characteristic of the prediction model 10, that is, the output result of the prediction model 10 changes depending on whether the age is 35 or over. The explanatory text also includes an explanation encouraging confirmation as to whether this characteristic violates the "XXX Law" (laws in the fields of human resources and recruitment) that stipulates that equal employment opportunities be provided to all ages. Also provided below the explanation are a confirmation button 30 for confirming that there are no problems, and a start adjustment button 31 for starting adjustment of the prediction model 10.

[0091] For example, suppose that the user checks the "XXX method" for the characteristics of the prediction model 10 and determines that it violates the method. In this case, by pressing the adjustment start button 31, a process of adjusting the prediction model 10 so that the hiring decision rate is uniform regardless of age is started. In this way, by determining the output (output rule 4) of the prediction model 10 based on the rule database 18, it is possible to point out problems that the user is not aware of, for example.

[0092] When the adjustment start button 31 is selected, a selection screen such as that shown in Fig. 12C is displayed. On the selection screen, two methods for uniformly adjusting the hiring decision rate are displayed, one for increasing the hiring rate for those aged 35 or younger, and the other for decreasing the hiring rate for those aged 35 or older, so that the selection screen can be selected. Also provided below the adjustment method are a confirmation button 30 for confirming that there are no problems, and a start adjustment button 31 for starting adjustment of the prediction model 10.

[0093] For example, by using an adjustment method that lowers the hiring rate for those aged 35 or older, it is possible to adjust the overall hiring rate lower. Conversely, by using an adjustment method that raises the hiring rate for those aged 35 or younger, it is possible to adjust the overall hiring rate higher. The user can start the adjustment process using the selected adjustment method by checking the check box of one of the adjustment methods and selecting the adjustment start button. In this manner, in the present embodiment, a plurality of adjustment methods for the output of the prediction model 10 are presented in a selectable manner. Then, the learning process of the prediction model 10 is adjusted based on a method selected by the user from among the plurality of adjustment methods. This makes it possible to adjust the prediction model 10 to suit the situation in which it is applied.

[0094] 6, when the user completes the input operation via the check screen, the check result for check item 5 is read (step 105). Then, based on the check result, it is determined whether or not there is a region (data range) in the output of the prediction model 10 that does not satisfy a predetermined standard, that is, whether or not there is a violation rule (step 106). For example, if any of check items 5 (conditions 1 to 3) shown in FIG. 11B is checked, or if adjustment start button 31 shown in FIG. 12B is selected, it is determined that there is an area that does not satisfy the predetermined criteria (Yes in step 106). Furthermore, for example, if all check items 5 are not checked, it is determined that all areas of the output of the predictive model 10 do not violate laws and regulations, and that there are no areas that do not meet the specified criteria (No in step 106), and the process of creating the predictive model 10 is completed.

[0095] If there is an area that does not satisfy the predetermined criteria, the adjustment processing unit 26 extracts the learning data 1 corresponding to that area (step 107). Specifically, information (such as the ID of the learning data 1) specifying the learning data 1 that matches the output rule 4 (violating rule) that the user has checked after determining that the output rule violates laws and regulations is extracted. For example, if it is determined that the rule that the number of job changes is three or more and the age is between 30 and 45 (condition 2 shown in FIG. 11B) is a violation rule, the ID of learning data 1 that satisfies this condition is read.

[0096] When the learning data 1 that matches the violating rule is extracted, the adjustment processing unit 26 generates an adjustment instruction for the prediction model 10 (step 108). First, a prediction value that violates a rule (the output of the prediction model 10 for the learning data 1 that matches the violating rule) is compared with a prediction value that does not violate a rule, and an adjustment policy is calculated that indicates what the prediction value that violates the rule should do to avoid violating the rule.

[0097] When a classification model is used as the prediction model 10, the learning process of the prediction model 10 is adjusted so that the predicted value of the prediction model 10 in the area specified by the violation rule approximately matches the predicted value of the prediction model 10 in the area specified by the output rule 4 that satisfies a specified criterion. For example, for prediction model 10 that predicts the classification of "adopt / not adopt", the proportion (prediction value) of "adopt" is calculated for each of the areas that violate the rules and the areas that do not violate the rules. Then, a policy is selected that matches the proportion of "adopt" predicted in the areas that violate the rules to the proportion of "adopt" predicted in the areas that do not violate the rules. At this time, the degree to which the prediction model 10 is adjusted (adjustment amount, etc.) is calculated so that the proportion of predicted "adopted" results is approximately the same for each region.

[0098] Next, an adjustment instruction for the prediction model 10 is generated according to a policy and an adjustment amount for avoiding a rule violation. Specifically, an adjustment instruction is generated as appropriate according to an adjustment method for adjusting the learning process of the prediction model 10. There are multiple methods for adjusting the prediction model 10 (classification model).

[0099] As a method for adjusting the prediction model 10, a method for adjusting the weighting coefficient of the learning data 1 may be used. For example, for a violation rule that has a high probability of predicting "not adopt," the weighting coefficient of learning data 1 included in the area specified by the violation rule is increased. This increases the rate at which the prediction "adopt" is obtained in the area specified by the violation rule, and as a result, it becomes possible to make the predicted value approximately match the area specified by the output rule 4 other than the violation rule. Such a weighting coefficient is generated as an adjustment instruction. In this embodiment, the weighting coefficient is an example of a parameter that adjusts the output of a prediction model regarding training data.

[0100] Furthermore, for example, as a method for adjusting the prediction model 10, a method for adjusting a judgment threshold for judging a prediction may be used. For example, for a violation rule with a high probability of predicting "adopt," the probability of "adopt" is calculated for each piece of learning data 1 included in the area specified by the violation rule, and the decision threshold for determining "adopt" is set higher than for other areas. For example, in an area specified by output rule 4 other than the violation rule, if the prediction model 10 outputs "should be adopted with a 50% probability," it is determined to be "adopt." In contrast, in an area specified by a violation rule, the decision threshold is set high, and it is determined to be "adopt" only when it outputs "should be adopted with a probability of 80% or more." This makes it possible to lower the rate at which the violating rule is predicted to be "adopted", and as a result, it becomes possible to bring it closer to the rate at which the other output rule 4 is predicted to be "adopted". Such a judgment coefficient is generated as an adjustment instruction. In this embodiment, the decision threshold is an example of a parameter that adjusts the output of a predictive model regarding training data.

[0101] Furthermore, for example, as a method for adjusting the prediction model 10, a method for thinning out the learning data 1 may be used. For example, for a violation rule that has a high probability of predicting "adopt", learning data 1 labeled with "adopt" in the area specified by the violation rule is thinned out, and the learning process is performed using the reduced number of data pieces of learning data 1. Therefore, this adjustment method can be said to be a process of reducing the number of learning data pieces that cause the violation rule to not satisfy a predetermined standard, among the learning data 1 included in the area specified by the violation rule. This makes it possible to lower the rate at which the violating rule predicts "to be adopted," and as a result, to bring it closer to the rate at which the other output rules 4 predict "to be adopted." In this case, the ID or the like of the learning data 1 to be thinned out is generated as an adjustment instruction.

[0102] In this manner, in this embodiment, at least one of the learning data 1 of the prediction model 10 and the learning parameters of the prediction model 10 is adjusted based on the region (data range) specified by the violation rule. This makes it possible to appropriately adjust the output of the prediction model 10 that is causing an ethical violation or the like.

[0103] When the adjustment instruction is generated, step 101 is executed again, and the prediction model learning unit 21 starts a new learning process for recreating the prediction model 10 so as to reduce predictions that violate the rules. By repeating such a process, it is possible to finally build a prediction model 10 that does not violate laws and regulations.

[0104] [How to adjust the regression model] 13 is an example of a check screen displayed when a regression model is used. Here, a process will be described in which a regression model that predicts an item value of a target item is used as the prediction model 10. A summary model 11 that summarizes the regression model generates output rules 4 that divide, for example, a map of predicted values ​​that is the output of the prediction model 10 into a plurality of regions (data ranges). In the data analysis device 100, for each region specified by the output rules 4, it is evaluated whether the distribution of predicted values, etc., satisfies laws and regulations, etc.

[0105] FIG. 13 shows a check screen for a prediction model 10 that predicts the expected length of service of a subject. The graph shown at the top of the figure is a histogram of the predicted length of service by gender. The horizontal axis of the graph is the predicted length of service, and the vertical axis is the number of people predicted for each length of service. In this histogram, the center of the distribution of men's years of service is higher than the center of the distribution of women's years of service. The explanatory text shown at the bottom of the figure is a check item that prompts the user to confirm the distribution of predicted values ​​for years of service.

[0106] When predicting years of service, for example, the output of prediction model 10 is calculated separately for the output of learning data 1 for men and the output of learning data 1 for women. Then, it is confirmed whether the predicted years of service for men and women are "equal." Length of service is expressed as a number and cannot be described as a probability. In this case, as shown in Figure 13, the distribution (histogram) of predicted length of service for men and women is calculated, and each distribution is compared using parameters for comparing distributions (e.g., Histogram Intersection, KL Divergence, JS Divergence, etc.). For example, if the difference in distribution between men and women exceeds a certain threshold, the area where such a distribution is calculated (output rule 4) is likely to be an ethically questionable violation of a rule. For this reason, the data analysis device 100 outputs check item 5 such as "There is a discrepancy between the predicted value outputs for men and women. Is this okay?" In addition, in the area specified by the output rule 4 other than the violating rule, that is, in the area where no violation occurs, for example, a prediction result in which the distribution of males and females is approximately the same is obtained.

[0107] In the example shown in FIG. 13, the output distribution differs for "gender," and an explanatory message (check item 5) is displayed to prompt the user to check whether the "Equal Employment Opportunity Law" is violated. If the user determines that there is a problem with the difference in distribution, adjustment of the prediction model 10 is started by selecting the adjustment start button 31.

[0108] When a regression model is used as the prediction model 10, the learning process of the prediction model 10 is adjusted so that the distribution of predicted values ​​of the prediction model 10 in the area specified by the violation rule approximately matches the distribution of predicted values ​​of the prediction model 10 in the area specified by the output rule 4 that satisfies a specified criterion. For example, the prediction model 10 that predicts "years of service" is adjusted so that the distribution shape of the predicted "years of service" becomes equal for each of the rule violation area and the rule non-violation area. For example, a comparison parameter such as KL divergence (Kullback-Leibler distance) regarding the years of service between the rule violation area and the rule non-violation area is calculated. The degree of adjustment of the prediction model 10 (adjustment amount, etc.) is calculated so that this comparison parameter becomes smaller.

[0109] Next, an adjustment instruction for the prediction model 10 is generated according to the calculated adjustment amount. Specifically, an adjustment instruction is generated as appropriate according to an adjustment method for adjusting the learning process of the prediction model 10. There are multiple methods for adjusting the prediction model 10 (regression model).

[0110] As a method for adjusting the prediction model 10, a method for adjusting a parameter for adjusting the loss function of the prediction model 10 may be used. In general, the prediction model 10 learns so that the value of the loss function becomes small. Therefore, for example, by setting a penalty such that the loss function becomes large when a predicted value that violates the rule is predicted for an area specified by a violation rule, it is possible to reduce the rate at which predicted values ​​that violate the rule are predicted. For example, for a violation rule that predicts years of service too long, quantile regression is set in the loss function used to train the prediction model 10 only in that area, and the loss function is shifted. At this time, by increasing the penalty when predicting years of service too long in an area where the rule is violated, it is possible to bring predictions in the area where the rule is violated closer to areas where the rule is not violated. A parameter for adjusting such a loss function is generated as an adjustment instruction.

[0111] Furthermore, for example, a method of thinning out the learning data 1 may be used as a method of adjusting the prediction model 10. Specifically, among the learning data 1 included in the area specified by the violation rule, the learning data 1 that causes the violation rule to not satisfy a predetermined standard is thinned out. For example, for a violation rule that has a high probability of predicting longevity, learning data 1 is thinned out in ascending order of longevity in the area specified by the violation rule, and learning processing is performed using the reduced learning data 1. In this case, the ID, etc. of the learning data 1 to be thinned out is generated as an adjustment instruction. This makes it possible to reduce the rate at which the length of service is predicted to be excessively long in the area specified by the violation rule.

[0112] Furthermore, for example, as a method for adjusting the prediction model 10, a method of adding dummy learning data 1 (dummy data) that does not actually exist may be used. For example, for a violation rule that predicts excessively long years of employment for men, data on short years of employment for men is generated and used to train the prediction model 10. Therefore, this adjustment method can be said to be a process of adding dummy data adjusted to satisfy a predetermined standard to the area specified by the violation rule as the learning data 1. This shortens the years of service predicted in the violation rule, and as a result, it becomes possible to bring the distribution of years of service closer to the distribution predicted in the other output rule 4. In this case, together with the information on the dummy data, an instruction to add the dummy data to the learning data 1 is generated as an adjustment instruction.

[0113] In this way, even when a regression model is used as the predictive model 10, it is possible to easily adjust the output that is determined to be an ethical violation or to have a problem with fairness, and to resolve the problem. The above-mentioned adjustment method for the classification model may be applied to the adjustment method for the regression model, and conversely, the adjustment method for the regression model may be applied to the adjustment method for the classification model. Each adjustment method may be used alone or in combination. The parameters (weighting coefficients, etc.) used for each adjustment may be calculated according to the violation content, or may be input by the user. In addition, the adjustment method for the prediction model 10 is not limited, and any adjustment method that can resolve the violation content may be used.

[0114] In the following, a process for constructing the prediction model 10 will be specifically described for each application example of the data analysis device 100.

[0115] [Application example 1] An example of a human resources manager (user) creating a predictive model10 that predicts whether a candidate will be hired or not based on "gender," "age," "qualifications obtained," and "number of job changes." First, a start screen, a setting screen, etc. are displayed, and the learning data 1 to be used, target items, etc. are designated by the user. A learning model (for example, a neural network) is trained based on the specified learning data 1, etc., and a prediction model 10 is created. A summary model 11 is created using an algorithm for summarizing the prediction model 10 (such as rule fitting, which creates a model that combines multiple rules), and the output of the prediction model 10 is summarized as a combination of multiple output rules 4.

[0116] Here, it is assumed that the output rule 4, "if the age is between 50 and 60 years old, then they should be hired", is created from the summary model 11. At this time, for the gender set as an item to be careful of, the predicted hiring rate for each gender is calculated. For example, in the above output rule 4, if the ratio of males hired is higher than the ratio of females hired, check item 5 is generated along with an explanation such as "In the age range of 50-60 years old, we are hiring excessively many males. Is this a problem?" and displayed on the check screen. Alternatively, a simple explanatory statement explaining the "employee" ratio of men and women may be used.

[0117] If the user looks at the above check item 5 and judges that output rule 4 is unethical, check item 5 is checked and output rule 4 becomes a violating rule. In the adjustment process of the prediction model 10, the weighting (weighting coefficient) for the learning data 1 that corresponds to the violation rule (data corresponding to males aged 50 to 60 years old) is set to be light, and the learning process of the prediction model 10 is executed again. This makes it possible to lower the proportion of males aged 50 to 60 years old that are determined to be "hired."

[0118] Once re-learning of the predictive model 10 is complete, a number of output rules 4 that approximate the re-trained predictive model 10 using the summary model 11 are generated, and each output rule 4 is displayed as a check item 5 together with an explanatory text. If the user confirms that there are no ethical issues with respect to all output rules 4, the latest prediction model 10 at that point in time is determined to be the model to be used for future predictions.

[0119] [Application example 2] An example in which a human resources manager (user) creates a predictive model10 that predicts whether an applicant will be hired or not based on "gender," "age," "qualifications obtained," and "number of times a job has been changed," and checks whether the model violates laws and regulations. First, as in the first application example, a prediction model 10 and a summary model 11 are generated. It is assumed that output rule 4, which states that "if the person is between 75 and 90 years old and has a driver's license, he / she should not be hired," is generated from summary model 11. It is also assumed that the user has determined that there is no ethical problem with output rule 4. In this case, this output rule 4 is read as an output rule 4 that is not a violating rule.

[0120] At this time, data analysis apparatus 100 generates a query such as the one below and makes an inquiry to rule database 18. Query: (domain=human resources, problem setting=classification, attention item=age) As a result, condition A is referenced as a rule that satisfies the query condition, and condition A is specified as (comparison method item=equal, comparison area=all areas). This means that the field that the prediction model 10 deals with is a field that should be treated equally regardless of age. In this case, based on the database, data for those 65 and over should be treated equally, but it is unnatural that there is a different distribution only for those aged 75 to 90, so the user is notified that output rule 4 may violate ordinance A. For example, a check item 5 is generated to prompt the user to confirm, "Does the area between 75 and 90 years old violate ordinance A?"

[0121] In this way, the data analysis device 100 can automatically determine outputs that are ethically questionable or unfair, and warn the user about them. This makes it possible to reliably check rule violations that the user may have overlooked, and makes it possible to easily build a highly reliable prediction model 10.

[0122] [Application example 3] An example of creating a predictive model10 to determine ticket prices when dynamic pricing tickets for sports events. Here, a method for adjusting a prediction model 10 (regression model) that deals with a regression problem of predicting in real time the appropriate selling price of a spectator ticket, the price of which fluctuates, will be described. First, the user specifies the learning data 1 to be used, the target items (sales price), etc. The learning data 1 is, for example, personal data such as the gender and age of the customer who will purchase the ticket, and data related to the ticket such as the history of tickets the customer has previously purchased, the time of the game for which the ticket is about to be purchased, the opposing team, etc. From such learning data 1, a predictive model 10 is created that predicts at what price a customer will be willing to buy a ticket, i.e., predicts the price at which a customer is expected to purchase the ticket.

[0123] For example, the opposing team is designated as an "item to be noted," and the creation of the prediction model 10 is started. As in the first application example, the prediction model 10 and the summary model 11 are generated. By specifying the opposing team as an "item to note," it becomes possible to make adjustments to suppress differences in ticket prices depending on the opposing team, that is, fluctuations in ticket prices depending on the opposing team. Assume that summary model 11 produces output rule 4, for example, "For tickets after 6:00 p.m., ticket price (yen) = 5,000 - (capacity of the venue) + (adjustment based on personal data)."

[0124] As mentioned above, since the team names are specified as "items to note," here the distributions for "tickets after 6:00 PM" are compared for Team A and Team B. Specifically, a price histogram is created for each team, and the distance between each distribution is compared using KL divergence, etc. As a result, it turns out that there is a large difference in the distribution of predicted ticket sales prices when the competing teams are Team A and Team B. In this case, an alert (check item 5, etc.) is output to inform the user that the sales price will vary between Team A and Team B. When the alert is displayed, the user checks the distribution of ticket prices for each team. For example, if the user determines that the distribution of ticket prices is not fair, the user selects the adjustment start button 31 to correct the distribution bias.

[0125] For example, assume that the price is predicted to be excessively high when team B is the opposing team. In this case, if the ticket price of team B is predicted to be too high for tickets after 18:00 p.m., the loss function used in training the prediction model 10 is adjusted so that the loss is increased. The adjusted loss function is used to create the prediction model 10 again. The output distribution of the newly created prediction model 10 is displayed, and if it is confirmed that there are no problems, the creation of the prediction model 10 is completed. In this way, in the prediction model 10 that solves the regression problem, it is possible to allow the user to check rules that would make the predicted value excessively large (or small). This makes it possible to easily build a prediction model 10 that makes fair price predictions.

[0126] As described above, in the control unit 16 according to this embodiment, a learning process is performed between a predetermined learning model (prediction model 10 and authentication model) and the summary model 11 that converts the output of the learning model. The summary model 11 converts the output of the prediction model 10 into a set of rules described in a format that can be interpreted by the user. The learning process of the learning model is adjusted using evaluation information that evaluates this set of rules according to a predetermined standard. This makes it possible to easily build a learning model that meets the required standards.

[0127] In recent years, predictive analysis has become problematic due to predictive models that produce ethically questionable output due to bias in past data or the way data is collected. For example, suppose a model is used to predict how many years a person will stay at a company when they are hired. In this case, because there have been many male employees in the past, it is possible that the model will predict an excessively long length of stay for only men. The creator of the predictive model may not be aware of such a problem. Even if the creator notices the problem, it is often difficult to solve it.

[0128] FIG. 14 is a map of the output of a predictive model given as a comparative example. The horizontal axis of the map is the number of times a person has changed jobs, and the vertical axis is age. The gray areas of the map are areas where the predictive model predicted "will hire," and the white areas are areas where the predictive model predicted "will not hire." Each data point corresponds to an individual piece of training data. As shown in FIG. 14, the output of the predictive model (distribution of hires and non-hires) is complex. For this reason, it is difficult to check whether there are any cases of ethical violations under specific conditions, and depending on the predictive model (such as neural networks), even experts may find it difficult to check detailed cases. Furthermore, if the creator of the model is not an expert, he or she may not know how to adjust the parameters of the predictive model, making it difficult to adjust the predictive model so as to correct problems. Furthermore, it was necessary to prepare all rules that must not be violated (laws and regulations that must be observed) in advance, which meant that checking for violations took time and there was a risk of something being overlooked.

[0129] In this embodiment, a summary model 11 is used to replace the prediction model 10 with a decision tree or a set of rules that can be understood by humans. This makes it possible to display an explanation of the output rule 4 that can be interpreted by a user, such as "If A and B, then there is a bias towards men." This allows the user to easily understand the output of the prediction model. As a result, even a user who does not understand the algorithm of the prediction model 10 can check whether there is a problem with the output of the prediction model.

[0130] The explanation of the output rule 4 is displayed as a check item for the user to check whether or not the rule or ethics is violated. The prediction model 10 is adjusted based on the content checked in this check item 5. For example, an output rule 4 (violating rule) that the user has pointed out as problematic is compared with data that matches an output rule 4 that the user has deemed problematic. Then, depending on the comparison result, the data that matches the violating rule is automatically adjusted so that it approaches the prediction of another output rule 4. Alternatively, the learning parameters for using each data are appropriately adjusted. In this manner, in this embodiment, it is possible to easily realize correction of the output of the prediction model 10 through the summary model 11 with respect to areas that may be ethically questionable.

[0131] In this embodiment, a set of laws and rules to be observed for each usage scenario of the prediction model 10 is stored in advance as a rule database 18. Then, a query is generated to inquire the rule database 18 about the positions of the decision boundaries of the prediction model 10 and the summary model 11, and the necessary standards (rules) are appropriately read. This makes it possible to automatically check whether there are any ethical issues. As a result, the time required to check for violations is shortened, and it is possible to sufficiently prevent the occurrence of user oversights, etc. This makes it possible to easily build a learning model that meets the necessary standards.

[0132] <Other embodiments> The present technology is not limited to the above-described embodiment, and various other embodiments can be realized.

[0133] In the above embodiment, a method for adjusting a prediction model has been mainly described. The present technology is not limited to a prediction model, and can be applied to any learning model. As an application example, a case of correcting the error rate of an authentication system using a camera will be described below. Here, an authentication model trained using image data generated by a camera as learning data will be taken as an example. The authentication model is an example of a predetermined learning model. In the authentication model, image data is used as training data instead of tabular training data such as CSV.

[0134] Let us assume that we are creating an authentication model that performs person authentication using a camera installed at the gate of a company. In the following, we will create an authentication model that authenticates only people wearing company uniforms. First, image data of various people wearing company uniforms is prepared as data of people to be authenticated, and image data of various people wearing casual clothes is prepared as data of people not to be authenticated. These image data are used as training data to train an authentication model.

[0135] An example of a summary model that summarizes an authentication model is a random forest model that uses BoF (Bag Of Features). Here, BoF is a method for creating a model using fragments of elements contained in an image as features. An index called VW (Visual Words) is generated by clustering features extracted from multiple images. For example, matching features extracted from a target image with VW makes it possible to perform person authentication, etc. A random forest is a learning model that randomly combines decision tree algorithms, for example. As a summary model, a 0 or 1 vector is generated to represent whether the VW created by BoF exists in the image, and this is input into a random forest to create a model that classifies whether the person is wearing a uniform or not. In this case, it is possible to summarize the authentication model as a set of conditions for whether each VW is included or not.

[0136] FIG. 15 is an example of a check screen that is displayed when the authentication model is used. By using the above-mentioned summary model, it is possible to calculate the importance (contribution) of each feature in authentication. This makes it possible to detect VWs that were important when making predictions. 15 shows, for each image feature 8 (VW) that contributed to the prediction, a check item 5 (conditions 1 and 2) indicating the accuracy rate of authentication for image data including that image feature 8. These check items 5 are examples of output rules that explain the output of the authentication model. For example, in condition 1, it is explained that in images containing image feature 8 of clothing with few wrinkles, 90% of the subjects were wearing uniforms, and the accuracy rate of recognition was 90%. Also, in condition 2, it is explained that in images containing image feature 8 of clothing with many wrinkles, 80% of the subjects were wearing uniforms, and the accuracy rate of recognition was 40%. From this, we can see that the presence of image feature 8, which was created based on a "wrinkled uniform," reduces the probability of authentication.

[0137] In this case, from among multiple training data, training data that contains image features created using BoF that are similar to "wrinkled uniforms" is extracted.Then, to adjust the authentication model, only when training is performed using the extracted training data, the parameter update range is increased, i.e., the classification is performed more carefully than with other training data. After the adjusted authentication model and its summary model are created, an explanatory text is displayed explaining the summary results, etc. If the user determines that the classification accuracy for people wearing uniforms with many wrinkles has improved, the creation of the authentication model is complete. This makes it possible to easily build an authentication model with high detection accuracy.

[0138] In the above embodiment, the learning process of the prediction model is adjusted mainly using the check results of the user for the check items as evaluation information. However, the present invention is not limited to this, and the learning process of the prediction model may be adjusted using evaluation information automatically generated by the explanation generating unit, for example. Specifically, the explanation generating unit may generate, as the evaluation information, information about the output rule that is determined not to satisfy a predetermined standard. In this case, for example, by referring to a rule database, the output rule that is determined to violate laws and regulations is treated as a violating rule, and the prediction model is automatically adjusted. This makes it possible to automatically build a prediction model that does not violate laws and regulations.

[0139] In the above, a single control unit 16 is exemplified as an embodiment of the information processing device according to the present technology. However, the information processing device according to the present technology may be realized by any computer configured separately from the control unit 16 and connected to the control unit 16 via wired or wireless communication. For example, the information processing method according to the present technology may be executed by a cloud server. Alternatively, the control unit 16 may be linked with another computer to execute the information processing method according to the present technology.

[0140] That is, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing are both systems.

[0141] The information processing method and program execution according to the present technology by a computer system includes both cases where the process of training a predetermined learning model, the process of training a summary model, the acquisition of evaluation information, and the adjustment of the learning process of the learning model are executed by a single computer, and cases where each process is executed by a different computer. Furthermore, the execution of each process by a predetermined computer includes causing another computer to execute a part or all of the process and acquiring the result.

[0142] In other words, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by multiple devices via a network.

[0143] It is also possible to combine at least two of the characteristic parts of the present technology described above. That is, the various characteristic parts described in each embodiment may be arbitrarily combined without distinction between the embodiments. In addition, the various effects described above are merely examples and are not limited thereto, and other effects may be exhibited.

[0144] In this disclosure, the terms "same", "equal", "orthogonal", etc. are concepts that include "substantially the same", "substantially equal", "substantially orthogonal", etc. For example, a state that falls within a predetermined range (e.g., a range of ±10%) based on "completely the same", "completely equal", "completely orthogonal", etc. is also included.

[0145] The present technology can also be configured as follows. (1) a first learning unit that learns a predetermined learning model; a second learning unit that learns a conversion model that converts an output of the predetermined learning model into a set of rules described in a format that can be interpreted by a user; an evaluation unit that acquires evaluation information obtained by evaluating the rule group according to a predetermined criterion; an adjustment unit that adjusts the learning process of the predetermined learning model based on the evaluation information; An information processing device comprising: (2) An information processing device according to (1), The predetermined criteria include at least one of criteria set by laws and regulations and criteria set by the user. Information processing device. (3) An information processing device according to (1) or (2), The learning model is a predictive model that predicts a target item. Information processing device. (4) An information processing device according to (3), the set of rules includes at least one output rule that describes an output of the predictive model; The evaluation unit generates at least one of an explanatory text and a diagram for each of the output rules. Information processing device. (5) An information processing device according to (4), The evaluation unit generates a check item for causing the user to check whether or not each of the output rules satisfies the predetermined criterion. Information processing device. (6) An information processing device according to (5), The evaluation unit reads, as the evaluation information, a check result of the user for the check item. Information processing device. (7) An information processing device according to (4) or (5), The evaluation unit generates the check item targeting a data item designated by the user among a plurality of data items included in the learning data of the prediction model. Information processing device. (8) An information processing device according to any one of (4) to (7), A storage unit is provided in which a database relating to the predetermined criteria is stored, The evaluation unit determines whether the output rule satisfies the predetermined criterion based on the database. Information processing device. (9) An information processing device according to (8), The evaluation unit generates a check item for the output rule that is determined not to satisfy the predetermined criterion, the check item prompting the user to check whether the output rule satisfies the predetermined criterion. Information processing device. (10) An information processing device according to (8) or (9), The evaluation unit generates, as the evaluation information, information regarding the output rule that is determined not to satisfy the predetermined criterion. Information processing device. (11) An information processing device according to any one of (4) to (10), the evaluation information includes information regarding a violation rule, which is the output rule that does not satisfy the predetermined criterion; The adjustment unit adjusts at least one of the learning data of the prediction model or the learning parameters of the prediction model based on a data range specified by the violation rule. Information processing device. (12) The information processing device according to (11), The adjustment unit performs at least one of a process of reducing the number of pieces of learning data that cause the violation rule to not satisfy the predetermined criterion among the learning data included in a data range specified by the violation rule, and a process of adding dummy data adjusted to satisfy the predetermined criterion to the data range specified by the violation rule as the learning data. Information processing device. (13) An information processing device according to (11) or (12), The learning parameters include at least one of a parameter for adjusting an output of the prediction model regarding the training data and a parameter for adjusting a loss function of the prediction model. Information processing device. (14) An information processing device according to any one of (11) to (13), the prediction model is a classification model in which a classification regarding the target item is a predicted value, The adjustment unit adjusts a learning process of the prediction model so that a predicted value of the prediction model in a data range specified by the violation rule substantially matches a predicted value of the prediction model in a data range specified by the output rule that satisfies the predetermined criterion. Information processing device. (15) An information processing device according to any one of (11) to (13), the prediction model is a regression model in which the value of the target item is a predicted value; The adjustment unit adjusts a learning process of the prediction model so that a distribution of predicted values ​​of the prediction model in a data range specified by the violation rule substantially matches a distribution of predicted values ​​of the prediction model in a data range specified by the output rule that satisfies the predetermined criterion. Information processing device. (16) An information processing device according to any one of (3) to (15), The evaluation unit presents a plurality of adjustment methods related to an output of the prediction model in a selectable manner; The adjustment unit adjusts the learning process of the prediction model based on a method selected by the user from among the plurality of adjustment methods. Information processing device. (17) An information processing device according to any one of (1) to (16), The second learning unit learns the conversion model according to the predetermined criterion. Information processing device. (18) An information processing device according to any one of (1) to (17), The conversion model is a learning model using at least one of a decision tree and a rule fitting algorithm. Information processing device. (19) Training a predetermined learning model; training a conversion model that converts the output of the predetermined learning model into a set of rules described in a format that can be interpreted by a user; acquiring evaluation information that evaluates the rule group according to a predetermined criterion; adjust the learning process of the predetermined learning model based on the evaluation information The information processing method is executed by a computer system. (20) a step of training a predetermined learning model; A step of training a conversion model that converts the output of the predetermined learning model into a set of rules described in a format that can be interpreted by a user; acquiring evaluation information obtained by evaluating the rule group according to a predetermined criterion; adjusting a learning process of the predetermined learning model based on the evaluation information; A program that causes a computer system to execute the above. [Explanation of symbols]

[0146] 1. Training data 2…Condition area 4. Output rules 5. Check items 10. Prediction model 11…Summary model 15...Storage section 16...Control section 17…Learning database 18…Rule Database 20...UI generation section 21…Prediction model learning section 22…Characteristics evaluation section 23…Summary model learning section 24...Explanation generation section 25…Query generation section 26...Adjustment processing section 100...Data analysis equipment

Claims

1. a first learning unit that learns a predetermined learning model that calculates a target item from input data based on a plurality of learning data; a second learning unit that uses an algorithm that summarizes the correspondence relationships to configure a conversion model that converts an output of the predetermined learning model, which indicates a correspondence relationship between the target items calculated by the predetermined learning model using a plurality of test data including the plurality of learning data or a plurality of other learning data as input, and a set of rules in which the correspondence relationships are described in a format that can be interpreted by a user, and that trains the conversion model using the target items calculated by the predetermined learning model as correct answer data; an evaluation unit that acquires evaluation information obtained by evaluating the rule group according to a predetermined criterion, the evaluation information including a determination result as to whether or not the rule group satisfies the predetermined criterion; an adjustment unit that adjusts at least one of the learning data for training the predetermined learning model or a learning parameter of the predetermined learning model, based on the evaluation information, when the set of rules does not satisfy the predetermined criterion, so that the correspondence indicated by the set of rules satisfies the predetermined criterion; Equipped with the set of rules includes at least one output rule that explains the correspondence relationship between the target item and the feature amount represented by an output of the predetermined learning model; the evaluation information includes information regarding a violation rule, which is the output rule that does not satisfy the predetermined criterion; The adjustment unit adjusts at least one of the learning data or the learning parameters based on a data range related to the feature amount specified by the violation rule. Information processing device.

2. 2. The information processing device according to claim 1, The predetermined criteria include at least one of criteria set by laws and regulations and criteria set by the user. Information processing device.

3. 2. The information processing device according to claim 1, The learning model is a predictive model that predicts the target item. Information processing device.

4. 4. The information processing device according to claim 3, The evaluation unit generates at least one of an explanatory text and a diagram for each of the output rules. Information processing device.

5. 5. The information processing device according to claim 4, The evaluation unit generates a check item for causing the user to check whether or not each of the output rules satisfies the predetermined criterion. Information processing device.

6. 6. The information processing device according to claim 5, The evaluation unit reads, as the evaluation information, a check result of the user for the check item. Information processing device.

7. 6. The information processing device according to claim 5, the learning data includes a plurality of data items as data items; The evaluation unit generates the check item targeted at a data item designated by the user among the plurality of data items. Information processing device.

8. The information processing device according to claim 4, further comprising: A storage unit is provided in which a database relating to the predetermined criteria is stored, The evaluation unit determines whether the output rule satisfies the predetermined criterion based on the database. Information processing device.

9. 9. The information processing device according to claim 8, The evaluation unit generates a check item for the output rule that is determined not to satisfy the predetermined criterion, the check item prompting the user to check whether the output rule satisfies the predetermined criterion. Information processing device.

10. 9. The information processing device according to claim 8, The evaluation unit generates, as the evaluation information, information regarding the output rule that is determined not to satisfy the predetermined criterion. Information processing device.

11. 5. The information processing device according to claim 4, The adjustment unit performs at least one of a process of reducing the number of pieces of learning data that cause the violation rule to not satisfy the predetermined criterion among the pieces of learning data included in the data range specified by the violation rule, and a process of adding dummy data adjusted to satisfy the predetermined criterion to the data range specified by the violation rule as the learning data. Information processing device.

12. 5. The information processing device according to claim 4, The learning parameters include at least one of a parameter for adjusting an output of the prediction model regarding the training data and a parameter for adjusting a loss function of the prediction model. Information processing device.

13. 5. The information processing device according to claim 4, the prediction model is a classification model in which a classification regarding the target item is a predicted value, The adjustment unit adjusts at least one of the learning data or the learning parameters so that a predicted value of the prediction model in the data range specified by the violation rule substantially matches a predicted value of the prediction model in the data range specified by the output rule that satisfies the predetermined criterion. Information processing device.

14. 5. The information processing device according to claim 4, the prediction model is a regression model in which the value of the target item is a predicted value; The adjustment unit adjusts at least one of the learning data or the learning parameters so that a distribution of predicted values ​​of the prediction model in the data range specified by the violation rule substantially matches a distribution of predicted values ​​of the prediction model in the data range specified by the output rule that satisfies the predetermined criterion. Information processing device.

15. 4. The information processing device according to claim 3, The evaluation unit presents a plurality of adjustment methods related to an output of the prediction model in a selectable manner; The adjustment unit adjusts at least one of the learning data or the learning parameters based on a method selected by the user from among the plurality of adjustment methods. Information processing device.

16. 2. The information processing device according to claim 1, The second learning unit learns the conversion model according to the predetermined criterion. Information processing device.

17. 2. The information processing device according to claim 1, The conversion model is a learning model using at least one of a decision tree and a rule fitting algorithm. Information processing device.

18. A predetermined learning model for calculating a target item from input data is trained based on a plurality of learning data; a conversion model that converts an output of the predetermined learning model, which represents a correspondence relationship between the target items calculated by the predetermined learning model using a plurality of test data including the plurality of learning data or a plurality of other learning data as input, and a set of rules in which the correspondence relationship is described in a format interpretable by a user, by using an algorithm that summarizes the correspondence relationship; and training the conversion model using the target items calculated by the predetermined learning model as correct answer data; acquiring evaluation information obtained by evaluating the rule group according to a predetermined criterion, the evaluation information including a determination result as to whether or not the rule group satisfies the predetermined criterion; If the set of rules does not satisfy the predetermined criterion, at least one of the learning data for training the specified learning model or the learning parameters of the specified learning model is adjusted based on the evaluation information so that the correspondence indicated by the set of rules satisfies the predetermined criterion. An information processing method executed by a computer system, comprising: the set of rules includes at least one output rule that explains the correspondence relationship between the target item and the feature amount represented by an output of the predetermined learning model; the evaluation information includes information regarding a violation rule, which is the output rule that does not satisfy the predetermined criterion; The step of adjusting at least one of the learning data and the learning parameters includes adjusting at least one of the learning data and the learning parameters based on a data range related to the feature amount specified by the violation rule. Information processing methods.

19. A step of learning a predetermined learning model for calculating a target item from input data based on a plurality of learning data; a step of configuring a conversion model that converts an output of the predetermined learning model, which represents a correspondence relationship between the target items calculated by the predetermined learning model using a plurality of test data including the plurality of learning data or a plurality of other learning data as an input, and a set of rules in which the correspondence relationship is described in a format interpretable by a user, by using an algorithm that summarizes the correspondence relationship, and training the conversion model using the target items calculated by the predetermined learning model as correct answer data; acquiring evaluation information obtained by evaluating the rule group according to a predetermined criterion, the evaluation information including a determination result as to whether or not the rule group satisfies the predetermined criterion; if the set of rules does not satisfy the predetermined criterion, adjusting at least one of the learning data for training the specified learning model or a learning parameter of the specified learning model based on the evaluation information so that the correspondence indicated by the set of rules satisfies the predetermined criterion; A program for causing a computer system to execute the above, the set of rules includes at least one output rule that explains the correspondence relationship between the target item and the feature amount represented by an output of the predetermined learning model; the evaluation information includes information regarding a violation rule, which is the output rule that does not satisfy the predetermined criterion; The step of adjusting at least one of the learning data and the learning parameters includes adjusting at least one of the learning data and the learning parameters based on a data range related to the feature amount specified by the violation rule. program.

Images