Computer system and model learning method

Abstract

To provide a computer system that generates a model that performs inferences based on a decision rule commonly used in business.SOLUTION: A computer system holds a plurality of pieces of input data, a plurality of rules for classifying classes, and class ratio information that stores a probability distribution of the classes. The computer system performs inference for each combination of the rules and the input data; stores them as inference result information; calculates, on the basis of inference result information, first probability that an inference result based on one rule is in the first class; calculates, on the basis of the inference result information, second probability that the inference result based on one rule is in a second class and an inference result based on another rule is in a third class; calculates, on the basis of the class ratio information, the first probability, and the second probability, when a correct class is a fourth class, third probability that the inference result based on the rule is in a fifth class; and stores a model as model information that defines the model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system and method for presenting data useful for interpreting the basis for prediction results of an AI that makes inferences based on a model constructed by machine learning. [Background technology]

[0002] In recent years, services utilizing AI have emerged in various fields, including healthcare and finance. For example, in the healthcare field, AI is used to predict the incidence of human diseases and identify human symptoms. In the financial field, AI is used to predict the incidence of loan defaults for applicants.

[0003] Advances in technological development aimed at improving the accuracy of AI are accelerating the black-box nature of AI models. This has led to problems in which users of AI cannot trust the inferences made by AI. Against this background, there is a growing demand for the explainability of AI. The technology described in Patent Document 1 is known as a technology that ensures the explainability of AI.

[0004] Patent Document 1 describes a computer-implemented method for training a predictive model from data including a plurality of features, each feature associated with a real-value and a time component, the method comprising: executing the following steps in a processor of the computer, the steps including: a) defining a number of predicates as binary functions operating on a time series of the features or logical operations on the time series of the features; and b) iteratively training a boosting model by performing the following operations: 1) generating a predetermined number of new random predicates; and 2) using weighted information regarding class labels associated with the predictions of the boosting model. 3) selecting a predetermined number of the new random predicates with the highest weighted information gains and adding the selected predicates to the boosting model; 4) calculating weights for all the random predicates in the boosting model; 5) removing one or more of the selected new predicates with the highest weighted information gains from the boosting model in response to input from an operator; and 6) repeating the performance of steps 1, 2, 3, 4, and 5 multiple times, thereby generating a final iteratively trained boosting model. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2020-529058 [Patent Document 2] Japanese Patent Application Publication No. 03-121534 [Non-patent literature]

[0006] [Non-Patent Document 1] Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Advances in Neural Information Processing Systems (pp. 4765-4774). 2017. Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 describes generating a model using predicates. A predicate corresponds to a type of decision rule. It is also important in terms of increasing explainability that the AI ​​decision rule is the same as the decision rule generally used in business. However, with the technology described in Patent Document 1, the decision rule may not be the one generally used in business.

[0008] On the other hand, the technology described in Patent Document 2 is known as a technology for performing inference using decision rules that are generally used in business.

[0009] Patent Document 2 describes a rule-based inference device characterized in that "in an expert system comprising a knowledge base (13) storing rules, an inference engine (12) that applies given initial facts to the rules to derive new facts, and a certainty calculation means (14), the device comprises a certainty calculation switching device (19), the certainty calculation means (14) comprises a plurality of different types of certainty calculation devices (14A, 14B, etc.), the knowledge base (13) comprises a plurality of rules corresponding to the certainty calculation devices, the certainty calculation switching device (19) selects an appropriate certainty calculation device (14A, 14B, etc.) and an appropriate rule corresponding to the certainty calculation device, inputs and outputs them to the inference engine (12), and calculates certainty using a specified calculation method when proceeding with inference."

[0010] The technology described in Patent Document 2 requires the user to assign a degree of certainty to a rule. However, it is difficult to set the degree of certainty of a rule in advance. In particular, it is difficult to set the degree of certainty for rules regarding rare cases, and the rule itself may be incorrect.

[0011] The present invention provides a system and method for generating a model that reduces the workload of a user and performs inference based on decision rules that are commonly used in business. [Means for solving the problem]

[0012] A representative example of the invention disclosed in the present application is as follows: That is, a computer system for learning a model that performs inference regarding class classification includes at least one computer having a processor, a storage device connected to the processor, and a network interface connected to the processor, and holds a plurality of input data including a plurality of item values, a plurality of rules for performing class classification, and class ratio information that stores a probability distribution of classes, wherein the processor performs inference for each combination of the rules and the input data, stores the inference result information in the storage device, calculates a first probability that a result of inference based on one of the rules will be the first class based on the inference result information, calculates a second probability that a result of inference based on one of the rules will be the second class and a result of inference based on another of the rules will be the third class based on the class ratio information, the first probability, and the second probability, calculates a third probability that a result of inference based on one of the rules will be the fifth class when a correct class is the fourth class, and stores the third probability for each of the plurality of rules in the storage device as model information that defines the model. [Effects of the Invention]

[0013] According to the present invention, it is possible to reduce the workload of the user and generate a model that performs inference based on decision rules that are commonly used in business. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates an example of a system configuration according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of a computer that constitutes the inference system of the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a data structure of a dataset according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a data structure of function information according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a data structure of class ratio information according to the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a data structure of rule data according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a data structure of a feature-added dataset according to the first embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a data structure of a rule application matrix according to the first embodiment. [Figure 9] 1 is a flowchart illustrating an example of processing executed by the inference system of the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a screen displayed on a user terminal according to the first embodiment. [Figure 11] A figure showing an example of the data structure of inference model information in Example 1. [Figure 12] FIG. 10 is a diagram illustrating an example of a data structure of inference result information according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a data structure of rule reliability information according to the first embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of a screen displayed on a user terminal according to the first embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of a system configuration according to a second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a data structure of basis information according to the second embodiment. [Figure 17] 10 is a flowchart illustrating an example of processing executed by the inference system of the second embodiment. [Figure 18] FIG. 11 is a diagram illustrating an example of a screen displayed on a user terminal according to the second embodiment. [Figure 19] FIG. 11 is a diagram illustrating an example of a screen displayed on a user terminal according to the second embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of a system configuration according to a third embodiment. [Figure 21] FIG. 11 is a diagram illustrating an example of a data structure of correct answer information according to the third embodiment. [Figure 22] FIG. 11 is a diagram illustrating an example of a data structure of target value information according to the third embodiment. [Figure 23] FIG. 11 is a diagram illustrating an example of a data structure of weight information according to the third embodiment. [Figure 24] 11 is a flowchart illustrating an example of processing executed by the inference system of the third embodiment. [Figure 25] FIG. 11 is a diagram illustrating an example of a screen displayed on a user terminal according to a third embodiment. [Figure 26] FIG. 11 is a diagram illustrating an example of the data structure of information generated during processing by the inference system of the third embodiment. [Figure 27] FIG. 11 is a diagram illustrating an example of the data structure of information generated during processing by the inference system of the third embodiment. [Figure 28] FIG. 11 is a diagram illustrating an example of the data structure of information generated during processing by the inference system of the third embodiment. [Figure 29] FIG. 11 is a diagram illustrating an example of a data structure of a rule list according to the third embodiment. [Figure 30] FIG. 11 is a diagram illustrating an example of a screen displayed on a user terminal according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0016] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.

[0017] In this specification, the terms "first," "second," "third," etc. are used to identify components and do not necessarily limit the number or order.

[0018] To facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings etc. [Example]

[0019] Fig. 1 is a diagram illustrating an example of the configuration of a system according to the first embodiment. Fig. 2 is a diagram illustrating an example of the hardware configuration of a computer that constitutes the inference system according to the first embodiment.

[0020] The system is composed of an inference system 100 and a user terminal 101. The inference system 100 and the user terminal 101 are connected directly or via a network. The network may be, for example, a wide area network (WAN) or a local area network (LAN), and the connection method may be either wired or wireless.

[0021] The inference system 100 generates a model for making inferences and uses the model to make inferences. The model is assumed to be a model for classifying classes. The user terminal 101 transmits various information to the inference system 100 and receives various information output from the inference system 100.

[0022] The inference system 100 is composed of, for example, a computer 200 as shown in Figure 2. The computer 200 has a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. The hardware elements are connected to each other via a bus. The computer 200 may have input devices such as a keyboard, a mouse, and a touch panel, and may also have output devices such as a display and a printer.

[0023] The processor 201 executes a program stored in the main memory device 202. The processor 201 executes processing in accordance with the program, thereby operating as a functional unit (module) that realizes a specific function. In the following description, when a processing is described using a functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.

[0024] The main memory device 202 is a storage device such as a DRAM (Dynamic Random Access Memory), and stores programs executed by the processor 201 and data used by the programs. The main memory device 202 is also used as a work area.

[0025] The secondary storage device 203 is a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) and permanently stores data. The programs and data stored in the main storage device 202 may be stored in the secondary storage device 203. In this case, the processor 201 reads the programs and data from the secondary storage device 203 and loads them into the main storage device 202. The network interface 204 performs communication via a network.

[0026] The inference system 100 includes a user input unit 110 , a rule input unit 111 , a feature calculation unit 112 , a rule application matrix generation unit 113 , an inference processing unit 114 , and a display unit 115 .

[0027] The user input unit 110 receives input of various information used for generating a model, etc. The user input unit 110 of the first embodiment receives input of a dataset 120, function information 121, and class ratio information 122 from the user terminal 101.

[0028] The dataset 120 includes multiple pieces of input data used for inference. The input data includes multiple item values. The function information 121 is definition information of a function for calculating features used for inference using the item values. The class ratio information 122 is information about the probability distribution of classes. Note that the correct label cannot be observed in advance.

[0029] The rule input unit 111 receives input of rules used in business to classify classes, and stores the received rules as rule data 123.

[0030] The feature calculation unit 112 calculates the feature of each piece of input data included in the dataset 120 using the function information 121. The feature calculation unit 112 outputs a feature-added dataset 130 made up of the input data to which the feature has been added.

[0031] The rule application matrix generation unit 113 performs inference (classification) on input data with features using rules corresponding to the rule data 123, and generates a rule application matrix 131. The rule application matrix 131 is data for managing the results of inference on input data with features using rules corresponding to the rule data 123.

[0032] The inference processing unit 114 generates an inference model using the rule application matrix 131 and the class ratio information 122, and outputs it as inference model information 132. The inference processing unit 114 calculates a reliability that indicates the degree of reliability of the rule in the inference, and outputs it as rule reliability information 134. The inference processing unit 114 also uses the inference model to perform inference on input data, and outputs it as inference result information 133.

[0033] The display unit 115 displays various information on the user terminal 101. Specifically, the display unit 115 displays inference model information 132, inference result information 133, rule reliability information 134, and the like on the user terminal 101.

[0034] Regarding the functional units of the inference system 100, multiple functional units may be combined into one functional unit, or one functional unit may be divided into multiple functional units for each function.

[0035] FIG. 3 is a diagram illustrating an example of the data structure of the data set 120 according to the first embodiment.

[0036] The dataset 120 stores entries each including a data ID 301 and a data value 302. One entry exists for each piece of input data. Note that the fields included in an entry are not limited to those described above.

[0037] The data ID 301 is a field that stores identification information of the input data. The data value 302 is a group of fields that stores item values ​​included in the input data. The data value 302 includes fields for the number of items.

[0038] FIG. 4 is a diagram illustrating an example of the data structure of the function information 121 according to the first embodiment.

[0039] The function information 121 stores entries each including a feature ID 401 and a function 402. One entry exists for one feature. Note that the fields included in the entry are not limited to those described above.

[0040] A feature ID 401 is a field for storing identification information of a feature. A function 402 is a field for storing a function for calculating a feature.

[0041] FIG. 5 is a diagram illustrating an example of the data structure of the class ratio information 122 according to the first embodiment.

[0042] The model of the first embodiment is assumed to classify input data into either class "1" or class "0." In this case, the class ratio information 122 includes a probability P(Y=1) and a probability P(Y=0). P(Y=1) is the probability of becoming class "1," and P(Y=0) is the probability of becoming class "0."

[0043] FIG. 6 is a diagram illustrating an example of the data structure of the rule data 123 according to the first embodiment.

[0044] The rule data 123 includes a rule ID 601 and a rule 602. The rule ID 601 is a field for storing identification information of a rule. The rule 602 is a field for storing a rule. For example, the rule 602 stores a function for outputting a value related to a label from a feature amount.

[0045] FIG. 7 is a diagram illustrating an example of the data structure of the feature-added dataset 130 according to the first embodiment.

[0046] The feature-attached dataset 130 stores entries each including a data ID 701, a data value 702, and a feature 703. One entry exists for one piece of input data. Note that the fields included in the entry are not limited to those described above.

[0047] The data ID 701 and the data value 702 are the same fields as the data ID 301 and the data value 302. The feature amount 703 is a group of fields that store feature amounts. The feature amount 703 includes as many fields as there are feature amounts.

[0048] FIG. 8 is a diagram showing an example of the data structure of the rule application matrix 131 according to the first embodiment.

[0049] The row direction of the rule application matrix 131 corresponds to the input data. The column direction of the rule application matrix 131 corresponds to the rules. (x i ,L j ) in the cell corresponding to rule L j Input data x using iThe result of inference (class) for the given variable is stored in the cell. In this embodiment, the cell stores either "0", "1", or "-1". "-1" is a value indicating that the class is unknown (Abstain).

[0050] The inference results do not have to be data in matrix format.

[0051] FIG. 9 is a flowchart illustrating an example of processing executed by the inference system 100 of the first embodiment. FIG. 10 is a diagram illustrating an example of a screen displayed on the user terminal 101 of the first embodiment. FIG. 11 is a diagram illustrating an example of the data structure of inference model information 132 of the first embodiment. FIG. 12 is a diagram illustrating an example of the data structure of inference result information 133 of the first embodiment. FIG. 13 is a diagram illustrating an example of the data structure of rule reliability information 134 of the first embodiment. FIG. 14 is a diagram illustrating an example of a screen displayed on the user terminal 101 of the first embodiment.

[0052] The inference system 100 displays a screen 1000 on the user terminal 101 and accepts user input (step S101).

[0053] The screen 1000 includes a dataset input field 1001 , a rule input field 1002 , a function input field 1003 , a class ratio input field 1004 , and an execution instruction field 1005 .

[0054] The dataset input field 1001 includes an input box 1010 and a register button 1011. The input box 1010 is a box for inputting a file corresponding to the dataset 120. The register button 1011 is an operation button for registering the dataset 120 in the inference system 100.

[0055] The rule input field 1002 includes an input box 1020, a register button 1021, and a display box 1022. The input box 1020 is a box for inputting a rule. The register button 1021 is an operation button for registering a rule. The display box 1022 is a box for displaying rules registered in the inference system 100.

[0056] The function input field 1003 includes an input box 1030, a register button 1031, and a display box 1032. The input box 1030 is a box for inputting a function. The register button 1031 is an operation button for registering a function. The display box 1032 is a box for displaying functions registered in the inference system 100.

[0057] The class ratio input field 1004 includes an input box 1040 and a register button 1041. The input box 1040 is a box for inputting a file corresponding to the class ratio information 122. The register button 1041 is an operation button for registering the class ratio information 122 in the inference system 100.

[0058] The execution instruction field 1005 includes an execution button 1050. The execution button 1050 is an operation button for issuing an instruction to execute a process using a user input.

[0059] When the inference system 100 receives an operation of the execute button 1050, it generates a feature-attached dataset 130 (step S102). Specifically, the feature calculation unit 112 uses the function information 121 to calculate the feature of each input data included in the dataset 120 and assigns the feature to the input data.

[0060] The inference system 100 generates the rule application matrix 131 (step S103). Specifically, the following process is executed.

[0061] (S103-1) The rule application matrix generation unit 113 initializes the rule application matrix 131.

[0062] (S103-2) The rule application matrix generation unit 113 selects one piece of input data from the feature-added dataset 130.

[0063] (S103-3) The rule application matrix generation unit 113 selects one rule data 123.

[0064] (S103-4) The rule application matrix generation unit 113 classifies the input data into classes based on the feature amounts assigned to the input data and the rules set in the rule 602 of the rule data 123. The rule application matrix generation unit 113 sets the classification results in predetermined cells of the rule application matrix 131.

[0065] (S103-5) The rule application matrix generation unit 113 determines whether or not processing has been completed for all rule data 123. If processing has not been completed for all rule data 123, the rule application matrix generation unit 113 returns to S103-3.

[0066] (S103-6) When processing has been completed for all rule data 123, the rule application matrix generation unit 113 determines whether processing has been completed for all input data. If processing has not been completed for all input data, the rule application matrix generation unit 113 returns to S103-2.

[0067] (S103-7) When the processing has been completed for all input data, the rule application matrix generation unit 113 ends the processing.

[0068] The inference system 100 generates an inference model using the class ratio information 122 and the rule application matrix 131 (step S104). Specifically, the following processing is executed.

[0069] (S104-1) The inference processing unit 114 deletes rows containing "-1" from the rule application matrix 131.

[0070] (S104-2) The inference processing unit 114 uses the values ​​of the columns (rules) of the rule application matrix 131 to determine the rule L i Class λ in i Ga l iCalculate the probability (first probability) that i is either "0" or "1." Here, the first probability is expressed as in equation (1).

[0071]

number

[0072] (S104-3) The inference processing unit 114 uses the values ​​of the columns (rules) of the rule application matrix 131 to determine the rule L i Class λ in i Ga l i And rule L j Class λ in j Ga l j Here, i and j are different values. The second probability is expressed as in equation (2).

[0073]

number

[0074] If each rule is defined from a different perspective, then class λ i can be considered as independent random variables. In this case, the second probability can be transformed into equation (3). The second probability, P(Y=1) and P(Y=0) are known values. Therefore, equation (3) can be expressed as the conditional probability P(λ i =l i |Y=y i ) is an equation with unknowns. The conditional probability P(λ i =l i |Y=y i ) is the correct class y i If ,then ... i Ga l i is the probability that

[0075]

number

[0076] (S104-4) The inference processing unit 114 calculates the conditional probability P(λ) from the simultaneous equations defined by the formula (3). i =l i |Y=y i ) is calculated, where y i is either "0" or "1". The inference processing unit 114 calculates the conditional probability P(λ i =l i |Y=y i ) is output as inference model information 132. For example, the inference model information 132 in a matrix format as shown in FIG. 11 is output. The rows of the matrix are i and the columns of the matrix correspond to Y.

[0077] The conditional probability P(λ i =l i |Y=y i ) corresponds to, for example, the weights of a neural network.

[0078] The inference system 100 performs inference for each input data using the inference model information 132 and outputs the inference result information 133 (step S105).

[0079] Specifically, the inference processing unit 114 calculates the input data x from the rule application matrix 131. k The row value of l ik The value of equation (4) is obtained and calculated as the inference result. Here, equation (4) can be transformed into equation (5). Each probability in equation (5) has already been calculated.

[0080]

number

[0081]

number

[0082] The inference processing unit 114 calculates the input data x kThe inference result is output as inference result information 133. For example, inference result information 133 as shown in FIG. 12 is output. Inference result information 133 stores entries including data ID 1201 and inference result 1202. One entry is stored for one piece of input data. Data ID 1201 is the same field as data ID 301. Inference result 1202 is a field that stores the inference result, i.e., the value of equation (5).

[0083] The inference system 100 calculates the reliability of each rule and outputs it as rule reliability information 134 (step S106).

[0084] Specifically, the inference processing unit 114 calculates the value of the formula (6) as a rule L i The reliability of the rule is calculated and output as rule reliability information 134. For example, rule reliability information 134 as shown in FIG. 13 is output. The rule reliability information 134 stores entries including a rule ID 1301 and a reliability 1302. One entry exists for one rule. The rule ID 1301 is the same field as the rule ID 601. The reliability 1302 is a field that stores the reliability that indicates the degree of reliability (validity) of the rule in inference, that is, the value of formula (6).

[0085]

number

[0086] The inference system 100 displays the inference model information 132, the inference result information 133, and the rule reliability information 134 on the user terminal 101 (step S107). For example, a screen 1400 as shown in FIG.

[0087] As described above, according to the first embodiment, the inference system 100 allows a user to input a dataset 120, rules (rule data 123), and class ratio information 122, thereby enabling the user to generate an inference model that performs rule-based inference. Since there is no need to set the confidence level of the rules, the user's workload can be reduced. Since the class ratio information 122 can be generated from business information using statistical techniques, the user's workload is not heavy. Furthermore, since the inference is performed based on rules used in business, the explainability of the inference can be improved.

[0088] Furthermore, the reasoning system 100 can present the reliability of the rules, which allows the user to recognize useful rules and to review rules with low reliability. [Example]

[0089] In the second embodiment, the inference system 100 calculates the contribution of a rule in inference of input data. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0090] 15 is a diagram showing an example of the system configuration of Example 2. The inference system 100 of Example 2 newly includes a contribution calculation unit 116. Note that the rule input unit 111, feature calculation unit 112, rule application matrix generation unit 113, and inference processing unit 114 are omitted because they are not directly related to the processing described in Example 2.

[0091] The user input unit 110 of the second embodiment receives a data ID specifying input data from the user terminal 101 and outputs it to the contribution degree calculation unit .

[0092] The contribution calculation unit 116 receives as input the row corresponding to the data ID of the rule application matrix 131, the inference model, and the inference result, calculates the contribution of the rule in the inference for the specified input data, and outputs the result as grounds information 135. For example, the contribution calculation unit 116 calculates the contribution based on the algorithm described in Non-Patent Document 1.

[0093] FIG. 16 is a diagram illustrating an example of the data structure of the basis information 135 according to the second embodiment.

[0094] The basis information 135 stores entries including a rule ID 1601 and a contribution degree 1602. One entry exists for one rule. The rule ID 1601 is the same field as the rule ID 601. The contribution degree 1602 is a field that stores the contribution degree.

[0095] Fig. 17 is a flowchart illustrating an example of processing executed by the inference system 100 of the second embodiment. Fig. 18 and Fig. 19 are diagrams illustrating an example of a screen displayed on the user terminal 101 of the second embodiment.

[0096] The inference system 100 displays a screen 1800 on the user terminal 101 and accepts user input (step S201).

[0097] The screen 1800 includes an input box 1801 and an execute button 1802. The input box 1801 is a box for inputting identification information of input data. The execute button 1802 is an operation button for instructing the execution of processing using user input.

[0098] When the inference system 100 receives an operation of the execute button 1802, it acquires the rule application matrix 131, the inference model information 132, and the inference result information 133 (step S202).

[0099] The inference system 100 calculates the contribution of the rule in the inference for the specified input data using the rule application matrix 131, the inference model information 132, and the inference result information 133 (step S203). The inference system 100 generates the calculation result as basis information 135.

[0100] The method for calculating the contribution degree is a well-known technique. For example, the contribution degree is calculated in the following procedure. First, the contribution degree calculation unit 116 extracts a row (row vector) corresponding to input data specified by the user from the rule application matrix 131. The contribution degree calculation unit 116 slightly changes the values ​​of the elements of the row vector and calculates the variation in the inference result. The contribution degree calculation unit 116 calculates the contribution degree of the rule based on the result of the variation.

[0101] The inference system 100 displays the basis information 135 on the user terminal 101 (step S204). On the user terminal 101, for example, a screen 1900 as shown in FIG.

[0102] According to the second embodiment, it is possible to present the degree of contribution of rules to inference on input data, thereby improving the explainability of inference. [Example]

[0103] In the third embodiment, the reasoning system 100 evaluates a combination of rules. The third embodiment will be described below, focusing on the differences from the first embodiment.

[0104] FIG. 20 is a diagram illustrating an example of a system configuration according to the third embodiment.

[0105] The inference system 100 includes a user input unit 110 , a rule input unit 111 , a feature calculation unit 112 , a rule application matrix generation unit 113 , a rule evaluation unit 117 , and a display unit 115 .

[0106] The user input unit 110, rule input unit 111, feature amount calculation unit 112, rule application matrix generation unit 113, and display unit 115 of the third embodiment are the same as those described in the first embodiment.

[0107] However, the user input unit 110 of the third embodiment accepts input of a dataset 120, function information 121, correct value information 124, target value information 125, and weight information 126 from the user terminal 101. Furthermore, the display unit 115 of the third embodiment displays a rule list 136 output from the rule evaluation unit 117 on the user terminal 101.

[0108] The dataset 120 and function information 121 of the third embodiment are the same as those of the first embodiment. Correct value information 124 is information related to the correct label of the input data. Target value information 125 is information related to the target value of the index used to select a rule. Weight information 126 is information related to the weight used when calculating the score for selecting a rule.

[0109] FIG. 21 is a diagram illustrating an example of the data structure of the correct answer information 124 according to the third embodiment.

[0110] The correct value information 124 stores entries each including a data ID 2101 and a correct value 2102. One entry exists for one piece of input data. Note that the fields included in the entry are not limited to those described above.

[0111] The data ID 2101 is the same field as the data ID 301. The correct answer value 2102 is a field that stores a value related to the correct label of the input data. In the third embodiment, the correct answer value 2102 stores the value of the probability P(Y=1).

[0112] FIG. 22 is a diagram illustrating an example of the data structure of the target value information 125 according to the third embodiment.

[0113] The target value information 125 includes a target AUC 2201 , a target correlation coefficient 2202 , and a target coverage 2203 .

[0114] Target AUC 2201 is a field that stores the target value of AUC (Area Under Curve), which is an evaluation index for the model. Target correlation coefficient 2202 is a field that stores the target value of the correlation coefficient that evaluates the correlation between rules. Target coverage 2203 is a field that stores the target value of the proportion (coverage) of input data pairs for which the classification result of either of the two rules takes a value other than "-1" with respect to all input data pairs.

[0115] FIG. 23 is a diagram illustrating an example of the data structure of the weight information 126 according to the third embodiment.

[0116] The weight information 126 stores entries each including an index type 2301 and a weight 2302. There is one entry for each index. Note that the fields included in the entry are not limited to those described above.

[0117] The index type 2301 is a field that stores the type of index. The weight 2302 is a field group that stores the weight for the index. In the third embodiment, two weights, "intercept" and "slope", are set.

[0118] FIG. 24 is a flowchart illustrating an example of processing executed by the inference system 100 of the third embodiment. FIG. 25 is a diagram illustrating an example of a screen displayed on the user terminal 101 of the third embodiment. FIGS. 26, 27, and 28 are diagrams illustrating an example of a data structure of information generated during processing by the inference system 100 of the third embodiment. FIG. 29 is a diagram illustrating an example of a data structure of the rule list 136 of the third embodiment. FIG. 30 is a diagram illustrating an example of a screen displayed on the user terminal 101 of the third embodiment.

[0119] The inference system 100 displays a screen 2500 on the user terminal 101 and accepts user input (step S301).

[0120] The screen 2500 includes a correct answer label input field 2501 , a rule input field 2502 , a function input field 2503 , a target value input field 2504 , a weight input field 2505 , and an execution instruction field 2506 .

[0121] The correct answer label input field 2501 includes an input box 2510 and a register button 2511. The input box 2510 is a box for inputting a file corresponding to the correct answer value information 124. The register button 2511 is an operation button for registering the correct answer value information 124 in the inference system 100.

[0122] The rule input field 2502 includes an input box 2520, a register button 2521, and a display box 2522. The input box 2520 is a box for inputting a rule. The register button 2521 is an operation button for registering a rule. The display box 2522 is a box for displaying rules registered in the inference system 100.

[0123] The function input field 2503 includes an input box 2530, a register button 2531, and a display box 2532. The input box 2530 is a box for inputting a function. The register button 2531 is an operation button for registering a function. The display box 2532 is a box for displaying functions registered in the inference system 100.

[0124] The target value input field 2504 includes an input box 2540 and a register button 2541. The input box 2540 is a box for inputting a file corresponding to the target value information 125. The register button 2541 is an operation button for registering the target value information 125 in the inference system 100.

[0125] The weight input field 2505 includes an input box 2550 and a register button 2551. The input box 2550 is a box for inputting a file corresponding to the weight information 126. The register button 2551 is an operation button for registering the weight information 126 in the inference system 100.

[0126] The execution instruction field 2506 includes an execution button 2560. The execution button 2560 is an operation button for issuing an instruction to execute a process using a user input.

[0127] When the inference system 100 receives an operation of the execute button 2560, it generates a feature-attached dataset 130 (step S302) and also generates a rule application matrix 131 (step S303). The processing of steps S302 and S303 is the same as the processing of steps S102 and S103. Note that if the feature-attached dataset 130 and the rule application matrix 131 have already been generated, the processing of steps S302 and S303 can be omitted.

[0128] The inference system 100 generates accuracy information 2600 using the correct answer information 124 and the rule application matrix 131 (step S304).

[0129] The accuracy information 2600 stores entries including a rule ID 2601 and an AUC 2602. There is one entry for one rule. The rule ID 2601 is the same field as the rule ID 601. The AUC 2602 is a field that stores the AUC, which is an evaluation index of the accuracy of the rule.

[0130] In step S304, the following process is executed.

[0131] (S304-1) The rule evaluation unit 117 initializes the accuracy information 2600. Specifically, empty accuracy information 2600 is generated.

[0132] (S304-2) The rule evaluation unit 117 selects one rule, and calculates the AUC by comparing the value of the row corresponding to the rule in the rule application matrix 131 with the correct answer value information 124.

[0133] (S304-3) The rule evaluation unit 117 adds an entry to the accuracy information 2600 and sets a value to the added entry.

[0134] (S304-4) The rule evaluation unit 117 determines whether or not processing has been completed for all rules. If processing has not been completed for all rules, the rule evaluation unit 117 returns to S304-2. If processing has been completed for all rules, the rule evaluation unit 117 ends the processing of step S304.

[0135] The inference system 100 uses the rule application matrix 131 to generate correlation information 2700 and coverage information 2800 (step S305).

[0136] The correlation information 2700 stores entries including a rule pair 2701 and a correlation coefficient 2702. There is one entry for each rule pair. The rule pair 2701 is a field that stores identification information of the rules that make up the pair, and the correlation coefficient 2702 is a field that stores a correlation coefficient that indicates the degree of correlation between the rules that make up the pair.

[0137] The coverage information 2800 stores entries including a rule pair 2801 and coverage 2802. One entry exists for one rule pair. The rule pair 2801 is the same field as the rule pair 2701. The coverage 2802 is a field that stores the proportion (coverage) of pairs of input data for which the classification results of both rules are a value other than Abstain.

[0138] In step S305, the following process is executed.

[0139] (S305-1) The rule evaluation unit 117 initializes the correlation information 2700 and the coverage information 2800. Specifically, empty correlation information 2700 and empty coverage information 2800 are generated.

[0140] (S305-2) The rule evaluation unit 117 generates a pair of rules.

[0141] (S305-3) The rule evaluation unit 117 selects one pair of rules and obtains a row corresponding to each rule from the rule application matrix 131.

[0142] (S305-4) The rule evaluation unit 117 compares the values ​​of the two rows obtained from the rule application matrix 131 to calculate the correlation coefficient between the rules.

[0143] (S305-5) The rule evaluation unit 117 adds an entry to the correlation information 2700 and sets a value to the added entry.

[0144] (S305-6) The rule evaluation unit 117 determines whether or not the processing has been completed for all rule pairs. If the processing has not been completed for all rule pairs, the rule evaluation unit 117 returns to S305-3.

[0145] (S305-7) When processing has been completed for all rule pairs, the rule evaluation unit 117 selects one rule pair and obtains the row corresponding to each rule from the rule application matrix 131.

[0146] (S305-8) The rule evaluation unit 117 calculates the ratio (coverage) of pairs of input data in which the values ​​of the two rows acquired from the rule application matrix 131 are other than "-1" to all pairs of input data.

[0147] (S305-9) The rule evaluation unit 117 adds an entry to the coverage information 2800 and sets a value to the added entry.

[0148] (S305-10) The rule evaluation unit 117 determines whether or not processing has been completed for all rule pairs. If processing has not been completed for all rule pairs, the rule evaluation unit 117 returns to S305-7. If processing has not been completed for all rule pairs, the rule evaluation unit 117 ends the processing of step S305.

[0149] The inference system 100 uses the target value information 125 to perform filtering of the accuracy information 2600, the correlation information 2700, and the coverage information 2800 (step S306). Specifically, the following processing is performed.

[0150] (S306-1) The rule evaluation unit 117 removes entries of the accuracy information 2600 whose AUC 2602 value is smaller than the target AUC 2201 value.

[0151] (S306-2) The rule evaluation unit 117 removes entries of the correlation information 2700 whose correlation coefficient 2702 value is smaller than the target correlation coefficient 2202 value.

[0152] (S306-3) The rule evaluation unit 117 removes entries of the coverage information 2800 whose coverage 2802 value is smaller than the target coverage 2203 value.

[0153] The inference system 100 uses the filtered accuracy information 2600, correlation information 2700, and coverage information 2800 to generate a group of rules consisting of three or more elements, and registers the group in the rule list 2900 (step S307).

[0154] Specifically, the rule evaluation unit 117 generates a group of rules based on the rules registered in the filtered accuracy information 2600 and pairs of rules registered in the correlation information 2700 and the coverage information 2800.

[0155] Here, an example will be described in which five rules L1, L2, L3, L4, and L5 are registered. Entries for L1, L3, L4, and L5 are registered in the accuracy information 2600, and entries for L2-L5 are registered in the correlation information 2700. 3、Assume that entries L2-L4, L2-L5, L3-L4, L3-L5, and L4-L5 are registered, and entries L1-L2, L1-L3, L1-L4, L1-L5, L2-L4, L2-L5, L3-L4, and L4-L5 are registered in coverage information 2800. In this case, a group is generated that does not include L2 as an element and does not include the combinations L1-L2, L1-L3, L2-L3, and L3-L5. That is, a group is generated that includes L1, L4, and L5 as elements.

[0156] The rule list 2900 stores entries including a rule group 2901, a number of rules 2902, an average AUC 2903, an average correlation coefficient 2904, an average coverage 2905, and a score 2906. One entry exists for one rule group. The rule group 2901 is a field that stores a list of identification information for the rules included in the group. The number of rules 2902 is a field that stores the number of rules included in the group. The average AUC 2903 is a field that stores the AUC of the group. The average correlation coefficient 2904 is a field that stores the correlation coefficient of the group. The average coverage 2905 is a field that stores the coverage of the group. The score 2906 is a field that stores a score that represents the effectiveness of the group in inference. At this point, the average AUC 2903, the average correlation coefficient 2904, the average coverage 2905, and the score 2906 are empty.

[0157] The inference system 100 calculates the average AUC, average correlation coefficient, and average coverage for each group (step S308). Specifically, the following processing is performed.

[0158] (S308-1) The rule evaluation unit 117 selects a group.

[0159] (S308-2) The rule evaluation unit 117 obtains the AUC of each rule included in the selected group from the accuracy information 2600. The rule evaluation unit 117 calculates the average AUC by dividing the total of the obtained AUCs by the number of elements. The rule evaluation unit 117 references the rule list 2900 and sets the calculated value as the average AUC 2903 of the entry corresponding to the selected group.

[0160] (S308-3) The rule evaluation unit 117 generates a pair of rules from the rules included in the selected group.

[0161] (S308-4) The rule evaluation unit 117 acquires the correlation coefficient of each pair from the correlation information 2700. The rule evaluation unit 117 calculates an average correlation coefficient by dividing the sum of the acquired correlation coefficients by the total number of pairs. The rule evaluation unit 117 refers to the rule list 2900 and sets the calculated value as the average correlation coefficient 2904 of the entry corresponding to the selected group.

[0162] (S308-5) The rule evaluation unit 117 acquires the coverage of each pair from the coverage information 2800. The rule evaluation unit 117 calculates the average coverage by dividing the total of the acquired coverages by the total number of pairs. The rule evaluation unit 117 references the rule list 2900 and sets the calculated value as the average coverage 2905 of the entry corresponding to the selected group.

[0163] (S308-6) The rule evaluation unit 117 determines whether or not processing has been completed for all groups. If processing has not been completed for all groups, the rule evaluation unit 117 returns to S308-1. If processing has been completed for all groups, the rule evaluation unit 117 ends the processing of step S308.

[0164] The inference system 100 calculates a score for each group using the number of rules, the average AUC, the average correlation coefficient, the average coverage, and the weight information 126 (step S309).

[0165] The inference system 100 displays the rule list 2900 on the user terminal 101 (step S310). Note that together with the rule list 2900, the accuracy information 2600, the correlation information 2700, and the coverage information 2800 may also be displayed. For example, a screen 3000 such as that shown in FIG. 30 is displayed on the user terminal 101.

[0166] As described above, according to the third embodiment, the inference system 100 can evaluate rule combinations that improve the accuracy of inference. The user can select the rules to use by referring to the evaluation results.

[0167] Note that reasoning system 100 may automatically select the rule combination to use based on the score.

[0168] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0169] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0170] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java (registered trademark).

[0171] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

[0172] In the above-described embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]

[0173] 100 Inference Systems 101 User terminal 110 User input section 111 Rule Input Section 112 Feature calculation unit 113 Rule application matrix generation unit 114 Inference processing unit 115 Display section 116 Contribution calculation unit 117 Rule Evaluation Unit 120 datasets 121 Function Information 122 Class Ratio Information 123 Rule Data 124 Correct answer information 125 Target Value Information 126 Weight Information 130 feature-based datasets 131 Rule Application Matrix 132 Inference Model Information 133 Inference result information 134 Rule reliability information 135 Basis Information 136 Rules List 200 calculator 201 processor 202 Main storage 203 Secondary storage device 204 Network Interface 1000, 1400, 1800, 2500, 3000 screens 2600 Accuracy Information 2700 Correlation Information 2800 Coverage Information 2900 Rules List

Claims

1. 1. A computer system for learning a model that makes inferences about class classification, comprising: at least one computer having a processor, a storage device connected to the processor, and a network interface connected to the processor; A plurality of input data including a plurality of item values, a plurality of rules for classifying classes, and class ratio information storing probability distributions of classes are held; The processor: performing an inference for each combination of the rule and the input data, and storing the inference result information in the storage device; calculating a first probability that the result of an inference based on one of the rules will be a first class based on the inference result information; calculating a second probability that the result of an inference based on one of the rules will be a second class and that the result of an inference based on another of the rules will be a third class based on the inference result information; calculating, for each rule, a third probability that the result of the inference based on the rule will be the fifth class when the correct class is the fourth class, based on the class ratio information, the first probability, and the second probability; The computer system is characterized in that the third probability of each of the plurality of rules is stored in the storage device as model information that defines the model.

2. 2. The computer system of claim 1, The processor calculates the third probability by solving simultaneous equations with the third probability as an unknown and the probability distribution of the class, the first probability, and the second probability as coefficients.

3. 2. The computer system of claim 1, The processor: calculating a reliability indicating a degree of reliability of the rule in the inference based on the class ratio information and the third probability of each of the plurality of rules; A computer system that generates information for displaying the reliability.

4. 2. The computer system of claim 1, The processor: performing an inference on each of the plurality of input data based on the model information; A computer system that generates information for displaying inference results.

5. 2. The computer system of claim 1, The processor: Accepting an instruction specifying the input data to be processed; Obtaining an inference result for the input data to be processed from the inference result information; calculating a degree of contribution indicating the degree of contribution of each of the plurality of rules in the inference of the input data to be processed based on the result of the inference of the input data to be processed; A computer system that generates information for displaying the degree of contribution.

6. 2. The computer system of claim 1, retaining correct answer value information that stores correct answer values ​​for inferences of each of the plurality of input data; The processor: calculating, for each of the plurality of rules, the accuracy of the rule in inference using the inference result information and the correct answer value information; calculating a correlation coefficient representing a degree of correlation between the rules for each pair of the rules using the inference result information and the correct answer information; Using the inference result information and the correct answer information, calculate, for each pair of rules, a coverage, which is the proportion of pairs of input data for which none of the results of inference based on the rules constituting the pair are unclassifiable, to all pairs of input data; generating a group consisting of three or more of said rules; calculating a score representing the effectiveness of the group in inference based on the accuracy of the rules included in the group and the correlation coefficients and coverages of pairs of the rules included in the group; A computer system that generates information for displaying the groups and the scores.

7. A method for learning a model that performs inference regarding class classification, executed by a computer system, comprising: The computer system at least one computer having a processor, a storage device connected to the processor, and a network interface connected to the processor; A plurality of input data including a plurality of item values, a plurality of rules for classifying classes, and class ratio information storing probability distributions of classes are held; The model learning method includes: a first step in which the processor performs inference for each combination of the rule and the input data, and stores the inference result information in the storage device; a second step in which the processor calculates a first probability that a result of an inference based on one of the rules will be a first class based on the inference result information; a third step in which the processor calculates, based on the inference result information, a second probability that the result of the inference based on one of the rules will be a second class and that the result of the inference based on the other of the rules will be a third class; a fourth step in which the processor calculates, for each rule, a third probability that a result of inference based on the rule will be a fifth class when a correct class is a fourth class, based on the class ratio information, the first probability, and the second probability; a fifth step in which the processor stores the third probability of each of the plurality of rules in the storage device as model information defining the model.

8. A method for training a model according to claim 7, comprising: The fourth step is a model training method characterized in that the processor calculates the third probability by solving simultaneous equations in which the third probability is an unknown and the probability distribution of the class, the first probability, and the second probability are coefficients.

9. A method for training a model according to claim 7, comprising: the processor calculating a confidence score representing a degree of reliability of the rule in the inference based on the class ratio information and the third probability of each of the plurality of rules; and generating information indicating the confidence level by the processor.

10. A method for training a model according to claim 7, comprising: the processor making an inference for each of the plurality of input data based on the model information; and generating information for displaying the results of the inference by the processor.

11. A method for training a model according to claim 7, comprising: receiving, by the processor, an instruction specifying the input data to be processed; The processor acquires a result of inference of the input data to be processed from the inference result information; a step of the processor calculating a degree of contribution indicating a degree of contribution of each of the plurality of rules in the inference of the input data to be processed based on a result of the inference of the input data to be processed; and generating information indicating the degree of contribution by the processor.

12. A method for training a model according to claim 7, comprising: the computer system holds correct answer value information that stores a correct answer value of an inference for each of the plurality of input data; The model learning method includes: the processor uses the inference result information and the correct answer information to calculate the accuracy of the rule in inference for each of the plurality of rules; a step of the processor calculating a correlation coefficient representing a degree of correlation between the rules for each pair of the rules using the inference result information and the correct answer information; the processor uses the inference result information and the correct answer information to calculate, for each pair of rules, a coverage, which is the proportion of pairs of input data for which none of the results of inference based on the rules constituting the pair is unclassifiable, to all pairs of input data; said processor generating a group consisting of three or more of said rules; The processor calculates a score representing the effectiveness of the group in inference based on the accuracy of the rules included in the group, and the correlation coefficient and the coverage of pairs of the rules included in the group; the processor generating information for displaying the groups and the scores.

Images