Classification class inference device, classification class inference method, and classification class inference program

Abstract

To provide a class inference apparatus, a class inference method, and a class inference program for classifying data to be subjected to inference, into an appropriate class, even if learning data constituting an AI model is insufficient.SOLUTION: A class inference apparatus 1 executes: AI model inference processing which outputs a class corresponding to designated data 113 to be subjected to inference by inputting the data 113 to an AI model 116; ensemble processing which combines a result output by alternative model inference processing with a result output by the AI model inference processing; and adjustment processing which adjusts the weight of each of the AI model 116 executing the ensemble processing and an alternative model (empirical rule model 115) on the basis of relationship data of the AI model 116 and relationship data of the alternative model (empirical rule model 115).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a classification class inference device, a classification class inference method, and a classification class inference program. [Background technology]

[0002] When operating a business system that uses AI (Artificial Intelligence) models There are cases where it is necessary to handle business data in a new format that differs from the format previously assumed. Typically, this requires accumulating a sufficient amount of business data in the new format and using that data as training data to rebuild the AI ​​model. However, accumulating a sufficient amount of business data in a business system takes years, making it difficult to quickly adapt to new formats.

[0003] To address these problems, Non-Patent Document 1 presents a method for constructing a generative model that generates pseudo-training data similar to the training data from the training data using an algorithm similar to that of a GAN (Generative Adversarial Network). From one training data, this generative model generates μ (where μ is an integer) pseudo-training data, and by using both the training data and the pseudo-training data, the available training data is increased by 1 + μ times to construct an AI model. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zhengping Che, et.al, “Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records,” IEEE International Conference on Data Mining, pp787-792, Nov. 2017. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, the technology described in Non-Patent Document 1 assumes the existence of training data when constructing a generative model to generate pseudo-training data. Therefore, the technology described in Non-Patent Document 1 cannot construct an AI model if there is no training data at all, and thus cannot handle new formats of business data. On the other hand, when the amount of training data for AI models increases with the operation of business systems, it may be more convenient to build the AI ​​model without using pseudo-training data like that described in Non-Patent Document 1. In other words, operating an AI model when the amount of training data is insufficient for its construction is a significant challenge.

[0006] This invention has been made in view of these circumstances, and its purpose is to provide a classification class inference device, a classification class inference method, and a classification class inference program that can classify data to be inferred into an appropriate class even when there is insufficient training data to construct an AI model. [Means for solving the problem]

[0007] One of the present inventions for solving the above problems is a classification class inference device that constructs an alternative model different from an AI model constructed based on training data, comprising a storage device that stores rules for determining a classification class corresponding to a given condition from a given condition relating to a business, and the rules relating to the business Generate pseudo learning data corresponding to the pieces, and construct the alternative model based on the pseudo learning data and the rules; and an alternative model inference process for outputting the classification class corresponding to the inference target data by inputting the specified inference target data into the alternative model; A processing device that executes the above is provided. A classification class inference device.

Advantages of the Invention

[0008] According to the present invention, even when the learning data for constructing the AI model is insufficient, the inference target data can be classified into appropriate classes. Configurations, effects, etc. other than those described above will be clarified by the description of the following embodiments.

Brief Description of Drawings

[0009] [Figure 1] It is a diagram showing an example of the hardware configuration of the classification class inference device in the present embodiment. [Figure 2] It is a diagram showing an example of the functional configuration of the classification class inference device in the present embodiment. [Figure 3] It is a diagram showing an example of learning data. [Figure 4] It is a diagram showing an example of parameters in the AI model. [Figure 5] It is a diagram showing an example of inference target data. [Figure 6] It is a diagram showing an example of an inference target data discrimination rule. [Figure 7] It is a diagram showing an example of an inference result. [Figure 8] It is a diagram showing an example of an experience rule. [Figure 9] It is a diagram showing an example of an experience rule model. [Figure 10] It is a diagram showing an example of achievement accuracy prediction information. [Figure 11] It is a diagram showing an example of a reconstruction instruction. [Figure 12] It is a processing flowchart for explaining the outline of the processing of the classification class inference device. [Figure 13] This is a process flow diagram that explains the details of the process for building an empirical rule model. [Figure 14] This diagram shows the data structure of LabelMatrix. [Figure 15] This is a processing flow diagram that explains the details of the empirical rule model inference process. [Figure 16] This is a processing flow diagram that explains the details of the process for predicting the accuracy achieved by the empirical rule model. [Figure 17] This figure shows the data structure of the pseudo-correct answer data. [Figure 18] This diagram shows the data structure of a pseudo-LabelMatrix. [Figure 19] This figure shows an example of related data. [Figure 20] This is a processing flow diagram that explains the details of the AI ​​model's accuracy prediction process. [Figure 21] This is a process flow diagram that explains the details of the process for issuing an instruction to rebuild the experience rule model. [Figure 22] This is a process flow diagram that explains the details of the AI ​​model reconstruction instruction issuance process. [Figure 23] This is a process flow diagram that explains the details of the automatic parameter adjustment process. [Figure 24] This figure shows an example of an empirical rule and training data input GUI. [Figure 25] This figure shows an example of a GUI for instructing model reconstruction. [Figure 26] This graph shows an example of the relationship between data volume and accuracy. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. The following description and drawings are illustrative for illustrating the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be carried out in various other forms. Unless otherwise specified, each component may be singular or plural. The positions, sizes, shapes, and ranges of the components shown in the drawings may not represent their actual positions, sizes, shapes, and ranges in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in the drawings. In the following explanation, various types of information may be described using terms such as "table," "list," and "queue," but these types of information may also be represented using other data structures. To indicate independence from data structure, "XX table," "XX list," etc., may be referred to as "XX information." When describing identification information, terms such as "identification information," "identifier," "name," "ID," and "number" will be used, but these terms are interchangeable. When there are multiple components with the same or similar function, they may be described using the same symbol but with different subscripts. However, if it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description. Furthermore, while the following explanation may describe the processes performed by executing a program, the processor (e.g., CPU, GPU) executes the program, performing defined processes using memory resources (e.g., memory) and / or interface devices (e.g., communication ports) as appropriate. Therefore, the processor may be the primary entity performing the processes. Similarly, the primary entity performing the processes by executing a program may be a controller, device, system, computer, or node having a processor. The primary entity performing the processes by executing a program may be an arithmetic unit, and may include dedicated circuits (e.g., FPGAs or ASICs) that perform specific processes. A program may be installed from its program source into a device such as a computer. The program source may be, for example, a program distribution server or a computer-readable storage medium. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. Furthermore, in the following description, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0011] Figure 1 shows an example of the hardware configuration of the classification class inference device 1 in this embodiment. The classification class inference device 1 is an information processing device that constructs an empirical rule model 115 (alternative model) which is a different model from the AI ​​model 116 built based on training data, and which is capable of inference similar to that of the AI ​​model 116. The classification class inference device 1 uses both the AI ​​model 116 and the empirical rule model 115 to infer the classification class of the specified data to be inferred 113.

[0012] In this embodiment, the AI ​​model 116 will be used to describe a classification class inference device 1 that predicts whether or not a customer will default (classification class) based on customer activity data (e.g., credit card usage history) at a financial institution or the like. In this example, the customer activity data is the data to be inferred 113. In this embodiment, the case where the classification class is FALSE (0) or TRUE (1), indicating whether or not a customer will default, will be described, but the classification class is not limited to these, and there may be three or more types.

[0013] The classification class inference device 1 comprises a first computer 10 and a second computer 20. The connection between the first computer 10 and the second computer 20 is, for example, via the Internet or a LAN (Local Area Network). Network, WAN (Wide Area Network), or wired or wireless communication network such as a dedicated line. It is connected by workpiece 30.

[0014] The second computer 20 constructs an AI model 116 and implements an AI business system that uses the AI ​​model 116 to infer the classification class of the data to be inferred 113. The second computer 20 is equipped with a second CPU 21 and a second memory 22 such as RAM and ROM. The second computer 20 is also connected to a second auxiliary storage device 23 such as an HDD or SSD.

[0015] The second CPU 21 (processing unit) executes the program stored in the second memory 22. This enables the implementation of the following: an inference target data discrimination means 104 that determines whether or not to estimate the classification class of the inference target data 113 using the AI ​​model 116; an AI model construction means 103 that constructs the AI ​​model 116; an AI model inference means 105 that estimates the classification class of the inference target data 113 using the AI ​​model 116; and an inference result correction request means 106 that requests the ensemble execution means 107 to correct the inference result of the AI ​​model 116. The second memory 22 stores programs for implementing the inference target data discrimination means 104, the AI ​​model construction means 103, the AI ​​model inference means 105, and the inference result correction request means 106, respectively. The second auxiliary storage device 23 (storage device) stores various data that are input and output when the second CPU 21 executes the program stored in the second memory 22. Specifically, the second auxiliary storage device 23 stores the inference target data 113, which is the target of inference by the AI ​​model 116 or the empirical rule model 115; the inference target data discrimination rule 117, which determines whether or not inference is possible on the inference target data 113 by the AI ​​model 116; the inference result 114 by the AI ​​model 116 or the empirical rule model 115; and the AI ​​model 116.

[0016] The first computer 10 constructs an empirical rule model 115 and implements a hybrid model building and inference platform that uses the empirical rule model 115 to infer the classification class of the data to be inferred 113. For example, when customer activity data is changed to a new data format and the AI ​​model 116 cannot handle the new format, the hybrid model building and inference platform performs inference on the activity data in the new format.

[0017] The first computer 10 comprises a first CPU (Central Processing Unit) 11 and a first memory 12 including RAM (Random Access Memory) and ROM (Read Only Memory). The first computer 10 is connected to a first auxiliary storage device 13, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0018] The first CPU 11 (processing unit) executes a program stored in the first memory 12 to implement the following functions: an empirical rule model construction means 101 for constructing an empirical rule model 115; an empirical rule model inference means 102 for inferring the classification class of the data to be inferred 113 using the empirical rule model 115; an ensemble execution means 107 for ensemble processing the inference results of the AI ​​model 116 and the inference results of the empirical rule model 115; and an achievement accuracy prediction means 108 for predicting the accuracy achieved by the AI ​​model 116 or the empirical rule model 115. The first memory 12 stores programs for implementing the empirical rule model construction means 101, the empirical rule model inference means 102, the ensemble execution means 107, and the achievement accuracy prediction means 108, respectively. The first auxiliary storage device 13 (storage device) stores various data that is input and output when the first CPU 11 executes the programs stored in the first memory 12. Specifically, the first auxiliary storage device 13 stores learning data 111 for constructing the AI model 116 or the experience rule model 115, experience rules 112 for determining classification classes corresponding to the conditions from business-related conditions, the experience rule model 115, the AI model 116, or a reconstruction instruction 119 for the experience rule model 115, and achievement accuracy prediction information 118 of the AI model 116 or the experience rule model 115 predicted by the achievement accuracy prediction means 108, respectively.

[0019] FIG. 2 is a diagram showing an example of the functional configuration of the classification class inference device 1 in the present embodiment. The classification class inference device 1 includes an experience rule model construction means 101, an experience rule model inference means 102, an AI model construction means 103, an inference target data discrimination means 104, an AI model inference means 105, an inference result correction request means 106, an ensemble execution means 107, and an achievement accuracy prediction means 108.

[0020] <AI business system> In the classification class inference device 1, an AI business system including the AI model construction means 103, the inference target data discrimination means 104, the AI model inference means 105, and the inference result correction request means 106 is operating. In the AI business model, the classification class of the inference target data 113 is inferred using the existing AI model 116.

[0021] The AI model construction means 103 takes the learning data 111 as input, constructs the AI model 116 using the learning data 111, and outputs it. The AI model 116 is a learned model for inferring the classification class of the inference target data 113. The AI model 116 is stored in the classification class inference device 1 in advance. The AI model 116 is constructed based on, for example, algorithms of neural networks, decision trees, random forests, and support vector machines (SVMs: Support Vector Machine).

[0022] (Training data) Figure 3 shows an example of training data 111. Each training data 111 contains a data ID 311 that uniquely identifies each training data 111, one or more features 312 (in the illustrated example, feature 1, feature 2, and feature 3), and a ground truth label 313 for the classification class. The ground truth label 313 contains the classification class. The training data 111 includes training data 111 with a ground truth label 313 and training data 111 without a ground truth label 313. Hereinafter, training data 111 with a ground truth label 313 will be referred to as "training data 111 with a ground truth label," and training data 111 without a ground truth label 313 will be referred to as "training data 111 without a ground truth label."

[0023] (AI model) Figure 4 shows an example of parameters in AI model 116. In the case of an AI model 116, for example a neural network model, a list of connections in each neural network is stored in the weight parameter 413, and the values ​​of those weights are stored in the weight value 414.

[0024] The data to be inferred discrimination means 104 takes the data to be inferred 113 as input and determines whether the input data to be inferred 113 can be inferred by the AI ​​model 116 according to the data to be inferred discrimination rule 117. If the data to be inferred discrimination means 104 determines that the data to be inferred 113 can be inferred using the AI ​​model 116, it outputs the data to be inferred 113 to the AI ​​model inference means 105. On the other hand, if the data to be inferred discrimination means 104 determines that the data to be inferred 113 cannot be inferred using the AI ​​model 116, it adds the inference result "Inferrable (-1)" to the data to be inferred 113 and outputs it to the inference result correction request means 106.

[0025] (Data to be inferred) Figure 5 shows an example of the inference data 113. Each inference target data 113 contains a data ID 311 that uniquely identifies each inference target data 113, and one or more feature quantities 312 (in the illustrated example, feature quantity 1, feature quantity 2, and feature quantity 3). In other words, the inference target data 113 has the same data structure as the training data 111, except that it does not contain the correct label 313.

[0026] (Inference target data discrimination rules) Figure 6 shows an example of the inference target data discrimination rule 117. The inference target data discrimination rule 117 has a rule definition 415. The rule definition 415 contains a data frame that stores the entire inference target data 113 and the inference to which the rule will be applied. A function is stored that takes the data ID of the target data 113 as an argument and returns TRUE or FALSE to indicate whether the inference target data 113 is subject to inference by the AI ​​model 116. The rule definition 415 sets conditions for determining whether the inference target data 113 can be inferred by the AI ​​model 116, such as conditions related to the attributes of the data. The inference target data discrimination rule 117 is set in advance by the user.

[0027] The AI ​​model inference means 105 takes the data to be inferred 113 as input and inputs the input data to be inferred 113 into the AI ​​model 116, thereby outputting a classification class (provisional inference result) corresponding to the data to be inferred 113. The AI ​​model inference means 105 adds the provisional inference result to the data to be inferred 113 and outputs it to the inference result correction request means 106.

[0028] The inference result correction request means 106 takes the inference target data 113 with the provisional inference result attached as input, and outputs the input inference target data 113 together with the provisional inference result to the ensemble execution means 107, thereby requesting the ensemble execution means 107 to correct the provisional inference result. The inference result correction request means 106 then outputs the corrected inference result 114 by the ensemble execution means 107.

[0029] (Inference result) Figure 7 shows an example of inference result 114. The inference result 114 stores the data ID 311 of the data to be inferred 113, and the inference result 317 for the data to be inferred 113.

[0030] <Hybrid Model Building and Inference Platform> The classification class inference device 1 also operates a hybrid model construction and inference platform comprising an empirical rule model construction means 101, an empirical rule model inference means 102, an ensemble execution means 107, and an achievement accuracy prediction means 108. In the hybrid model construction and inference platform, if the data to be inferred 113 cannot be inferred using the AI ​​model 116, or if the inference accuracy of the AI ​​model 116 is low, the classification class of the data to be inferred 113 is inferred using an empirical rule model 115, which is an alternative to the AI ​​model 116.

[0031] The experience rule model construction means 101 constructs and outputs an experience rule model 115 based on the training data 111 and the experience rule 112, or only the experience rule 112. The experience rule 112 is a rule that determines the classification class corresponding to a given condition from the conditions related to the business. In this embodiment, the experience rule 112 is an aggregation of business experience such as the activities of each customer handled by the financial institution and whether the customer defaulted as a result.

[0032] (Experience Rules) Figure 8 shows an example of the experience rule 112. Each experience rule 112 has a rule ID 314 that uniquely identifies each experience rule 112, and a rule definition 315 that shows the content of each experience rule 112. The rule definition 315 stores a data frame that stores the entire training data 111, and a function that takes the data ID of the training data 111 to which the rule applies as an argument, and returns as a value which classification class the training data 111 of that data ID should be classified into. This function is a rule that determines the classification class corresponding to the conditions related to the work. Experience rules 112 are set in advance by the user based on their own experience or knowledge, or that of the person in charge of the work. Hereinafter, if the rule ID 314 is "L i The empirical rule 112, which states "(i is an integer)", is sometimes referred to as empirical rule Li.

[0033] Furthermore, empirical rule 112 includes a class ratio 316. The class ratio 316 is the probability that each classification class should be determined. In other words, the class ratio 316 is the probability that each classification class should be determined. Data 113 indicates the proportion in which each classification class should be assigned. In the illustrated example, the inference data 113 is assigned to FALSE and TRUE in a ratio of 0.8 to 0.2. The class ratio 316 is set in advance by the user based on their own experience or knowledge or that of the person in charge of the work. For example, the user sets the class ratio 316 to the ratio of default activity data to non-default activity data from the total customer activity data accumulated so far.

[0034] (Empirical Rule Model) Figure 9 shows an example of the empirical rule model 115. The empirical rule model 115 stores estimated probability values ​​412 for each probability type 411. The probability type is defined by the condition that the latent variable Y (ground truth label) is FALSE (0) or TRUE (1), and the random variable L i A list of probabilities for the random variable L to be FALSE (0) or TRUE (1) is provided. i is the experience rule L iThe results of estimating the classification class (hereinafter referred to as "classification results") are shown. These probability values ​​allow us to determine the confidence level of each empirical rule 112.

[0035] The empirical rule model inference means 102 takes the inference target data 113 output from the ensemble execution means 107 as input, and inputs the input inference target data 113 into the empirical rule model 115, thereby outputting a classification class (provisional inference result) corresponding to the inference target data 113 to the ensemble execution means 107.

[0036] The ensemble execution means 107 performs an ensemble process that combines the provisional inference results from the empirical rule model inference means 102 and the provisional inference results from the AI ​​model inference means 105. Specifically, the ensemble execution means 107 performs an ensemble on the provisional inference results of the inference target data 113 obtained from the inference result correction request means 106 and the empirical rule model inference means 102, and outputs the result of the performed ensemble (corrected inference result) to the inference result correction request means 106. The ensemble execution means 107 performs the ensemble process using methods such as bagging, boosting, or stacking.

[0037] The accuracy prediction means 108 predicts the accuracy achieved by the empirical rule model 115 and the AI ​​model 116, respectively, based on the training data 111, the empirical rule 112, the empirical rule model 115, and the AI ​​model 116, and outputs the prediction result as accuracy prediction information 118.

[0038] (Predicted accuracy information) Figure 10 shows an example of the achievement accuracy prediction information 118. The accuracy prediction information 118 includes the model type 416 and the predicted AUC (Area Under the Curve) 417. The model type 416 indicates either the empirical rule model 115 or the AI ​​model 116. The predicted AUC 417 is the accuracy that the empirical rule model 115 or the AI ​​model 116 is predicted to have achieved.

[0039] Furthermore, the achievement accuracy prediction means 108 monitors the amount of data in the inference target data 113 on which the ensemble execution means 107 has performed ensemble processing. The achievement accuracy prediction means 108 also monitors the amount of data in the training data 111 with correct labels registered by the user. Furthermore, based on the achievement accuracy prediction information 118, the achievement accuracy prediction means 108 automatically adjusts the parameters that control the ensemble processing performed by the ensemble execution means 107. Furthermore, based on the achievement accuracy prediction information 118, the achievement accuracy prediction means 108 issues instructions to the user to rebuild the experience rule model 115 or the AI ​​model 116.

[0040] (Rebuild instruction) Figure 11 shows an example of a reconstruction instruction 119. The reconstruction instruction 119 stores a model type 416 indicating whether the model to be reconstructed is the experience rule model 115 or the AI ​​model 116.

[0041] <Process Overview> Figure 12 is a diagram illustrating the overview of the processing performed by the classification class inference device 1. The classification class inference device 1 executes an empirical rule model construction process S1 to construct an empirical rule model 115. Details of the empirical rule model construction process S1 will be described later. Then, the classification class inference device 1 executes an empirical rule model inference process S2, which infers the classification class of the data to be inferred 113 using the empirical rule model 115 constructed in the empirical rule model construction process S1. Details of the empirical rule model inference process S2 will be described later.

[0042] Meanwhile, the classification class inference device 1 executes process S8 to construct a new AI model 116 (or process to input additional training data 111 into the AI ​​model 116 and perform machine learning). The additional training data 111 is training data 111 with correct labels that has been newly registered by the user after the AI ​​model 116 has been constructed. This training data 111 increases as the business system is operated, and is registered and accumulated by the user or automatically. By inputting the registered additional training data 111 into the AI ​​model 116 and performing machine learning, the estimation accuracy of the AI ​​model 116 is improved. Then, the classification class inference device 1 executes an AI model inference process S9 that uses the AI ​​model 116 to infer the classification class of the data to be inferred 113.

[0043] Here, the classification class inference device 1 outputs an ensemble result (corrected inference result) by performing an ensemble process S10 that combines the provisional inference result from the empirical rule model inference process S2 and the provisional inference result from the AI ​​model inference process S9.

[0044] Furthermore, the classification class inference device 1 executes an empirical rule model achievement accuracy prediction process S3, which predicts the achievement accuracy of the empirical rule model 115 constructed in the empirical rule model construction process S1. Details of the empirical rule model achievement accuracy prediction process S3 will be described later. Similarly, the classification class inference device 1 executes an AI model achievement prediction process S4 to predict the achievement accuracy of the AI ​​model 116 constructed in S8. Details of the AI ​​model achievement accuracy prediction process S4 will be described later.

[0045] Here, the classification class inference device 1 executes an experience rule model reconstruction instruction issuance process S5, which issues an instruction to the user to rebuild the experience rule model 115, based on the achievement accuracy of the experience rule model 115 predicted in the experience rule model achievement accuracy prediction process S3, for example, when a certain amount of data for constructing the experience rule model 115 has been accumulated, as will be described later. Details of the experience rule model reconstruction instruction issuance process S5 will be described later. Furthermore, the classification class inference device 1 executes an AI model reconstruction instruction issuance process S6, which issues an instruction to rebuild the AI ​​model 116 to the user, based on the achievement accuracy of the empirical rule model 115 predicted in the empirical rule model achievement accuracy prediction process S3 and the achievement accuracy of the AI ​​model 116 predicted in the AI ​​model achievement accuracy prediction process S4. For example, when a certain amount of training data for the AI ​​model 116 has been accumulated, as will be described later. Details of the AI ​​model reconstruction instruction issuance process S6 will be described later.

[0046] Furthermore, the classification class inference device 1 operates the ensemble execution means 107 based on the achievement accuracy of the empirical rule model 115 predicted in the empirical rule model achievement accuracy prediction process S3 and the achievement accuracy of the AI ​​model 116 predicted in the AI ​​model achievement accuracy prediction process S4 (empirical rule model The automatic parameter adjustment process S7 is executed to automatically adjust the parameters that control the weighting of 115 and AI model 116. Details of the automatic parameter adjustment process S7 will be described later. Next, we will explain the details of each process described above.

[0047] <Experience Rule Model Construction Process (Alternative Model Construction Process)> Figure 13 is a processing flow diagram illustrating the details of the empirical rule model construction process S1 performed by the classification class inference device 1. The empirical rule model construction process is executed, for example, when a predetermined input is made by the user to the first computer 10 or the second computer 20, or at a predetermined timing (for example, a predetermined time, a predetermined time interval). The empirical rule model construction means 101 receives only the empirical rule 112, or the empirical rule 112 and training data 111 without a ground truth label, as input, and outputs the empirical rule model 115 by executing this empirical rule model construction process.

[0048] First, the experience rule model construction means 101 accepts input of either only the experience rule 112, or the experience rule 112 and training data 111 without a correct answer label (step S101). The experience rule model construction means 101 registers the input training data 111 without a correct answer label and the experience rule 112 by writing them to the first auxiliary storage device 13.

[0049] Next, the empirical rule model construction means 101 determines whether or not the input data entered in step S101 includes training data 111 without correct labels (step S102).

[0050] If the empirical rule model building means 101 determines that the input data includes training data 111 without correct labels (step S102: YES), it applies the registered empirical rule 112 to the training data 111 without correct labels and generates a LabelMatrix 501. Step S103).

[0051] (LabelMatrix) Figure 14 shows the data structure of LabelMatrix501. LabelMatrix501 has data ID 311 in the row, and experience rule 112 has rule ID 51 It has a matrix-like data structure with columns 1 (L1, L2, L3 in the illustrated example). LabelMatrix501 is either training data 111 without ground truth labels or inference data 11 3 stores information indicating whether the classification result obtained by applying each empirical rule 112 is FALSE (0) or TRUE (1). In other words, LabelMatrix501 contains information indicating no correct label. The classification results are stored in the fields corresponding to the data ID 311 of the training data 111 or the inference target data 113, and the rule ID 511 of the experience rule 112.

[0052] Specifically, the experience rule model construction means 101 provides the entire learning data 111 without correct labels and its data ID 311 as arguments to the function defined in the rule definition 315 of the registered experience rule 112, and stores the return value of the function in the corresponding field of the LabelMatrix 501. The experience rule model construction means 101 executes this process for each of all the registered learning data 111 without correct labels, applying all the registered experience rules 112. Then, it proceeds to the process of step S105. On the other hand, when the experience rule model construction means 101 determines that the input data does not include the learning data 111 without correct labels (step S102: NO), it generates a preset fixed amount of data for the LabelMatrix 501 using random numbers (step S104). That is,

[0053] when there is no learning data 111, the experience rule model construction means 101 generates pseudo-learning data by generating the LabelMatrix 501 using random numbers. Then, it proceeds to the process of step S105.

[0054] Following step S103 or step S104, the experience rule model construction means 101 calculates the following probability values by statistically processing the generated LabelMatrix 501 (step S105). Specifically, first, for all the experience rules 112, the experience rule model construction means 101 calculates the probability P(λ i = l i )(where l is 1 or 0) that the return value λ i of the experience rule L i becomes l i . Then, the experience rule model construction means calculates that the return value λ i of the experience rule L becomes l i (1 or 0), and at the same time, the return value λ i of the experience rule L j (where j is an integer other than i) becomes l and the return value λ j of the experience rule Lj The probability P(λ) is (1 or 0) i =l i ∧λ j =l j The empirical rule model construction means 101 calculates the probability P(λ) for all combinations of empirical rules 112. i =l i ∧λ j =l j Calculate ).

[0055] Next, the empirical rule model construction means 101 obtains from the class ratio 316 the probability P(Y=1) that the return value of empirical rule 112 is TRUE and the probability P(Y=0) that the return value of empirical rule 112 is FALSE. Then, the probability P(λ) calculated in step S105 i =l i ∧λ j =l j Substitute P(Y=1) and P(Y=0) into the following system of equations (1) to calculate a numerical solution, and output an empirical rule model 115 in which the calculated numerical solution is stored in probability value 412 (step S106).

[0056]

number

[0057] Next, the empirical rule model construction means 101 performs each process for each row of LabelMatrix501. The class classification results obtained by applying the empirical rule 112 are calculated, and the probability that the correct label for each data ID 311 is TRUE (1) is calculated based on the obtained class classification results. Specifically, the empirical rule model construction means 101 obtains the data ID "x" from LabelMatrix501. i "of Each value l stored in a row i1 ,l i2 ,…,l in (n is the number of columns in LabelMatrix501 (that is, Then, obtain the number of experience rule 112) and, according to the following formula (2), the data ID "x iThe inference result for "[ ]" is calculated and output (step S107).

[0058]

number

[0059] Next, the experience rule model construction means 101 uses each data ID "x i The variance of the inference result of " and the increment in the amount of data in training data 111 (i.e., the number of rows in LabelMatrix501) The rate of increase of the variance is calculated (step S108). The rate of increase of the variance calculated here is the speed at which the variance increases in response to the increase in the amount of data associated with the operation of the business system. This is based on the observation that when the amount of training data 111 is sufficient, the probability values ​​calculated in step S107 are separated around 0 and 1, and when the amount of training data 111 is insufficient, they are concentrated around 0.5.

[0060] Next, the empirical rule model construction means 101 determines whether the rate of increase of the calculated variance is less than or equal to a predetermined constant value (step S109). If it is determined that the rate of increase of the variance is less than or equal to a constant value (step S109: YES), the empirical rule model construction means 101 determines that the effect of further increasing the amount of training data 111 on improving accuracy is small, and terminates the empirical rule model construction process. On the other hand, if the rate of increase of the variance is greater than a constant value, If the determination is made (step S109: NO), the empirical rule model construction means 101 returns to the process in step S104 and adds the pseudo-training data to LabelMatrix501.

[0061] <Empirical Rule Model Inference Processing (Alternative Model Inference Processing)> Figure 15 is a processing flow diagram illustrating the details of the empirical rule model inference process S2 performed by the classification class inference device 1. The empirical rule model inference process is executed, for example, when the data to be inferred 113 is input to the empirical rule model inference means 102 via the ensemble execution means 107 from the inference result correction request means 106. When the data to be inferred 113 is input to the empirical rule model inference means 102, it executes the empirical rule model inference process shown below and outputs the inference result 114.

[0062] First, the empirical rule model inference means 102 receives input of the data to be inferred 113 (step S201).

[0063] Next, the empirical rule model inference means 102 applies each empirical rule 112 to each of the inference target data 113 in the same procedure as in step S103 described above to generate a LabelMatrix 501 (step S202).

[0064] Next, the empirical rule model inference means 102 uses the empirical rule model 115 to calculate the probability that the correct label for each data ID 311 in each row of the data to be inferred 113 is TRUE (1), in the same procedure as in step S107 described above (step S203). The empirical rule model inference means 102 then stores the calculated probability in the corresponding field of the provisional inference result. After that, the empirical rule model inference means 102 outputs the provisional inference result to the ensemble execution means 107 and terminates the empirical rule model inference process.

[0065] <Prediction process for achieving accuracy of empirical rule models (prediction process for achieving accuracy of alternative models)> Figure 16 is a processing flow diagram illustrating the details of the empirical rule model achievement accuracy prediction process S3 performed by the classification class inference device 1. The empirical rule model achievement accuracy prediction process is executed, for example, when a predetermined input is made by the user to the first computer 10 or the second computer 20, or at a predetermined timing (for example, a predetermined time, a predetermined time interval). The achievement accuracy prediction means 108 takes the empirical rule 112 and a very small amount of training data 111 without correct labels as input and outputs data showing the relationship between the amount of data used when constructing the empirical rule model 115 and the accuracy of the empirical rule model 115.

[0066] First, the accuracy prediction means 108 accepts input of training data 111 without correct labels (step S301).

[0067] Next, the achievement accuracy prediction means 108 determines the latent variables (the correct labels for each training data 111) by majority vote (step S302). Specifically, the achievement accuracy prediction means 108 generates a LabelMatrix 501 for the input training data 111 using the same procedure as in step S103 described above. Then, for each data ID 311, the achievement accuracy prediction means 108 determines the latent variables (the correct labels for each training data 111) The system determines whether there are more empirical rules 112 that classify a data row as FALSE (0) or more empirical rules 112 that classify it as TRUE (1). The accuracy prediction means 108 then stores the more frequent classification result (FALSE (0) or TRUE (1)) in the correct label 313 of the training data 111.

[0068] In this embodiment, the correct labels for the training data 111 without correct labels are automatically determined by majority vote. However, the system is not limited to this method, and the user may manually input the correct labels for each training data 111.

[0069] Next, the achievement accuracy prediction means 108 calculates the probability parameters of the empirical rule model 115 (step S303). Specifically, the achievement accuracy prediction means 108 uses the training data 111 and LabelMatrix 501 with correct labels generated in step S302 to calculate the probability parameters of the empirical rule model 115. The probability values ​​412 for each probability type 411 to be described in the rule model 115 are calculated by statistical processing. In other words, the achievement accuracy prediction means 108 calculates a probability value that indicates the probability that the classification class determined by the empirical rule model 115 matches the correct label by calculating the probability values ​​412 for each probability type 411.

[0070] Furthermore, the achievement accuracy prediction means 108 accepts input of the class ratio 316 of the experience rule 112 (step S304).

[0071] Next, the achievement accuracy prediction means 108 generates pseudo-correct data 701 using random numbers so that the class ratio 316 obtained in step S304 (step S305).

[0072] (Pseudo-correct data) Figure 17 shows the data structure of the pseudo-correct data 701. The pseudo-correct data 701 stores the correct label 313 (pseudo-correct label) value for each data ID 311. The data ID 311 and correct label 313 stored in the pseudo-correct data 701 are not from the actual training data 111, but are randomly generated data.

[0073] Next, the achievement accuracy prediction means 108 generates a pseudo-LabelMatrix 702 from the pseudo-correct data 701 generated in step S305 using random numbers, such that the rate at which the classification class determined by the rule matches the pseudo-correct label is the probability value calculated in step S303. (Step S306). In other words, the achievement accuracy prediction means 108 is a pseudo-LabelMatrix7 By generating 02, pseudo-training data is generated.

[0074] (pseudo LabelMatrix) Figure 18 shows the data structure of a pseudo-LabelMatrix702. The pseudo-LabelMatrix702 has a similar data structure to LabelMatrix501. However, the values ​​stored in pseudo-LabelMatrix702 are obtained by applying the empirical rule 112 to the training data 111. These are not values ​​obtained through a specific method, but rather values ​​generated using random numbers.

[0075] Next, the achievement accuracy prediction means 108 uses the pseudo-LabelMatrix 702 generated in step S306 to construct the empirical rule model 115 without ground truth labels. The relational data 703 is calculated that shows the relationship between the amount of training data 111 and the inference accuracy of the empirical rule model 115 (step S307).

[0076] (Related data 703) Figure 19 shows an example of related data 703. The relational data 703 stores a predicted value (hereinafter referred to as "predicted accuracy") of the level of accuracy that can be achieved when the amount of training data 111 used to construct the empirical rule model 115 or AI model 116 changes. In other words, the relational data 703 stores the predicted accuracy of each model when each empirical rule model 115 or AI model 116 is constructed using each amount of training data 111 (0, 500, 100, 1500, 2000 in the illustrated example).

[0077] Specifically, the accuracy prediction means 108 determines the amount of data (number of rows) of the pseudo-LabelMatrix 702. For example, by changing this to 0, 500, 1000, 1500, 2000, we construct an empirical rule model 115, and the inference results of the constructed empirical rule model 115 and the pseudo-correct data 701 are used. The degree of agreement is calculated. The achievement accuracy prediction means 108 then stores the calculated degree of agreement as the predicted achievement accuracy in the corresponding field of the relational data 703 related to the empirical rule model 115. After that, the empirical rule model achievement accuracy prediction process is terminated.

[0078] <AI Model Achievement Accuracy Prediction Process> FIG. 20 is a processing flowchart for explaining the details of the AI model achievement accuracy prediction process S4 performed by the classification class inference device 1. The AI model achievement accuracy prediction process is executed, for example, when a predetermined input is received from the user by the first computer 10 or the second computer 20, or at a predetermined timing (for example, a predetermined time, a predetermined time interval). The achievement accuracy prediction means 108 takes as input a very small amount of learning data 111 with correct labels, and outputs relationship data 703 indicating the relationship between the data amount used when constructing the AI model 116 and the accuracy of the AI model 116.

[0079] First, the achievement accuracy prediction means 108 receives the input of the learning data 111 with correct labels (step S401).

[0080] Subsequently, the achievement accuracy prediction means 108 uses a known algorithm such as Tabular GAN to generate pseudo-learning data from the input learning data 111 (step S402). The data structure of the pseudo-learning data is the same as that of the learning data 111.

[0081] Subsequently, the achievement accuracy prediction means 108 calculates relationship data 703 indicating the relationship between the data amount of the learning data 111 with correct labels used when constructing the AI model 116 and the inference accuracy of the AI model 116 using the pseudo-learning data generated in step S402 (step S403). Specifically, the achievement accuracy prediction means 108 constructs the AI model 116 while changing the data amount of the pseudo-learning data, for example, to 0, 500, 1000, 1500, 2000, and calculates the degree of coincidence between the inference result of the constructed AI model 116 and the correct label 313 of the pseudo-learning data. Then, the achievement accuracy prediction means 108 stores the calculated degree of coincidence in the corresponding field of the relationship data 703 regarding the AI model 116 as the predicted achievement accuracy. After that, this AI model achievement accuracy prediction process ends.

[0082] <Empirical Rule Model Reconstruction Instruction Issuance Process (Alternative Model Reconstruction Instruction Issuance Process)> Figure 21 is a processing flow diagram illustrating the details of the empirical rule model reconstruction instruction issuance process S5 performed by the classification class inference device 1. The empirical rule model reconstruction instruction issuance process is executed when the inference target data 113 is input to or output to the ensemble execution means 107, when a predetermined input is made by the user to the first computer 10 or the second computer 20, or at a predetermined timing (for example, a predetermined time, a predetermined time interval). The achievement accuracy prediction means 108 monitors the amount of related data 703 and the amount of training data 111 that can be used to construct the empirical rule model 115, and issues an instruction 119 to rebuild the empirical rule model 115 to the user at an appropriate timing.

[0083] First, the achievement accuracy prediction means 108 monitors the amount of data in the inference target data 113 that the ensemble execution means 107 has processed (step S501). Specifically, after the empirical rule model 115 is constructed, the achievement accuracy prediction means 108 counts the cumulative amount of data in the inference target data 113 that has been processed by the ensemble execution means 107 each time the ensemble execution means 107 performs the ensemble processing. The amount of data in the inference target data 113 that has been processed by the ensemble execution means 107 is the number of times the inference target data 113 and inference result 114 have been input and output between the inference result correction request means 106 and the empirical rule model inference means 102. In other words, the amount of data in the inference target data 113 that has been processed by the ensemble is the amount of training data 111 without ground truth labels that can be used to construct the empirical rule model 115.

[0084] Next, the accuracy prediction means 108 refers to the relational data 703 of the empirical rule model 115 and calculates the amount of data at which the accuracy of the empirical rule model 115 saturates (hereinafter referred to as the "accuracy saturation data amount") (step S502). "Accuracy saturating" means that the accuracy becomes almost constant at a predetermined value (steady-state value) (reaches a steady state). Therefore, the accuracy saturation data amount is the amount of unlabeled training data 111 used when constructing the empirical rule model 115 at which the accuracy reaches a steady state.

[0085] As in the data example of the relationship data 703 shown in FIG. 19, when the amount of learning data 111 without correct labels used for constructing the empirical rule model 115 reaches a certain amount (500 in the illustrated example) or more, the prediction achievement accuracy thereof reaches a steady state. In this example, the achievement accuracy prediction means 108 sets the accuracy saturation data amount to "500".

[0086] Subsequently, the achievement accuracy prediction means 108 compares the amount of learning data 111 monitored in step S501 with the accuracy saturation data amount calculated in step S502, and determines whether the amount of learning data 111 is greater than the accuracy saturation data amount (step S503). When it is determined that the amount of learning data 111 is less than or equal to the accuracy saturation data amount (step S503: NO), the achievement accuracy prediction means 108 ends the empirical rule model reconstruction instruction issuance process.

[0087] On the other hand, when it is determined that the amount of learning data 111 is greater than the accuracy saturation data amount (step S503: YES), the achievement accuracy prediction means 108 designates the empirical rule model 115 as the model type 416 of the reconstruction instruction 119, and outputs the reconstruction instruction 119 (step S504). Then, the empirical rule model reconstruction instruction issuance process is ended.

[0088] <AI Model Reconstruction Instruction Issuance Process> FIG. 22 is a processing flowchart for explaining the details of the AI model reconstruction instruction issuance process S6 performed by the classification class inference device 1. The AI model reconstruction instruction issuance process is executed at the timing when learning data 111 with correct labels is registered by the user, when a predetermined input is received from the user by the first computer 10 or the second computer 20, or at a predetermined timing (for example, a predetermined time, a predetermined time interval). The achievement accuracy prediction means 108 monitors the relationship data 703 and the amount of learning data 111 available for constructing the AI model 116, and issues a reconstruction instruction 119 for the AI model 116 to the user at an appropriate timing.

[0089] First, the achievement accuracy prediction means 108 monitors the amount of data in the training data 111 with correct labels that is registered in the classification class inference device 1 (step S601). Specifically, when training data 111 with correct labels is registered in the classification class inference device 1 by a user, the achievement accuracy prediction means 108 adds the amount of data in the registered training data 111 with correct labels to the amount of data in the training data 111 with correct labels that is already stored. The registered training data 111 with correct labels is training data 111 with correct labels that can be used to construct the AI ​​model 116.

[0090] Next, the accuracy prediction means 108 refers to the related data 703 and calculates the amount of data at which the accuracy of the empirical rule model 115 and the accuracy of the AI ​​model 116 are reversed (hereinafter referred to as the "accuracy reversal data amount"), and the amount of data at which the accuracy of the AI ​​model 116 saturates (hereinafter referred to as the "accuracy saturation data amount") (step S602).

[0091] As shown in the example of relational data 703 in Figure 19, the AI ​​model 116 reaches a steady state in prediction accuracy when the amount of ground-labeled training data 111 used in its construction exceeds a certain amount (1500 in the illustrated example). When the amount of training data 111 reaches a certain value (1500 in the illustrated example), the prediction accuracy of the empirical rule model 115 and the prediction accuracy of the AI ​​model 116 reverse. That is, when the amount of training data 111 used in construction reaches a certain value, the prediction accuracy of the AI ​​model 116 becomes higher than that of the empirical rule model 115. In this example, the accuracy prediction means 108 sets the accuracy saturation data amount and the accuracy reversal data amount to "1500".

[0092] Next, the accuracy prediction means 108 determines whether the amount of training data 111 with correct labels that it has been monitoring in step S601 exceeds the accuracy inversion data amount (step S603). If it determines that the amount of training data 111 exceeds the accuracy inversion data amount (step S603: YES), the accuracy prediction means 108 proceeds to the process in step S605.

[0093] On the other hand, if it is determined that the amount of training data 111 does not exceed the accuracy inversion amount (step S603: NO), the achievement accuracy prediction means 108 determines whether the amount of training data 111 with correct labels that was monitored in step S601 exceeds the accuracy saturation amount (step S604). If it is determined that the amount of training data 111 does not exceed the accuracy saturation amount (step S604: NO), the achievement accuracy prediction means 108 terminates the AI ​​model reconstruction instruction issuance process.

[0094] On the other hand, if it is determined that the amount of training data 111 exceeds the accuracy saturation data amount (step S604: YES), the accuracy prediction means 108 specifies the AI ​​model 116 as the model type 416 of the reconstruction instruction 119 and outputs the reconstruction instruction 119 (step S605). After that, the AI ​​model reconstruction instruction issuance process ends.

[0095] <Automatic parameter adjustment process> Figure 23 is a processing flow diagram illustrating the details of the automatic parameter adjustment process S7 performed by the classification class inference device 1. The automatic parameter adjustment process is executed at the same time that the ensemble execution means 107 executes the ensemble process.

[0096] First, the achievement accuracy prediction means 108 predicts the predicted achievement accuracy of the empirical rule model 115 and the AI ​​model 116 (step S701). Specifically, the achievement accuracy prediction means 108 obtains from the relational data 703 the predicted achievement accuracy corresponding to the amount of data in the training data 111 used by the empirical rule model construction means 101 when it constructed the empirical rule model 115. The achievement accuracy prediction means 108 also obtains from the relational data 703 the predicted achievement accuracy corresponding to the amount of data in the training data 111 used by the AI ​​model construction means 103 when it constructed the AI ​​model 116. Then, the achievement accuracy prediction means 108 stores the obtained predicted achievement accuracy values ​​in the corresponding fields of the predicted AUC 417 of the achievement accuracy prediction information 118.

[0097] Next, the ensemble execution means 107 sets the weight parameters of the empirical rule model 115 and the AI ​​model 116 using the inverse ratio of the predicted achievement accuracy stored in the achievement accuracy prediction information 118, and performs the ensemble processing (step S702). After that, this automatic parameter adjustment process is terminated. This automatic parameter adjustment process makes it possible to automatically adjust the inference execution to give more weight to the model with higher prediction accuracy.

[0098] Note that there are other possible methods for executing the ensemble processing shown in step S702. One of these other methods involves the achievement accuracy prediction means 108 dividing the feature space into multiple regions according to the values ​​taken by the features 312 of the training data 111 and pseudo-training data when outputting the achievement accuracy prediction information 118, calculating relational data 703 for each region, and outputting the achievement accuracy prediction information 118. The ensemble execution means 107 is attempting to execute the ensemble processing. The system determines which region of the feature space each feature 312 of the data to be inferred belongs to, sets the weight parameters of the empirical rule model 115 and the AI ​​model 116 using the inverse ratio of the prediction accuracy stored in the prediction AUC 417 corresponding to that region, and then performs ensemble processing.

[0099] The second alternative method, in addition to the first alternative method, performs ensemble processing only on specific regions in the feature space specified by the user. The user specifies whether or not to perform ensemble processing using a data structure similar to that of the inference target data discrimination rule 117. If ensemble processing is necessary, it is performed in the same manner as the first alternative method. If ensemble processing is not necessary, it is not performed, and the inference result of the AI ​​model 116 is used as the modified inference result.

[0100] (Experience rules and learning data input GUI (Graphical User Interface)) Figure 24 shows an example of the Experience Rule and Learning Data Input GUI 1201. The Experience Rule and Learning Data Input GUI 1201 is a screen that accepts input of Experience Rules 112 and Learning Data 111 from the user.

[0101] The Experience Rule / Learning Data Input GUI 1201 includes a Rule ID input section 1211, a Rule Definition input section 1212, a Class Ratio input section 1213, and a Rule Registration Button 1216. The Rule ID input section 1211 is an area that accepts input of Rule ID 314 from the user. The Rule Definition input section 1212 is an area that accepts input of Rule Definition 315 from the user. The Class Ratio input section 1213 is an area that accepts input of Class Ratio 316 from the user. The Rule Registration Button 1216 is an operation button that accepts instructions from the user to register the data entered in the Rule ID input section 1211, the Rule Definition input section 1212, and the Class Ratio input section 1213. After the user inputs data into the rule ID input unit 1211, the rule definition input unit 1212, and the class ratio input unit 1213, they can register an experience rule 112 having the entered rule ID 314, rule definition 315, and class ratio 316 by pressing the rule registration button 1216. When the experience rule model construction means 101 receives input indicating that the rule registration button 1216 has been pressed, it registers the experience rule 112 having the rule ID 314 entered in the rule ID input unit 1211, the rule definition 315 entered in the rule definition input unit 1212, and the class ratio 316 entered in the class ratio input unit 1213 by writing it to the first auxiliary storage device 13.

[0102] In this way, by accepting registration of experience rules 112 from users through the experience rule / learning data input GUI 1201, users can easily register any experience rules 112 based on their own experience or knowledge, or that of the person in charge of the work.

[0103] Furthermore, the experience rule / learning data input GUI 1201 further includes a learning data input section 1214 and a learning data registration button 1217. The learning data input section 1214 is an area that accepts input from the user for the file name of a file containing the learning data 111. The learning data registration button 1217 is an operation button that accepts from the user an instruction to register the learning data 111 contained in the file with the file name entered in the learning data input section 1214. After the user enters the file name of a file containing the learning data 111 into the learning data input section 1214, they can register the learning data 111 contained in the file with the entered file name by pressing the learning data registration button 1217. When the experience rule model construction means 101 receives input to press the learning data registration button 1217, it reads the learning data 111 from the file with the file name entered in the learning data input section 1214 and registers the read learning data 111 by writing it to the first auxiliary storage device 13.

[0104] In this way, the learning data 111 is registered using the experience rule / learning data input GUI 1201. By accepting input from the user, users can easily register training data 111 simply by specifying a file name.

[0105] Furthermore, the experience rule / learning data input GUI 1201 further includes a model construction instruction unit 1215 and a model construction button 1218. The model construction instruction unit 1215 is an area that receives input from the user regarding the type of model to be reconstructed (experience rule model 115 or AI model 116). The model construction button 1218 is an operation button that receives instructions from the user to reconstruct the model of the model type entered in the model construction instruction unit 1215. After entering the model type in the model construction instruction unit 1215, the user can instruct the reconstruction of the model of the entered model type by pressing the model construction button 1218. If the Model Construction Instruction Unit 1215 receives input to press the Model Construction Button 1218 while the Experience Rule Model 115 is specified, the Experience Rule Model Construction Means 101 constructs the Experience Rule Model 115 using the Experience Rules 112 registered in the Rule ID Input Unit 1211, Rule Definition Input Unit 1212, and Class Ratio Input Unit 1213, and the Training Data 111 registered in the Training Data Input Unit 1214. On the other hand, if the Model Construction Instruction Unit 1215 receives input to press the Model Construction Button 1218 while the AI ​​Model 116 is specified, the AI ​​Model Construction Means 103 constructs the AI ​​Model 116 using the Training Data 111 registered in the Training Data Input Unit 1214.

[0106] In this way, by receiving instructions from the user to build the experience rule model 115 or AI model 116 through the experience rule / learning data input GUI 1201, the user can build the experience rule model 115 or AI model 116 with simple operations.

[0107] (Model Reconstruction Instruction Output GUI) Figure 25 shows an example of the model reconstruction instruction output GUI 1301. The model reconstruction instruction output GUI 1301 is the screen displayed when the achievement accuracy prediction means 108 issues a reconstruction instruction 119.

[0108] The Model Reconstruction Instruction Output GUI 1301 is a screen that displays relational data 703, the amount of available training data, and the reconstruction instruction 119. The Model Reconstruction Instruction Output GUI 1301 comprises a data volume-accuracy relationship output unit 1311, an available data volume output unit 1312, and a reconstruction instruction output unit 1313. The data volume-accuracy relationship output unit 1311 displays the relational data 703 output by the accuracy prediction means 108. The available data volume output unit 1312 displays the amount of data of the monitored, available training data 111 without ground truth labels (inference target data 113 processed by the ensemble execution means 107) and the amount of data of the registered training data 111 with ground truth labels. The reconstruction instruction output unit 1313 displays the reconstruction instruction 119 output by the accuracy prediction means 108.

[0109] As the amount of training data increases with the operation of the business system, it becomes necessary to rebuild the AI ​​model 116. Generally, the AI ​​model 116, which uses more training data than the pseudo-learning model (empirical rule model 115), will have higher accuracy. Therefore, rebuilding the AI ​​model 116 when the amount of available training data increases can be expected to improve accuracy. However, it is difficult for users to determine the appropriate timing for rebuilding the AI ​​model 116. As a result, users may need to try rebuilding the AI ​​model 116 multiple times. To address this problem, the classification class inference device 1 according to this embodiment automatically determines the appropriate timing for rebuilding the AI ​​model 116 and the empirical rule model 115 and recommends rebuilding the AI ​​model 116 or the empirical rule model 115 to the user. This allows the user to rebuild the AI ​​model 116 or the empirical rule model 115 at the appropriate time. As a result, the user does not need to repeatedly try rebuilding the AI ​​model 116 or the empirical rule model 115 through trial and error.

[0110] Figure 26 is a graph showing an example of the relationship between data volume and accuracy. The horizontal axis of the graph shown in this figure represents the amount of data (number of data points) used when constructing the empirical rule model 115 or the AI ​​model 116. The vertical axis of the graph shown in this figure represents the prediction accuracy (AUC value) of the empirical rule model 115 or the AI ​​model 116. As shown in the figure, when the amount of data used in constructing the empirical rule model 115 and the AI ​​model 116 exceeds a certain amount, their prediction accuracy reaches a steady state. Also, when the amount of data used in constructing them exceeds a certain amount, the prediction accuracy of the AI ​​model 116 becomes higher than that of the empirical rule model 115.

[0111] Therefore, at timing T1, when the amount of training data 111 without ground truth labels that can be used to construct the empirical rule model 115 reaches the accuracy saturation data amount where the prediction accuracy of the empirical rule model 115 reaches a steady state, the classification class inference device 1 displays a command to rebuild the empirical rule model 115 119 on the model rebuild command output GUI 1301. In other words, at timing T1, when the prediction accuracy of the empirical rule model 115 saturates, the classification class inference device 1 automatically recommends rebuilding the empirical rule model 115 to the user. Similarly, at timing T2, when the amount of training data 111 with ground truth labels that can be used to construct the AI ​​model 116 reaches the accuracy saturation data amount where the prediction accuracy of the AI ​​model 116 reaches a steady state, the classification class inference device 1 displays a command to rebuild the AI ​​model 116 119 on the model rebuild command output GUI 1301. In other words, at timing T2, when the prediction accuracy of the AI ​​model 116 saturates, the classification class inference device 1 automatically recommends rebuilding the AI ​​model 116 to the user. Furthermore, at timing T3, when the amount of training data 111 with correct labels available for constructing the AI ​​model 116 becomes higher than the prediction accuracy of the empirical rule model 115, the classification class inference device 1 displays an instruction to rebuild the AI ​​model 116 119 on the model rebuild instruction output GUI 1301. In other words, at timing T3, when the prediction accuracy of the AI ​​model 116 saturates, the classification class inference device 1 automatically recommends rebuilding the AI ​​model 116 to the user.

[0112] In this way, by displaying the rebuilding instructions for the experience rule model 115 or AI model 116 on the model rebuilding instruction output GUI 1301, users can easily determine the appropriate timing for rebuilding the experience rule model 115 or AI model 116 without trial and error. Therefore, users will no longer need to attempt to rebuild the experience rule model 115 or AI model 116 multiple times. In other words, the number of rebuilds (man-hours) for the experience rule model 115 or AI model 116 can be reduced.

[0113] As described above, the classification class inference device 1 of this embodiment is a classification class inference device 1 that constructs an alternative model (experience rule model 115) that is different from the AI ​​model 116 constructed based on the training data 111, and stores rules (experience rules 112) that determine the classification class corresponding to the conditions related to the business from the conditions related to the business, generates pseudo-training data (LabelMatrix 501) corresponding to the conditions related to the business, and performs an alternative model construction process that constructs an alternative model based on the pseudo-training data (LabelMatrix 501) and the rules (experience rules 112), and an alternative model inference process that outputs a classification class (e.g., TRUE, FALSE) corresponding to the inference target data 113 by inputting the specified inference target data 113 into the alternative model.

[0114] According to the classification class inference device 1 of this embodiment, since experience rules 112 that determine the classification class corresponding to business conditions are stored in advance, an alternative model (experience rule model 115) that can replace the AI ​​model 116 can be constructed even if the amount of training data is insufficient. Then, the classification class of the data to be inferred can be inferred using the constructed alternative model. In a system, even if there is insufficient training data to build the AI ​​model 116, operations can continue using an alternative model (experience rule model 115). For example, in a financial institution's business system, even if the format of customer activity data changes to a new format that was not initially anticipated, the system can quickly adapt to the new format of the operational data.

[0115] Furthermore, the classification class inference device 1 of this embodiment stores the probability (class ratio 316) that each classification class should be determined, generates pseudo-correct labels for classification classes using random numbers so as to achieve that probability, calculates a probability value 412 that indicates the probability that the classification class determined by the rule (empirical rule 112) matches the correct label of that classification class, generates pseudo-training data (pseudo-LabelMatrix 702) from the pseudo-correct labels so that the rate at which the classification class determined by the rule (empirical rule 112) matches the pseudo-correct label becomes a probability value, and generates pseudo-training data Using (pseudo-LabelMatrix702), we construct an alternative model (empirical rule model 115). An alternative model achievement accuracy prediction process is executed to generate relational data 703 that shows the relationship between the amount of data used and the accuracy of the alternative model.

[0116] In this way, by using newly generated pseudo-training data (pseudo-LabelMatrix702) based on the probability that a classification class should be determined (class ratio 316), an alternative model (evolution) can be created. It is possible to accurately predict the relationship between the amount of data used when constructing the empirical rule model 115 and the accuracy of the alternative model (empirical rule model 115).

[0117] Furthermore, the classification class inference device 1 of this embodiment calculates the amount of data used to construct the alternative model (empirical rule model 115) at which the accuracy of the alternative model (empirical rule model 115) saturates, based on the relational data 703, and executes an alternative model reconstruction instruction issuance process to issue an instruction to rebuild the alternative model (empirical rule model 115) based on the amount of data available for constructing the alternative model (empirical rule model 115) and the amount of data at which the accuracy of the alternative model (empirical rule model 115) saturates.

[0118] In this way, by issuing a rebuilding instruction for the alternative model (empirical rule model 115) when the amount of data available for building the alternative model (empirical rule model 115) reaches the amount of data that saturates the accuracy of the alternative model (empirical rule model 115), it is possible to issue a rebuilding instruction for the alternative model (empirical rule model 115) at a time when an improvement in accuracy can be expected. As a result, the timing of rebuilding the alternative model (empirical rule model 115) can be automatically recommended to the user, so that the user can rebuild the alternative model (empirical rule model 115) at the appropriate time without having to attempt to rebuild it multiple times.

[0119] Furthermore, the classification class inference device 1 of this embodiment generates pseudo-training data from training data using machine learning, and performs an AI model achievement accuracy prediction process that uses this pseudo-training data to calculate relational data 703 that shows the relationship between the amount of training data used when constructing the AI ​​model 116 and the accuracy of the AI ​​model 116. This makes it possible to accurately predict the relationship between the amount of training data used when constructing the AI ​​model 116 and the accuracy of the AI ​​model 116.

[0120] Furthermore, the classification class inference device 1 of this embodiment calculates, based on the relational data 703 of the AI ​​model 116 and the relational data 703 of the alternative model (empirical rule model 115), the amount of training data used to construct the AI ​​model 116 at which the accuracy of the AI ​​model 116 and the accuracy of the alternative model (empirical rule model 115) are reversed, and the amount of training data used to construct the AI ​​model at which the accuracy of the AI ​​model 116 saturates. Based on the amount of available training data, the AI ​​model reconstruction instruction issuance process is executed to issue a reconstruction instruction for AI model 116.

[0121] This allows a command to rebuild the AI ​​model 116 when the amount of training data available for building the AI ​​model 116 reaches a point where the accuracy of the AI ​​model 116 reverses with that of the alternative model (empirical rule model 115), or when the accuracy of the AI ​​model 116 saturates. Therefore, a command to rebuild the AI ​​model 116 can be issued when the accuracy of the AI ​​model 116 and the alternative model (empirical rule model 115) reverses, or when an improvement in the accuracy of the AI ​​model 116 can be expected. This allows the system to automatically recommend the timing of the AI ​​model 116's reconstruction to the user, enabling the user to rebuild the AI ​​model 116 at the appropriate time without having to attempt to rebuild it multiple times.

[0122] Furthermore, the classification class inference device 1 of this embodiment performs the following: an AI model inference process that outputs a classification class corresponding to the specified inference target data 113 by inputting the specified inference target data 113 to the AI ​​model 116; an ensemble process that ensembles the output result of the alternative model inference process with the output result of the AI ​​model inference process; and an adjustment process that adjusts the respective weights of the AI ​​model 116 and the alternative model (empirical rule model 115) when executing the ensemble process, based on the relational data 703 of the AI ​​model 116 and the relational data 703 of the alternative model (empirical rule model 115).

[0123] In this way, by automatically adjusting the weight parameters of the ensemble processing, the system can perform appropriate ensemble processing according to the accuracy of each of the AI ​​model 116 and the alternative model (empirical rule model 115) without the user having to manually adjust the weight parameters. For example, when the amount of training data 111 increases due to the operation of a business system, and the AI ​​model 116 or the alternative model (empirical rule model 115) is reconstructed and the accuracy ratio changes, the weight parameters of the ensemble processing are automatically adjusted. This makes it possible to automatically improve the inference accuracy of the classification class. Therefore, the amount of effort required to adjust the parameters can be reduced.

[0124] Furthermore, in the alternative model construction process of the classification class inference device 1 of this embodiment, the amount of increase in the pseudo-training data (LabelMatrix501) is used to determine the alternative model (empirical route) of the pseudo-training data. An alternative model is constructed by adding pseudo-training data until the rate of increase of the variance of the inference result by the original model (115) falls below a predetermined value.

[0125] In this way, by building an alternative model while adding pseudo-training data (LabelMatrix501) until the rate of increase of the variance of the inference result falls below a predetermined constant value, This allows us to construct alternative models (empirical rule models 115) with higher estimation accuracy.

[0126] Furthermore, in the alternative model construction process, the classification class inference device 1 of this embodiment generates pseudo-training data (LabelMatrix501) using random numbers. By generating training data (LabelMatrix501), complex calculations can be easily performed. It is possible to generate pseudo-training data. Therefore, the processing load required to generate pseudo-training data can be reduced.

[0127] The present invention is not limited to the embodiments described above, and can be implemented using any components without departing from its spirit. The embodiments and modifications described above are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments that can be considered within the scope of the technical idea of ​​the present invention are also the present invention. It is included within the range.

[0128] For example, some of the hardware components of each device in this embodiment may be provided in other devices.

[0129] Furthermore, each program of the classification class inference device 1 may be provided in other devices, a program may consist of multiple programs, or multiple programs may be integrated into a single program. [Explanation of Symbols]

[0130] 1 Classification class inference device, 101 empirical rule model construction means, 102 empirical rule model inference means, 103 AI model construction means, 104 inference target data discrimination means, 105 AI model inference means, 106 inference result correction request means, 107 ensemble execution means, 108 achievement accuracy prediction means

Claims

1. A classification class inference device that constructs an alternative model different from the AI ​​model built based on training data, A memory device that stores rules for determining a classification class corresponding to a given condition based on the conditions related to the business, and the probability that each classification class should be determined. An AI model inference process that outputs the classification class corresponding to the specified data to be inferred by inputting the specified data to be inferred into the AI ​​model, An alternative model construction process that generates pseudo-training data corresponding to the conditions related to the aforementioned business, and constructs the alternative model based on the pseudo-training data and the rules, The alternative model includes an alternative model inference process that outputs a classification class corresponding to the specified data to be inferred by inputting the specified data to be inferred into the alternative model, An ensemble process that combines the output results of the alternative model inference process and the output results of the AI ​​model inference process, A processing unit that performs the following: Equipped with, The aforementioned processing apparatus is To achieve the aforementioned probabilities, pseudo-correct labels for classification classes are generated using random numbers. A probability value is calculated that indicates the probability that the classification class of the condition determined by the aforementioned rule matches the correct label of that classification class. The rate at which the classification class determined by the rule matches the pseudo-correct label is such that the probability value is obtained from the pseudo-correct label, and pseudo-training data for the alternative model is generated from the pseudo-correct label. Using the aforementioned pseudo-training data, an alternative model achievement accuracy prediction process is performed to generate relational data showing the relationship between the amount of data used when constructing the alternative model and the accuracy of the alternative model. Classification class inference system.

2. The aforementioned processing apparatus is Based on the aforementioned relationship data, the amount of data used to construct the alternative model at which the accuracy of the alternative model saturates is calculated, and based on the amount of data available for constructing the alternative model and the amount of data at which the accuracy of the alternative model saturates, an alternative model reconstruction instruction is issued. A classification class inference device according to claim 1, which performs the following:

3. The aforementioned processing apparatus is AI model achievement accuracy prediction process: Generates pseudo-training data for the AI ​​model from the aforementioned training data using machine learning, and uses the pseudo-training data to calculate relational data showing the relationship between the amount of data in the training data used when constructing the AI ​​model and the accuracy of the AI ​​model. A classification class inference device according to claim 1, which performs the following:

4. The aforementioned processing apparatus is Based on the relational data of the AI ​​model and the relational data of the alternative model, the AI ​​model reconstruction instruction issuance process calculates the amount of training data used to construct the AI ​​model at which the accuracy of the AI ​​model and the accuracy of the alternative model are reversed, and the amount of training data used to construct the AI ​​model at which the accuracy of the AI ​​model saturates. Based on the amount of data at which the accuracy is reversed, the amount of data at which the accuracy of the AI ​​model saturates, and the amount of training data available for constructing the AI ​​model, the AI ​​model reconstruction instruction issuance process issues an instruction to rebuild the AI ​​model. A classification class inference device according to claim 3, which performs the following:

5. The aforementioned processing apparatus is An adjustment process to adjust the weights of the AI ​​model and the alternative model when performing the ensemble processing, based on the relational data of the AI ​​model and the relational data of the alternative model. A classification class inference device according to claim 3, which performs the following:

6. The processing device constructs the alternative model by adding pseudo-training data in the alternative model construction process until the rate of increase of the variance of the inference result of the pseudo-training data by the alternative model relative to the increase in the amount of pseudo-training data for the alternative model falls below a predetermined constant value. The classification class inference device according to claim 1.

7. The processing device generates pseudo-training data for the alternative model using random numbers in the alternative model construction process. The classification class inference device according to claim 1.

8. An information processing device that constructs an alternative model different from the AI ​​model built based on training data, An alternative model construction process that generates pseudo-training data corresponding to business conditions, and constructs the alternative model based on rules that determine the classification class corresponding to the conditions from the pseudo-training data and the business conditions, An AI model inference process that outputs the classification class corresponding to the specified data to be inferred by inputting the specified data to be inferred into the AI ​​model, The alternative model includes an alternative model inference process that outputs the classification class corresponding to the specified data to be inferred by inputting the specified data to be inferred into the alternative model, An ensemble process is performed which combines the output result of the alternative model inference process and the output result of the AI ​​model inference process. A pseudo-correct label for each classification class is generated using random numbers so that each classification class has a probability of being determined. A probability value is calculated that indicates the probability that the classification class of the condition determined by the aforementioned rule matches the correct label of that classification class. The rate at which the classification class determined by the rule matches the pseudo-correct label is such that the probability value is obtained from the pseudo-correct label, and pseudo-training data for the alternative model is generated from the pseudo-correct label. Using the aforementioned pseudo-training data, an alternative model achievement accuracy prediction process is performed to generate relational data showing the relationship between the amount of data used when constructing the alternative model and the accuracy of the alternative model. A classification class inference method.

9. An information processing device that constructs an alternative model different from the AI ​​model built based on training data, An alternative model construction process that generates pseudo-training data corresponding to business conditions, and constructs the alternative model based on rules that determine the classification class corresponding to the conditions from the pseudo-training data and the business conditions, An AI model inference process that outputs the classification class corresponding to the specified data to be inferred by inputting the specified data to be inferred into the AI ​​model, The alternative model includes an alternative model inference process that outputs the classification class corresponding to the specified data to be inferred by inputting the specified data to be inferred into the alternative model, The output result of the alternative model inference process and the output result of the AI ​​model inference process are combined in an ensemble process, Random numbers are used to generate pseudo-correct labels for each classification class, such that each classification class has a probability of being determined. The system calculates a probability value that indicates the probability that the classification class of the condition determined by the rule matches the correct label of that classification class. The rate at which the classification class determined by the rule matches the pseudo-correct label is such that the probability value is obtained by generating pseudo-training data for the alternative model from the pseudo-correct label. Using the aforementioned pseudo-training data, an alternative model achievement accuracy prediction process is executed to generate relational data showing the relationship between the amount of data used when constructing the alternative model and the accuracy of the alternative model. A classification class inference program.

Images