Estimation system, estimation method, and estimation program

Abstract

To provide an estimation system, an estimation method, and an estimation program that appropriately estimate the effect of introducing a trained model for detection. [Solution] An information processing system 100 for estimating the introduction effect, which is the effect on detecting fraudulent transactions, of introducing a first model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the first model is generated based on financial transaction related information, which is information about financial transactions at financial institutions other than the target financial institution, and the information processing system 100 comprises a first acquisition means for acquiring first region-related information, which is information about financial transactions based on a region related to the target financial institution, a second acquisition means for acquiring second region-related information, which is information about financial transactions based on a region related to the other financial institution, and an estimation means for estimating the introduction effect based on the first region-related information and the second region-related information.

Description

[Technical Field]

[0001] The present invention relates to an estimation system, an estimation method, and an estimation program. [Background technology]

[0002] Conventionally, techniques for detecting fraudulent transactions have been known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-015000 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there has been a demand for technology to estimate the effect of introducing a trained model for detection.

[0005] The present invention has been made in consideration of the above, and aims to provide an estimation system, an estimation method, and an estimation program that appropriately estimate the effect of introducing a trained model for detection. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the estimation system described in claim 1 is an estimation system for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at a financial institution other than the target financial institution, and the estimation system comprises: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution; and an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means. The estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input.

[0007] The estimation system according to claim 2 is the estimation system according to claim 1, The region-related trained model includes a first region-related trained model that outputs a first indicator corresponding to a precision rate based on the fraudulent transaction as the estimation indicator, and a second region-related trained model that outputs a second indicator corresponding to a recall rate based on a financial account associated with the fraudulent transaction as the estimation indicator.

[0008] The estimation system according to claim 3 comprises: An estimation system for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at financial institutions other than the target financial institution, and the estimation system includes a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution, a second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution, and a first acquisition means for acquiring the trained detection model. The system includes a third acquisition means that acquires feature-related information that indicates features of a model, and an estimation means that estimates the introduction effect based on the first region-related information acquired by the first acquisition means, the second region-related information acquired by the second acquisition means, and the feature-related information acquired by the third acquisition means, wherein the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information, the second region-related information, and the feature-related information are input.

[0009] The estimation system according to claim 4 comprises: An estimation system for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at a financial institution other than the target financial institution, and the estimation system includes: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution; an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means; and a determination means for determining whether or not the trained detection model should be a target for estimating the introduction effect, based on the first region-related information acquired by the first acquisition means. the estimation means estimates the introduction effect for the detection trained model determined by the determination means to be the estimation target, and the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input, and the region-related trained model is also configured to output a judgment index, which is used to determine whether the detection trained model is to be the estimation target for the introduction effect, when only one of the first region-related information or the second region-related information is input, and the determination means determines whether the detection trained model is to be the estimation target based on the region-related trained model.

[0010] The estimation system according to claim 5 comprises: In the estimation system described in any one of claims 1 to 4, the first region-related information includes information indicating the ratio of the number of account holders of the target financial institution who have addresses in an area related to the target financial institution to the total number of account holders of the target financial institution, information indicating the ratio of the number of specialized fraud cases that occurred in an area related to the target financial institution in a first period to the total number of specialized fraud cases that occurred in the first period, or information indicating the ratio of the number of specialized fraud cases by type that occurred in an area related to the target financial institution in a second period to the total number of specialized fraud cases that occurred in the second period, and the second region-related information includes information indicating the ratio of the number of account holders of the other financial institution who have addresses in an area related to the other financial institution to the total number of account holders of the other financial institution, information indicating the ratio of the number of specialized fraud cases that occurred in an area related to the other financial institution in a third period to the total number of specialized fraud cases that occurred in the third period, or information indicating the ratio of the number of specialized fraud cases by type that occurred in an area related to the other financial institution in the fourth period to the total number of specialized fraud cases that occurred in the fourth period.

[0011] The estimation according to claim 6 The method is an estimation method for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a detection trained model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at a financial institution other than the target financial institution, and the estimation method includes: a first acquisition step in which a first acquisition means acquires first region-related information, which is information about financial transactions based on a region related to the target financial institution; a second acquisition step in which a second acquisition means acquires second region-related information, which is information about financial transactions based on a region related to the other financial institution; and an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step and the second region-related information acquired in the second acquisition step, and the estimation means estimates the introduction effect based on a region-related trained model, which is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model when the first region-related information and the second region-related information are input.

[0012] The estimation method according to claim 7 comprises: An estimation method for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at financial institutions other than the target financial institution, and the estimation method includes a first acquisition step in which a first acquisition means acquires first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquisition step in which a second acquisition means acquires second region-related information, which is information on financial transactions based on a region related to the other financial institution; and a third acquisition step in which a third acquisition means acquires the first region-related information, which is information on financial transactions based on a region related to the other financial institution, The method includes a third acquisition step of acquiring feature-related information indicating features of a trained model, and an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step, the second region-related information acquired in the second acquisition step, and the feature-related information acquired in the third acquisition step, wherein the estimation means estimates the introduction effect based on the region-related trained model, which is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information, the second region-related information, and the feature-related information are input.

[0013] The estimation according to claim 8 The method is an estimation method for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at a financial institution other than the target financial institution, and the estimation method includes a first acquisition step in which a first acquisition means acquires first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquisition step in which a second acquisition means acquires second region-related information, which is information on financial transactions based on a region related to the other financial institution; an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step and the second region-related information acquired in the second acquisition step; and a determination means determines the introduction effect of the trained detection model based on the first region-related information acquired by the first acquisition means. and a determination step of determining whether or not the detection trained model is to be a target for estimation of the introduction effect, wherein the estimation means estimates the introduction effect for the detection trained model determined by the determination means to be the target for estimation, and the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input, and the region-related trained model is also configured to output a determination index, which is used to determine whether or not the detection trained model is to be a target for estimation of the introduction effect, when only one of the first region-related information or the second region-related information is input, and the determination means determines whether or not the detection trained model is to be the target for estimation, based on the region-related trained model. The estimation program described in claim 9 is an estimation program for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a detection trained model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the detection trained model is generated based on financial transaction related information, which is information about financial transactions at a financial institution other than the target financial institution, and the estimation program causes a computer to function as: a first acquisition means for acquiring first region-related information, which is information about financial transactions based on a region related to the target financial institution; a second acquisition means for acquiring second region-related information, which is information about financial transactions based on a region related to the other financial institution; and an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means, and the estimation means estimates the introduction effect based on a region-related trained model, which is configured to output an estimation index, which is an index used to estimate the introduction effect of the detection trained model when the first region-related information and the second region-related information are input. The estimation program according to claim 10 is an estimation program for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at financial institutions other than the target financial institution, and the estimation program includes a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution, and a second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution. a third acquisition means for acquiring feature-related information indicating features of the trained model for detection; and an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means, the second region-related information acquired by the second acquisition means, and the feature-related information acquired by the third acquisition means, wherein the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the trained model for detection, when the first region-related information, the second region-related information, and the feature-related information are input. The estimation program described in claim 11 is an estimation program for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the trained detection model is generated based on financial transaction related information, which is information on financial transactions at a financial institution other than the target financial institution, and the estimation program includes a computer including: first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution; estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means; and a computer including: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; and a determination means for determining whether the detection trained model is to be a target for estimation of the effect of introduction, wherein the estimation means estimates the effect of introduction for the detection trained model determined by the determination means to be the target for estimation, the estimation means estimates the effect of introduction based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the effect of introduction for the detection trained model, when the first region-related information and the second region-related information are input, the region-related trained model is also configured to output a determination index, which is used to determine whether the detection trained model is to be a target for estimation of the effect of introduction, when only one of the first region-related information or the second region-related information is input, and the determination means determines whether the detection trained model is to be the target for estimation, based on the region-related trained model. [Effects of the Invention]

[0014] Claim 1, 3、4 The estimation system according to claim 6~8 The estimation method according to claim 9~11 According to the estimation program described in the above, by estimating the introduction effect based on the first region-related information and the second region-related information, it is possible to estimate the introduction effect taking into account the characteristics of the region, for example, and therefore it is possible to appropriately estimate the introduction effect of the trained model for detection.

[0015] Claim 1 The estimation system according to , the estimation method according to claim 6, and the estimation program according to claim 9. According to the method, by estimating the introduction effect based on a regionally related trained model, it becomes possible to easily estimate the introduction effect of, for example, a trained model for detection.

[0016] Claim 2 According to the estimation system described in the above, by including a first region-related trained model that outputs a first indicator corresponding to a precision rate based on fraudulent transactions as an estimation indicator, and a second region-related trained model that outputs a second indicator corresponding to a recall rate based on a financial account associated with the fraudulent transactions as an estimation indicator, it becomes possible to more appropriately estimate the effect of introducing a detection trained model by using, for example, two types of region-related trained models.

[0017] Claim 3 The estimation system according to , the estimation method according to claim 7, and the estimation program according to claim 10. According to the method, by estimating the introduction effect based on the first region-related information, the second region-related information, and the feature-related information, it is possible to take into account, for example, the features of the trained model for detection, thereby making it possible to more appropriately estimate the introduction effect of the trained model for detection.

[0018] The estimation system according to claim 4 , the estimation method according to claim 8, and the estimation program according to claim 11.According to the method, by determining whether or not to use a trained model for detection as a target for estimating the introduction effect, it is possible to narrow down the targets for estimation and efficiently estimate the introduction effect, for example, by determining whether or not to use a trained model for detection as a target for estimating the introduction effect, thereby narrowing down the targets for estimation and enabling efficient estimation of the introduction effect.

[0019] The estimation system according to claim 4, the estimation method according to claim 8, and the estimation program according to claim 11. According to the method, the regional-related trained model is configured to output an estimation indicator or an assessment indicator, so that it is possible to use the regional-related trained model, for example, to both determine whether to include it in estimation and estimate the effects of its introduction. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a block diagram of an information processing system according to a first embodiment. [Figure 2] FIG. 1 is an explanatory diagram of a part of an information processing system. [Figure 3] FIG. 10 is a diagram illustrating fraudulent transaction-related information. [Figure 4] FIG. 10 is a diagram illustrating genuine transaction related information. [Figure 5] FIG. 10 is an explanatory diagram of a learned combination model for fraud detection. [Figure 6] FIG. 1 is an explanatory diagram of a first model. [Figure 7] FIG. 10 is an explanatory diagram of a second model. [Figure 8] FIG. 10 is a diagram illustrating an example of region-related explanatory variable information. [Figure 9] FIG. 10 is a diagram illustrating bank-derived related explanatory variable information. [Figure 10] FIG. 10 is a diagram illustrating explanatory variable information related to bank rankings. [Figure 11] FIG. 10 is a diagram illustrating an example of model-derived related explanatory variable information. [Figure 12] FIG. 10 is a diagram illustrating related explanatory variable information derived from other bank models. [Figure 13] FIG. 10 is a diagram illustrating exemplary response variable related information. [Figure 14] FIG. 10 is an explanatory diagram of a learned combination model for fraud detection. [Figure 15] 10 is a flowchart of a first storage process. [Figure 16] FIG. 10 is a diagram illustrating the detection result of a fraudulent transaction. [Figure 17] FIG. 1 is an explanatory diagram of a trading unit AUC. [Figure 18] 10 is a flowchart of a second storage process. [Figure 19] 10 is a flowchart of a region model generation process. [Figure 20] FIG. 1 is an explanatory diagram of a region model. [Figure 21] FIG. 10 is an explanatory diagram of explanatory variables. [Figure 22] FIG. [Figure 23] 10 is a flowchart of a region model relevance determination process. [Figure 24] FIG. 1 is an explanatory diagram of a part of an information processing system. [Figure 25] FIG. 10 is a diagram illustrating an example of model-derived related explanatory variable information. [Figure 26] FIG. 10 is a diagram illustrating related explanatory variable information derived from other bank models. [Figure 27] FIG. [Figure 28] FIG. [Figure 29] 10 is a flowchart of another bank model selection process. [Figure 30] FIG. [Figure 31] FIG. [Figure 32] FIG. 10 is an explanatory diagram of a learned combination model for fraud detection. [Figure 33] FIG. 10 is a block diagram of an information processing system according to a second embodiment. [Figure 34] FIG. 10 is a diagram illustrating fraudulent transaction-related information. [Figure 35] FIG. 10 is a diagram illustrating genuine transaction related information. [Figure 36] FIG. 1 is an explanatory diagram of a trained model for detection. [Figure 37]10 is a flowchart of an AML-specialized F-value specification process. [Figure 38] FIG. 1 is an explanatory diagram of an AML-specialized F value. [Figure 39] FIG. 10 is a diagram illustrating the detection result of a fraudulent transaction. [Figure 40] FIG. 10 is a diagram illustrating an example of calculation of each index. [Figure 41] 10 is a flowchart of an AML-specialized AUC determination process. [Figure 42] FIG. 1 is an explanatory diagram of AML-specialized AUC. [Figure 43] 10 is a display example of a first display screen. [Figure 44] 10 is a display example of a second display screen. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the embodiments. Here, embodiments 1 and 2 will be described.

[0022] (Embodiment 1) First, a description will be given of embodiment 1. Embodiment 1 relates to an estimation system, an estimation method, and an estimation program.

[0023] (Basic concept) First, the basic concept of this embodiment will be described. The estimation system according to the present invention is a system for estimating the introduction effect of a trained detection model, specifically, a system for estimating the introduction effect, which is the effect on the detection of fraudulent transactions of introducing the trained detection model into a target financial institution. This estimation system includes, for example, a first acquisition means, a second acquisition means, and an estimation means.

[0024] A "trained model for detection" is a model for detecting fraudulent financial transactions, and is a concept that includes, for example, trained models generated by performing predetermined machine learning. This trained model for detection is generated, for example, based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution. In other words, as an example, it is a trained model generated by performing machine learning using the financial transaction-related information as training data.

[0025] "Financial transactions" is a concept that refers to financial transactions, and more specifically, to transactions conducted using a computer. Financial transactions include, for example, transactions conducted using an ATM (Automated Teller Machine) installed in a specific store, and transactions conducted using a network-based transaction function including so-called Internet banking functions, and examples include deposits, withdrawals, and transfers of currency. Furthermore, financial transactions may be interpreted as including, for example, transactions related to currency as well as transactions related to financial products other than currency, such as stocks or futures. While these "financial transactions" are generally genuine transactions, fraudulent transactions are also anticipated.

[0026] "Fraudulent transactions" are fraudulent financial transactions, and include, for example, transactions that are not in line with the original intentions of the transactors, and include, for example, transactions related to criminal activities such as fraud (including bank transfer fraud) and illegal lending. Incidentally, fraudulent transactions may also be interpreted as meaning illegal transactions, for example.

[0027] A "genuine transaction" is a genuine (legitimate) transaction in financial transactions, and is a concept that includes, for example, transactions that are in line with the original intentions of the transactors, and one example is a concept that includes transactions that are not related to criminal activity. Note that a genuine transaction may also be interpreted as indicating a lawful (or legal) transaction, for example.

[0028] The "first acquisition means" is a means for acquiring first region-related information. The "second acquisition means" is a means for acquiring second region-related information. The "estimation means" is a means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means.

[0029] "First region-related information" refers to information on financial transactions based on a region related to the target financial institution, and "second region-related information" refers to information on financial transactions based on a region related to other financial institutions.

[0030] "Region" is a concept that indicates an area separated by specified criteria, and includes, for example, prefectures, cities, towns, villages, regional divisions (Kyushu region, Chugoku region, etc.), or areas separated by specified criteria set in advance by the user.

[0031] In the following embodiment, the case where the "region" is a prefecture will be described as an example.

[0032] (composition) First, an information processing system according to the present embodiment will be described. Fig. 1 is a block diagram of an information processing system according to the embodiment of the present invention, and Fig. 2 is an explanatory diagram of a part of the information processing system.

[0033] The information processing system 100 in FIG. 1 is an estimation system, and includes, for example, a management terminal device 1 and a server device 2.

[0034] (Configuration-Management Terminal Device) The management terminal device 1 in FIG. 1 is a device that manages financial transactions, for example, a device that manages financial transactions processed by a transaction terminal device (not shown) of each financial institution, and as an example, a device that detects fraudulent transactions in the financial transactions, and is installed on the side of each financial institution.

[0035] The number of management terminal devices 1 is arbitrary, but in this embodiment, the description will focus on management terminal devices 101-104 for managing the financial transactions of Bank A, Bank B, Bank C, and Bank D. Furthermore, when there is no need to distinguish between these management terminal devices 101-104, they will be collectively referred to as management terminal device 1. Furthermore, management terminal devices 102-104 also have the same components as management terminal device 101, but for ease of explanation, they are not shown in the illustration.

[0036] A transaction terminal device (not shown) is a device (including a computer) on the side of each financial institution, specifically a device that is capable of communicating with each financial institution's management terminal device 1, such as a computer for conducting financial transactions, and one example is a device that is capable of communicating with an ATM or a terminal used by the transactor (such as a personal computer, tablet terminal, or smartphone) and processes information related to financial transactions.

[0037] The management terminal device 1 in FIG. 1 includes, for example, a communication unit 11, a recording unit 12, and a control unit 13.

[0038] (Configuration - Management terminal device - Communication unit) 1 is a communication means for communicating with an external device (for example, the server device 2). The specific type and configuration of this communication unit 11 are arbitrary, but it can be configured using, for example, a known communication circuit or the like (the same applies to communication units of other devices).

[0039] (Configuration - Management terminal device - Recording unit) 1 is a recording means (storage means) that records programs and various data necessary for the operation of the management terminal device 1, and is configured using, for example, a hard disk or flash memory (not shown) as an external recording device (the same applies to the recording units of other devices). However, instead of or in addition to the hard disk or flash memory, any other recording medium can be used, including a magnetic recording medium such as a magnetic disk, or an optical recording medium such as a DVD or Blu-ray disc (the same applies to the recording units of other devices).

[0040] The recording unit 12 stores, for example, fraudulent transaction related information, genuine transaction related information, and a trained combination model for fraud detection.

[0041] (Configuration - Management terminal device - Recording unit - Fraudulent transaction related information) Figure 3 is a diagram illustrating fraudulent transaction-related information. Note that in Figure 3, for the sake of convenience, some information is omitted and indicated by "..." (the same applies to other figures). Also, the items in Figure 3 are merely examples, and some items may be omitted or other items may be added (the same applies to other figures). Also, the information shown for each item in Figure 3 is for the sake of convenience (the same applies to other figures).

[0042] ===Information about fraudulent transactions=== "Fraudulent transaction related information" refers to financial transaction related information that is information related to financial transactions, and specifically, various information corresponding to fraudulent transactions at each bank. For example, the fraudulent transaction related information of management terminal device 101 in Figure 1 is various information corresponding to fraudulent transactions at Bank A, the fraudulent transaction related information of management terminal device 102 in Figure 1 is various information corresponding to fraudulent transactions at Bank B, the fraudulent transaction related information of management terminal device 103 in Figure 1 is various information corresponding to fraudulent transactions at Bank C, and the fraudulent transaction related information of management terminal device 104 in Figure 1 is various information corresponding to fraudulent transactions at Bank D.

[0043] In the fraudulent transaction related information, for example, the information items shown in FIG. 3 are mutually associated.

[0044] The transaction ID in FIG. 3 is transaction identification information (hereinafter, the identification information will be referred to as "ID") that uniquely identifies a financial transaction that corresponds to a fraudulent transaction (such as "F001" in FIG. 3).

[0045] The account ID in Figure 3 is an account ID that identifies the financial account associated with the financial transaction identified by the transaction ID (e.g., "A001" in Figure 3). Note that the "financial account associated with the financial transaction" may be interpreted as indicating, for example, the financial account in which the financial transaction was made.

[0046] The transaction content information in Figure 3 is information that indicates the content of the financial transaction identified by the transaction ID (Figure 3 shows examples such as the transaction amount being "20,000" yen, the transaction method being "ATM", and the transaction time being "23:05").

[0047] ===Description=== The information at the top of Figure 3 indicates that the financial transaction identified by "F001" is a fraudulent transaction, that the financial transaction was carried out in the financial account identified by "A001," and that the details of the financial transaction correspond to "Amount: 20,000, Method: ATM, Time: 23:05, ...."

[0048] Figure 3 also shows that the financial transactions identified by "F001," "F002," "F003," and "F004" are fraudulent transactions, and that each of these financial transactions was conducted in the financial account identified by "A001."

[0049] FIG. 3 also shows that the financial transactions identified by "F005" are fraudulent transactions, and that each of the financial transactions is carried out in a financial account identified by "A002."

[0050] That is, FIG. 3 also shows that the financial accounts identified by "A001" and "A002" are financial accounts in which fraudulent transactions were made.

[0051] ===Storage Method=== The method for storing such fraudulent transaction-related information in Figure 3 is arbitrary, and may be, for example, by inputting information related to actual fraudulent transactions reported from actual financial transactions at each financial institution, or by any other method. Note that in this embodiment, the fraudulent transaction-related information is configured to be accumulated or updated periodically (for example, daily, weekly, etc.) so as to reflect the latest information related to fraudulent transactions.

[0052] (Configuration - Management terminal device - Recording unit - Genuine transaction related information) FIG. 4 is a diagram illustrating genuine transaction related information.

[0053] ===Genuine Transaction Information=== "Genuine transaction related information" refers to financial transaction related information that is information related to financial transactions, and specifically, various information corresponding to genuine transactions at each bank. For example, the genuine transaction related information of management terminal device 101 in Figure 1 is various information corresponding to genuine transactions at Bank A, the genuine transaction related information of management terminal device 102 in Figure 1 is various information corresponding to genuine transactions at Bank B, the genuine transaction related information of management terminal device 103 in Figure 1 is various information corresponding to genuine transactions at Bank C, and the genuine transaction related information of management terminal device 104 in Figure 1 is various information corresponding to genuine transactions at Bank D.

[0054] The transaction ID in FIG. 4 is a transaction ID that uniquely identifies a financial transaction that is a genuine transaction (such as "C006" in FIG. 4).

[0055] The account ID in FIG. 4 is an account ID that identifies the financial account associated with the financial transaction identified by the transaction ID (such as "A003" in FIG. 4).

[0056] The transaction content information in FIG. 4 is information indicating the content of the financial transaction identified by the transaction ID (in FIG. 4, the transaction amount is exemplified as "5000" yen, etc.).

[0057] ===Description=== The information at the top of Figure 4 indicates that the financial transaction identified by "C006" is a genuine transaction, that the financial transaction was carried out in the financial account identified by "A003," and that the content of the financial transaction corresponds to "Amount: 5000, ...."

[0058] FIG. 4 also shows that the financial transactions identified by "C006" and "C007" are genuine transactions, and that each of the financial transactions is carried out in the financial account identified by "A003."

[0059] FIG. 4 also shows that the financial transactions identified by "C008" are genuine transactions, and that each of the financial transactions is carried out in the financial account identified by "A004."

[0060] That is, FIG. 4 also shows that the financial accounts identified by "A003" and "A004" are financial accounts in which genuine transactions have been made.

[0061] ===Storage Method=== The method for storing the genuine transaction related information in Fig. 4 is arbitrary, and for example, it may be stored by inputting information related to actual genuine transactions confirmed from actual financial transactions at each financial institution, or it may be stored by any other arbitrary method. Note that in this embodiment, the genuine transaction related information is accumulated or updated periodically (for example, daily, weekly, etc.) so as to reflect the latest information related to genuine transactions.

[0062] (Configuration - Management terminal device - Recording unit - Learned combination model for fraud detection) Fig. 5 is an explanatory diagram of the learned combination model for fraud detection, Fig. 6 is an explanatory diagram of the first model, and Fig. 7 is an explanatory diagram of the second model. Note that Fig. 6 and Fig. 7 are descriptions for convenience of explanation, and the number of hidden layers is not limited to two.

[0063] The "trained combination model for fraud detection" is a trained model for detecting fraudulent transactions, and is a model that outputs an AI score when transaction content information is input, as shown in Figure 5, and is a model in which the first model and the second model are combined with each other.

[0064] ===Transaction Details=== As explained in FIG. 3 and FIG. 4, "transaction content information" is information indicating the content of the financial transaction, such as the amount, method, and time.

[0065] ===AI Score=== An "AI score" is information that indicates the likelihood that a financial transaction corresponding to the transaction content information input into the trained model is fraudulent or genuine.

[0066] The specific content of this AI score is arbitrary, but for example, we will explain using numerical information ranging from "0" to "1" indicating the likelihood of a fraudulent transaction from lowest to highest. In detail, for example, the larger the AI ​​score value, the higher the likelihood of a fraudulent transaction, i.e., the lower the likelihood of a genuine transaction. Also, for example, the smaller the AI ​​score value, the lower the likelihood of a fraudulent transaction, i.e., the higher the likelihood of a genuine transaction.

[0067] For example, if there is a financial transaction with an "AI score" of "0.3" ("First Financial Transaction"), a financial transaction with an "AI score" of "0.5" ("Second Financial Transaction"), and a financial transaction with an "AI score" of "0.9" ("Third Financial Transaction"), the third financial transaction is most likely to be fraudulent, the second financial transaction is second most likely to be fraudulent, and the first financial transaction is least likely to be fraudulent.

[0068] In this embodiment, a detection method is described in which a threshold value (a predetermined number between "0" and "1") is set to be compared with the AI ​​score, the AI ​​score is compared with the threshold value, and fraudulent transactions are detected based on the magnitude relationship of the AI ​​score relative to the threshold value. Specifically, the detection method is set so that if the AI ​​score is below the threshold value, the transaction is considered to be genuine and no fraudulent transaction is detected, and if the AI ​​score is above the threshold value, the transaction is considered to be fraudulent and a fraudulent transaction is detected.

[0069] In this detection method, if the threshold is, for example, "0.85," the AI ​​scores of the first and second financial transactions mentioned above are "0.3" and "0.5," respectively, which are below the threshold of "0.85," and therefore these first and second financial transactions will not be detected as fraudulent transactions. On the other hand, the AI ​​score of the third financial transaction mentioned above is "0.9," which is above the threshold of "0.85," and therefore this third financial transaction will be detected as fraudulent. The threshold is arbitrary and may be set to "0.5."

[0070] ===First Model=== The "first model" is a trained model for detection to detect fraudulent transactions in financial transactions, specifically, a trained model that constitutes a trained combination model for fraud detection, and is a trained model that outputs an AI score when transaction content information is input, for example, as shown in Figure 6. This first model is a first model (first model included in the first model group) stored in the recording unit 22 of the server device 2 in Figure 1, which is generated by machine learning at another bank using the other bank's fraudulent transaction-related information and genuine transaction-related information as training data.

[0071] ===Second Model=== The "second model" is a trained model for detecting fraudulent financial transactions, specifically, a trained model that constitutes a trained combination model for fraud detection, and is a model that outputs an AI score when transaction content information and the AI ​​score output from the first model are input, as shown in Figure 7. This second model is a trained model generated by the bank through machine learning using the bank's own fraudulent transaction-related information and genuine transaction-related information as training data. ===Our bank and other banks===

[0072] "Own bank" and "other banks" are concepts that indicate banks based on the management terminal device 1, which is the processing entity. "Own bank" refers to a bank (e.g., Bank A) whose financial transactions are managed by the management terminal device 1 (e.g., management terminal device 101), and "other banks" are concepts that indicate banks whose financial transactions are not managed by the management terminal device 1 (e.g., management terminal device 101) (e.g., Bank B, Bank C, Bank D whose financial transactions are managed by other management terminal devices 102-104).

[0073] For example, if the learned combination model for fraud detection shown in Figure 5 is stored in the management terminal device 101 (Figure 1), then, based on this management terminal device 101, "own bank" will indicate "Bank A," and "other bank" will indicate banks other than "Bank A" (such as "Bank B," "Bank C," or "Bank D").

[0074] In this case, since the second model in Figure 5 is generated by machine learning using the transaction-related information of Bank A as training data, the weights a21, a22, bias b21, etc., which are the model parameters of the second model in Figure 7, are determined by the transaction-related information of Bank A. Also, since the first model in Figure 5 is generated by machine learning using the transaction-related information of a bank other than Bank A (e.g., "Bank B") as training data, the weights a11, a12, bias b11, etc., which are the model parameters of the first model in Figure 6, are determined by the transaction-related information of Bank B.

[0075] ===Storage Method=== A method for storing such a trained combination model for fraud detection as shown in Figure 5 will be described later.

[0076] (Configuration - Management terminal device - Control unit) 1 is a control means for controlling the management terminal device 1, and is specifically a computer that includes a CPU, various programs that are interpreted and executed on the CPU (including basic control programs such as an OS and application programs that are started on the OS and realize specific functions), and an internal memory such as RAM for storing programs and various data (the same applies to the control units of other devices). In particular, the program according to the embodiment is installed in the management terminal device 1 via any recording medium or a network, thereby substantially configuring each unit of the control unit 13.

[0077] The control unit 13 includes, for example, a first acquisition means, a second acquisition means, a third acquisition means, a determination means, and an estimation means.

[0078] ===First acquisition method=== As described above, the first acquisition means is a means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution.

[0079] ===Second acquisition means=== As described above, the second acquisition means is a means for acquiring second region-related information, which is information on region-based financial transactions related to other financial institutions.

[0080] ===Third acquisition means=== The third acquisition means is a means for acquiring feature-related information indicating the features of the trained model for detection.

[0081] ===Judgment means=== The determination means is means for determining whether or not to use the trained model for detection as a target for estimating the introduction effect based on the first region-related information acquired by the first acquisition means. Also, the determination means determines whether or not to use the trained model for detection as a target for estimating the introduction effect based on, for example, the region-related trained model.

[0082] A "region-related trained model" is configured to output an estimation index, which is an index used to estimate the introduction effect of a detection trained model, when first region-related information and second region-related information are input, and includes, for example, a first region-related trained model and a second region-related trained model.

[0083] The "first region-related trained model" is a model that outputs a first indicator corresponding to the precision rate based on fraudulent transactions as an inference indicator. The "second region-related trained model" is a model that outputs a second indicator corresponding to the recall rate based on financial accounts associated with fraudulent transactions as an inference indicator.

[0084] The aforementioned "region-related trained model" is also configured to output a judgment index used to determine whether or not to use the detection trained model as a target for estimating the introduction effect when, for example, only one of the first region-related information or the second region-related information is input.

[0085] ===Estimation means=== As described above, the estimation means is a means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means. The estimation means also estimates the introduction effect, for example, based on a region-related trained model. The estimation means also estimates the introduction effect, for example, based on the first region-related information acquired by the first acquisition means, the second region-related information acquired by the second acquisition means, and the feature-related information acquired by the third acquisition means. The estimation means also estimates the introduction effect for the detection trained model determined by the determination means to be the estimation target.

[0086] The specific processing performed by the control unit 13 will be described later.

[0087] (Configuration - Server Device) The server device 2 in FIGS. 1 and 2 is a device that generates a first region model and a second region model (described later), and includes, for example, a communication unit 21, a recording unit 22, and a control unit 23 shown in FIG.

[0088] (Configuration - Server Device - Communication Unit) The communication unit 21 in FIG. 1 is a communication means for communicating with an external device (for example, the management terminal device 1).

[0089] (Configuration - Server Device - Recording Unit) The recording unit 22 in Figure 1 is a recording means (storage means) that records programs and various data necessary for the operation of the server device 2, and stores, for example, a first model group, explanatory variable related information, and target variable related information.

[0090] (Configuration - Server Device - Recording Unit - First Model Group) The "first model group" in Fig. 1 is a concept indicating a plurality of first models. In this embodiment, it is assumed that the first model group including "Bank A_1," "Bank A_2," "Bank A_3," etc. in Fig. 2 is stored.

[0091] "Bank A_1," "Bank A_2," "Bank A_3," etc. in Figure 2 are multiple first models generated by machine learning in the management terminal device 101 at Bank A using each transaction-related information of Bank A as training data.

[0092] "Bank B_1," "Bank B_2," "Bank B_3," etc. in Figure 2 are multiple first models generated by machine learning in the management terminal device 102 at Bank B using each transaction-related information of Bank B as training data.

[0093] "Bank C_1," "Bank C_2," "Bank C_3," etc. in Figure 2 are multiple first models generated by machine learning at the management terminal device 103 of Bank C using each transaction-related information of Bank C as training data.

[0094] "Bank D_1," "Bank D_2," "Bank D_3," etc. in Figure 2 are multiple first models generated by machine learning at the management terminal device 104 of Bank D using each transaction-related information of Bank A as training data.

[0095] ===Storage Method=== The method for storing this first model group is arbitrary, but for example, the control unit 13 of each bank's management terminal device 1 generates first models by performing machine learning using fraudulent transaction-related information and genuine transaction-related information stored in its own recording unit 12 as training data, and then transmits the generated first models to the server device 2 and stores them, repeating this process periodically (for example, daily, weekly, etc.), thereby storing a first model group consisting of multiple first models.

[0096] Note that the specific method of machine learning is arbitrary, but machine learning may be applied using a large number of combinations of transaction content information of fraudulent transactions (transaction content information of fraudulent training data) and the AI ​​score "1" corresponding to that transaction content information, and a large number of combinations of transaction content information of genuine transactions (transaction content information of genuine training data) and the AI ​​score "0" corresponding to that transaction content information. Note that the fraudulent transaction-related information in Figure 3 is also referred to as "fraudulent training data," and the genuine transaction-related information in Figure 4 is also referred to as "genuine training data."

[0097] =Bank A= Here, for example, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 performs machine learning using the fraudulent transaction-related information and genuine transaction-related information stored in its own recording unit 12 as training data to generate a first model, "Bank A_1," and transmits the generated "Bank A_1" to the server device 2 to store it as a first model group. Furthermore, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 periodically repeats this process to generate and transmit "Bank A_2," "Bank A_3," etc., and stores these "Bank A_2" and "Bank A_3" in the server device 2 as well as the first model group.

[0098] As mentioned above, for these first models ("Bank A_1," "Bank A_2," "Bank A_3," etc.), the transaction-related information (Figures 3 and 4) used as training data is accumulated or updated periodically (e.g., daily, weekly, etc.), so the information is different each time machine learning is performed, resulting in trained models with different model parameters (weights a11, a12, bias b11, etc. in Figure 6).

[0099] These first models ("Bank A_1," "Bank A_2," "Bank A_3," etc.) are configured so that the date and time when the first model was stored can be known by storing them in association with date and time information indicating the date and time when the first model was stored.

[0100] =Bank B~D= 1 also performs the same processing as the management terminal device 101 of Bank A. That is, the control units of the management terminal devices 102 to 104 of Banks B to D in Fig. 1 periodically perform processing to generate and transmit a first model using each piece of transaction-related information stored in their own recording units as training data, thereby storing each first model in Fig. 2 as a first model group.

[0101] (Configuration - Server Device - Recording Unit - Explanatory Variable Related Information) The explanatory variable related information in Figure 1 is information corresponding to the explanatory variables of the first region model and the second region model (Figure 2), and includes the region related explanatory variable information, bank-derived related explanatory variable information, bank ranking related explanatory variable information, model-derived related explanatory variable information, and other bank model-derived related explanatory variable information in Figure 1. Each region model will be described later.

[0102] (Configuration - Server Device - Recording Unit - Explanatory Variable Related Information - Region Related Explanatory Variable Information) 8 is a diagram illustrating an example of region-related explanatory variable information. Note that, although information on all prefectures may be used as the region-related explanatory variable information, in this embodiment, Saitama Prefecture, Chiba Prefecture, Tokyo, and Kanagawa Prefecture, which are closely related to Banks A to D illustrated in FIG. 1, are illustrated and the explanation will focus on these prefectures.

[0103] "Region-related explanatory variable information" refers to the aforementioned first region-related information and second region-related information, specifically information based on the region (region) associated with a financial institution. This region-related explanatory variable information is information corresponding to the region-related explanatory variables, and is, for example, information that reflects the characteristics of financial transactions in each region. Examples include information showing the proportion of accounts in each prefecture (i.e., each region) (Figure 8(a)), the proportion of the total number of special frauds in each prefecture (Figure 8(b)), and the proportion of the number of fraud cases by type in each prefecture (Figure 8(c)).

[0104] ===Percentage of accounts by prefecture=== The "proportion of accounts in each prefecture" in FIG. 8(a) indicates, for example, the proportion of the number of account holders at each bank who have addresses in each prefecture to the total number of account holders at each bank.

[0105] The information at the top of Figure 8(a) shows, for example, that the percentages of account holders with addresses in Saitama, Chiba, Tokyo, and Kanagawa prefectures relative to the total number of account holders at Bank A are 85%, 2%, 4%, and 9%, respectively. In other words, if the total number of account holders at Bank A is 1,000, for example, it shows that 850 of them live in Saitama prefecture and 20 live in Chiba prefecture.

[0106] ===Percentage of total special fraud cases in each prefecture=== The "proportion of the total number of special frauds in each prefecture" in Figure 8(b) indicates, for example, the proportion of the total number of special frauds (cases) that occurred in each prefecture to the total number of special frauds (total number of cases in Saitama, Chiba, Tokyo, and Kanagawa prefectures) that occurred in a given period (for example, the most recent three months or one year). Note that special fraud refers to frauds related to financial transactions, and is a concept that includes, for example, "it's me" fraud, deposit and savings fraud, and various fee billing fraud.

[0107] Figure 8(b) shows that the percentages of the total number of special fraud cases that occurred in Saitama, Chiba, Tokyo, and Kanagawa prefectures out of the total number of special fraud cases that occurred during a given period are 30%, 20%, 15%, and 35%. In other words, if the total number of special fraud cases (the total number of special fraud cases in Saitama, Chiba, Tokyo, and Kanagawa prefectures) is 1,000, for example, 300 of those cases occurred in Saitama prefecture and 200 in Chiba prefecture.

[0108] ===Proportion of fraud cases by type in each prefecture=== The "proportion of cases by type of fraud in each prefecture" in Figure 8(c) indicates, for example, the proportion of cases by type of fraud that occurred in each prefecture to the total number of special frauds that occurred in each prefecture over a specified period (e.g., the last three months or one year).

[0109] The information at the top of Figure 8(c) shows that, for example, the percentages of "it's me" fraud, "deposit and savings fraud," and "fictitious fee billing fraud" cases out of the total number of special frauds that occurred in Saitama prefecture during a given period are 28%, 22.5%, and 49.4%, respectively. In other words, if the total number of special frauds that occurred in Saitama prefecture was 300, 28% (=84 cases) of those were "it's me" frauds.

[0110] ===Storage Method=== The region-related explanatory variable information in Fig. 8 may be stored by any method, for example, an administrator may periodically (e.g., daily, weekly, etc.) collect information from financial institutions and related organizations (police-related organizations, etc.) and input the information to store it, or may be stored by any other method. Note that since information is input periodically, the region-related explanatory variables in Fig. 8 will reflect the latest information.

[0111] (Configuration - Server Device - Recording Unit - Explanatory Variable Related Information - Bank-Derived Related Explanatory Variable Information) FIG. 9 is a diagram illustrating bank-derived related explanatory variable information.

[0112] "Bank-derived related explanatory variable information" is information corresponding to bank-derived related explanatory variables, specifically statistical information related to banks, such as information reflecting the characteristics of each bank. For example, as shown in Figure 9, this bank-derived related explanatory variable information is information indicating the proportion of corporate accounts, the proportion of online banking operations, the number of reports to the police in a three-month period, the number of overseas remittances in a three-month period, and the total amount of overseas remittances in a three-month period, etc., for each bank.

[0113] The "proportion of corporate accounts" in Figure 9 indicates the proportion of corporate accounts to the total number of accounts (personal accounts and corporate accounts) at each bank (such as the "30%" at the top of Figure 9, which indicates that Bank A's proportion of corporate accounts is 30%).

[0114] The "proportion of online banking services" in Figure 9 indicates the proportion of online banking services (services provided via the Internet, not over the counter) to the total number of services provided by each bank (e.g., "25%" at the top of Figure 9, which indicates that Bank A's proportion of online banking services is 25%). Note that this proportion may also be calculated based on the number of times over a predetermined period (e.g., three months).

[0115] The "Number of reports to the police in the last three months" in Figure 9 indicates the number of financial transactions that each bank reported to the police for a specified reason (such as the discovery of fraudulent transactions) in the last three months (such as the "20" at the top of Figure 9, which indicates that Bank A reported 20 financial transactions to the police for a specified reason in the last three months).

[0116] The "Number of overseas remittances in three months" in Figure 9 indicates the number of financial transactions in which each bank conducted overseas remittances in the most recent three months (such as the "10" in the top row and second column from the right in Figure 9, which indicates that Bank A conducted 10 financial transactions in which it conducted overseas remittances in the most recent three months).

[0117] The "Total amount of overseas remittances over the past three months" in Figure 9 indicates the total amount of overseas remittances made by each bank over the past three months (such as the "10" in the top row and rightmost column of Figure 9, which indicates that the total amount of overseas remittances made by Bank A over the past three months is 1 billion yen).

[0118] ===Storage Method=== The method for storing such bank-derived related explanatory variable information in Fig. 9 is arbitrary, but for example, the administrator may periodically (for example, daily, weekly, etc.) collect information from each financial institution and input the information to store it, or any other method may be used to store it. Note that since information is input periodically, the bank-derived related explanatory variable information in Fig. 9 will reflect the latest information.

[0119] (Configuration - Server Device - Recording Unit - Explanatory Variable Related Information - Bank Ranking Related Explanatory Variable Information) FIG. 10 is a diagram illustrating explanatory variable information related to bank rankings.

[0120] "Bank ranking-related explanatory variable information" is information corresponding to bank ranking-related explanatory variables, specifically, information on rankings (orders) related to banks, for example, information that reflects the characteristics of each bank. For example, as shown in Figure 10, this bank ranking-related explanatory variable information is information that indicates the ranking of each bank in terms of the number of monitoring operations in the most recent month and the number of reports to the police in the most recent three months.

[0121] The "Ranking of the number of monitoring operations in the past month" in Figure 10 shows the ranking of each bank in terms of the number of monitoring operations in the past month. Note that monitoring operations refer to bank operations such as checking the details of financial transactions detected as fraudulent, stopping the transactions as necessary, and reporting or filing a police report.

[0122] Figure 10 shows that among Banks A to D, Bank A has the highest number of monitoring operations, Bank B has the second highest number of monitoring operations, Bank C has the third highest number of monitoring operations, and Bank D has the lowest number of monitoring operations.

[0123] The "Ranking of the number of reports to the police in a three-month period" in Figure 10 shows the ranking of each bank in terms of the number of financial transactions (number of reports) that were reported to the police for a specific reason (such as the discovery of fraudulent transactions) in a specific three-month period (such as the most recent three months).

[0124] Figure 10 shows that among Banks A to D, Bank B has the most notifications, Bank A has the second most notifications, Bank C has the third most notifications, and Bank D has the least number of notifications.

[0125] ===Storage Method=== The method for storing such bank ranking-related explanatory variable information in Fig. 10 is arbitrary, but for example, an administrator may periodically (e.g., daily, weekly, etc.) collect information from each financial institution to determine each ranking, and then input information indicating the ranking to store it, or any other method may be used to store it. Note that, since information is input periodically, the bank ranking-related explanatory variable information in Fig. 10 will reflect the latest information.

[0126] (Configuration - Server Device - Recording Unit - Explanatory Variable Related Information - Model-Derived Related Explanatory Variable Information) Fig. 11 is a diagram illustrating model-derived related explanatory variable information. For ease of explanation, Fig. 11 illustrates only information related to a portion of the first model, and the following description will focus on this information (the same applies to Figs. 12 and 13 described below).

[0127] "Model-derived related explanatory variable information" is information (feature-related information) indicating the features of the first model (trained model for detection). This model-derived related explanatory variable information is information corresponding to the model-derived related explanatory variables, and is, for example, information reflecting the features related to the structure of each of the first models recorded as the first model group in the recording unit 22 in Figures 1 and 2, and an example is information indicating the number of accounts in the fraudulent training data and the number of decision trees.

[0128] The "Number of accounts in fraudulent training data" in Figure 11 indicates the number of financial accounts associated with financial transactions indicated by the transaction content information indicated in the fraudulent transaction-related information (Figure 3) used as training data in the machine learning to generate the first model.

[0129] In this case, when counting the number of financial accounts, duplicate financial accounts will be counted as one. For example, in the information specifically illustrated in Figure 3, four "A001"s are shown, but since these relate to the common "A001", they will be counted as one financial account.

[0130] The "number of decision trees" in FIG. 11 indicates the number of decision trees corresponding to the first model.

[0131] The information at the top of Figure 11 indicates that the number of financial accounts associated with financial transactions indicated in the fraudulent training data used in the machine learning to generate the first model, "Bank A_1" (Figure 2), is 150, and also indicates that the number of decision trees corresponding to "Bank A_1" is 80. Note that while the number of financial accounts is counted as "number of accounts," it may also be counted as "number of items," and can be interpreted as indicating that the number of financial accounts mentioned above is 150.

[0132] ===Storage Method=== The method for storing such model-derived related explanatory variable information in Figure 11 is arbitrary, but with regard to the information indicating the "number of accounts in fraudulent training data," for example, when each bank's management terminal device 1 generates and transmits a first model to store it in the recording unit 22 of Figure 2, it may be configured to count the number of financial accounts associated with the financial transactions indicated in the fraudulent training data used in the machine learning when generating the first model, and transmit count information indicating the number of financial accounts counted together with the first model to the server device 2, and then store the information based on that count information.

[0133] Furthermore, the information indicating the "number of decision trees" may be configured, for example, so that when the management terminal device 1 of each bank generates and transmits a first model, it transmits number information (information indicating the number of decision trees of the generated first model) along with the first model to the server device 2, and the information is then stored based on the number information. Alternatively, the control unit 23 of the server device 2 may be configured to identify the number of decision trees by analyzing the first model recorded in the recording unit 22, and the information indicating the identified number may be stored.

[0134] Alternatively, the system may be configured so that an administrator identifies the "number of accounts with fraudulent training data" and the "number of decision trees" based on the generation history of the first model at each bank, and then inputs information indicating the identified "number of accounts with fraudulent training data" and "number of decision trees" to store them.

[0135] (Configuration - Server Device - Recording Unit - Explanatory Variable Related Information - Other Bank Model-Derived Related Explanatory Variable Information) FIG. 12 is a diagram illustrating the related explanatory variable information derived from other bank models.

[0136] "Other bank model-derived related explanatory variable information" is information (feature-related information) indicating the features of the first model (trained model for detection). This other bank model-derived related explanatory variable information is information corresponding to the other bank model-derived related explanatory variables, and is, for example, information reflecting the features related to the performance of each of the first models recorded as the first model group in the recording unit 22 in Figures 1 and 2, and is, for example, information indicating the transaction-based precision rate, account-based recall rate, and transaction-based AUC.

[0137] 12 is information indicating the performance of a trained model (here, for example, the first model) and is an evaluation index for evaluating a trained model from the perspective of performance. This transaction-based precision is information based on financial transactions, and is, for example, information indicating the ratio of the number of financial transactions detected as fraudulent (detected fraudulent transactions) to the number of financial transactions that were actually fraudulent among the detected fraudulent transactions, with a larger value indicating higher performance.

[0138] The "account-based recall" in Figure 12 is information indicating the performance of a trained model (here, for example, the first model) and is an evaluation index for evaluating a trained model from the perspective of performance. This account-based recall is information based on financial accounts, and is, for example, information indicating the ratio of the number of financial accounts in which financial transactions detected as fraudulent transactions were made among the financial accounts in which fraudulent transactions were actually made to the number of financial accounts in which fraudulent transactions were actually made, with a larger value indicating higher performance. Note that duplicate financial accounts are counted as one.

[0139] 12, "transaction unit AUC" is information indicating the performance of a trained model (here, for example, the first model) and is an evaluation index for evaluating the trained model from the performance perspective. This transaction unit AUC is information corresponding to the AUC (Area Under the Curve) based on the false positive rate and true positive rate, and the larger the value, the higher the performance.

[0140] The information at the top of Figure 12 shows that for the first model, "Bank A_1" (Figure 2), the transaction-based precision rate is "65%", the account-based recall rate is "50%," and the "transaction-based AUC" is "0.90".

[0141] ===Storage Method=== The method for storing the other bank model-derived related explanatory variable information in FIG. 12 may be to execute the first storage process described below, or may be to store the information using any other method.

[0142] (Configuration - Server Device - Recording Unit - Objective Variable Related Information) FIG. 13 is a diagram illustrating the response variable related information.

[0143] The objective variable-related information in Figure 1 is information corresponding to the objective variables of the first region model and the second region model (Figure 2), and includes, for example, information on "use of other bank models" (Figure 13(a)) and information on "not use of other bank models" (Figure 13(b)).

[0144] ===Utilizing other bank models=== The information in Figure 13(a) "Other bank models used" is information that indicates the characteristics of the trained combination model for fraud detection (Figure 5), for example, information that reflects the performance characteristics of each trained combination model for fraud detection, and one example is information that indicates the transaction-based precision rate and account-based recall rate.

[0145] Figure 14 is an explanatory diagram of a trained combination model for fraud detection. The information in the "Introducing Bank Model Name" column in Figure 13(a) indicates the second model (Figure 5), and the information in the "Other Bank Model Name" column indicates the other bank's first model (Figure 5), and the combination of these indicates the trained combination model for fraud detection (Figure 5). Here, for example, we will focus on "Bank A_1(2)" and "Bank B_1" in the top row of Figure 13(a) and explain their contents.

[0146] As shown in Figure 14, for example, "Bank A_1(2)" and "Bank B_1" represent a learned combination model for fraud detection that includes the second model generated by machine learning using the training data based on the generated first model "Bank B_1" and the training data of Bank A's transaction-related information (i.e., the transaction-related information stored in the recording unit 12 of the management terminal device 101 in Figure 1), and the aforementioned first model.

[0147] In detail, the "Bank A" in "Bank A_1(2)" indicates that the transaction-related information, which is the training data used in the machine learning to generate the second model, belongs to "Bank A," the "_1" is a convenient notation for unique identification, and the "(2)" indicates that it is the second model. Also, "Bank B_1" indicates that the first model included in the trained combination model for fraud detection is "Bank B_1."

[0148] The information in the "Trading-level precision" column of the "Objective variable" in Figure 13(a) indicates the performance of a trained model (here, for example, a trained combination model for fraud detection), and is an evaluation index for evaluating the trained model from the performance perspective. The definition of this trading-level precision is the same as that explained in Figure 12.

[0149] The information in the "Account-level recall" column of the "Objective variable" in Figure 13(a) is information indicating the performance of a trained model (here, for example, a trained combination model for fraud detection) and is an evaluation index for evaluating the trained model from the performance perspective. The definition of this account-level recall is the same as that explained in Figure 12.

[0150] The information in the top row of Figure 13(a) shows that the transaction-based precision rate is "66%" and the account-based recall rate is "52%" for the trained combination model for fraud detection, which is a combination of the first model "Bank B_1" and the second model "Bank A_1(2)" generated by machine learning based on the first model "Bank B_1" and the training data for each transaction-related information of Bank A.

[0151] Furthermore, the information in the fourth row of Figure 13 shows that the transaction-based precision rate is "50%" and the account-based recall rate is "48%" for the trained combination model for fraud detection, which is a combination of the first model "Bank A_1" and the second model "Bank B_1(2)" generated by machine learning based on the first model "Bank A_1" and the training data for each transaction-related information of Bank B.

[0152] Furthermore, in the information in the seventh row of Figure 13 (information such as "Bank C_1(2)") and the information in the eighth row (information such as "Bank C_2(2)"), the first model ("Bank A_1") and the bank ("Bank C") of each transaction-related information that constitutes the training data are the same, but the timing at which the machine learning was performed is different, and each transaction-related information is periodically updated or accumulated, with at least some of the information differing depending on the timing. As a result, the second models (i.e., trained combination models for fraud detection) that are generated are different, and therefore the transaction-based precision rates and account-based recall rates are different. The same is true for the information in the bottom row and the second-lowest row of Figure 13.

[0153] ===No use of other bank models=== The information in Figure 13(b) for "Other bank models not used" is information (feature-related information) indicating the characteristics of the first model (trained model for detection), and is, for example, information reflecting the characteristics related to the performance of each of the first models recorded as the first model group in the recording unit 22 in Figures 1 and 2, and an example is information indicating the transaction-based precision rate and account-based recall rate.

[0154] The information in the "Introducing Bank Model Name" column in Figure 13(b) indicates the first model. Note that the information in the "Other Bank Model Name" column does not indicate anything ("None" in Figure 13(b)).

[0155] The information in the "Trading unit precision" column of the "Objective variable" in Figure 13(b) indicates the performance of the trained model (here, for example, the first model) and is an evaluation index for evaluating the trained model from the performance perspective. The definition of this trading unit precision is the same as that explained in Figure 12.

[0156] The information in the "Account-level recall" column of the "Objective variable" in Figure 13(b) indicates the performance of the trained model (here, for example, the first model) and is an evaluation index for evaluating the trained model from the performance perspective. The definition of this account-level recall is the same as that explained in Figure 12.

[0157] The information in the top row of Figure 13(b) shows that for the first model, "Bank A_1" (Figure 2), the transaction-based precision rate is "65%" and the account-based recall rate is "50%."

[0158] ===Storage Method=== The method of storing the objective variable-related information in FIG. 13 may be performed by executing a first storage process and a second storage process, which will be described later, or may be performed by any other method.

[0159] (Configuration - Server Device - Control Unit) The control unit 23 in FIG. 1 is a control means for controlling the server device 2.

[0160] (process) Next, we will explain the processes performed by the information processing system 100 configured in this manner, such as financial institution side fraud detection process, first storage process, second storage process, region model generation process, region model related determination process, and other bank model selection process.

[0161] (Processing - Financial institution side fraud detection processing) The fraud detection process on the financial institution side will now be described. The fraud detection process on the financial institution side is a process for detecting fraudulent transactions (fraud detection), and is a process that is repeatedly executed by, for example, each of the management terminal devices 101 to 104 of each bank in FIGS.

[0162] The fraud detection process on the financial institution side can be similar to known processes, so only an outline will be explained.

[0163] For example, when an actual financial transaction is carried out at each bank, each transaction terminal device (computer for carrying out financial transactions) (not shown) of each bank transmits executed financial transaction information (e.g., information including information corresponding to each item in Figures 3 and 4) indicating the content of the financial transaction to the management terminal device 1 of each bank, and the management terminal device 1 is able to grasp the content of the financial transaction that was actually carried out based on the executed financial transaction information.

[0164] When a financial transaction is carried out, the control unit 13 of the management terminal device 1 receives the executed financial information from the transaction terminal device (not shown) and detects the fraudulent transaction using the learned combination model for fraud detection (Figure 5) recorded in the recording unit 12.

[0165] For example, let us consider a case where "0.85" is set as the threshold (threshold to be compared with the AI ​​score). In this case, the control unit 13 of the management terminal device 1 inputs the transaction content information included in the executed financial information received from the transaction terminal device (not shown) into the trained combination model for fraud detection, and obtains the AI ​​score output from the trained combination model for fraud detection.

[0166] Next, the acquired AI score is compared with the set threshold of "0.85," and if the acquired AI score is less than "0.85," the financial transaction corresponding to the transaction content information input into the trained model for detection is deemed to be genuine and is not detected as a fraudulent transaction. On the other hand, if the acquired AI score is "0.85" or higher, the financial transaction corresponding to the transaction content information input into the trained model for detection is deemed to be fraudulent and is detected as a fraudulent transaction.

[0167] Subsequent processing is optional, but for example, the control unit 13 of the management terminal device 1 may record information indicating the detected fraudulent financial transaction in the recording unit 12, or may notify an administrator, etc. This concludes the explanation of the fraud detection processing on the financial institution side.

[0168] (Processing - First storage process) Next, the first storage process will be described. Figure 15 is a flowchart of the first storage process (hereinafter, each step will be referred to as "S"). The first storage process is a process for storing the other bank model-derived related explanatory variable information in Figure 12 and the information "other bank model not utilized" in the target variable related information in Figure 13(b), and is a process executed mainly by, for example, the management terminal device 1 of each bank.

[0169] The timing for executing this first storage process is arbitrary, but for example, it may be started when the management terminal device 1 generates the first model recorded in the recording unit 22 of the server device 2 in Fig. 2, or at another predetermined timing, and the explanation will begin from the point where execution starts. Note that the explanation will be given mainly using the processing at Bank A as an example (the same applies to the second storage process, the region model related determination process, and the other bank model selection process).

[0170] ===SA1=== In SA1 of Figure 15, the control unit 13 of the management terminal device 1 acquires the generated first model when the bank's first model, which is sent to and stored in the server device 2 of Figure 2, is generated by performing machine learning using the transaction-related information of Figures 3 and 4 stored in its own recording unit 12 as training data.

[0171] Note that "our bank's first model" is a concept that refers to a first model generated by machine learning using each transaction-related information of our bank as training data.

[0172] 3 and 4 is used, for example, as training data for performing machine learning to generate a first model, and as verification data for evaluating and verifying the generated first model. While common information among the transaction-related information may be used as training data and verification data, this embodiment will exemplify a case in which some of the information in the transaction-related information in FIGS. 3 and 4 is used as training data, and the other information (information not used as training data) is used as verification data (the same applies to other processes).

[0173] For ease of explanation, the explanation will be given using the information specifically illustrated in Figures 3 and 4, but in actual processing, the processing will be performed using the training data and verification data described above (the same applies to other processing).

[0174] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in FIG. 1 generates and acquires "Bank A_1" which is the first model of the bank itself.

[0175] ===SA2=== In SA2 of Figure 15, the control unit 13 of the management terminal device 1 acquires its own transaction-related information (i.e., the transaction-related information of Figures 3 and 4 recorded in its own recording unit 12 (Figure 1)) as verification data for evaluating and verifying the first model acquired in SA1.

[0176] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in FIG. 1 acquires each piece of transaction-related information shown in FIGS.

[0177] ===SA3=== In SA3 of Figure 15, the control unit 13 of the management terminal device 1 detects fraudulent transactions using the first model and threshold value obtained in SA1 for each financial transaction indicated by each transaction-related information obtained in SA2, using the above-mentioned detection method (a method of detecting by comparing the AI ​​score with a threshold value).

[0178] Figure 16 is a diagram illustrating the results of fraudulent transaction detection. Note that Figure 16 illustrates the results of fraudulent transaction detection performed using the first model for financial transactions indicated by the transaction-related information in Figures 3 and 4. The following explanation will be based on the results for the eight financial transactions illustrated in Figure 16.

[0179] The transaction ID and account ID in Figure 16 are the same as the information with the same names in Figures 3 and 4. The "Detection Results" column in Figure 16 shows examples of the detection results of fraudulent transactions performed using the first model, and shows either "Fraud" indicating that the transaction was detected as fraudulent, or "Genuine" indicating that the transaction was not detected as fraudulent (i.e., the transaction was determined to be genuine).

[0180] For example, when the control unit 13 of the management terminal device 101 of Bank A in FIG. 1 inputs the transaction content information in the top row of FIG. 3 to the first model "Bank A_1" acquired by SA1 with respect to the financial transaction indicated by the information in the top row of FIG. 3, the control unit 13 outputs an AI score of, for example, "0.95" from the first model. If the threshold value used in the first model is, for example, "0.85," the AI ​​score is equal to or greater than the threshold, and therefore a fraudulent transaction is detected (see "Fraud" in the "Detection Results" column in the top row of FIG. 16). Similar processing is then performed on other financial transactions, thereby detecting fraudulent transactions as shown in the "Detection Results" column of FIG. 16.

[0181] That is, for example, using the first model "Bank A_1," we will explain an example in which five financial transactions with "Transaction ID" = "F001," "F002," "F005," "C006," and "C007" are detected as fraudulent transactions, while the other three financial transactions are not detected as fraudulent transactions.

[0182] ===SA4=== In SA4 of FIG. 15, the control unit 13 of the management terminal device 1 identifies the trade unit conformance rate of the first model acquired in SA1 based on the detection result in SA3.

[0183] Specifically, the calculation is performed by dividing the number of detected fraudulent financial transactions that were actually fraudulent by the number of detected fraudulent financial transactions, and the result is identified as the transaction-level matching rate. Note that the "financial transactions that were actually fraudulent among detected fraudulent transactions" correspond to financial transactions with a "detection result" of "fraudulent" in Figure 16 and a transaction ID of "F~~" (i.e., financial transactions indicated in the fraudulent transaction-related information in Figure 3).

[0184] Here, in the case of Figure 16, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 determines that the transaction unit conformance rate for the first model, "Bank A_1", is "3 (financial transactions of F001, F002, F005)" ÷ "5 (financial transactions of F001, F002, F005, C006, C007)" = 60%.

[0185] ===SA5=== In SA5 of FIG. 15, the control unit 13 of the management terminal device 1 identifies the account-based recall rate of the first model acquired in SA1 based on the detection result in SA3.

[0186] Specifically, the calculation is performed by dividing the number of financial accounts in which fraudulent transactions were detected among the financial accounts in which fraudulent transactions actually occurred by the number of financial accounts in which fraudulent transactions actually occurred, and the result of this calculation is identified as the account-level recall rate. Note that a financial account associated with at least one detected fraudulent transaction is defined as a "financial account in which a financial transaction detected as fraudulent was detected," and a financial account associated with at least one fraudulent transaction is defined as a "financial account in which a fraudulent transaction actually occurred."

[0187] Here, in the case of Figure 16, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 determines that the account-based reproducibility rate for the first model, "Bank A_1", is "2 (both of the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" ÷ "2 (financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" = 100%.

[0188] ===SA6=== 15, the control unit 13 of the management terminal device 1 identifies the trading unit AUC of the first model acquired in SA1 based on the detection result in SA3. Specifically, the following first to third steps are performed.

[0189] ==Step 1== In the first step, we identify the transaction-level false positive rate based on the detection results of SA3.

[0190] The "transaction-based false positive rate" is information that indicates the performance of a trained model (here, for example, the first model) and is an evaluation metric for evaluating trained models from the perspective of performance. This transaction-based false positive rate is information based on financial transactions, and indicates, for example, the ratio of the number of financial transactions that were not actually fraudulent but were mistakenly detected as fraudulent to the number of financial transactions that were not actually fraudulent (financial transactions that were actually genuine), with a higher value indicating lower performance.

[0191] Specifically, the calculation is performed by dividing the number of financial transactions that were mistakenly detected as fraudulent transactions despite not actually being fraudulent by the number of financial transactions that were not actually fraudulent (financial transactions that were actually genuine), and the result is identified as the transaction-level false positive rate.

[0192] Here, in the case of Figure 16, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 determines that the transaction-based false positive rate for the first model, "Bank A_1", is "2 (financial transactions of C006 and C007)" ÷ "3 (financial transactions of C006, C007, and C008)" = 66.7%.

[0193] ==Second Step== In the second step, we determine the transaction-level true positive rate based on the detection results of SA3.

[0194] The "transaction-level true positive rate" is information indicating the performance of a trained model (here, for example, the first model) and is an evaluation index for evaluating trained models from the performance perspective. This transaction-level true positive rate is information based on financial transactions, and indicates, for example, the ratio of the number of financial transactions detected as fraudulent to the number of actual fraudulent transactions, with a higher value indicating higher performance.

[0195] Specifically, the calculation is performed by dividing the number of financial transactions detected as fraudulent among actual fraudulent transactions by the number of actual fraudulent transactions, and the result is identified as the transaction-level true positive rate.

[0196] Here, in the case of Figure 16, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 determines that the transaction-based true positive rate for the first model, "Bank A_1", is "3 (F001, F002, F003, which are associated with the "detection result" = "fraud" in the financial transactions of "F~~")" ÷ "5 (F001 to F005, which are financial transactions of "F~~")" = 60%.

[0197] ==Third Step== Figure 17 is an explanatory diagram of transaction-level AUC. In the third step, the threshold (the threshold compared with the AI ​​score output from the first model) is changed in multiple stages (for example, the threshold is increased from "0" to "1" in increments of "0.0001"), and SA3 to SA6 (first and second steps) are repeated multiple times to identify multiple combinations of transaction-level false positive rates and transaction-level true positive rates. An ROC curve is then generated by connecting the coordinates consisting of the combinations of transaction-level false positive rates and transaction-level true positive rates in the coordinate system shown in Figure 17, and the AUC (numerical value) corresponding to the area under the identified ROC curve is identified as the transaction-level AUC.

[0198] In this case, for example, the control unit 13 of the management terminal device 101 of bank A in FIG. 1 specifies "0.90" as the transaction unit AUC.

[0199] ===SA7=== 15, the control unit 13 of the management terminal device 1 transmits information indicating the transaction-unit precision, account-unit recall, and transaction-unit AUC identified in SA4 to SA6 (also referred to as "first index-related information") to the server device 2 along with the first model acquired in SA1. In this case, the control unit 13 of the server device 2 acquires the first index-related information and the first model transmitted from the management terminal device 1, records the acquired first model in the first model group of the recording unit 22, stores information corresponding to the transaction-unit precision, account-unit recall, and transaction-unit AUC indicated in the acquired first index-related information as other bank model-derived related explanatory variable information in FIG. 12, and stores information corresponding to the transaction-unit precision and account-unit recall indicated in the acquired first index-related information as information of "other bank model not utilized" in the objective variable-related information in FIG. 13(b).

[0200] Here, although the numerical values ​​differ from those described above using Figure 16, for example, if the control unit 13 of the management terminal device 101 of Bank A in Figure 1 acquires the first model "Bank A_1" at SA1 in Figure 15, and identifies the transaction unit precision rate = "65%", the account unit recall rate = "50%", and the transaction unit AUC = "0.90" at SA4 to SA6, then at SA7, it transmits the first index-related information indicating these and "Bank A_1" to the server device 2, and "Bank A_1" is stored in the recording unit 22 in Figure 2, the information shown in the top row of Figure 12, and the information shown in the top row of Figure 13(b). Note that the first model "Bank A_1" may be configured to be stored at a different timing than the above.

[0201] ===Other Bank Processing=== 12 and 13(b) are stored by the management terminal devices 1 of other banks other than Bank A. For example, the information in the second row of FIGS. 12 and 13(b) is stored by the processing of the management terminal device 102 of Bank B, and the information in the third and fourth rows of FIGS. 12 and 13(b) is stored by the processing of the management terminal device 103 of Bank C.

[0202] === Variations === 15, the management terminal device 1 in FIG. 2 may be configured to communicate with the server device 2 to receive and acquire its own first model that has already been recorded as a first model group in the recording unit 22, and then perform the above-mentioned processes. This concludes the explanation of the first storage process.

[0203] (Processing - Second storage process) Next, the second storage process will be described. Fig. 18 is a flowchart of the second storage process. The second storage process is a process for storing information on "other bank model utilized" in the objective variable related information of Fig. 13(a), and is a process executed mainly by the management terminal device 1 of each bank, for example.

[0204] The timing for executing this second process is arbitrary, but for example, it may be started when the first model of each bank is stored in the recording unit 22 of the server device 2 in Figure 2, or at some other predetermined timing, and the explanation will begin from the point where execution has started.

[0205] ===SB1=== In SB1 of FIG. 18, the control unit 13 of the management terminal device 1 communicates with the server device 2 to receive and acquire the first models of other banks that have already been recorded in the recording unit 22 as a first model group.

[0206] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in FIG. 1 acquires "Bank B_1" which is the first model of another bank.

[0207] ===SB2=== At SB2 in FIG. 18, the control unit 13 of the management terminal device 1 acquires transaction-related information of its own bank as training data.

[0208] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in FIG. 1 acquires each piece of transaction-related information in FIGS.

[0209] ===SB3=== At SB3 in Figure 18, the control unit 13 of the management terminal device 1 generates a learned combination model for fraud detection (Figure 14) by performing machine learning to generate a second model using the first model acquired at SB1 and the transaction-related information acquired as training data at SB2.

[0210] Note that the specific method of machine learning is arbitrary, but machine learning may be applied using a large number of combinations of transaction content information of fraudulent transactions (transaction content information of fraudulent training data) and the AI ​​score "1" corresponding to that transaction content information, and a large number of combinations of transaction content information of genuine transactions (transaction content information of genuine training data) and the AI ​​score "0" corresponding to that transaction content information. In this case, as shown in Figure 14, a configuration is configured in which transaction content information is input to a first model and the transaction content information and the AI ​​score output from the first model are input to a second model, and then a second model is generated by machine learning, thereby generating a trained combination model for fraud detection.

[0211] Here, only the second model of the first or second models is the subject of machine learning, so the model parameters of the already generated first model (weights a11, a12, bias b11, etc. in Figure 6) are not changed, and only the model parameters of the second model (weights a21, a22, bias b21, etc. in Figure 7) are determined here.

[0212] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 performs machine learning using the first model ``Bank B_1'' obtained in SB1 and the transaction-related information obtained in SB2 to generate the second model ``Bank A_1(2)'' in Figure 14, and generates a learned combination model for fraud detection.

[0213] ===SB4=== In SB4 of FIG. 18, the control unit 13 of the management terminal device 1 acquires transaction-related information of its own bank as verification data for evaluating and verifying the trained combination model for fraud detection generated in SB3.

[0214] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in FIG. 1 acquires each piece of transaction-related information shown in FIGS.

[0215] ===SB5=== At SB5 in Fig. 18, the control unit 13 of the management terminal device 1 detects fraudulent transactions for each financial transaction indicated by each piece of transaction-related information acquired at SB4 using the fraud detection trained combination model and threshold generated at SB3, using the aforementioned detection method (a method of detecting fraud by comparing the AI ​​score with a threshold). Specifically, it performs the same process as SA3 in Fig. 15.

[0216] ===SB6=== 18, the control unit 13 of the management terminal device 1 determines the trade-unit fitness rate of the trained combination model for fraud detection generated in SB3 based on the detection result in SB5. Specifically, the control unit 13 performs the same process as SA4 in FIG.

[0217] ===SB7=== 18, the control unit 13 of the management terminal device 1 determines the account-based recall rate of the fraud detection trained combination model generated in SB3 based on the detection result in SB5. Specifically, the same process as SA5 in FIG. 15 is performed.

[0218] ===SB8=== 18, the control unit 13 of the management terminal device 1 transmits model identification information indicating the trained combination model for fraud detection generated in SB3 (for example, information indicating that the first model constituting the trained combination model for fraud detection is "Bank B_1" and the second model is "Bank A_1(2)"), and information indicating the transaction-based precision rate and account-based recall rate identified in SB6 to SB7 (also referred to as "second index-related information") to the server device 2. In this case, the control unit 13 of the server device 2 acquires the model identification information and second index-related information transmitted from the management terminal device 1, and records the information indicated by the acquired model identification information and second index-related information as information of "other bank model utilized" in the objective variable-related information in FIG. 13(a).

[0219] Here, for example, when the control unit 13 of the management terminal device 101 of Bank A in Figure 1 generates the learned combination model for fraud detection of Figure 14 at SB3 in Figure 18, and identifies a transaction unit precision rate of "66%" and an account unit recall rate of "52%" at SB6 to SB7, it sends model identification information and second indicator-related information indicating these to the server device 2 at SB8, and the information shown in the top row of Figure 13(a) is stored.

[0220] ===Iteration=== The management terminal device 1 of each bank, including the management terminal device 1 of banks other than Bank A, then repeats the above process. In particular, SB1 acquires a different first model each time it is repeated and performs processing, thereby storing each piece of information in FIG. 13(a). For example, the information in the second and third rows of FIG. 13(a) is stored by processing in the management terminal device 101 of Bank A, the information in the fourth to sixth rows of FIG. 13(a) is stored by processing in the management terminal device 102 of Bank B, and the information in the seventh to tenth rows of FIG. 13(a) is stored by processing in the management terminal device 103 of Bank C. This concludes the explanation of the second storage process.

[0221] (Processing - Region model generation processing) Next, the region model generation process will be described. Fig. 19 is a flowchart of the region model generation process. The region model generation process is a process for generating a region model, and is a process that is executed mainly by the server device 2, for example.

[0222] The timing of executing this region model generation process is arbitrary, but for example, it may be repeated periodically (e.g., every six months or every year), or when an execution operation is performed by an administrator, or at other specified times, and the explanation will begin from the point where execution has started.

[0223] ===Region Model=== Fig. 20 is an explanatory diagram of a region model, and Fig. 21 is an explanatory diagram of explanatory variables. The "region model" is the aforementioned region-related trained model, which is a trained model generated by machine learning, and includes, for example, a first region model and a second region model as shown in the "region model" column in Fig. 20.

[0224] 20, the explanatory variables and response variables of each region model are shown in the "Explanatory Variables" and "Response Variables" columns, and for the convenience of explaining each variable, the number used to explain each variable is shown in the "Number" column. The explanatory variables and response variables of each region model shown in Fig. 20 are the explanatory variables corresponding to the explanatory variable information shown in Figs. 8 to 12 and the response variables corresponding to the response variable-related information in Figs. 13(a) and (b).

[0225] Details of the explanatory variables in FIG. 20 are shown in FIG.

[0226] That is, for example, the explanatory variables for each of the numbers "1," "5," "10," and "14" in Figure 20, which are "region-related explanatory variables," include the explanatory variables for numbers "1" to "8" in Figure 21 (such as "Saitama Prefecture" in "Account Proportion by Prefecture," and "Chiba Prefecture" in "Account Proportion by Prefecture").

[0227] Furthermore, for example, the explanatory variables for each of numbers "2," "6," "11," and "15" in Figure 20, which are "bank-related explanatory variables," include explanatory variables numbered "9" to "13" in Figure 21 (such as "proportion of corporate accounts" and "proportion of online banking operations").

[0228] Furthermore, for example, the explanatory variables for numbers "3," "7," "12," and "16" in Figure 20, which are "bank ranking-related explanatory variables," include the explanatory variables for numbers "14" to "15" in Figure 21 ("ranking in the number of monitoring operations in the past month," "ranking in the number of reports to the police in the past three months").

[0229] Furthermore, for example, the explanatory variables for each of the numbers "4," "8," "13," and "17" in Figure 20, which are "model-derived related explanatory variables," include the explanatory variables for the numbers "16" to "17" in Figure 21 ("number of accounts in fraudulent training data" and "number of decision trees").

[0230] Furthermore, for example, the explanatory variables for numbers "9" and "18" in Figure 20, which are the "related explanatory variables derived from other bank models," include the explanatory variables for numbers "18" to "20" in Figure 21 ("transaction unit precision rate," "account unit recall rate," and "transaction unit AUC").

[0231] ==Introduction line and other lines== The "introducing bank" and "other banks" shown in Figure 20 are concepts similar to the aforementioned "own bank" and "other banks," but are bank concepts based on the management terminal device 1 of each bank (i.e., the bank that is trying to introduce another bank's first model) that uses the region model to execute the processes related to the introduction of another bank's first model (the region model related determination process and other bank model selection process described below) when using the region model.

[0232] "Introducing bank" refers to a bank (e.g., Bank A) whose financial transactions are managed by the management terminal device 1 (e.g., management terminal device 101), which is the processing entity for the region model related determination process and the other bank model selection process, and "other banks" is a concept that refers to banks whose financial transactions are not managed by the management terminal device 1 (e.g., management terminal device 101) (e.g., Bank B, Bank C, Bank D, whose financial transactions are managed by other management terminal devices 102-104).

[0233] ==First Region Model== The "first region model" is a trained model related to the first region, and is a trained model that uses the explanatory variables indicated by numbers "1" to "9" in Figure 20 as explanatory variables and the trade unit conformance rate as the objective variable (see the "Objective Variable" column in Figure 20). In other words, the first region model is a trained model that outputs information (numerical information) corresponding to the trade unit conformance rate as the objective variable when information corresponding to each explanatory variable is input.

[0234] In addition, the "explanatory variables for the introduction row" numbered "1" to "4" in Figure 20 indicate explanatory variables into which each piece of information about the introduction row is input, and the specific contents of the explanatory variables are as described in the columns for each number and in Figure 21.

[0235] Furthermore, the "explanatory variables related to other banks" numbered "5" to "8" in Figure 20 indicate explanatory variables into which information about other banks is input, and the specific contents of the explanatory variables are as described in the columns for each number and in Figure 21. The explanatory variables numbered "1" to "4" and the explanatory variables numbered "5" to "8" are similar to each other except for the difference between using information from the introducing bank and using information from other banks.

[0236] In addition, the "explanatory variable related only to other banks" numbered "9" in Figure 20 indicates that this is an explanatory variable into which information about other banks is entered, and the specific contents of the explanatory variable are as described in the columns of each number and in Figure 21.

[0237] ==Second Region Model== The "second region model" is a second region-related trained model, and is a trained model that uses the explanatory variables indicated by numbers "10" to "18" in Figure 20 as explanatory variables (i.e., the same explanatory variables as the explanatory variables of the first region model) and the account-based recall rate as the objective variable (see the "Objective Variable" column in Figure 20). In other words, the first region model is a trained model that, when information corresponding to each explanatory variable is input, outputs information (numerical information) corresponding to the transaction-based precision rate according to the objective variable.

[0238] ===SC1=== 19, the control unit 23 of the server device 2 acquires the explanatory variable related information and the response variable related information stored in the recording unit 22 of FIG.

[0239] Here, for example, the region-related explanatory variable information in FIG. 8, the bank-derived related explanatory variable information in FIG. 9, the bank ranking related explanatory variable information in FIG. 10, the model-derived related explanatory variable information in FIG. 11, the other bank model-derived related explanatory variable information in FIG. 12, and the objective variable related information in FIG. 13 are acquired.

[0240] ===SC2=== In SC2 of FIG. 19, the control unit 23 of the server device 2 generates a first regional model and a second regional model by performing machine learning using the explanatory variable related information and the response variable related information acquired in SC1 as training data.

[0241] Fig. 22 is an explanatory diagram of the process. In Fig. 22, the explanatory variable-related information and response variable-related information used as training data, as exemplified in Figs. 8 to 13, are shown in the columns for explanatory variables and response variables. Since there are many variables for each, they are shown in five rows.

[0242] In Figure 22, the "Number" column shows numbers for ease of explanation. The "Introducing Bank Model Name" column and the "Other Bank Model Name" column show the same content as the columns of the same names in Figures 13(a) and (b). The "Introducing Bank" column shows the bank corresponding to the assumed introducing bank (here, for example, the bank where the second model or the first model shown in the "Introducing Bank Model Name" column was generated), and the "Other Bank" column shows the bank corresponding to the assumed other bank (here, for example, the bank where the first model shown in the "Other Bank Model Name" column was generated).

[0243] The "Explanatory Variables" columns in the first to fifth rows of Figure 22 show the explanatory variable-related information (information in Figures 8 to 12 obtained in SC1) used as training data corresponding to each explanatory variable described in Figures 20 and 21.

[0244] The "Objective Variable" column in the fifth row of FIG. 22 shows objective variable-related information (the information in FIG. 13 acquired in SC1) used as training data corresponding to each objective variable described in FIG.

[0245] == Training data == As the training data, for example, the combination of each piece of information numbered "1" shown in FIG. 22, the combination of each piece of information numbered "2", the combination of each piece of information numbered "3" to "5", etc. are used. In reality, a large amount of information, including information other than that shown in FIG. 22, is used as training data. In the following, the "number=" in each figure will be omitted as appropriate, and only numbers such as "1" will be used for explanation.

[0246] == Training data (case "1" in Figure 22) == We will explain the training data consisting of the combination of each piece of information ``1'' shown in Figure 22 (i.e., the training data corresponding to the introducing bank = ``Bank A'', introducing bank model name = ``Bank A_1(2)'', other bank = ``Bank B'', other bank model name = ``Bank B_1'').

[0247] =1st Region Model= The following information is used as training data for generating the first regional model.

[0248] <Training data for the objective variable (trading unit conformance rate)> The information on the "transaction unit conformance rate" in the top row of Figure 13(a) is used as training data for the "objective variable" in the fifth row of Figure 22 (see the "objective variable" column corresponding to the "first region model" in Figure 20).

[0249] <Training data for explanatory variables ("Region-related explanatory variables" in "Introduction row explanatory variables")> Information about "Bank A," the introducing bank in the region-related explanatory variable information in Figure 8, is used as training data for the "region-related explanatory variables" in the "explanatory variables related to introducing banks" in the first row of Figure 22 ("1" in Figure 20, "1" to "8" in Figure 21).

[0250] In detail, for each explanatory variable ("1" to "4" in Figure 21) of "Account ratio by prefecture" in the first row of Figure 22, information about "Bank A" in the top row of Figure 8(a) is used as training data.

[0251] Furthermore, for the "proportion of the total number of special frauds in each prefecture" in the first row of Figure 22 ("5" in Figure 21), the information on "Saitama Prefecture," which is the region with the highest degree of association with "Bank A" among the information in Figure 8(b), is used as training data. The degree of association between the introducing bank and the region may be determined based on any criteria, but in this embodiment, we will illustrate a case where it is determined based on the "proportion of accounts in each prefecture" in Figure 8(a). According to this criteria, "Saitama Prefecture" has the highest "proportion of accounts in each prefecture" for "Bank A," so "Saitama Prefecture" is determined as the region with the highest degree of association with "Bank A."

[0252] In addition, for each explanatory variable ("6" to "8" in Figure 21) of "Proportion of the number of fraud cases by type in each prefecture" in the first row of Figure 22, information on "Saitama Prefecture," which is the region with the highest correlation with "Bank A" among the information in Figure 8(c), is used as training data.

[0253] <Training data for explanatory variables ("Bank-related explanatory variables" in "Explanatory variables related to adopting banks")> As training data for the "bank-derived related explanatory variables" in the "explanatory variables related to the introducing bank" in the second row of Figure 22 ("2" in Figure 20, "9" to "13" in Figure 21), information about "Bank A," the introducing bank in the bank-derived related explanatory variable information in Figure 9 (information in the top row) is used.

[0254] <Training data for explanatory variables ("Bank ranking-related explanatory variables" in "Explanatory variables related to participating banks")> As training data for the "bank ranking related explanatory variables" in the "explanatory variables related to the introducing bank" in the second row of Figure 22 ("3" in Figure 20, "14" to "15" in Figure 21), information about "Bank A," the introducing bank in the bank ranking related explanatory variable information in Figure 10 (information in the top row) is used.

[0255] <Training data for explanatory variables ("Model-derived related explanatory variables" in "Explanatory variables for introduction rows")> As training data for the "model-derived related explanatory variables" ("4" in Figure 20, "16" to "17" in Figure 21) of the "explanatory variables related to the introducing bank" in the second row of Figure 22, information on "Bank A_1" (information in the top row), which is the model of the introducing bank (a model generated with "Bank A" as the introducing bank) in the model-derived related explanatory variable information in Figure 11, is used.

[0256] In addition, if there are multiple models in the introduction row in the model-derived related explanatory variable information in Figure 11, information about any one model may be used as training data, or statistical values ​​(e.g., mean values, median values, etc.) of information about each of the multiple models may also be used as training data.

[0257] <Training data for explanatory variables ("region-related explanatory variables" in "explanatory variables related to other banks")> Information about "Bank B," another bank in the region-related explanatory variable information in Figure 8, is used as training data for the "region-related explanatory variables" ("5" in Figure 20, "1" to "8" in Figure 21) in the "explanatory variables related to other banks" in the third row of Figure 22.

[0258] In detail, for each explanatory variable ("1" to "4" in Figure 21) of "Account ratio by prefecture" in the third row of Figure 22, information about "Bank B" in the second row of Figure 8(a) is used as training data.

[0259] Furthermore, for the "proportion of the total number of special frauds in each prefecture" in the third row of Figure 22 ("5" in Figure 21), information on "Chiba Prefecture," which is the region with the highest correlation with "Bank B" among the information in Figure 8(b), is used as training data.

[0260] In addition, for each explanatory variable ("6" to "8" in Figure 21) of "Proportion of the number of fraud cases by type in each prefecture" in the third row of Figure 22, information on "Chiba Prefecture," which is the region with the highest correlation with "Bank B" among the information in Figure 8(c), is used as training data.

[0261] <Training data for explanatory variables ("Bank-related explanatory variables" in "Other Bank-related explanatory variables")> As training data for the "bank-derived related explanatory variables" in the "explanatory variables related to other banks" in the fourth row of Figure 22 ("6" in Figure 20, "9" to "13" in Figure 21), information about "Bank B," which is another bank in the bank-derived related explanatory variable information in Figure 9 (information in the second row) is used.

[0262] <Training data for explanatory variables ("Bank ranking-related explanatory variables" in "Other banks' explanatory variables")> As training data for the "bank ranking related explanatory variables" in the "explanatory variables related to other banks" in the fourth row of Figure 22 ("7" in Figure 20, "14" to "15" in Figure 21), information about "Bank B," another bank in the bank ranking related explanatory variable information in Figure 10 (information in the second row) is used.

[0263] <Training data for explanatory variables ("Model-derived related explanatory variables" for "explanatory variables related to other banks")> As training data for the "model-derived related explanatory variables" of the "explanatory variables related to other banks" in the fourth row of Figure 22 ("8" in Figure 20, "16" to "17" in Figure 21), information on "Bank B_1," which is the model of the other bank (the same model as the model shown in the "Other bank model name" column) in the model-derived related explanatory variable information in Figure 11 (information in the second row) is used.

[0264] <Training data for explanatory variables ("Other bank model-derived related explanatory variables" in "Other bank-only explanatory variables")> As training data for the "relevant explanatory variables derived from other bank models" in the fifth row of Figure 22 ("9" in Figure 20, "18" to "20" in Figure 21), information on "Bank B_1," which is the other bank model (the same model as the model shown in the "other bank model name" column) in the relevant explanatory variable information derived from other bank models in Figure 12 (information in the second row) is used.

[0265] =Second Region Model= The following information is used as training data for generating the second region model.

[0266] <Training data for objective variable (account-level recall)> The information on the "account-level recall" in the top row of Figure 13(a) is used as training data for the "objective variable" in the fifth row of Figure 22 (see the "objective variable" column corresponding to the "second region model" in Figure 20).

[0267] <Training data for explanatory variables> The training data for the explanatory variables of the second region model uses the same information as that of the first region model.

[0268] == Training data (cases "2" to "5" in Figure 22) == In the cases of "2" to "4" in FIG. 22, similarly to the case of "1" in FIG. 22 described above, each variable-related information is used as training data for each variable of each region model.

[0269] As shown in "5" in Figure 22, if a model for "Other Bank" and "Other Bank Model Name" does not exist, predetermined information (e.g., "-1") is used as training data for each variable corresponding to the non-existent model for "Other Bank" and "Other Bank Model Name." In other words, the "-1" entered into the variables shown in each figure is for convenience and indicates that no information has been entered (the same applies to Figure 27 described below).

[0270] A large amount of information, including information other than that shown in Fig. 22, is used as training data. In other words, various information is used as training data, including information in which "Bank B," "Bank C," and "Bank D" are the adopting banks, information in which "Bank A," "Bank C," and "Bank D" are the other banks, information in which various first models are used as other bank models, etc.

[0271] ==Machine Learning== The explanatory variable related information and the response variable related information acquired in SC1 are used as training data for generating the first region model, and machine learning is performed to generate the first region model.

[0272] In addition, the explanatory variable related information and the target variable related information obtained in SC1 are used as training data for generating a second region model, and machine learning is performed to generate a second region model.

[0273] Here, for example, the model parameters (weights, biases, etc.) of the first and second region models shown in Figures 20 and 21 are determined based on the training data, and the first and second region models are generated.

[0274] As mentioned above, various information is used as training data to generate two models, the first region model and the second region model, and these two models can be used by any bank in Figure 1.

[0275] ===SC3=== At SC3 in FIG. 19, the control unit 23 of the server device 2 transmits the first and second region models generated at SC2 to the management terminal devices of the banks.

[0276] 2, for example, the first region model and the second region model are transmitted to the management terminal devices 101-104 of banks A-D, respectively. Then, the management terminal devices 101-104 receive the first region model and the second region model and store them in their own recording units 12, making them available for use as needed. This concludes the explanation of the region model generation process.

[0277] (Processing - Region model related determination processing) Next, the region model related determination process will be explained. Fig. 23 is a flowchart of the region model related determination process. The region model related determination process is a process for making various determinations regarding each region model generated and transmitted in the region model generation process (Fig. 19) and stored in each management terminal device 1, and is a process that is mainly executed by the management terminal device 1 of each bank, for example.

[0278] Since the region model generation process in Figure 19 is executed periodically, it is possible that multiple copies of each region model are stored. In this case, various determinations may be made regarding the most recent region model that was most recently stored.

[0279] The various judgments here include, for example, determining whether or not to use the first model included in the first model group in the recording unit 22 (Figure 2) of the server device 2 as the target for estimating the introduction effect, and determining whether or not the stored regional model is suitable for the introduction row.

[0280] In addition, the "implementation effect" is a concept that indicates the effect on the detection of fraudulent transactions of introducing, for example, the first model included in the first model group in the recording unit 22 of the server device 2 to the introducing bank (target financial institution), in other words, a concept regarding the performance of fraudulent transactions by introducing the first model.

[0281] The timing for executing this region model related determination process is arbitrary, but for example, it may be started after the region model generation process is executed and before the other bank model selection process described below is executed, and the description will begin from the point where execution has started.

[0282] Here, for example, a case will be described in which the target financial institution is "Bank A" and the other financial institutions are "Bank B," "Bank C," and "Bank D" (the same applies to the other bank model selection process described below).

[0283] ===Assumptions=== Fig. 24 is an explanatory diagram of a part of the information processing system, Fig. 25 is a diagram illustrating model-derived related explanatory variable information, and Fig. 26 is a diagram illustrating other bank model-derived related explanatory variable information. Note that Figs. 24 to 26 show the same configuration as Figs. 2 and 11 to 12, but the stored information is different.

[0284] It is assumed that the first and second region models have already been stored in the management terminal device 1 of each bank. Also, it is assumed that a relatively long time has passed since the execution of the region model association determination process of Fig. 19, and that a first model such as "Bank A_o" that is different from the time when each region model was generated (Fig. 2) is stored in the recording unit 22 of the server device 2 of Fig. 24, and that information corresponding to a timing different from the time when each region model was generated is also stored as explanatory variable association information in the recording unit 22 of the server device 2 of Fig. 1.

[0285] In addition, in Figure 24, "Bank A_o" indicates the first model generated by Bank A, "Bank B_m" and "Bank B_n" indicate the first model generated by Bank B, "Bank C_p" and "Bank C_q" indicate the first model generated by Bank C, and "Bank D_r" and "Bank D_s" indicate the first model generated by Bank D.

[0286] Furthermore, with regard to explanatory variable-related information, region-related explanatory variable information (FIG. 8), bank-derived related explanatory variable information (FIG. 9), and bank ranking-related explanatory variable information (FIG. 10) depend on the characteristics of the region and bank and may change through periodic updates, but the degree of change is generally not that great. Therefore, for the sake of convenience, this embodiment will be described assuming that the information shown in each of these figures is stored. Furthermore, it is assumed that the information in FIG. 25 is stored as model-derived related explanatory variable information, and that the information in FIG. 26 is stored as other bank model-derived related explanatory variable information.

[0287] ===SD1=== 23, the control unit 13 of the management terminal device 1 communicates with the server device 2 and acquires multiple first models included in the first model group in the recording unit 22. Specific examples are arbitrary, but may include acquiring first models generated by the bank itself and first models generated by other banks. For example, the models may be acquired randomly, or a predetermined number of models recently generated based on the date and time of generation may be acquired.

[0288] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 acquires "Bank A_o," "Bank B_m," "Bank B_n," "Bank C_p," "Bank C_q," "Bank D_r," and "Bank D_s" in Figure 24.

[0289] ===SD2=== In SD2 in Fig. 23, the control unit 13 of the management terminal device 1 communicates with the server device 2 and acquires explanatory variable related information from the recording unit 22 in Fig. 1. Note that, here, it may be configured to acquire all information, or to acquire only information related to the first model acquired in SD1.

[0290] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 acquires the region-related explanatory variable information in Figure 8, the bank-derived related explanatory variable information in Figure 9, the bank ranking-related explanatory variable information in Figure 10, the model-derived related explanatory variable information in Figure 25, and the other bank model-derived related explanatory variable information in Figure 26.

[0291] ===SD3=== In SD3 of Figure 23, the control unit 13 of the management terminal device 1 determines the predicted values ​​of the transaction unit conformance rate and account unit recall rate based on the first region model and second region model recorded therein and the information acquired in SD1 and SD2.

[0292] ==Illustration== Fig. 27 is an explanatory diagram of the process. In Fig. 27, the information input to each explanatory variable is shown in the corresponding explanatory variable column in the same format as Fig. 22. Since there are many variables, the diagram is divided into five columns.

[0293] In Figure 27, the "Introducing Bank Model Name" column shows the first model (the first model obtained in SD1), and the "Introducing Bank" column shows the bank that generated the first model (that is, the bank that identified each evaluation indicator in the related explanatory variable information derived from other bank models in Figure 26 for the first model). The "Other Bank" column and the "Other Bank Model Name" column do not show any information ("None" in Figure 27).

[0294] The "explanatory variables" columns in the first to fifth rows of Fig. 27 show the information related to each explanatory variable (information acquired by SD2) that is input to each explanatory variable explained in Fig. 20 and Fig. 21. Note that, as there is no particular information here for the "explanatory variables related to other banks" and "explanatory variables related only to other banks" in the third to fifth rows of Fig. 27, for example, "-1" (information corresponding to the predetermined information used when generating the region model (see "-1" in the third to fifth rows of "5" in Fig. 22)) is used.

[0295] ==Deviation information, reference values, and forecast values== The deviation information, the reference value, and the predicted value will be explained. Fig. 28 is an explanatory diagram of the process. Fig. 28(a) shows an example of the reference value, the predicted value, the error, and the deviation information, and Fig. 28(b) shows an example of the deviation information and whether the deviation is acceptable or not.

[0296] "Deviation information" is information used to make the various judgments mentioned above, and is, for example, numerical information indicating the degree of deviation between the reference values ​​of the transaction-unit conformance rate and account-unit recall rate for the first model and the predicted values ​​of the transaction-unit conformance rate and account-unit recall rate.

[0297] The "reference value" is a numerical value that serves as a standard for identifying information. For example, we will explain the case where the transaction-based precision rate and account-based recall rate shown in the related explanatory variable information derived from other bank models in Figure 26 are used as the reference value.

[0298] A "predicted value" is a numerical value obtained using the first region model and the second region model. For example, we will explain the case where the transaction unit precision rate and account unit recall rate, which are output as information corresponding to the objective variable when information about the first model is input into the explanatory variables of the first region model and the second region model, are used as predicted values.

[0299] ==About processing== Regarding the processing of SD3 in Figure 23, the control unit 13 of the management terminal device 1 inputs the explanatory variable related information acquired in SD2 for the first model acquired in SD1 into the "explanatory variables for the introducing bank" (numbers "1" to "4" in Figure 20) of the first region model, acquires information indicating the transaction unit conformance rate output in the objective variable of the first region model, and identifies the transaction unit conformance rate indicated by the acquired information as the predicted value. Note that "-1" is input into the "explanatory variables for other banks" and "explanatory variables for only other banks" of the first region model (the same applies to the second region model).

[0300] Here, for example, when the control unit 13 of Bank A's management terminal device 101 inputs explanatory variable related information regarding "Bank A_o" into the first region model as shown at "1" in Figure 27, if the first region model outputs "65%" as the transaction unit conformance rate, it identifies this "65%" as the predicted value of the transaction unit conformance rate as shown at "1" in Figure 28(a).

[0301] In addition, the control unit 13 of the management terminal device 1 inputs explanatory variable related information obtained in SD2 regarding the first model obtained in SD1 for the ``explanatory variables related to the introduced bank'' (numbers ``10'' to ``13'' in Figure 20) of the second region model, obtains information indicating the account-based recall output in the objective variable of the second region model, and identifies the account-based recall indicated by the obtained information as a predicted value.

[0302] Here, for example, when the control unit 13 of the management terminal device 101 of Bank A inputs explanatory variable related information regarding "Bank A_o" into the second region model as shown at "1" in Figure 27, if "55%" is output from the second region model as the account unit reproducibility, the control unit 13 identifies this "55%" as the predicted value of the account unit reproducibility as shown at "1" in Figure 28(a).

[0303] Then, similar processing is performed for other first models such as "Bank B_m" to identify the information shown in the "Predicted Value" column in FIG.

[0304] The explanatory variable related information (input information) for "Bank A_o" is the same as that for the training data in FIG. 22, but the following information is used.

[0305] <Region-related explanatory variables> The input information for the "region-related explanatory variables" in the "explanatory variables related to the introducing bank" in the first row of Figure 27 is information about "Bank A," the bank that generated "Bank A_o" in the region-related explanatory variable information in Figure 8.

[0306] Specifically, for each explanatory variable of "account ratio by prefecture" in the first row of FIG. 27, information about "Bank A" in the top row of FIG. 8(a) is used as input information.

[0307] In addition, for the "Proportion of total number of special frauds in each prefecture" in the first row of Figure 27, the information for "Saitama Prefecture," which is the region with the highest degree of association with "Bank A" among the information in Figure 8(b), is used as input information.

[0308] In addition, for each explanatory variable of the "proportion of fraud cases by type in each prefecture" in the first row of Figure 27, the information of "Saitama Prefecture," which is the region with the highest correlation with "Bank A" among the information in Figure 8(c), is used as input information.

[0309] <Bank-related explanatory variables> The information about "Bank A," the bank that generated "Bank A_o" in the bank-derived related explanatory variable information in Figure 9 (information in the top row), is used as input information for the "Bank-derived related explanatory variables" in the "Explanatory variables related to introducing banks" in the second row of Figure 27.

[0310] <Bank ranking-related explanatory variables> The information about "Bank A," the bank that generated "Bank A_o" in the bank ranking-related explanatory variable information in Figure 10 (information in the top row), is used as input information for the "bank ranking-related explanatory variables" in the "explanatory variables related to the introducing bank" in the second row of Figure 27.

[0311] <Model-derived related explanatory variables> As input information for the "model-derived related explanatory variables" of the "explanatory variables related to the introducing bank" in the second row of Figure 27, information about "Bank A_o," which is the model of the introducing bank in the model-derived related explanatory variable information in Figure 25 (information in the top row) is used.

[0312] <Other explanatory variables> As input information for the other explanatory variables in the third to fifth rows of FIG. 27, "-1" is used.

[0313] ==Interpretation of terms== As explained here, when the target financial institution is "Bank A," the information corresponding to "1" in the "region-related explanatory variables" in the top row of Figure 27 (such as "Saitama Prefecture" = "85%," "Chiba Prefecture" = "2.0"%) can be interpreted as corresponding to "first region-related information," and the information corresponding to "2" to "7" in the "region-related explanatory variables" in the top row of Figure 27 can be interpreted as corresponding to "second region-related information."

[0314] In this case, in the process of SD3 in Fig. 23, it can be interpreted that only one of the first region-related information or the second region-related information is input to each region model. That is, for example, in the case of "1" in Fig. 27, it can be interpreted that the first region-related information is input to each region model, and in the cases of "2" to "7" in Fig. 27, the second region-related information is input to each region model.

[0315] Furthermore, the deviation information described below is identified based on the predicted values ​​of the transaction-based conformance rate and account-based recall rate output from each regional model (SD5 described below), and various judgments (such as determining whether or not to use the first model as a target for estimating the introduction effect) (SD6 described below) are made based on the deviation information. Therefore, the transaction-based conformance rate and account-based recall rate can be said to be used to determine whether or not to use the first model as a target for estimating the introduction effect, and can be interpreted as corresponding to the "judgment index."

[0316] ===SD4=== In SD4 of Figure 23, the control unit 13 of the management terminal device 1 determines the standard values ​​for the transaction unit conformance rate and account unit recall rate based on the first region model and second region model recorded therein and the information acquired in SD1 and SD2.

[0317] Specifically, from the relevant explanatory variable information derived from other bank models obtained in SD2, information corresponding to the first model obtained in SD1 is obtained, and the transaction-level precision rate and account-level recall rate indicated by the obtained information are identified as reference values.

[0318] Here, for example, the control unit 13 of the management terminal device 101 of Bank A acquires the top row of "70%" and "51%" related to "Bank A_o" in FIG. 26, and identifies the acquired "70%" and "51%" as the transaction unit conformance rate and account unit recall rate shown in "1" in FIG. 28(a) as the reference values.

[0319] Then, similar processing is performed for other first models such as "Bank B_m" to identify the information shown in the "Reference Value" column in FIG.

[0320] ===SD5=== In SD5 of FIG. 23, the control unit 13 of the management terminal 1 identifies deviation information relating to the first model acquired in SD1 based on the processing results of SD3 and SD4.

[0321] Specifically, the error (absolute value of the difference) between each predicted value identified in SD3 and each reference value identified in SD4 is calculated, the root mean square of each calculated error is calculated, and the calculation result is identified as deviation information.

[0322] For example, as shown in "1" in Figure 8(a), the control unit 13 of Bank A's management terminal device 101 calculates "5%" as the error between the transaction unit conformance rate of "65%" (predicted value specified in SD3) and "70%" (reference value specified in SD4), and calculates "4%" as the error between the account unit recall rate of "55%" (predicted value specified in SD3) and "51%" (reference value specified in SD4).Then, it calculates "0.045", which is the square root of "(5% x 5% + 4% x 4%) / 2", and identifies this as deviation information.

[0323] Then, by performing similar processing for other first models such as "Bank B_m", the information shown in the "Error" column of 28 is calculated and the information shown in the "Deviation Information" column is identified. Note that the calculation formula shown here is an example, and any other calculation may be performed as long as it meets the definition of deviation information.

[0324] ===SD6=== In SD6 of FIG. 23, the control unit 13 of the management terminal 1 makes various decisions based on the processing results of SD5.

[0325] Specifically, based on deviation information regarding other banks' first models, a decision is made as to whether or not to use the first model as the target for estimating the effects of implementation (also referred to as a "decision regarding the target for estimation"), and based on deviation information regarding the bank's own first model, a decision is made as to whether or not the regional model is suitable for the adopting bank (the bank itself) (also referred to as a "decision regarding the regional model").

[0326] ==Determination of the Estimation Target== Regarding the determination of the estimation target, the first model of another bank (the first model generated by the other bank) is identified from the first models obtained in SD1, the deviation information identified in SD5 is identified regarding the identified first model of the other bank, and the identified deviation information is compared with a threshold value (a value to be compared with the deviation information, which is a predetermined threshold value).

[0327] If the deviation information is less than the threshold, it is determined that the degree of deviation between the predicted value and the reference value is relatively small and acceptable, and it is then determined that the first model described above should be used to estimate the introduction effect.If the deviation information is equal to or greater than the threshold, it is determined that the degree of deviation between the predicted value and the reference value is relatively large and unacceptable, and it is then determined that the first model described above should not be used to estimate the introduction effect.

[0328] The threshold used here may be determined by any method, for example, by an administrator through any simulation or experiment, or by the following method. Specifically, a large amount of deviation information may be calculated through a simulation based on past information, a 95% confidence interval for this large amount of deviation information may be identified, and a numerical value corresponding to the identified 95% confidence interval may be used as the threshold. However, the present invention is not limited to this, and the threshold may be set based on experience, or may be set by other calculations.

[0329] Here, for example, if "0.09" is set as the threshold information, the control unit 13 of Bank A's management terminal device 101 identifies the deviation information "2" to "7" in Figure 28(a) as the deviation information of the first model of other banks, compares each deviation information with the threshold indicated by the threshold information, and determines that the deviation information for "Bank B_m," "Bank B_n," and "Bank C_p," as shown at "2" to "4" in Figure 28(b), is above the threshold and therefore unacceptable (i.e., "TRUE" in Figure 28(b)), and therefore does not include them in the estimation of the introduction effect.

[0330] Furthermore, for example, as shown in "5" to "7" in Figure 28(b), for "Bank C_q," "Bank D_r," and "Bank D_s," the deviation information is below the threshold, so it is determined to be acceptable (i.e., "FALSE" in Figure 28(b)), and it is decided that they will be included in the estimation of the introduction effect.

[0331] ==Region Model Decisions== For the judgment regarding the regional model, the bank's first model (the first model generated by another bank) is identified from the first models obtained in SD1, and the deviation information identified in SD5 regarding the identified bank's first model is identified, and the identified deviation information is compared with the threshold value.

[0332] If the deviation information is less than the threshold, the degree of deviation between the predicted value and the reference value is determined to be relatively small and acceptable, and the first and second region models stored in the bank are determined to be suitable for the bank that is introducing the system. If the deviation information is equal to or greater than the threshold, the degree of deviation between the predicted value and the reference value is determined to be relatively large and unacceptable, and the first and second region models stored in the bank are determined to be unsuitable for the bank that is introducing the system.

[0333] The threshold value used here may be set as described above, and may be the same as or different from the threshold value used in the "determination regarding the estimation target."

[0334] Here, for example, if the threshold information is set to "0.09", the control unit 13 of Bank A's management terminal device 101 identifies the deviation information of "1" in Figure 28(a) as the deviation information of its own bank's first model, compares the deviation information with the threshold indicated by the threshold information, and determines that the deviation information is acceptable (i.e., "FALSE" in Figure 28(b)) because it is less than the threshold, as shown by "1" in Figure 28(b), and then determines that the first region model and second region model stored in its own bank are suitable for the adopting bank (its own bank).

[0335] If there are multiple first models in the bank, each region model may be determined to be suitable only when it is determined that all of the first models are acceptable, or each region model may be determined to be suitable only when it is determined that a predetermined percentage or more of the first models are acceptable (for example, if there are 10 first models, 8 or more of the first models, which is 80% or more). This concludes the explanation of the region model association determination process.

[0336] (Processing - Other Line Model Selection Processing) Next, the other bank model selection process will be described. Figure 29 is a flowchart of the other bank model selection process. The other bank model selection process is a process that estimates the introduction effect on the detection of fraudulent transactions of introducing the first model of another bank to the introducing bank, and selects the first model to be introduced. For example, this process is mainly executed by the management terminal device 1 of each bank.

[0337] The timing for executing this region model relation determination process is arbitrary, but for example, it is assumed that the execution starts after the region model relation determination process has been executed, and the description will begin from the point where the execution starts.

[0338] Although not limited to this, if it is desired to use each region model that is suitable for the introducing bank, for example, in the judgment regarding the region model in SD6 of Figure 23, if it is determined that the first region model and the second region model are suitable for the introducing bank (own bank), the execution of the other bank model selection process may be started, and processing may be performed using the first region model and the second region model that are determined to be suitable.

[0339] If the region model is not determined to be suitable in SD6 of FIG. 23, a region model generation request signal may be sent to the server device 2. When the server device 2 receives the signal, the server device 2 may execute the region model generation process of FIG. 19 to newly generate each region model. In this case, the variable-related information acquired in SC1 may be at least partially different from the information acquired previously, thereby generating each region model different from the previous one. Alternatively, if multiple region models are stored in each management terminal device 1, the region model association determination process of FIG. 23 may be performed for other region models. If a region model is determined to be suitable for the introducing bank, the other bank model selection process may be started, and processing may be performed using the first and second region models determined to be suitable.

[0340] ===SE1=== 29, the control unit 13 of the management terminal device 1 communicates with the server device 2, and acquires the first model (the first model of another bank) to be used as the target for estimating the introduction effect from among the first model group in the recording unit 22. Specifically, in SD6 of Fig. 23, the first model determined to be used as the target for estimating the introduction effect is acquired.

[0341] Here, for example, the control unit 13 of the management terminal device 101 of Bank A acquires “Bank C_q,” “Bank D_r,” and “Bank D_s” (see “5” to “7” in FIG. 28(b)) from the first model group in FIG. 24.

[0342] ===SE2=== 29, the control unit 13 of the management terminal device 1 communicates with the server device 2 in the same manner as SD2 of FIG. 23, and acquires the explanatory variable related information of the recording unit 22 of FIG.

[0343] Here, for example, the control unit 13 of the management terminal device 101 of Bank A in Figure 1 acquires the region-related explanatory variable information in Figure 8, the bank-derived related explanatory variable information in Figure 9, the bank ranking-related explanatory variable information in Figure 10, the model-derived related explanatory variable information in Figure 25, and the other bank model-derived related explanatory variable information in Figure 26.

[0344] In reality, it is expected that the information will differ from what is shown in each figure depending on the timing of execution of the other bank model selection process in Figure 29, but for convenience of explanation, we will use what is shown in the figure.

[0345] ===SE3=== In SE3 of Figure 23, the control unit 13 of the management terminal device 1 determines the predicted values ​​of the transaction unit conformance rate and account unit recall rate based on the first region model and second region model recorded therein and the information acquired in SE1 and SE2.

[0346] ==Illustration== Fig. 30 is an explanatory diagram of the process. In Fig. 30, information to be input for each explanatory variable is shown in the corresponding column for each explanatory variable in a format similar to that of Fig. 22 or 27. Since there are many variables, the diagram is divided into five columns.

[0347] In Figure 30, the information in the "Implementing bank" column indicates the implementing bank (own bank), the "Implementing bank's model name" column indicates the ungenerated second model of the own bank, the "Other bank's model name" column indicates the other bank's first model (the first model obtained in SE1), and the "Other bank" column indicates the bank that generated the first model (i.e., the bank that identified each evaluation indicator of the other bank's model-derived related explanatory variable information in Figure 26 for the first model).

[0348] Note that the "ungenerated second model of the bank" is referred to as "Bank A_o(2)" for the sake of convenience, and is one that has not yet been generated. For the sake of convenience, it is shown in the "Introducing bank model name" column in Figure 30, but this "Introducing bank model name" column may be omitted.

[0349] The "explanatory variable" columns in the first to fifth rows of FIG. 30 show each explanatory variable related information (information acquired in SE2) input to each explanatory variable explained in FIG. 20 and FIG.

[0350] ==Baseline value, predicted value, improvement rate== The deviation information, reference value, and predicted value will be explained. FIG. 31 is an explanatory diagram of the process. Note that FIG. 31 illustrates the reference value, predicted value, improvement rate of trade-unit precision rate, improvement rate of account-unit recall rate, and overall improvement rate. Note that the reference value and predicted value are the same as those explained in FIG. 28(a). Furthermore, each improvement rate will be described later.

[0351] ==About processing== Regarding the processing of SE3 in Figure 29, the control unit 13 of the management terminal device 1 inputs explanatory variable related information obtained in SE2 regarding the first model obtained in SE1 for all explanatory variables of the first region model (numbers "1" to "9" in Figure 20), obtains information indicating the trading unit conformance rate output in the objective variable of the first region model, and identifies the trading unit conformance rate indicated by the obtained information as a predicted value.

[0352] Here, for example, when the control unit 13 of the management terminal device 101 of Bank A inputs explanatory variable related information regarding "Bank C_q" into the first region model as shown at "1" in Figure 30, if "71%" is output from the first region model as the transaction unit conformance rate, this "71%" is identified as the predicted value of the transaction unit conformance rate as shown at "1" in Figure 31.

[0353] In addition, the control unit 13 of the management terminal device 1 inputs explanatory variable related information acquired in SE2 regarding the first model acquired in SE1 for all explanatory variables of the second region model (numbers "10" to "18" in Figure 20), acquires information indicating the account-based recall output in the objective variable of the second region model, and identifies the account-based recall indicated by the acquired information as a predicted value.

[0354] Here, for example, when the control unit 13 of the management terminal device 101 of Bank A inputs explanatory variable related information regarding "Bank C_q" into the second region model as shown at "1" in Figure 30, if "53%" is output from the second region model as the account unit reproducibility, the control unit 13 identifies this "53%" as the predicted value of the account unit reproducibility as shown at "1" in Figure 31.

[0355] Then, similar processing is performed on the first models of "Bank D_r" and "Bank D_s" to identify the information shown in the "Predicted Value" column in FIG.

[0356] The explanatory variable related information (input information) for "Bank C_q" is the same as the training data in FIG. 22 or the input information in FIG. 27, but the following information is used.

[0357] <Explanatory variables for introduction rows - Region-related explanatory variables> As input information for the "region-related explanatory variables" in the "explanatory variables related to the introducing bank" in the first row of Figure 30, information about "Bank A," the introducing bank (own bank) in the region-related explanatory variable information in Figure 8, is used.

[0358] Specifically, for each explanatory variable of "account ratio by prefecture" in the first row of FIG. 30, information about "Bank A" in the top row of FIG. 8(a) is used as input information.

[0359] In addition, for the "Proportion of total number of special frauds in each prefecture" in the first row of Figure 30, the information for "Saitama Prefecture," which is the region with the highest correlation with "Bank A" among the information in Figure 8(b), is used as input information.

[0360] In addition, for each explanatory variable of the "proportion of fraud cases by type in each prefecture" in the first row of Figure 30, the information on "Saitama Prefecture," which is the region with the highest correlation with "Bank A" among the information in Figure 8(c), is used as input information.

[0361] <Explanatory variables for introducing banks - bank-related explanatory variables> As input information for the "bank-derived related explanatory variables" in the "explanatory variables related to the adopting bank" in the second row of Figure 30, information about "Bank A," the adopting bank (own bank) in the bank-derived related explanatory variable information in Figure 9 (information in the top row) is used.

[0362] <Explanatory variables related to participating banks - explanatory variables related to bank rankings> The information about "Bank A," the introducing bank (own bank) in the bank ranking-related explanatory variable information in Figure 10 (information in the top row) is used as input information for the "bank ranking-related explanatory variables" in the "explanatory variables related to the introducing bank" in the second row of Figure 30.

[0363] <Explanatory variables for the introduction line - Model-derived related explanatory variables> As input information for the "model-derived related explanatory variables" of the "explanatory variables related to the introducing bank" in the second row of Figure 30, information (top row information) related to "Bank A_o," which is the first model related to "Bank A," the introducing bank (own bank) in the model-derived related explanatory variable information in Figure 25, is used.

[0364] In addition, in the model-derived related explanatory variable information in Figure 25, if there are multiple first models related to "Bank A," information about any one of them may be used, or a statistical value (e.g., average value, median value, etc.) may be used (the same applies to the processing in SE4 described below).

[0365] <Explanatory variables for other banks - Region-related explanatory variables> As input information for the "region-related explanatory variables" of the "explanatory variables related to other banks" in the third row of FIG. 30, information about "Bank C," which is the other bank in the region-related explanatory variable information in FIG. 8, is used.

[0366] Specifically, for each explanatory variable of "account ratio by prefecture" in the third row of FIG. 30, information about "Bank C" in the third row of FIG. 8(a) is used as input information.

[0367] Furthermore, for the "Proportion of total number of special frauds in each prefecture" in the third row of Figure 30, the information for "Kanagawa Prefecture," which is the region with the highest correlation with "Bank C" among the information in Figure 8(b), is used as input information.

[0368] In addition, for each explanatory variable of "Proportion of fraud cases by type in each prefecture" in the third row of Figure 30, the information of "Kanagawa Prefecture," which is the region with the highest correlation with "Bank C" among the information in Figure 8(c), is used as input information.

[0369] <Explanatory variables related to other banks - explanatory variables related to banks> The information about "Bank C," which is another bank in the bank-derived related explanatory variable information in Figure 9 (information in the third row) is used as input information for the "bank-derived related explanatory variables" in the "explanatory variables related to other banks" in the fourth row of Figure 30.

[0370] <Explanatory variables related to other banks - explanatory variables related to bank rankings> The information about "Bank C," which is another bank in the bank ranking-related explanatory variable information in Figure 10 (information in the third row) is used as input information for the "bank ranking-related explanatory variables" of the "explanatory variables related to other banks" in the fourth row of Figure 30.

[0371] <Explanatory variables for other banks - Model-derived related explanatory variables> As input information for the "model-derived related explanatory variables" of the "explanatory variables related to other banks" in the fourth row of Figure 30, information on "Bank C_q" (information in the fifth row), which is the model of the other bank (the same model as the model shown in the "Other bank model name" column) in the model-derived related explanatory variable information in Figure 25, is used.

[0372] <Explanatory variables related to other banks only - Relevant explanatory variables derived from other banks' model> As input information for the "relevant explanatory variables derived from other bank models" in the "relevant explanatory variables related only to other banks" in the fifth row of Figure 30, information on "Bank C_q" (information in the fifth row), which is the model of the other bank (the same model as the model shown in the "other bank model name" column) in the relevant explanatory variable information derived from other bank models in Figure 26, is used.

[0373] ==Interpretation of terms== As explained here, when the target financial institution is "Bank A," the information corresponding to the "region-related explanatory variables" in the "explanatory variables for the adopting bank" in the top row of Figure 30 (such as "Saitama Prefecture" = "85%," "Chiba Prefecture" = "2.0"%) can be interpreted as corresponding to the "first region-related information," and the information corresponding to the "region-related explanatory variables" in the "explanatory variables for other banks" in the third row of Figure 30 can be interpreted as corresponding to the "second region-related information."

[0374] In this case, in the process of SE3 in FIG. 29, it can be interpreted that both the first region-related information and the second region-related information are input to each region model.

[0375] Furthermore, because the improvement rate described below is determined based on the predicted values ​​of the transaction-based precision rate and account-based recall rate output from each regional model (see section SE5 below), the transaction-based precision rate and account-based recall rate can be said to be used to estimate the implementation effect and can be interpreted as corresponding to "estimation indicators." In this case, the transaction-based precision rate can be interpreted as corresponding to the "first indicator corresponding to the precision rate based on fraudulent transactions," and the account-based recall rate can be interpreted as corresponding to the "second indicator corresponding to the recall rate based on financial accounts associated with fraudulent transactions."

[0376] ===SE4=== In SE4 of Figure 29, the control unit 13 of the management terminal device 1 determines the standard values ​​for the transaction unit conformance rate and account unit recall rate based on the first region model and second region model recorded therein and the information acquired in SE1 and SE2.

[0377] Specifically, from the related explanatory variable information derived from other bank models obtained in SE2, information corresponding to the first model of the introducing bank (own bank) is obtained, and the transaction-based conformance rate and account-based recall rate indicated by the obtained information are identified as reference values.

[0378] Here, for example, the control unit 13 of the management terminal device 101 of Bank A acquires "70%" and "51%" in the top row related to "Bank A_o" in FIG. 26, and identifies the acquired "70%" and "51%" as the transaction unit conformance rate and account unit recall rate shown in "1" in FIG. 31 as reference values.

[0379] Then, similar processing is performed on the first models of "Bank D_r" and "Bank D_s" to identify the information shown in the "Reference Value" column of FIG.

[0380] ===SE5=== In SE5 of Figure 29, the control unit 13 of the management terminal device 1 determines the improvement rate of the transaction-unit precision rate, the improvement rate of the account-unit recall rate, and the overall improvement rate for the first model obtained in SE1 based on the processing results of SE3 and SE4.

[0381] ==Improvement rate of trading unit accuracy== The "improvement rate of transaction-unit conformance rate" is information indicating the estimated implementation effect of introducing another bank's first model, for example, information indicating the degree of improvement in transaction-unit conformance rate.

[0382] Specifically, the process of identifying the improvement rate of the transaction unit conformance rate is to calculate "(predicted value - standard value) / standard value" for the predicted value and standard value of the transaction unit conformance rate identified in SE3 and SE4, and identify the result of the calculation as the improvement rate of the transaction unit conformance rate.

[0383] Here, for example, as shown at "1" in Figure 31, the control unit 13 of Bank A's management terminal device 101 calculates "(71%-70%)÷70%" for the predicted value of the transaction unit conformance rate = "71%" and the reference value = "70%", and identifies the calculation result of "0.01428..." (i.e., 1.43%) as the improvement rate of the transaction unit conformance rate.

[0384] Since the transaction-unit matching rate indicates performance, this process can be said to estimate the introduction effect of, for example, introducing the first model, "Bank C_q," to the adopting bank, Bank A, such that the transaction-unit matching rate improves by "1.43%."

[0385] ==Improvement rate of account-level recall== The "improvement rate of account-based recall" is information indicating the estimated implementation effect of introducing another bank's first model, for example, information indicating the degree of improvement in account-based recall.

[0386] Specifically, the process of identifying the improvement rate of the account-level recall rate involves calculating "(predicted value - reference value) / reference value" for the predicted value and reference value of the account-level recall rate identified in SE3 and SE4, and identifying the result of the calculation as the improvement rate of the transaction-level precision rate.

[0387] Here, for example, as shown at "1" in Figure 31, the control unit 13 of the management terminal device 101 of Bank A calculates "(53%-51%)÷51%" for the predicted value of the account-based recall rate = "53%" and the reference value = "51%, and identifies the calculation result of "0.03921..." (i.e., 3.92%) as the improvement rate of the account-based recall rate.

[0388] Since the account-level recall rate indicates performance, this process can be said to estimate the introduction effect of, for example, introducing the first model, "Bank C_q," to the adopting bank, Bank A, such that the account-level recall rate improves by "3.92%."

[0389] ==Overall improvement rate== The "overall improvement rate" is information indicating the estimated implementation effect of introducing another bank's first model, for example, information indicating the degree of overall improvement taking into account both the transaction-level matching rate and the account-level recall rate.

[0390] Specifically, regarding the process of determining the overall improvement rate, a calculation is performed to find the average of the improvement rate of the transaction-based conformance rate and the improvement rate of the account-based recall rate determined above (i.e., for example, "(improvement rate of transaction-based conformance rate + improvement rate of account-based recall rate) ÷ 2"), and the result of the calculation is determined as the overall improvement rate.

[0391] Here, for example, as shown in "1" in Figure 31, the control unit 13 of Bank A's management terminal device 101 calculates "(1.43% + 3.92%) ÷ 2" for the improvement rate of transaction unit conformance rate = "1.43%" and the improvement rate of account unit recall rate = "3.92%, " and identifies the calculated result of "2.68%" as the overall improvement rate.

[0392] Since the transaction-based matching rate and account-based recall rate indicate performance, this process can be said to estimate the introduction effect of, for example, introducing the first model, "Bank C_q," to the adopting bank, Bank A, such that the combined performance of the transaction-based matching rate and account-based recall rate improves by 3.92%.

[0393] Then, by performing similar processing on the first models of "Bank D_r" and "Bank D_s", the information shown in the "Improvement rate of transaction-unit precision rate", "Improvement rate of account-unit recall rate", and "Overall improvement rate" columns in Figure 31 is identified.

[0394] The calculation formulas for each improvement rate shown here are merely examples, and any other calculation may be performed as long as it satisfies the definition of the improvement rate.

[0395] ===SE6=== In SE6 of FIG. 29, the control unit 13 of the management terminal device 1 selects one first model from the first models of other banks identified in SE1, based on the overall improvement rate identified in SE5.

[0396] Here, for example, the control unit 13 of the management terminal device 101 of Bank A focuses on the overall improvement rate shown in the "Overall Improvement Rate" column of Figure 31 and selects "Bank D_r", which is the first model with the largest overall improvement rate.

[0397] The selection criteria here are arbitrary. For example, the selection may be based on the recall rate of another type among the three types of improvement rates shown in FIG. 31, or the selection may be based on the improvement rates of any two or all types (three types).

[0398] Subsequent processing is optional, but for example, the control unit 13 of the management terminal device 101 of Bank A generates a second model by performing machine learning using the selected first model "Bank D_r" and the transaction-related information (FIGS. 3 and 4) of Bank A, the introducing bank (own bank), as training data, to generate a trained combination model for fraud detection (FIG. 5). The generated trained combination model for fraud detection may then be recorded in the recording unit 12 so that it can be used in the aforementioned fraud detection processing on the financial institution side. This concludes the description of the other bank model selection processing.

[0399] (Transaction-related information) In each of the above processes, the transaction-related information in Figures 3 and 4 is not sent to other banks and is used only within each bank, so transaction-related information that also falls under personal information of transactors is not shared among multiple banks and is used appropriately. Assuming that such transaction-related information is used appropriately, it becomes possible to estimate the introduction effect of Model 1 at other banks.

[0400] (Effects of the embodiment) According to this embodiment, by estimating the introduction effect based on the first region-related information and the second region-related information, it is possible to estimate the introduction effect taking into account, for example, the characteristics of the region, and therefore it is possible to appropriately estimate the introduction effect of the first model (trained model for detection).

[0401] Furthermore, by estimating the introduction effect based on each region model (region-related trained model), it becomes possible to easily estimate the introduction effect of, for example, the first model.

[0402] In addition, by including a first region model (first region-related trained model) that outputs a first indicator corresponding to the precision rate based on fraudulent transactions as an estimation indicator, and a second region model (second region-related trained model) that outputs a second indicator corresponding to the recall rate based on financial accounts associated with fraudulent transactions as an estimation indicator, it becomes possible to more appropriately estimate the effect of introducing the first model, for example, by using two types of region models.

[0403] Furthermore, by estimating the introduction effect based on the first region-related information, the second region-related information, and the feature-related information, it is possible to take into account, for example, the features of the first model, thereby making it possible to more appropriately estimate the introduction effect of the first model.

[0404] Furthermore, by determining whether or not to use the first model as a target for estimating the introduction effect, it is possible to narrow down the targets for estimation and efficiently estimate the introduction effect, for example, by using only appropriate first models as a target for estimating the introduction effect.

[0405] In addition, since each region model is configured to output an estimation indicator or a judgment indicator, it is possible to use each region model, for example, to both determine whether to make it an estimation target and estimate the introduction effect.

[0406] [Modification of the First Embodiment] Although the first embodiment of the present invention has been described above, the specific configuration and means of the present invention can be modified and improved as desired within the scope of the technical concept of the present invention as set forth in the claims. Such modifications will be described below.

[0407] (About the problem to be solved and the effects of the invention) First, the problems to be solved by the invention and the effects of the invention are not limited to the above, and may vary depending on the implementation environment of the invention and the details of the configuration, and only some of the above problems may be solved or only some of the above effects may be achieved. Furthermore, the problems solved by the technology of the above embodiments may be interpreted as the problems to be solved by the invention.

[0408] (Regarding decentralization and integration) Furthermore, the electrical components described above are functional concepts and do not necessarily have to be physically configured as shown in the drawings. In other words, the specific form of distribution or integration of each part is not limited to that shown in the drawings, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, the term "device" in this application is not limited to a single device, but includes a device configured from multiple devices.

[0409] (shape, numbers, structure, time series) The components illustrated in the embodiments and drawings may be modified and improved as desired within the scope of the technical concept of the present invention in terms of shape, numerical value, or the structure or chronological relationship of multiple components.

[0410] (About explanatory variables and target variables) Furthermore, the explanatory variables and response variables in FIGS. 20 and 21 described in the above embodiments are merely examples, and other variables may be used as explanatory variables or response variables, or some variables may be omitted.

[0411] (Region Model (Part 1)) In the above-mentioned embodiment 1, as shown in FIG. 20, a case where two region models, a first region model and a second region model, are used is described, but this is not limited to this, and it is also possible to configure the system to use only one of the region models, or to configure the system to use three or more region models including other region models.

[0412] (Region Model (Part 2)) Furthermore, each regional model generated by the regional model generation process of Figure 19 in the first embodiment can be used by banks in regions other than the region in which the information was used as training data.

[0413] For example, each regional model may be generated using only information related to Saitama, Chiba, and Kamikawa prefectures, and then the regional model may be used to estimate the introduction effect of the first model generated at a bank that is highly related to Tokyo. That is, for example, explanatory variables for the regional model for all prefectures in Japan may be set, and then each regional model may be generated using explanatory variable-related information and objective variable-related information of the regional model for some prefectures as training data, and then the introduction effect of the first model generated at a bank that is highly related to prefectures that were not used as training data may be estimated.

[0414] (Regarding processes and steps) Furthermore, the processes or steps described in the first embodiment may be modified, omitted, or added. For example, the region model related determination process in Fig. 23 may be omitted, and the other bank model selection process in Fig. 29 may be performed without making a determination regarding the estimation target and a determination regarding the region model. In this case, SE1 in Fig. 29 may be configured to acquire a plurality of arbitrary first models (first models of other banks) from the first model group in the recording unit 22 and then perform each process.

[0415] Furthermore, SD3 in Fig. 23 may be configured to perform processing assuming that the introducing bank (own bank) is indicated in the information in the "Introducing Bank" column in Fig. 27. In this case, if the control unit 13 of Bank A's management terminal device 101 is the processing subject, all of the "Introducing Bank" columns in Fig. 27 will be "Bank A," and the information in the "Region-related explanatory variables" in the first row, the information in the "Bank origin-related explanatory variables" in the second row, and the information in the "Bank ranking-related explanatory variables" will all be the same as when they are all "1."

[0416] (About the trained combination model for fraud detection) FIG. 32 is an explanatory diagram of a trained combination model for fraud detection. In the first embodiment, a different form of trained combination model for fraud detection may be used instead of the trained combination model for fraud detection of FIG. 5. For example, a first model may be added to the trained combination model for fraud detection of FIG. 5, and the AI ​​scores of multiple first models may also be input to a second model, as shown in FIG. 32(a). In this case, the number of first models is not limited to three, but may be two, four, or more. Alternatively, as shown in FIG. 32(b), a second model may be configured to have the same input and output as the first model, and these may be used in combination. That is, in the "calculation" of FIG. 32, a predetermined calculation (e.g., a calculation to calculate an average value) may be performed on multiple input AI scores, and the calculation result may be output. The number of first models and second models in the trained combination model for fraud detection is not limited to those shown in each figure, and may be arbitrary.

[0417] (Embodiment 2) Next, a description will be given of a second embodiment. The second embodiment relates to an evaluation system, an evaluation program, and an evaluation method.

[0418] (Basic concept) First, a basic concept: The evaluation system of the present invention is a system for evaluating detection systems for detecting fraudulent transactions in financial transactions associated with financial accounts.

[0419] A "detection system" is a system for detecting fraudulent transactions in financial transactions associated with a financial account. Specifically, it is a concept that includes various elements for detecting fraudulent transactions, such as a system that detects fraudulent transactions using a trained detection model, and a system that detects fraudulent transactions using any program other than a trained detection model (i.e., a rule-based system that detects fraudulent transactions using specified rules).

[0420] "Evaluating a detection system" may be interpreted as, for example, evaluating the accuracy of all or some of the elements of a detection system.

[0421] A "trained model for detection" is a model for detecting fraudulent transactions, and is a concept that includes, for example, models generated by machine learning.

[0422] "Financial transactions" is a concept that refers to financial transactions, and more specifically, to transactions conducted using a computer. Financial transactions include, for example, transactions conducted using an ATM (Automated Teller Machine) installed in a specific store, and transactions conducted using a network-based transaction function including so-called Internet banking functions, and examples include deposits, withdrawals, and transfers of currency. Furthermore, financial transactions may be interpreted as including, for example, transactions related to currency as well as transactions related to financial products other than currency, such as stocks or futures. While these "financial transactions" are generally genuine transactions, fraudulent transactions are also anticipated.

[0423] "Fraudulent transactions" are fraudulent financial transactions, and include, for example, transactions that are not in line with the original intentions of the transactors, and include, for example, transactions related to criminal activities such as fraud (including bank transfer fraud) and illegal lending. Incidentally, fraudulent transactions may also be interpreted as meaning illegal transactions, for example.

[0424] A "genuine transaction" is a genuine (legitimate) transaction in financial transactions, and is a concept that includes, for example, transactions that are in line with the original intentions of the transactors, and one example is a concept that includes transactions that are not related to criminal activity. Note that a genuine transaction may also be interpreted as indicating a lawful (or legal) transaction, for example.

[0425] In the following embodiment, an example will be described in which the accuracy of fraud detection (detection of fraudulent transactions) using a trained model for detection is evaluated.

[0426] (composition) First, an information processing system according to this embodiment will be described. Fig. 33 is a block diagram of an information processing system according to this embodiment.

[0427] The information processing system 9100 is a system including an evaluation system, and includes, for example, a transaction terminal device 91, a management terminal device 92, and a server device 93.

[0428] (Configuration - Transaction Terminal Device) The transaction terminal device 91 in Figure 33 is a device (including a computer) on the financial institution's side, for example, a computer for conducting financial transactions, and one example is a device that is capable of communicating with an ATM installed in a convenience store or other location not shown, or a terminal used by the transactor (a personal computer, tablet terminal, smartphone, etc.) and processes information regarding financial transactions.

[0429] (Configuration-Management Terminal Device) The management terminal device 92 in Figure 33 is a device that manages financial transactions (e.g., a device including a computer having a recording unit and a control unit), for example, a device that manages financial transactions processed by the transaction terminal device 91, and as an example, a device that detects fraudulent transactions in the financial transactions.

[0430] The transaction terminal device 91 and the management terminal device 92 described here are merely examples and may be changed as desired. For example, the transaction terminal device 91 and the management terminal device 92 may be integrated, or a known system configuration provided in an existing financial institution may be applied. Also, some or all of the functions of the transaction terminal device 91 and the management terminal device 92 may be incorporated into a server device 93, which will be described later.

[0431] (Configuration - Server Device) The server device 93 is an evaluation system, and includes, for example, a communication unit 931, a recording unit 932, and a control unit 933.

[0432] (Configuration - Server Device - Communication Unit) 33 is a communication means for communicating with an external device (for example, the management terminal device 92 or another device not shown). The specific type and configuration of this communication device 931 are arbitrary, but it can be configured using, for example, a known communication circuit or the like.

[0433] (Configuration - Server Device - Recording Unit) 33 is a recording means (storage means) that records programs and various data required for the operation of the server device 93, and is configured using, for example, a hard disk or flash memory (not shown) as an external recording device (the same applies to recording units of other devices). However, instead of or in addition to the hard disk or flash memory, any other recording medium including a magnetic recording medium such as a magnetic disk, or an optical recording medium such as a DVD or Blu-ray disc can be used (the same applies to recording units of other devices).

[0434] The recording unit 932 includes, for example, a fraudulent transaction related information database (hereinafter, the database is referred to as "DB") 9321 and a genuine transaction related information DB 9322.

[0435] (Configuration - Server device - Recording unit - Fraudulent transaction related information DB) The fraudulent transaction related information DB 9321 in FIG. 33 is fraudulent transaction related information storage means for storing fraudulent transaction related information.

[0436] Figure 34 is a diagram illustrating fraudulent transaction-related information. Note that in Figure 34, for the sake of convenience, some information is omitted and shown as "..." (the same applies to other figures). Also, the items in Figure 34 are merely examples, and some items may be omitted or other items may be added (the same applies to other figures). Also, the information shown for each item in Figure 34 is for the sake of convenience (the same applies to other figures).

[0437] ===Information about fraudulent transactions=== "Fraudulent transaction related information" refers to various information related to fraudulent transactions, and for example, the information items shown in FIG. 34 are mutually associated.

[0438] The transaction ID in FIG. 34 is transaction identification information (hereinafter, the identification information will be referred to as "ID") that uniquely identifies a financial transaction that corresponds to a fraudulent transaction (such as "F001" in FIG. 34).

[0439] The account ID in Figure 34 is an account ID that identifies the financial account associated with the financial transaction identified by the transaction ID (e.g., "A001" in Figure 34). Note that the "financial account associated with the financial transaction" may be interpreted as indicating, for example, the financial account in which the financial transaction was made.

[0440] The transaction content information in Figure 34 is information that indicates the content of the financial transaction identified by the transaction ID (Figure 34 shows examples such as the transaction amount being "20,000" yen, the transaction method being "ATM", and the transaction time being "23:05").

[0441] ===Description=== The information at the top of Figure 34 indicates that the financial transaction identified by "F001" is a fraudulent transaction, that the financial transaction was carried out in the financial account identified by "A001," and that the details of the financial transaction correspond to "Amount: 20,000, Method: ATM, Time: 23:05, ...."

[0442] Figure 34 also shows that the financial transactions identified by "F001," "F002," "F003," and "F004" are fraudulent transactions, and that each of these financial transactions was conducted in the financial account identified by "A001."

[0443] FIG. 34 also shows that the financial transactions identified by "F005" are fraudulent transactions, and that each of the financial transactions was carried out in the financial account identified by "A002."

[0444] That is, FIG. 34 also shows that the financial accounts identified by "A001" and "A002" are financial accounts in which fraudulent transactions have been made.

[0445] ===Storage Method=== The method for storing the fraudulent transaction-related information in Figure 34 is arbitrary, and may be, for example, stored by inputting information related to actual fraudulent transactions reported from actual financial transactions at a specified financial institution (for example, the financial institution illustrated in Figure 33), or may be stored using any other method.

[0446] (Configuration - Server Device - Recording Unit - Genuine Transaction Related Information DB) The genuine transaction related information DB 9322 in Fig. 33 is a genuine transaction related information storage means for storing genuine transaction related information. Fig. 35 is a diagram showing an example of genuine transaction related information.

[0447] ===Genuine Transaction Information=== "Genuine transaction related information" refers to various information related to genuine transactions, and for example, the information items shown in FIG. 35 are mutually associated.

[0448] The transaction ID in FIG. 35 is a transaction ID that uniquely identifies a financial transaction that is a genuine transaction (such as "C006" in FIG. 34).

[0449] The account ID in FIG. 35 is an account ID that identifies the financial account associated with the financial transaction identified by the transaction ID (such as "A003" in FIG. 35).

[0450] The transaction content information in FIG. 35 is information indicating the content of the financial transaction identified by the transaction ID (in FIG. 35, the transaction amount is exemplified as "5000" yen, etc.).

[0451] ===Description=== The information at the top of Figure 35 indicates that the financial transaction identified by "C006" is a genuine transaction, that the financial transaction was carried out in the financial account identified by "A003," and that the content of the financial transaction corresponds to "Amount: 5000, ...".

[0452] FIG. 35 also shows that the financial transactions identified by "C006" and "C007" are genuine transactions, and that each of the financial transactions is carried out in the financial account identified by "A003."

[0453] FIG. 35 also shows that the financial transactions identified by "C008" are genuine transactions, and that each of the financial transactions is carried out in the financial account identified by "A004."

[0454] That is, FIG. 35 also shows that the financial accounts identified by "A003" and "A004" are financial accounts in which genuine transactions have been made.

[0455] ===Storage Method=== The method for storing the genuine transaction-related information in Figure 35 is arbitrary, and may be, for example, by inputting information related to actual genuine transactions confirmed from actual financial transactions at a specified financial institution (e.g., the financial institution illustrated in Figure 33), or may be stored using any other method.

[0456] (Configuration - Server Device - Control Unit) 33 is a control means for controlling the server device 93, and specifically, is a computer including a CPU, various programs interpreted and executed on the CPU (including basic control programs such as an OS and application programs that are started on the OS and realize specific functions), and an internal memory such as a RAM for storing programs and various data (the same applies to control units of other devices). In particular, the program according to the embodiment is installed in the server device 93 via an arbitrary recording medium or a network, thereby substantially configuring each unit of the control unit 933.

[0457] The control unit 933 includes, for example, a first identification unit, a second identification unit, a third identification unit, and an evaluation unit.

[0458] ===First identification means=== The first identification means is a means for identifying a first evaluation index based on a detected fraudulent transaction, which is a financial transaction detected as a fraudulent transaction by the detection system, and is used to evaluate the detection system.

[0459] The first identification means identifies, for example, an evaluation index corresponding to a matching rate regarding the detection of a fraudulent transaction by the detection system, based on the fraudulent transaction, as the first evaluation index.

[0460] The first identification means identifies, for example, an evaluation index corresponding to a false positive rate regarding the detection of a fraudulent transaction by the detection system, based on the fraudulent transaction, as the first evaluation index.

[0461] ===Second identification means=== The second identification means is a means for identifying a second evaluation indicator based on the detected fraudulent financial account, which is a financial account associated with the detected fraudulent transaction, and for evaluating the detection system.

[0462] The second identification means identifies, for example, an evaluation index corresponding to a recall rate based on a financial account regarding the detection of fraudulent transactions by the detection system as the second evaluation index.

[0463] The second identification means identifies, for example, an evaluation index corresponding to a true positive rate based on a financial account regarding the detection of fraudulent transactions by the detection system as the second evaluation index.

[0464] ===Third identification means=== The third identification means is a means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being based on detected fraudulent transactions and detected fraudulent financial accounts, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means.

[0465] The third identification means identifies, for example, an evaluation index corresponding to an F value based on precision and recall as the third evaluation index.

[0466] The third identification means identifies, for example, an evaluation index corresponding to an AUC based on a false positive rate and a true positive rate as the third evaluation index.

[0467] ===Evaluation Method=== The evaluation means is means for evaluating the detection system based on the third evaluation index identified by the third identification means.

[0468] The processes performed by each means of the control unit 933 will be described later.

[0469] (process) Next, we will explain, for example, financial institution-side fraud detection processing, AML-specialized F value identification processing, and AML-specialized AUC identification processing as processing performed by the information processing system 9100 configured as described above. Note that "AML" may be interpreted as an abbreviation for "Anti-Money Laundering."

[0470] (Processing - Financial institution side fraud detection processing) The fraud detection process on the financial institution side will now be described. The fraud detection process on the financial institution side is a process for detecting fraudulent transactions (fraud detection), and is a process that is repeatedly executed by the management terminal device 92 in FIG.

[0471] (Processing - Financial institution's fraud detection processing - Pre-trained detection model) First, a trained model for detection used in fraud detection will be described. Fig. 36 is an explanatory diagram of the trained model for detection. Note that Fig. 36 is a diagram for convenience of explanation, and the number of intermediate layers is not limited to two.

[0472] As mentioned above, the "trained model for detection" is a model for detecting fraudulent transactions, and is a model that outputs an AI score when transaction content information is input, for example, as shown in Figure 36.

[0473] As explained in FIG. 34 and FIG. 35, "transaction content information" is information indicating the content of the financial transaction, such as the amount, method, and time.

[0474] The "AI score" is information that indicates the likelihood that a financial transaction corresponding to the transaction content information input into the trained model for detection is fraudulent or genuine.

[0475] The specific content of this AI score is arbitrary, but for example, we will explain using numerical information ranging from "0" to "1" indicating the likelihood of a fraudulent transaction from lowest to highest. In detail, for example, the larger the AI ​​score value, the higher the likelihood of a fraudulent transaction, i.e., the lower the likelihood of a genuine transaction. Also, for example, the smaller the AI ​​score value, the lower the likelihood of a fraudulent transaction, i.e., the higher the likelihood of a genuine transaction.

[0476] For example, if there is a financial transaction with an "AI score" of "0.3" ("First Financial Transaction"), a financial transaction with an "AI score" of "0.5" ("Second Financial Transaction"), and a financial transaction with an "AI score" of "0.9" ("Third Financial Transaction"), the third financial transaction is most likely to be fraudulent, the second financial transaction is second most likely to be fraudulent, and the first financial transaction is least likely to be fraudulent.

[0477] In this embodiment, a detection method is described in which a threshold value (a predetermined number between "0" and "1") is set to be compared with the AI ​​score, the AI ​​score is compared with the threshold value, and fraudulent transactions are detected based on the magnitude relationship of the AI ​​score relative to the threshold value. Specifically, the detection method is set so that if the AI ​​score is below the threshold value, the transaction is considered to be genuine and no fraudulent transaction is detected, and if the AI ​​score is above the threshold value, the transaction is considered to be fraudulent and a fraudulent transaction is detected.

[0478] In this detection method, if the threshold is, for example, "0.85," the AI ​​scores of the first and second financial transactions mentioned above are "0.3" and "0.5," respectively, which are below the threshold of "0.85," and therefore these first and second financial transactions will not be detected as fraudulent transactions. On the other hand, the AI ​​score of the third financial transaction mentioned above is "0.9," which is above the threshold of "0.85," and therefore this third financial transaction will be detected as fraudulent.

[0479] The learning method for such a trained detection model is arbitrary, but it may be generated by learning using machine learning with teacher data, for example. Specifically, it may be generated by learning using machine learning using a large number of combinations of transaction content information of fraudulent transactions and the AI ​​score "1" corresponding to that transaction content information, and a large number of combinations of transaction content information of genuine transactions and the AI ​​score "0" corresponding to that transaction content information.

[0480] (Processing - Financial institution's fraud detection processing - Processing details) Next, the contents of the fraud detection process on the financial institution side will be explained, but since this process can be similar to known processes, only an outline will be explained.

[0481] For example, it is assumed that the learned detection model of Figure 36 is stored in the recording unit of the management terminal device 92 of Figure 33. Furthermore, for example, when an actual financial transaction is carried out, each transaction terminal device 91 of Figure 33 transmits executed financial transaction information indicating the content of the financial transaction (e.g., information including information corresponding to each item of Figures 34 and 35) to the management terminal device 92, and the content of the financial transaction that was actually carried out can be ascertained on the management terminal device 92 side based on the executed financial transaction information.

[0482] When a financial transaction is carried out, the control unit of the management terminal device 92 receives the executed financial information from the transaction terminal device 91 and uses the learned detection model recorded in the recording unit to detect fraudulent transactions using the detection method described above.

[0483] For example, let us consider a case where the threshold value is set to "0.85." In this case, the management terminal device 92 inputs the transaction content information contained in the executed financial information received from the transaction terminal device 91 into the trained model for detection and obtains the AI ​​score output from the trained model for detection.

[0484] Next, the acquired AI score is compared with the set threshold of "0.85," and if the acquired AI score is less than "0.85," the financial transaction corresponding to the transaction content information input into the trained model for detection is deemed to be genuine and is not detected as a fraudulent transaction. On the other hand, if the acquired AI score is "0.85" or higher, the financial transaction corresponding to the transaction content information input into the trained model for detection is deemed to be fraudulent and is detected as a fraudulent transaction.

[0485] Subsequent processing is optional, but for example, the control unit of the management terminal device 92 may record information indicating the detected fraudulent financial transaction in a recording unit, or may notify an administrator or the like.

[0486] ===Interpretation of terms=== In addition, since the detection trained model and threshold (not shown) in Figure 36 can be interpreted as elements for detecting fraudulent transactions, both or one of these may be interpreted as corresponding to a "detection system."

[0487] (Processing - AML specialized F value specific processing) Next, the AML-specialized F-value identification process will be described. Fig. 37 is a flowchart of the AML-specialized F-value identification process (hereinafter, each step will be referred to as "S"). The AML-specialized F-value identification process is a process of identifying and evaluating an AML-specialized F-value, and is a process executed by the server device 93 of Fig. 33, for example.

[0488] The timing for executing this AML-specialized F-value identification process is arbitrary, but for example, execution will begin when the administrator performs a specified operation to execute the process, and the explanation will begin from the point where execution begins (the same applies to the AML-specialized AUC identification process described below).

[0489] Also, here, for example, a case where an AML-specified F-measure is specified and evaluated for the first detection trained model and the second detection trained model will be described as an example. Note that the first detection trained model (also referred to as the "first example model") and the second detection trained model (also referred to as the "second example model") are the aforementioned "detection trained models" and are detection trained models generated using mutually different training data. In other words, the explanatory variables and objective variables of the first example model and the second example model are the transaction content information and AI score shown in FIG. 36, which are mutually common, but the weights, biases, etc. are mutually different.

[0490] ===AML specialized F value=== Fig. 38 is an explanatory diagram of the AML-specified F-value. The "AML-specified F-value" is a third evaluation index for evaluating a detection system, and specifically, is an evaluation index corresponding to the F-value based on precision and recall, and is, for example, a numerical value calculated by the calculation formula shown in Fig. 38. The AML-specified F-value will be described in detail later.

[0491] ===SF1=== 37, the control unit 933 of the server device 93 acquires target model-related information (information related to the trained model for detection to be evaluated). Specifically, although this is optional, for example, the target model-related information may be recorded in the recording unit 932, and the recorded target model-related information may be acquired, or the target model-related information input by the administrator may be acquired.

[0492] Here, for example, information indicating the above-mentioned first and second example models and the threshold values ​​used in each of the models is acquired as the target model related information.

[0493] ===SF2=== In SF2 of Figure 37, the control unit 933 of the server device 93 acquires fraudulent transaction related information and genuine transaction related information as information for evaluating the detection trained model indicated by the target model related information acquired in SF1.

[0494] Here, for example, the fraudulent transaction related information in FIG. 34 and the genuine transaction related information in FIG. 35 are acquired.

[0495] ===SF3=== 37, the control unit 933 of the server device 93 uses the fraudulent transaction-related information and genuine transaction-related information acquired in SF2 to identify the AML-specialized F-measure for the detection trained model indicated by the target model-related information acquired in SF1. Specifically, the following first to fourth steps are performed.

[0496] ==Step 1== In the first step, for each financial transaction indicated by each transaction-related information obtained in SF2, fraudulent transactions are detected using the aforementioned detection method (a method of detecting by comparing the AI ​​score with a threshold) using the detection-use trained model and threshold indicated by the target model-related information obtained in SF1.

[0497] Figure 39 is a diagram illustrating the results of fraudulent transaction detection. Figure 39 illustrates the results of fraudulent transaction detection performed using the first and second example models for financial transactions indicated by the transaction-related information in Figures 34 and 35. The following explanation will be based on the results for the eight financial transactions illustrated in Figure 39.

[0498] The transaction ID and account ID in Figure 39 are the same as the information with the same names in Figures 34 and 35. The "First Example Model" column in Figure 39 shows an example of the detection results of fraudulent transactions performed using the first example model, with either "Fraud" indicating that the transaction was detected as fraudulent, or "Genuine" indicating that the transaction was not detected as fraudulent (i.e., the transaction was determined to be genuine). The "Second Example Model" column in Figure 39 shows an example of the detection results of fraudulent transactions performed using the second example model, with the same information as in the case of the first example model.

[0499] For example, when the transaction content information in the top row of Figure 34 is input to the first exemplary model indicated by the target model-related information acquired in SF1 for the financial transaction indicated by the information in the top row of Figure 34, an AI score of, for example, "0.95" is output from the first exemplary model. If the threshold used in the first exemplary model indicated by the target model-related information is, for example, "0.85," the AI ​​score is equal to or greater than the threshold, and therefore a fraudulent transaction is detected (see the "First exemplary model" column in the top row of Figure 39). Similar processing is then performed on the second exemplary model and other financial transactions, thereby detecting fraudulent transactions, as shown in the "First exemplary model" and "Second exemplary model" columns of Figure 39.

[0500] That is, for example, when the first exemplary model is used, five financial transactions with "transaction ID" = "F001," "F002," "F005," "C006," and "C007" are detected as fraudulent transactions, while the other three financial transactions are not detected as fraudulent transactions.

[0501] Furthermore, for example, when the second exemplary model is used, six financial transactions with "transaction ID" = "F001," "F002," "F003," "F004," "C006," and "C007" are detected as fraudulent transactions, while the other two financial transactions are not detected as fraudulent transactions.

[0502] ==Second Step== In the second step, a trade unit matching rate is determined based on the detection results in the first step.

[0503] The "transaction-based relevance rate" is a first evaluation index based on detected fraudulent transactions, which are financial transactions detected as fraudulent by a detection system, and is the first evaluation index for evaluating a detection system. Specifically, the "transaction-based relevance rate" is an evaluation index corresponding to the relevance rate based on the fraudulent transactions detected by the detection system.

[0504] The content of the transaction-based conformance rate is arbitrary as long as it meets the above definition. For example, the transaction-based conformance rate may be the ratio of the number of financial transactions that were actually fraudulent to the number of financial transactions detected as fraudulent (detected fraudulent transactions).

[0505] In processing, the transaction-level conformance rate is calculated as the result of dividing the number of financial transactions that were actually fraudulent among the detected fraudulent transactions by the number of financial transactions detected as fraudulent (detected fraudulent transactions).The "financial transactions that were actually fraudulent among the detected fraudulent transactions" correspond to the transaction IDs "F~~" in Figure 39 (i.e., financial transactions indicated by the fraudulent transaction-related information in Figure 34).

[0506] Figure 40 is a diagram showing an example of calculation of each index. Here, for example, the transaction unit conformance rate for the first exemplary model is specified as "3 (financial transactions of F001, F002, F005)" ÷ "5 (financial transactions of F001, F002, F005, C006, C007)" = 60% (see the "First exemplary model" and "Transaction unit conformance rate" columns in Figure 40).

[0507] Furthermore, for example, the transaction unit conformance rate for the second example model is determined to be "4 (financial transactions of F001, F002, F003, F004)" ÷ "6 (financial transactions of F001, F002, F003, F004, C006, C007)" = 66.7% (see the "Second example model" and "Transaction unit conformance rate" columns in Figure 40).

[0508] ==Third Step== In the third step, the account-level recall rate is determined based on the detection results in the first step.

[0509] The "account-based recall rate" is a second evaluation metric based on the detected fraudulent financial account, which is a financial account associated with a detected fraudulent transaction, and is a second evaluation metric for evaluating a detection system. Specifically, the "account-based recall rate" is an evaluation metric corresponding to the recall rate based on a financial account for the detection of fraudulent transactions by a detection system.

[0510] The content of the account-based recall rate is arbitrary as long as it meets the above definition, but for example, the account-based recall rate will be the ratio of the number of financial accounts in which financial transactions detected as fraudulent (detected fraudulent transactions) occurred to the number of financial accounts in which fraudulent transactions actually occurred. Note that duplicate financial accounts will be counted as one.

[0511] Regarding processing, the account-level recall rate is calculated as the result of the calculation: "the number of financial accounts (detected fraudulent financial accounts) in which financial transactions detected as fraudulent (detected fraudulent transactions) were made among the financial accounts in which fraudulent transactions actually occurred" divided by "the number of financial accounts in which fraudulent transactions actually occurred." Note that a financial account associated with at least one detected fraudulent transaction is defined as a "financial account (detected fraudulent financial account) in which a financial transaction detected as fraudulent (detected fraudulent transaction) was made," and a financial account associated with at least one fraudulent transaction is defined as a "financial account in which a fraudulent transaction actually occurred."

[0512] Here, for example, the account-based reproducibility rate for the first example model is determined as "2 (both of the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" ÷ "2 (the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" = 100% (see the "First example model" and "Account-based reproducibility" columns in Figure 40).

[0513] Furthermore, for example, the transaction-level accuracy rate for the second example model is determined as "1 (only the financial account A001 among the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" ÷ "2 (the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" = 50% (see the "Second example model" and "Account-level recall" columns in Figure 40).

[0514] ==Fourth Step== In the fourth step, an AML-specified F-value is determined based on the processing results of the second and third steps.

[0515] The "AML-specific F-score" is a third evaluation metric based on the number of detected fraudulent transactions and fraudulent financial accounts, and is used to evaluate detection systems. Specifically, the "AML-specific F-score" is an evaluation metric corresponding to the F-score based on precision and recall.

[0516] The content of the AML-specialized F value is arbitrary as long as it satisfies the above definition, but for example, a case where the calculation result of the calculation formula shown in Figure 38 is used as the AML-specialized F value will be illustrated. In Figure 38, "β" is a predetermined numerical value greater than 0, but if emphasis is placed on the account-based recall rate, a value greater than "1" can be used. The specific value of "β" is set by an administrator, etc., and for convenience, the following description will be given using an example where "β" = "1".

[0517] For the processing, the transaction unit matching rate identified in the second step and the account unit matching rate identified in the third step are applied to the calculation formula in Figure 38 to perform a calculation, and the calculation result is identified as the AML-specialized F value.

[0518] Here, for example, the AML-specialized F-value for the first example model is specified as 75% (see the "First example model" and "AML-specialized F-value" columns in Figure 40), and the AML-specialized F-value for the second example model is specified as 57% (see the "Second example model" and "AML-specialized F-value" columns in Figure 40).

[0519] ===SF4=== In SF4 of FIG. 37, the control unit 933 of the server device 93 evaluates the detection trained model, etc. indicated by the target model related information, based on the AML-specialized F-measure identified in SF3.

[0520] The specific evaluation method is arbitrary and not limited, but for example, a case where a larger AML-specified F value is evaluated as being better will be described as an example.

[0521] Here, for example, as shown in the "AML-specialized F-value" column in Fig. 40, the first exemplary model has a larger AML-specialized F-value than the second exemplary model, and therefore the first exemplary model may be evaluated as being superior. In this case, the combination of the first exemplary model and the thresholds used for the first exemplary model may be evaluated as being superior to the combination of the second exemplary model and the thresholds used for the second exemplary model.

[0522] Alternatively, for example, an F threshold (a predetermined numerical value) to be compared with the AML-specified F value may be determined in advance, and a model having an AML-specified F value greater than the F threshold may be evaluated as superior.

[0523] The subsequent processing is optional, but for example, the detection trained model evaluated as superior, or the combination of the detection trained model and threshold, may be sent to the management terminal device 92 at the financial institution illustrated in Figure 33, and fraudulent transactions may be detected at the financial institution based on the transmitted information (the same applies to the AML-specialized AUC identification processing described below).

[0524] (Processing - AML-specific AUC specific processing) Next, the AML-specialized AUC identification process will be described. Fig. 41 is a flowchart of the AML-specialized AUC identification process. The AML-specialized AUC identification process is a process for identifying and evaluating the AML-specialized AUC, and is a process executed by the server device 93 in Fig. 33, for example.

[0525] Also, here, for example, a case where the AML-specialized AUC is specified and evaluated for the first exemplary model and the second exemplary model will be described as an example.

[0526] ===AML specific AUC=== Fig. 42 is an explanatory diagram of the AML-specialized AUC. The "AML-specialized AUC" is a third evaluation index for evaluating a detection system, and specifically, is an evaluation index corresponding to the AUC based on the false positive rate and the true positive rate, such as the AUC (area under the ROC curve) shown in Fig. 42. The AML-specialized AUC will be described in detail later.

[0527] ===SG1=== In SG1 of FIG. 41, the control unit 933 of the server device 93 acquires target model related information (information related to the trained model for detection to be evaluated).

[0528] Here, for example, information indicating the first example model and the second example model is acquired as the target model related information.

[0529] ===SG2=== In SG2 in FIG. 41, the same processing as in SF2 in FIG. 37 is performed.

[0530] ===SG3=== 41, the control unit 933 of the server device 93 uses the fraudulent transaction-related information and genuine transaction-related information acquired in SG2 to identify the AML-specialized AUC for the detection trained model indicated by the target model-related information acquired in SG1. Specifically, the following first to fourth steps are performed.

[0531] ==Step 1== In the first step, for each financial transaction indicated by each transaction-related information acquired in SG2, fraudulent transactions are detected using the aforementioned detection method (a method of detecting by comparing the AI ​​score with a threshold) using the detection-trained model indicated by the target model-related information acquired in SG1. Here, for example, detection is performed as shown in Figure 39.

[0532] ==Second Step== In the second step, the transaction-level false positive rate is determined based on the detection results in the first step.

[0533] The "transaction-based false positive rate" is a first evaluation metric based on detected fraudulent transactions, which are financial transactions detected as fraudulent by a detection system, and is the first evaluation metric for evaluating a detection system. Specifically, the "transaction-based false positive rate" is an evaluation metric corresponding to the false positive rate based on the fraudulent transactions detected by the detection system.

[0534] The content of the transaction-level false positive rate is arbitrary as long as it meets the above definition. For example, the transaction-level false positive rate may be the ratio of the number of financial transactions that were not actually fraudulent but were mistakenly detected as fraudulent to the number of financial transactions that were not actually fraudulent (financial transactions that were actually genuine).

[0535] Regarding processing, the transaction-level false positive rate is calculated as the result of dividing the number of financial transactions that were mistakenly detected as fraudulent despite not actually being fraudulent by the number of financial transactions that were not actually fraudulent (financial transactions that were actually genuine).

[0536] Here, in the case of FIG. 39, the transaction-based false positive rate for the first example model is determined to be "2 (financial transactions of C006 and C007)" ÷ "3 (financial transactions of C006, C007, and C008)" = 66.7%.

[0537] In the case of FIG. 39, the transaction-based false positive rate for the second example model is also determined to be "2 (financial transactions of C006 and C007)" ÷ "3 (financial transactions of C006, C007, and C008)" = 66.7%.

[0538] ==Third Step== In the third step, the account-level true positive rate is determined based on the detection results in the first step.

[0539] The "account-level true positive rate" is a second evaluation metric based on detected fraudulent financial accounts, which are financial accounts associated with detected fraudulent transactions, and is a second evaluation metric for evaluating a detection system. Specifically, the "account-level true positive rate" is an evaluation metric corresponding to the true positive rate based on financial accounts regarding the detection of fraudulent transactions by a detection system.

[0540] The content of the account-level true positive rate is arbitrary as long as it meets the above definition, but for example, similar to the account-level recall rate described above, we will use the ratio of the number of financial accounts in which financial transactions detected as fraudulent (detected fraudulent transactions) occurred to the number of financial accounts in which fraudulent transactions actually occurred as the account-level true positive rate. Note that duplicate financial accounts will be counted as one.

[0541] The processing is the same as in the case of the account-based true recall described above.

[0542] ==Fourth Step== In the fourth step, the threshold (the threshold compared with the AI ​​score output from the detection trained model) is changed in multiple stages (for example, the threshold is increased from "0" to "1" in increments of "0.0001"), and steps 1 to 3 are repeated multiple times to identify multiple combinations of transaction-level false positive rates and account-level true positive rates. An ROC curve is then identified by connecting the coordinates consisting of combinations of transaction-level false positive rates and account-level true positive rates in the coordinate system shown in Figure 42, and the AUC corresponding to the area under the identified ROC curve is identified as the AML-specialized AUC.

[0543] Here, for example, although not specifically shown in Figure 42, the AUC corresponding to the ROC curve for the first example model is identified as the AML-specialized AUC of the first example model, and the AUC corresponding to the ROC curve for the second example model is identified as the AML-specialized AUC of the second example model.

[0544] ===SG4=== In SG4 of Figure 41, the control unit 933 of the server device 93 evaluates the detection trained model indicated by the target model related information, etc., based on the AML-specialized AUC identified in SG3.

[0545] The specific evaluation method is arbitrary and not limited, but for example, a case where the larger the value of the AML-specialized AUC, the better the evaluation will be explained as an example.

[0546] Here, for example, if the AML-specialized AUC of the first exemplary model is greater than that of the second exemplary model, the first exemplary model may be evaluated as being superior.

[0547] Alternatively, for example, an AUC threshold to be compared with the AML-specialized AUC may be determined in advance, and a model whose AML-specialized AUC is greater than the AUC threshold may be evaluated as superior.

[0548] Subsequent processing is optional, but for example, the detection trained model evaluated as superior, or the combination of the detection trained model and a threshold, may be sent to the management terminal device 92 at the financial institution illustrated in Figure 33, and fraudulent transactions may be detected at the financial institution based on the information sent.

[0549] (Effects of the embodiment) Thus, according to this embodiment, by specifying the AML-specialized F-value and AML-specialized AUC, which are the third evaluation indices based on detected fraudulent transactions and detected fraudulent financial accounts, it is possible to use evaluation indices that take into account, for example, both financial transactions (detected fraudulent transactions) and financial accounts (detected fraudulent financial accounts), thereby making it possible to appropriately evaluate the detection system.

[0550] Furthermore, by specifying an AML-specific F-measure, which is an evaluation index corresponding to an F-measure based on the precision rate based on fraudulent transactions and the recall rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation index that takes into account, for example, both the precision rate for financial transactions and the recall rate for financial accounts, thereby making it possible to appropriately evaluate detection systems.

[0551] By specifying an AML-specific AUC, which is an evaluation metric that corresponds to an AUC based on the false positive rate based on fraudulent transactions and the true positive rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation metric that takes into account, for example, both the false positive rate for financial transactions and the true positive rate for financial accounts, thereby making it possible to appropriately evaluate detection systems.

[0552] Furthermore, by evaluating the detection system based on the AML-specific F-score and AML-specific AUC, which are the third evaluation indexes, it becomes possible to appropriately evaluate the detection system, for example.

[0553] [Modification of the second embodiment] Although the second embodiment of the present invention has been described above, the specific configuration and means of the present invention can be modified and improved as desired within the scope of the technical concept of the present invention as set forth in the claims. Such modifications will be described below.

[0554] (About the problem to be solved and the effects of the invention) First, the problems to be solved by the invention and the effects of the invention are not limited to those described above and may vary depending on the implementation environment and details of the configuration of the invention. Therefore, only some of the problems described above may be solved or only some of the effects described above may be achieved. Furthermore, the problems solved by the techniques of the above-described embodiments may be interpreted as problems to be solved by the invention. For example, in the case of embodiment 2, the problem may be interpreted as providing an evaluation system, an evaluation program, and an evaluation method that enable appropriate evaluation of a detection system.

[0555] (Regarding decentralization and integration) Furthermore, the electrical components described above are functional concepts and do not necessarily have to be physically configured as shown in the drawings. In other words, the specific form of distribution or integration of each part is not limited to that shown in the drawings, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, the term "device" in this application is not limited to a single device, but includes a device configured from multiple devices.

[0556] (shape, numbers, structure, time series) The components illustrated in the embodiments and drawings may be modified and improved as desired within the scope of the technical concept of the present invention in terms of shape, numerical value, or the structure or chronological relationship of multiple components.

[0557] (Evaluation (Part 1)) In the above-mentioned second embodiment, a case has been described in which a detection system is evaluated based on the AML-specialized F-value identified by SF3 in Figure 37 or the AML-specialized AUC identified by SG3 in Figure 41, but the detection system may be configured to be evaluated using only the AML-specialized F-value, or the detection system may be configured to be evaluated using only the AML-specialized AUC, or the detection system may be configured to be evaluated using both the AML-specialized F-value and the AML-specialized AUC.

[0558] (Evaluation (Part 2)) Alternatively, the "first exemplary model" described in the second embodiment may be a trained detection model currently used by a financial institution, as illustrated in FIG. 33, and the "second exemplary model" may be a candidate for new application to the financial institution. The server device 93 may then use the method described in the embodiment to evaluate which of the first exemplary model and the second exemplary model is superior. If the server device 93 determines that the second exemplary model is superior to the first exemplary model, the server device 93 transmits the second exemplary model to the management terminal device 92, thereby applying the second exemplary model to the financial institution. The management terminal device 92 then performs a process of detecting fraudulent transactions using the second exemplary model. If the server device 93 determines that the first exemplary model is superior to the second exemplary model, the server device 93 does not apply the second exemplary model.

[0559] (Evaluation (Part 3)) Furthermore, as the "detection trained model that is a candidate for new application to financial institutions" described as the "second example model" in "(Regarding Evaluation (Part 2))" above, a detection trained model generated by re-training the "first example model" (i.e., a tuned detection trained model) may be adopted, or a detection trained model newly generated separately from the "first example model" may be adopted.

[0560] (Application examples) The AML-specialized AUC of the second embodiment may be used to terminate machine learning of a trained model for detection early. Specifically, the loss may be calculated for each of the training teacher data and the verification teacher data, and the training may be terminated when the difference between the two calculated loss values ​​exceeds a predetermined threshold. The specific configuration of the training teacher data and the verification teacher data is arbitrary as long as they are information related to financial transactions. For example, data having a configuration similar to that of the fraudulent transaction-related information in FIG. 34 and the genuine transaction-related information in FIG. 35 may be used. The loss may be arbitrary as long as the AML-specialized AUC is used. For example, the calculation result of "w1 × ordinary cross entropy error (CrossEntropyLoss) + w2 × AML-specialized AUC" may be used as the loss. Note that w1 and w2 are predetermined values ​​set by an administrator.

[0561] (About output methods) Furthermore, the control unit 933 of the server device 93 in the second embodiment may be configured to include output means. The "output means" refers to means for outputting information indicating the third evaluation index identified by the third identification means. The term "outputting information" includes, for example, displaying and outputting information, outputting information as sound or voice, outputting information in the form of a log file or a database table (i.e., saving information), printing and outputting information on an external medium such as paper, recording and outputting information on a recording medium such as an external memory, and transmitting information.

[0562] Here, for example, a case will be described in which the server device 93 displays and outputs information indicating the third evaluation index. Fig. 43 is a display example of the first display screen, and Fig. 44 is a display example of the second display screen.

[0563] ===1st display screen=== The "first display screen" in Figure 43 is a screen that displays the AML-specialized F-value, and more specifically, is a concept that shows the change over time in the AML-specialized F-value. In this first display screen, the AML-specialized F-value for each date is displayed in the form of a line graph, with the horizontal axis representing the date and the vertical axis representing the AML-specialized F-value.

[0564] The display method for this first display screen is arbitrary, but for example, the server device 93 may be configured to store the detection trained model and thresholds used by the financial institution illustrated in Figure 33, and to periodically and repeatedly transmit or input information corresponding to each piece of transaction-related information in Figures 34 and 35 from the financial institution to the server device 93, thereby accumulating new information as each piece of transaction-related information in Figures 34 and 35. For example, the financial institution may transmit or input information regarding actual fraudulent transactions and genuine transactions in a single day to the server device 93 at a predetermined time on that day, thereby accumulating new information as each piece of transaction-related information in Figures 34 and 35.

[0565] Then, on a specific date (for example, a date at seven-day intervals, such as June 1, 2024 or June 8, 2024), the control unit 933 of the server device 93 executes SF1 to SF3 of Figure 37 using the transaction-related information of Figures 34 and 35 for the seven days immediately preceding that date to identify an AML-specialized F-value, generates screen information for displaying a first display screen showing the identified AML-specialized F-value, and sends the generated screen information to an arbitrary display device (for example, an administrator's terminal device (personal computer, smartphone, tablet terminal, etc.)), thereby displaying the first display screen on that display device. Here, for example, the first display screen of Figure 43 is displayed.

[0566] The above process may be modified as desired. For example, the AML-specialized F value may be identified using the transaction-related information in Figures 34 and 35 for the most recent specified number of days for each specified number of days, so that the first display screen includes the change over time in the AML-specialized F value, including the latest AML-specialized F value (the same applies to the second display screen).

[0567] Furthermore, the display format of the AML-specialized F value is not limited to a line graph format, and may be configured to display in any other graph format, or may be configured to display a display format other than a graph format (for example, numerical text information, etc.) (the same applies to the second display screen).

[0568] ===Second display screen=== The "second display screen" in Figure 44 is a screen that displays the AML-specialized AUC, and more specifically, it is a concept that shows the change in the AML-specialized AUC over time. In this second display screen, the AML-specialized AUC for each date is displayed in the form of a line graph, with the horizontal axis representing the date and the vertical axis representing the AML-specialized AUC.

[0569] The display method for this second display screen is the same as the display method for the first display screen described above.

[0570] That is, on a specific date (for example, a date at seven-day intervals, such as June 1, 2024 or June 8, 2024), the control unit 933 of the server device 93 executes SG1 to SG3 in Figure 41 using the transaction-related information in Figures 34 and 35 for the seven days immediately preceding that date to identify an AML-specialized AUC, generates screen information for displaying a second display screen showing the identified AML-specialized AUC, and transmits the generated screen information to an arbitrary display device, thereby displaying the second display screen on that display device. Here, for example, the second display screen in Figure 44 is displayed.

[0571] By configuring in this manner, it is possible to provide information useful for evaluating the detection system, for example, by outputting information indicating the third evaluation index, the AML-specific F-value and the AML-specific AUC.

[0572] (About AML-specialized F-value) Furthermore, in the second embodiment, the calculation formula for the AML-specialized F value is described using the formula shown in Fig. 38, but the present invention is not limited to this. For example, any modification may be made, such as omitting "β" in Fig. 38 or multiplying by another weight coefficient (numerical value).

[0573] (About the pre-trained detection model) The trained detection model in Figure 36 may also be modified as desired. For example, it may be configured so that the objective variable is information indicating whether the financial transaction corresponding to the transaction content information is fraudulent (for example, "fraudulent" indicating a fraudulent transaction or "genuine" indicating a genuine transaction).

[0574] (Interpretation of terms) In addition, since the transaction-level relevance rate and account-level relevance rate in Figure 38 are used to identify the AML-specific F value, these can also be interpreted as evaluation indicators for evaluating detection systems.

[0575] Additionally, because the transaction-level false positive rate and account-level true positive rate in Figure 42 are used to identify the AML-specific AUC, they can also be interpreted as evaluation metrics for evaluating detection systems.

[0576] [Modifications of the First and Second Embodiments] (About the indicators) Furthermore, in the above-mentioned first embodiment, a case has been described in which the transaction-based precision rate, account-specific recall rate, and transaction-based AUC are used as the other bank model-derived related explanatory variable information in Figure 12, but this is not limited to this. For example, other indicators may be used in addition to these, or in place of at least some of these. For example, a configuration may be made to use the account-based AUC, the AML-specialized F value, the AML-specialized AUC described in the second embodiment, etc. In this case, the account-based AUC, the AML-specialized F value, the AML-specialized AUC, etc. may also be used as the objective variable-related information in Figure 13.

[0577] (About the indicators) Furthermore, the first model (FIG. 6), the second model (FIG. 7), the fraud detection trained combination model (FIG. 5) of the first embodiment, or the thresholds for detecting fraudulent transactions used therein may be used as the detection system of the second embodiment. In other words, the evaluation system of the second embodiment may be configured to evaluate each of these models or thresholds.

[0578] (Regarding the processing entity) Furthermore, for processes where the processing entity is not specified, it may be interpreted that the control unit of the related device is the processing entity.

[0579] (Regarding combinations, etc.) Furthermore, the features of the first embodiment, the features of the second embodiment, and the features of the modification may be combined in any manner.

[0580] (Interpretation of terms) Furthermore, the meanings of the terms in the above-mentioned embodiment 1 may be interpreted using the terms described in the above-mentioned embodiment 2, so long as they are consistent. Furthermore, the meanings of the terms in the above-mentioned embodiment 2 may be interpreted using the terms described in the above-mentioned embodiment 1, so long as they are consistent.

[0581] ===First Note=== The following describes a first supplementary note corresponding to the first embodiment and the effect of the first supplementary note.

[0582] (First Supplement) The estimation system of Appendix 1 is an estimation system for estimating the introduction effect, which is the effect on the detection of fraudulent transactions of introducing a detection trained model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the detection trained model is generated based on financial transaction related information, which is information on financial transactions at financial institutions other than the target financial institution, and the estimation system comprises: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution; and an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means.

[0583] The estimation system of Supplementary Note 2 is the estimation system described in Supplementary Note 1, wherein the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input.

[0584] The estimation system of Supplementary Note 3 is the estimation system described in Supplementary Note 2, wherein the region-related trained model includes a first region-related trained model that outputs, as the estimation indicator, a first indicator corresponding to a precision rate based on the fraudulent transaction, and a second region-related trained model that outputs, as the estimation indicator, a second indicator corresponding to a recall rate based on a financial account associated with the fraudulent transaction.

[0585] The estimation system of Supplementary Note 4 is the estimation system described in any one of Supplementary Notes 1 to 3, further comprising a third acquisition means for acquiring feature-related information indicating features of the trained model for detection, and the estimation means estimates the introduction effect based on the first region-related information acquired by the first acquisition means, the second region-related information acquired by the second acquisition means, and the feature-related information acquired by the third acquisition means.

[0586] The estimation system of Appendix 5 is the estimation system described in Appendix 1, further comprising a determination means for determining whether or not the trained model for detection is to be a target for estimation of the introduction effect based on the first region-related information acquired by the first acquisition means, and the estimation means estimates the introduction effect for the trained model for detection determined by the determination means to be a target for estimation.

[0587] The estimation system of Supplementary Note 6 is the estimation system described in Supplementary Note 5, wherein the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input, and the region-related trained model is also configured to output a judgment index, which is used to determine whether the detection trained model is to be the estimation target for the introduction effect, when only one of the first region-related information or the second region-related information is input, and the judgment means judges whether the detection trained model is to be the estimation target, based on the region-related trained model.

[0588] The estimation method of Appendix 7 is an estimation method for estimating an introduction effect, which is the effect on the detection of fraudulent transactions, of introducing a detection trained model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the detection trained model is generated based on financial transaction related information, which is information regarding financial transactions at a financial institution other than the target financial institution, and the estimation method includes: a first acquisition step in which a first acquisition means acquires first region-related information, which is information regarding financial transactions based on a region related to the target financial institution; a second acquisition step in which a second acquisition means acquires second region-related information, which is information regarding financial transactions based on a region related to the other financial institution; and an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step and the second region-related information acquired in the second acquisition step.

[0589] The estimation program of Appendix 8 is an estimation program for estimating the introduction effect, which is the effect on the detection of fraudulent transactions of introducing a detection trained model for detecting fraudulent transactions in financial transactions into a target financial institution, wherein the detection trained model is generated based on financial transaction related information, which is information on financial transactions at financial institutions other than the target financial institution, and the estimation program causes a computer to function as: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquisition means for acquiring second region-related information, which is information on financial transactions based on a region related to the other financial institution; and an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means.

[0590] (Effect of the first supplementary note) According to the estimation system described in Appendix 1, the estimation method described in Appendix 7, and the estimation program described in Appendix 8, by estimating the introduction effect based on the first region-related information and the second region-related information, it is possible to estimate the introduction effect taking into account the characteristics of the region, for example, and therefore it is possible to appropriately estimate the introduction effect of the trained model for detection.

[0591] According to the estimation system described in Appendix 2, by estimating the introduction effect based on a regionally related trained model, it becomes possible to easily estimate, for example, the introduction effect of a trained model for detection.

[0592] According to the estimation system described in Appendix 3, by including a first region-related trained model that outputs, as an estimation indicator, a first indicator corresponding to a precision rate based on fraudulent transactions, and a second region-related trained model that outputs, as an estimation indicator, a second indicator corresponding to a recall rate based on a financial account associated with the fraudulent transactions, it becomes possible, for example, to more appropriately estimate the effect of introducing a detection trained model by using two types of region-related trained models.

[0593] According to the estimation system described in Appendix 4, by estimating the introduction effect based on the first region-related information, the second region-related information, and the feature-related information, it is possible to take into account, for example, the features of the trained model for detection, thereby making it possible to more appropriately estimate the introduction effect of the trained model for detection.

[0594] According to the estimation system described in Appendix 5, by determining whether or not to use a trained model for detection as a target for estimating the introduction effect, it is possible to narrow down the targets for estimation and efficiently estimate the introduction effect, for example, by determining whether or not to use a trained model for detection as a target for estimating the introduction effect.

[0595] According to the estimation system described in Appendix 6, the regional-related trained model is configured to output estimation indicators and judgment indicators, so that, for example, it is possible to use the regional-related trained model to both determine whether to include it in estimation and estimate the effects of its introduction.

[0596] ===Second Note=== The second supplementary note corresponding to the second embodiment and the effect of the second supplementary note will be described below.

[0597] (Second Note) The evaluation system of Appendix 1 is an evaluation system for evaluating a detection system for detecting fraudulent transactions in financial transactions associated with a financial account, and comprises: a first identification means for identifying a first evaluation indicator based on a detected fraudulent transaction, which is the financial transaction detected as the fraudulent transaction by the detection system, and for evaluating the detection system; a second identification means for identifying a second evaluation indicator based on a detected fraudulent financial account, which is the financial account associated with the detected fraudulent transaction, and for evaluating the detection system; and a third identification means for identifying a third evaluation indicator based on the detected fraudulent transaction and the detected fraudulent financial account, based on the first evaluation indicator identified by the first identification means and the second evaluation indicator identified by the second identification means, and for evaluating the detection system.

[0598] The evaluation system of Appendix 2 is the evaluation system described in Appendix 1, wherein the first identification means identifies, as the first evaluation index, an evaluation index corresponding to a precision rate based on the fraudulent transaction regarding the detection of the fraudulent transaction by the detection system, the fraudulent transaction itself; the second identification means identifies, as the second evaluation index, an evaluation index corresponding to a recall rate based on the financial account regarding the detection of the fraudulent transaction by the detection system; and the third identification means identifies, as the third evaluation index, an evaluation index corresponding to an F value based on the precision rate and the recall rate.

[0599] The evaluation system of Appendix 3 is the evaluation system described in Appendix 1, wherein the first identification means identifies, as the first evaluation index, an evaluation index corresponding to a false positive rate based on the fraudulent transaction regarding the detection of the fraudulent transaction by the detection system, the second identification means identifies, as the second evaluation index, an evaluation index corresponding to a true positive rate based on the financial account regarding the detection of the fraudulent transaction by the detection system, and the third identification means identifies, as the third evaluation index, an evaluation index corresponding to an AUC based on the false positive rate and the true positive rate.

[0600] The evaluation system of Supplementary Note 4 is the evaluation system according to Supplementary Note 1, further comprising an output means for outputting information indicating the third evaluation index identified by the third identification means.

[0601] The evaluation system of Supplementary Note 5 is the evaluation system according to Supplementary Note 1, further comprising an evaluation means for evaluating the detection system based on the third evaluation index identified by the third identification means.

[0602] The evaluation program of Appendix 6 is an evaluation program for evaluating a detection system for detecting fraudulent transactions in financial transactions associated with a financial account, and causes a computer to function as a first identification means for identifying a first evaluation indicator based on a detected fraudulent transaction, which is the financial transaction detected as the fraudulent transaction by the detection system, and for evaluating the detection system; a second identification means for identifying a second evaluation indicator based on a detected fraudulent financial account, which is the financial account associated with the detected fraudulent transaction, and for evaluating the detection system; and a third identification means for identifying a third evaluation indicator based on the detected fraudulent transaction and the detected fraudulent financial account, and for evaluating the detection system, based on the first evaluation indicator identified by the first identification means and the second evaluation indicator identified by the second identification means.

[0603] The evaluation method of Supplementary Note 7 is an evaluation method for evaluating a detection system for detecting fraudulent transactions in financial transactions associated with a financial account, and includes a first identification step of identifying a first evaluation indicator based on a detected fraudulent transaction, which is the financial transaction detected as the fraudulent transaction by the detection system, and for evaluating the detection system; a second identification step of identifying a second evaluation indicator based on a detected fraudulent financial account, which is the financial account associated with the detected fraudulent transaction, and for evaluating the detection system; and a third identification step of identifying a third evaluation indicator based on the detected fraudulent transaction and the detected fraudulent financial account, and for evaluating the detection system, based on the first evaluation indicator identified in the first identification step and the second evaluation indicator identified in the second identification step.

[0604] (Effect of the second supplementary note) According to the evaluation system described in Appendix 1, the evaluation program described in Appendix 6, and the evaluation method described in Appendix 7, by identifying a third evaluation indicator based on detected fraudulent transactions and detected fraudulent financial accounts, it is possible to use evaluation indicators that take into account both financial transactions (detected fraudulent transactions) and financial accounts (detected fraudulent financial accounts), thereby making it possible to appropriately evaluate the detection system.

[0605] According to the evaluation system described in Appendix 2, by specifying an evaluation index corresponding to an F-value based on the precision rate based on fraudulent transactions and the recall rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation index that takes into account both the precision rate for financial transactions and the recall rate for financial accounts, thereby making it possible to appropriately evaluate the detection system.

[0606] According to the evaluation system described in Appendix 3, by specifying an evaluation metric corresponding to the AUC based on the false positive rate based on fraudulent transactions and the true positive rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation metric that takes into account both the false positive rate for financial transactions and the true positive rate for financial accounts, thereby making it possible to appropriately evaluate detection systems.

[0607] According to the evaluation system described in Supplementary Note 4, by outputting information indicating the third evaluation index, it becomes possible to provide information useful for evaluating the detection system, for example.

[0608] According to the evaluation system described in Supplementary Note 5, by evaluating the detection system based on the third evaluation index, it becomes possible to appropriately evaluate the detection system, for example. [Explanation of symbols]

[0609] 1. Management terminal 2. Server device 11 Communications Department 12 Recording section 13 Control Unit 21 Communications Department 22 Recording section 23 Control Unit 91 Transaction terminal equipment 92 Management terminal 93 Server equipment 100 Information Processing Systems 101 Management terminal 102 Management terminal 103 Management terminal 104 Management terminal 931 Communications Department 932 Recording Department 933 Control Unit 9100 Information Processing Systems 9321 Fraudulent Transaction Information Database 9322 Genuine Transaction Related Information DB

Claims

1. An estimation system for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation system includes: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquiring means for acquiring second region-related information, which is information on financial transactions based on regions related to the other financial institutions; an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means, the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input; Estimation system.

2. The region-related trained model is a first region-related trained model that outputs, as the estimation indicator, a first indicator corresponding to a precision rate based on the fraudulent transactions; a second region-related trained model that outputs, as the estimation indicator, a second indicator corresponding to a recall rate based on a financial account associated with the fraudulent transaction; The estimation system of claim 1 .

3. An estimation system for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation system includes: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquiring means for acquiring second region-related information, which is information on financial transactions based on regions related to the other financial institutions; a third acquisition means for acquiring feature-related information indicating features of the trained model for detection; an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means, the second region-related information acquired by the second acquisition means, and the feature-related information acquired by the third acquisition means, the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information, the second region-related information, and the feature-related information are input; Estimation system.

4. An estimation system for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation system includes: a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquiring means for acquiring second region-related information, which is information on financial transactions based on regions related to the other financial institutions; an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means; a determination means for determining whether or not the detection trained model is to be used as a target for estimating the introduction effect, based on the first region-related information acquired by the first acquisition means; the estimation means estimates the introduction effect for the detection trained model determined by the determination means to be the estimation target; the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input; the region-related trained model is also configured to output a determination index used to determine whether or not the detection trained model is to be a target for estimating the introduction effect when only one of the first region-related information and the second region-related information is input; The determination means determines whether the detection trained model is to be the estimation target based on the area-related trained model. Estimation system.

5. The first region-related information is information indicating the ratio of the number of account holders of the target financial institution who have addresses in a geographic area associated with the target financial institution to the total number of account holders of the target financial institution; Information indicating the ratio of the number of special fraud cases that occurred in the area related to the target financial institution during the first period to the total number of special fraud cases that occurred during the first period; or information indicating the ratio of the number of cases of special fraud by type that occurred in the area related to the target financial institution during the second period to the total number of cases of special fraud that occurred during the second period; The second region-related information is information indicating the ratio of the number of account holders at the other financial institution who reside in a geographic area associated with the other financial institution to the total number of account holders at the other financial institution; Information indicating the ratio of the number of special fraud cases that occurred in the area related to the other financial institution during the third period to the total number of special fraud cases that occurred during the third period; or information indicating the ratio of the number of cases of special fraud by type that occurred in the fourth period in an area related to the other financial institution to the total number of cases of special fraud that occurred in the fourth period; The estimation system according to any one of claims 1 to 4.

6. An estimation method for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, comprising: the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation method includes: a first acquisition step in which a first acquisition means acquires first region-related information, which is information regarding financial transactions based on a region related to the target financial institution; a second acquisition step in which second acquisition means acquires second region-related information, which is information regarding financial transactions based on regions related to the other financial institutions; an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step and the second region-related information acquired in the second acquisition step, the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input; Estimation method.

7. An estimation method for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, comprising: the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation method includes: a first acquisition step in which a first acquisition means acquires first region-related information, which is information regarding financial transactions based on a region related to the target financial institution; a second acquisition step in which second acquisition means acquires second region-related information, which is information regarding financial transactions based on regions related to the other financial institutions; a third acquisition step in which third acquisition means acquires feature-related information indicating features of the trained model for detection; an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step, the second region-related information acquired in the second acquisition step, and the feature-related information acquired in the third acquisition step, the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information, the second region-related information, and the feature-related information are input; Estimation method.

8. An estimation method for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, comprising: the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation method includes: a first acquisition step in which a first acquisition means acquires first region-related information, which is information regarding financial transactions based on a region related to the target financial institution; a second acquisition step in which second acquisition means acquires second region-related information, which is information regarding financial transactions based on regions related to the other financial institutions; an estimation step in which an estimation means estimates the introduction effect based on the first region-related information acquired in the first acquisition step and the second region-related information acquired in the second acquisition step; a determination step in which a determination means determines whether or not the detection trained model is to be a target for estimating the introduction effect, based on the first region-related information acquired by the first acquisition means; the estimation means estimates the introduction effect for the detection trained model determined by the determination means to be the estimation target; the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input; the region-related trained model is also configured to output a determination index used to determine whether or not the detection trained model is to be a target for estimating the introduction effect when only one of the first region-related information and the second region-related information is input; The determination means determines whether the detection trained model is to be the estimation target based on the area-related trained model. Estimation method.

9. An estimation program for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation program Computer, a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquiring means for acquiring second region-related information, which is information on financial transactions based on regions related to the other financial institutions; and causing the device to function as an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means; the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input; Estimation program.

10. An estimation program for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation program Computer, a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquiring means for acquiring second region-related information, which is information on financial transactions based on regions related to the other financial institutions; a third acquisition means for acquiring feature-related information indicating features of the trained model for detection; and causing the device to function as an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means, the second region-related information acquired by the second acquisition means, and the feature-related information acquired by the third acquisition means; the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information, the second region-related information, and the feature-related information are input; Estimation program.

11. An estimation program for estimating an introduction effect, which is an effect on the detection of fraudulent transactions, of introducing a trained detection model for detecting fraudulent transactions in financial transactions into a target financial institution, the detection trained model is generated based on financial transaction-related information, which is information about financial transactions at financial institutions other than the target financial institution; The estimation program Computer, a first acquisition means for acquiring first region-related information, which is information on financial transactions based on a region related to the target financial institution; a second acquiring means for acquiring second region-related information, which is information on financial transactions based on regions related to the other financial institutions; an estimation means for estimating the introduction effect based on the first region-related information acquired by the first acquisition means and the second region-related information acquired by the second acquisition means; a determination means for determining whether or not to use the trained model for detection as a target for estimating the introduction effect, based on the first region-related information acquired by the first acquisition means; the estimation means estimates the introduction effect for the detection trained model determined by the determination means to be the estimation target; the estimation means estimates the introduction effect based on a region-related trained model that is configured to output an estimation index, which is an index used to estimate the introduction effect for the detection trained model, when the first region-related information and the second region-related information are input; the region-related trained model is also configured to output a determination index used to determine whether or not the detection trained model is to be a target for estimating the introduction effect when only one of the first region-related information and the second region-related information is input; The determination means determines whether the detection trained model is to be the estimation target based on the area-related trained model. Estimation program.

Images