Evaluation system, evaluation program, and evaluation method

Abstract

To provide an evaluation system, an evaluation program, and an evaluation method that enable appropriate evaluation of a detection system. [Solution] The system comprises a first identification means for identifying a first evaluation index for evaluating the detection system, the first evaluation index being based on a detected fraudulent transaction, which is a financial transaction detected as a fraudulent transaction by the detection system; a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being based on a detected fraudulent financial account, which is a financial account associated with the detected fraudulent transaction; and a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being based on the detected fraudulent transaction and the detected fraudulent financial account, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means.

Description

[Technical Field]

[0001] The present invention relates to an evaluation system, an evaluation program, and an evaluation method. [Background technology]

[0002] Conventionally, techniques for detecting fraudulent transactions have been known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-015000 A Summary of the Invention [Problem to be solved by the invention]

[0004] There was a need for a technology that properly evaluates factors (i.e., detection systems) for detecting fraudulent transactions.

[0005] The present invention has been made in view of the above, and has an object to provide an evaluation system, an evaluation program, and an evaluation method that enable appropriate evaluation of a detection system. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the evaluation system according to claim 1 is a system for detecting fraudulent transactions in financial transactions associated with a financial account. or the fraudulent transaction within a genuine transaction An evaluation system for evaluating a detection system for detecting a storage means for storing first transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information, and second transaction-related information including second transaction content information indicating the content of the genuine financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; and a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transactions by the control means. To evaluate the detection system No. A first identification means for identifying one evaluation index; Based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the detection of the fraudulent transaction by the control means, To evaluate the detection system No. 2. Evaluation indicators the second evaluation index being different from the first evaluation index a second specifying means for specifying the first evaluation index specified by the first specifying means and the second evaluation index specified by the second specifying means,、 To evaluate the detection system No. 3. Evaluation Indicators a third evaluation index different from the first evaluation index and the second evaluation index, A third specifying means for specifying the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-unit conformance rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system that correspond to the financial transactions indicated by the first transaction content information, among the financial transactions detected as fraudulent by the control means using the detection system, to the number of financial transactions detected as fraudulent by the control means using the detection system, and identifies the transaction-unit conformance rate, which is the calculation result of the first calculation, as the first evaluation index; and the second identification means, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means, to determine a ratio of the number of financial accounts associated with the financial transactions detected as fraudulent by the control means using the detection system, among the financial accounts identified by the first account identification information, to the number of financial accounts identified by the first account identification information. Account-level recall rate and performing a second calculation to obtain the result of the second calculation. Account-level recall rate as the second evaluation index, and the third specifying means determines the transaction unit conformance rate and the Account-level recall rate The product of the above and the above trading unit conformance rate is used as the dividend. Account-level recall rate and the sum of the two is used as the divisor, and division result information indicating the division result is identified as the third evaluation index.

[0007] The evaluation system according to claim 2 is an evaluation system for evaluating a detection system for detecting fraudulent transactions among fraudulent transactions and genuine transactions in financial transactions associated with a financial account, the evaluation system comprising: a storage means for storing first transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; and second transaction-related information including second transaction content information indicating the content of the genuine financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; and a first identification means for identifying a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and a result of the fraudulent transaction detection by the control means. a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the fraudulent transaction detection result by the control means; and a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means, wherein the first identification means performs a first calculation to determine a transaction-based false positive rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system among the financial transactions indicated by the second transaction content information, to the number of financial transactions indicated by the second transaction content information, based on the first transaction-related information and the second transaction-related information stored in the storage means and the fraudulent transaction detection result by the control means, and identifies the transaction-based false positive rate indicating the calculation result of the first calculation as the first evaluation index, and the second identification meansa second calculation is performed based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transaction by the control means to determine an account-based true positive rate, which is the ratio of the number of financial accounts associated with the financial transaction detected as the fraudulent transaction by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and the account-based true positive rate indicating the calculation result of the second calculation is specified as the second evaluation index; the detection system has standards for detecting the fraudulent transactions, and the first identification means changes the standards of the detection system in multiple stages; the second identification means performs the second calculation to determine a plurality of the account-based true positive rates for each of the plurality of stages when the criteria of the detection system are changed into a plurality of stages, and the third identification means identifies an ROC curve generated based on coordinates consisting of a combination of each of the plurality of transaction-based false positive rates that are the calculation results of the first calculation and each of the plurality of account-based true positive rates that are the calculation results of the second calculation in a coordinate system with the transaction-based false positive rate as the horizontal axis and the account-based true positive rate as the vertical axis, and identifies an AUC corresponding to the area under the identified ROC curve as the third evaluation index.

[0008] The evaluation system according to claim 3 includes: 3. The evaluation system according to 1 or 2, further comprising an output unit that outputs information indicating the third evaluation index identified by the third identification unit.

[0009] The evaluation system according to claim 4 includes the evaluation system according to claim 1. or 2 In the evaluation system described in The system further includes an evaluation unit that evaluates the detection system by comparing the third evaluation index identified by the third identification unit with a predetermined threshold value.

[0010] The evaluation program according to claim 5 is an evaluation program for evaluating a detection system for detecting fraudulent transactions among fraudulent transactions and genuine transactions in financial transactions associated with a financial account, the evaluation program comprising: a computer; a storage means for storing first transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; and second transaction-related information including second transaction content information indicating the content of the genuine financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; and a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the results of the fraudulent transaction detection by the control means. a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the fraudulent transaction detection result by the control means; and a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means, and the first identification means performs a first calculation to determine a transaction unit conformance rate, which is the ratio of the number of financial transactions that fall under the financial transactions indicated by the first transaction content information among the financial transactions detected as fraudulent by the control means using the detection system to the number of financial transactions detected as fraudulent by the control means using the detection system, based on the first transaction-related information and the second transaction-related information stored in the storage means and the fraudulent transaction detection result by the control means;The second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-based reproducibility ratio, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-based reproducibility ratio, which is the result of the second calculation, as the second evaluation index; and the third identification means performs a division using the product of the transaction-based reproducibility ratio and the account-based reproducibility ratio as the dividend and the sum of the transaction-based reproducibility ratio and the account-based reproducibility ratio as the divisor, and identifies division result information, which is the result of the division, as the third evaluation index.

[0011] The evaluation program according to claim 6 an evaluation program for evaluating a detection system for detecting fraudulent transactions among fraudulent transactions and genuine transactions among financial transactions associated with a financial account, the evaluation program comprising: a computer; a storage means for storing first transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; and second transaction-related information including second transaction content information indicating the content of the genuine financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification means for identifying a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and a result of the fraudulent transaction detection by the control means; and a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means, the third evaluation index being different from the first evaluation index and the second evaluation index identified by the second identification means, wherein the first identification means performs a first calculation to determine a transaction-based false positive rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system among the financial transactions indicated by the second transaction content information, to the number of financial transactions indicated by the second transaction content information, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means, and identifies the transaction-based false positive rate indicating the calculation result of the first calculation as the first evaluation index, and the second identification meansa second calculation is performed based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transaction by the control means to determine an account-based true positive rate, which is the ratio of the number of financial accounts associated with the financial transaction detected as the fraudulent transaction by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and the account-based true positive rate indicating the calculation result of the second calculation is specified as the second evaluation index; the detection system has standards for detecting the fraudulent transactions, and the first identification means changes the standards of the detection system in multiple stages; the second identification means performs the second calculation to determine a plurality of the account-based true positive rates for each of the plurality of stages when the criteria of the detection system are changed into a plurality of stages, and the third identification means identifies an ROC curve generated based on coordinates consisting of a combination of each of the plurality of transaction-based false positive rates that are the calculation results of the first calculation and each of the plurality of account-based true positive rates that are the calculation results of the second calculation in a coordinate system with the transaction-based false positive rate as the horizontal axis and the account-based true positive rate as the vertical axis, and identifies an AUC corresponding to the area under the identified ROC curve as the third evaluation index.

[0012] The evaluation method according to claim 7 comprises: An evaluation method using an evaluation system for evaluating a detection system for detecting fraudulent transactions among fraudulent transactions and genuine transactions in financial transactions associated with a financial account, the evaluation system comprising: a storage means for storing first transaction-related information including first transaction content information indicating content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; and second transaction-related information including second transaction content information indicating content of the genuine financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information, the evaluation method comprising: a detection step in which a control means of the evaluation system detects the fraudulent transaction from the financial transaction indicated by the first transaction content information and the financial transaction indicated by the second transaction content information, based on the first transaction-related information and the second transaction-related information stored in the storage means; a first specifying step of specifying a first evaluation index for evaluating the detection system based on the fraudulent transaction detection result by the control means; a second specifying step of specifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, by a second specifying means of the evaluation system, based on the first transaction-related information and the second transaction-related information stored in the storage means and the fraudulent transaction detection result by the control means; and a third specifying step of specifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, by a third specifying means of the evaluation system, based on the first evaluation index specified by the first specifying means and the second evaluation index specified by the second specifying means, wherein the first specifying means calculates a ratio of the number of financial transactions detected as fraudulent by the control means using the detection system to the number of financial transactions detected as fraudulent by the control means, based on the first transaction-related information and the second transaction-related information stored in the storage means and the fraudulent transaction detection result by the control means.The detection system performs a first calculation to obtain a transaction unit conformance rate, which is the ratio of the number of financial transactions that correspond to the financial transactions indicated by the first transaction content information among the financial transactions detected as fraudulent by the control means, and identifies the transaction unit conformance rate, which indicates the calculation result of the first calculation, as the first evaluation index. The second identification means calculates the first transaction unit conformance rate relative to the number of financial accounts identified by the first account identification information based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means. A second calculation is performed to obtain an account unit reproducibility rate, which is the proportion of the number of financial accounts associated with the financial transaction detected as the fraudulent transaction by the control means, using the detection system among the financial accounts identified by the account identification information, and the account unit reproducibility rate indicating the calculation result of the second calculation is identified as the second evaluation index, and the third identification means performs division using the product of the transaction unit conformance rate and the account unit reproducibility rate as the dividend and the sum of the transaction unit conformance rate and the account unit reproducibility rate as the divisor, and identifies division result information indicating the division result of the division as the third evaluation index. The evaluation method according to claim 8 is an evaluation method using an evaluation system for evaluating a detection system for detecting fraudulent transactions among fraudulent transactions and genuine transactions in financial transactions associated with a financial account, the evaluation system comprising: a storage means for storing first transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; and second transaction-related information including second transaction content information indicating the content of the genuine financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information, the evaluation method comprising: a detection step in which a control means of the evaluation system detects the fraudulent transaction from among the financial transaction indicated by the first transaction content information and the financial transaction indicated by the second transaction content information, based on the first transaction-related information and the second transaction-related information stored in the storage means; a first specifying step of specifying a first evaluation index for evaluating the detection system based on the transaction-related information and the second transaction-related information and the result of the fraudulent transaction detection by the control means; a second specifying step of specifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, by a second specifying means of the evaluation system, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; and a third specifying step of specifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, by a third specifying means of the evaluation system, based on the first evaluation index specified by the first specifying means and the second evaluation index specified by the second specifying means, wherein the first specifying means calculates a ratio of the number of financial transactions indicated by the second transaction content information to the number of financial transactions indicated by the second transaction content information, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means.the second identification means performs a first calculation to obtain a transaction-based false positive rate, which is the ratio of the number of financial transactions detected as fraudulent transactions by the control means using the detection system among the financial transactions indicated by the second transaction content information, and identifies the transaction-based false positive rate indicating the calculation result of the first calculation as the first evaluation index; the second identification means performs a second calculation to obtain an account-based true positive rate, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information, to the number of financial accounts identified by the first account identification information, based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transactions by the control means; and identifies the account-based true positive rate indicating the calculation result of the second calculation as the second evaluation index. the detection system has standards for detecting the fraudulent transactions, the first identification means performs the first calculation to determine a plurality of transaction-level false positive rates for each of the multiple stages when the standards of the detection system are changed into multiple stages, the second identification means performs the second calculation to determine a plurality of account-level true positive rates for each of the multiple stages when the standards of the detection system are changed into multiple stages, and the third identification means identifies an ROC curve generated based on coordinates consisting of a combination of each of the multiple transaction-level false positive rates that are the calculation results of the first calculation and each of the multiple account-level true positive rates that are the calculation results of the second calculation in a coordinate system with the transaction-level false positive rate on the horizontal axis and the account-level true positive rate on the vertical axis, and identifies an AUC corresponding to the area under the identified ROC curve as the third evaluation index. [Effects of the Invention]

[0013] Claim 1 、2 The evaluation system according to claim 5、 6. The evaluation program according to claim 7. 、8 According to the evaluation method described in ,example For example , Examination This makes it possible to properly evaluate the distribution system.

[0016] Claim 3 According to the evaluation system described in the above, by outputting information indicating the third evaluation index, it becomes possible to provide information useful for evaluating the detection system, for example. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating fraudulent transaction-related information. [Figure 3] FIG. 10 is a diagram illustrating genuine transaction related information. [Figure 4] FIG. 1 is an explanatory diagram of a trained model for detection. [Figure 5] 10 is a flowchart of an AML-specialized F-value specification process. [Figure 6] FIG. 1 is an explanatory diagram of an AML-specialized F value. [Figure 7] FIG. 10 is a diagram illustrating the detection result of a fraudulent transaction. [Figure 8] FIG. 10 is a diagram illustrating an example of calculation of each index. [Figure 9] 10 is a flowchart of an AML-specialized AUC determination process. [Figure 10] FIG. 1 is an explanatory diagram of AML-specialized AUC. [Figure 11] 10 is a display example of a first display screen. [Figure 12] 10 is a display example of a second display screen. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the evaluation system, evaluation program, and evaluation method according to the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the embodiments. Here, the basic concepts and terminology will be explained, and then specific embodiments will be described.

[0020] (Basic concept) First, a basic concept: The evaluation system of the present invention is a system for evaluating detection systems for detecting fraudulent transactions in financial transactions associated with financial accounts.

[0021] A "detection system" is a system for detecting fraudulent transactions in financial transactions associated with a financial account. Specifically, it is a concept that includes various elements for detecting fraudulent transactions, such as a system that detects fraudulent transactions using a trained detection model, and a system that detects fraudulent transactions using any program other than a trained detection model (i.e., a rule-based system that detects fraudulent transactions using specified rules).

[0022] "Evaluating a detection system" may be interpreted as, for example, evaluating the accuracy of all or some of the elements of a detection system.

[0023] A "trained model for detection" is a model for detecting fraudulent transactions, and is a concept that includes, for example, models generated by machine learning.

[0024] "Financial transactions" is a concept that refers to financial transactions, and more specifically, to transactions conducted using a computer. Financial transactions include, for example, transactions conducted using an ATM (Automated Teller Machine) installed in a specific store, and transactions conducted using a network-based transaction function including so-called Internet banking functions, and examples include deposits, withdrawals, and transfers of currency. Furthermore, financial transactions may be interpreted as including, for example, transactions related to currency as well as transactions related to financial products other than currency, such as stocks or futures. While these "financial transactions" are generally genuine transactions, fraudulent transactions are also anticipated.

[0025] "Fraudulent transactions" are fraudulent financial transactions, and include, for example, transactions that are not in line with the original intentions of the transactors, and include, for example, transactions related to criminal activities such as fraud (including bank transfer fraud) and illegal lending. Incidentally, fraudulent transactions may also be interpreted as meaning illegal transactions, for example.

[0026] A "genuine transaction" is a genuine (legitimate) transaction in financial transactions, and is a concept that includes, for example, transactions that are in line with the original intentions of the transactors, and one example is a concept that includes transactions that are not related to criminal activity. Note that a genuine transaction may also be interpreted as indicating a lawful (or legal) transaction, for example.

[0027] In the following embodiment, an example will be described in which the accuracy of fraud detection (detection of fraudulent transactions) using a trained model for detection is evaluated.

[0028] (composition) First, an information processing system according to the present embodiment will be described. Fig. 1 is a block diagram of an information processing system according to the present embodiment.

[0029] The information processing system 100 includes an evaluation system, and includes, for example, a transaction terminal device 1, a management terminal device 2, and a server device 3.

[0030] (Configuration - Transaction Terminal Device) The transaction terminal device 1 in Figure 1 is a device (including a computer) on the financial institution's side, for example, a computer for conducting financial transactions, and as an example, is a device that is capable of communicating with an ATM installed in a convenience store or other location not shown, or a terminal used by the transactor (a personal computer, tablet terminal, smartphone, etc.) and processes information regarding financial transactions.

[0031] (Configuration-Management Terminal Device) The management terminal device 2 in Figure 1 is a device that manages financial transactions (e.g., a device including a computer having a recording unit and a control unit), for example, a device that manages financial transactions processed by the transaction terminal device 1, and as an example, a device that detects fraudulent transactions in the financial transactions.

[0032] The transaction terminal device 1 and management terminal device 2 described here are merely examples and may be changed as desired. For example, the transaction terminal device 1 and management terminal device 2 may be integrated, or a known system configuration provided in an existing financial institution may be applied. Also, some or all of the functions of the transaction terminal device 1 and management terminal device 2 may be incorporated into a server device 3, which will be described later.

[0033] (Configuration - Server Device) The server device 3 is an evaluation system, and includes, for example, a communication unit 31, a recording unit 32, and a control unit 33.

[0034] (Configuration - Server Device - Communication Unit) 1 is a communication means for communicating with an external device (for example, the management terminal device 2 or another device not shown). The specific type and configuration of this communication device 31 are arbitrary, but it can be configured using, for example, a known communication circuit or the like.

[0035] (Configuration - Server Device - Recording Unit) 1 is a recording means (storage means) that records programs and various data required for the operation of the server device 3, and is configured using, for example, a hard disk or flash memory (not shown) as an external recording device (the same applies to recording units of other devices). However, instead of or in addition to the hard disk or flash memory, any other recording medium including a magnetic recording medium such as a magnetic disk, or an optical recording medium such as a DVD or Blu-ray disc can be used (the same applies to recording units of other devices).

[0036] The recording unit 32 includes, for example, a fraudulent transaction related information database (hereinafter, the database will be referred to as "DB") 321 and a genuine transaction related information DB 322.

[0037] (Configuration - Server device - Recording unit - Fraudulent transaction related information DB) The fraudulent transaction related information DB 321 in FIG. 1 is a fraudulent transaction related information storage means for storing fraudulent transaction related information.

[0038] Figure 2 is a diagram illustrating fraudulent transaction-related information. Note that in Figure 2, for the sake of convenience, some information is omitted and shown as "..." (the same applies to other figures). Also, the items in Figure 2 are merely examples, and some items may be omitted or other items may be added (the same applies to other figures). Also, the information shown for each item in Figure 2 is for the sake of convenience (the same applies to other figures).

[0039] ===Information about fraudulent transactions=== "Fraudulent transaction related information" refers to various types of information related to fraudulent transactions, and for example, the information items shown in FIG. 2 are mutually associated.

[0040] The transaction ID in FIG. 2 is transaction identification information (hereinafter, the identification information will be referred to as "ID") that uniquely identifies a financial transaction that corresponds to a fraudulent transaction (such as "F001" in FIG. 2).

[0041] The account ID in Figure 2 is an account ID that identifies the financial account associated with the financial transaction identified by the transaction ID (e.g., "A001" in Figure 2). Note that the "financial account associated with the financial transaction" may be interpreted as indicating, for example, the financial account in which the financial transaction was made.

[0042] The transaction content information in Figure 2 is information that indicates the content of the financial transaction identified by the transaction ID (Figure 2 shows examples such as the transaction amount being "20,000" yen, the transaction method being "ATM", and the transaction time being "23:05").

[0043] ===Description=== The information at the top of Figure 2 indicates that the financial transaction identified by "F001" is a fraudulent transaction, that the financial transaction was carried out in the financial account identified by "A001," and that the details of the financial transaction correspond to "Amount: 20,000, Method: ATM, Time: 23:05, ...."

[0044] Figure 2 also shows that the financial transactions identified by "F001," "F002," "F003," and "F004" are fraudulent transactions, and that each of these financial transactions was conducted in the financial account identified by "A001."

[0045] FIG. 2 also shows that the financial transactions identified by "F005" are fraudulent transactions, and that each of the financial transactions is carried out in a financial account identified by "A002."

[0046] That is, FIG. 2 also shows that the financial accounts identified by "A001" and "A002" are financial accounts in which fraudulent transactions were made.

[0047] ===Storage Method=== The method for storing the fraudulent transaction-related information in Figure 2 is arbitrary, and may be, for example, by inputting information related to actual fraudulent transactions reported from actual financial transactions at a specified financial institution (e.g., a financial institution illustrated in Figure 1), or may be stored using any other method.

[0048] (Configuration - Server Device - Recording Unit - Genuine Transaction Related Information DB) The genuine transaction related information DB 322 in Fig. 1 is a genuine transaction related information storage means for storing genuine transaction related information. Fig. 3 is a diagram showing an example of genuine transaction related information.

[0049] ===Genuine Transaction Information=== "Genuine transaction related information" refers to various information related to genuine transactions, and for example, the information items shown in FIG. 3 are mutually associated.

[0050] The transaction ID in FIG. 3 is a transaction ID that uniquely identifies a financial transaction that is a genuine transaction (such as "C006" in FIG. 2).

[0051] The account ID in FIG. 3 is an account ID that identifies the financial account associated with the financial transaction identified by the transaction ID (such as "A003" in FIG. 3).

[0052] The transaction content information in FIG. 3 is information indicating the content of the financial transaction identified by the transaction ID (in FIG. 3, the transaction amount is exemplified as "5000" yen, etc.).

[0053] ===Description=== The information at the top of Figure 3 indicates that the financial transaction identified by "C006" is a genuine transaction, that the financial transaction was carried out in the financial account identified by "A003," and that the content of the financial transaction corresponds to "Amount: 5000, ...."

[0054] FIG. 3 also shows that the financial transactions identified by "C006" and "C007" are genuine transactions, and that each of the financial transactions is carried out in the financial account identified by "A003."

[0055] FIG. 3 also shows that the financial transactions identified by "C008" are genuine transactions, and that each of the financial transactions is carried out in the financial account identified by "A004."

[0056] That is, FIG. 3 also shows that the financial accounts identified by "A003" and "A004" are financial accounts in which genuine transactions have been made.

[0057] ===Storage Method=== The method for storing the genuine transaction-related information in Figure 3 is arbitrary, and may be, for example, by inputting information related to actual genuine transactions confirmed from actual financial transactions at a specified financial institution (e.g., the financial institution illustrated in Figure 1), or may be stored using any other method.

[0058] (Configuration - Server Device - Control Unit) 1 is a control means for controlling the server device 3, and is specifically a computer including a CPU, various programs interpreted and executed on the CPU (including basic control programs such as an OS and application programs that are started on the OS and realize specific functions), and an internal memory such as a RAM for storing programs and various data (the same applies to control units of other devices). In particular, the program according to the embodiment is installed on the server device 3 via an arbitrary recording medium or a network, thereby substantially configuring each unit of the control unit 33.

[0059] The control unit 33 includes, for example, a first specifying means, a second specifying means, a third specifying means, and an evaluation means.

[0060] ===First identification means=== The first identification means is a means for identifying a first evaluation index based on a detected fraudulent transaction, which is a financial transaction detected as a fraudulent transaction by the detection system, and is used to evaluate the detection system.

[0061] The first identification means identifies, for example, an evaluation index corresponding to a matching rate regarding the detection of a fraudulent transaction by the detection system, based on the fraudulent transaction, as the first evaluation index.

[0062] The first identification means identifies, for example, as the first evaluation index, an evaluation index corresponding to a false positive rate regarding the detection of a fraudulent transaction by the detection system, based on the fraudulent transaction.

[0063] ===Second identification means=== The second identification means is a means for identifying a second evaluation indicator based on the detected fraudulent financial account, which is a financial account associated with the detected fraudulent transaction, and is used to evaluate the detection system.

[0064] The second identification means identifies, for example, an evaluation index corresponding to a recall rate based on a financial account regarding the detection of fraudulent transactions by the detection system as the second evaluation index.

[0065] The second identification means identifies, for example, an evaluation index corresponding to a true positive rate based on a financial account regarding the detection of fraudulent transactions by the detection system as the second evaluation index.

[0066] ===Third identification means=== The third identification means is a means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being based on detected fraudulent transactions and detected fraudulent financial accounts, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means.

[0067] The third identification means identifies, for example, an evaluation index corresponding to an F value based on precision and recall as the third evaluation index.

[0068] The third identification means identifies, for example, an evaluation index corresponding to an AUC based on a false positive rate and a true positive rate as the third evaluation index.

[0069] ===Evaluation Method=== The evaluation means is means for evaluating the detection system based on the third evaluation index identified by the third identification means.

[0070] The processes performed by each means of the control unit 33 will be described later.

[0071] (process) Next, we will explain, for example, financial institution-side fraud detection processing, AML-specialized F value identification processing, and AML-specialized AUC identification processing as processing performed by the information processing system 100 configured as described above. Note that "AML" may be interpreted as an abbreviation for "Anti-Money Laundering."

[0072] (Processing - Financial institution side fraud detection processing) The fraud detection process on the financial institution side will now be described. The fraud detection process on the financial institution side is a process for detecting fraudulent transactions (fraud detection), and is a process that is repeatedly executed by, for example, the management terminal device 2 of FIG.

[0073] (Processing - Financial institution's fraud detection processing - Pre-trained detection model) First, a trained model for detection used in fraud detection will be described. Fig. 4 is an explanatory diagram of the trained model for detection. Note that Fig. 4 is a diagram for convenience of explanation, and the number of intermediate layers is not limited to two.

[0074] As mentioned above, the "trained model for detection" is a model for detecting fraudulent transactions, and as shown in Figure 4, for example, it is a model that outputs an AI score when transaction content information is input.

[0075] As explained in FIG. 2 and FIG. 3, "transaction content information" is information indicating the content of the financial transaction, such as the amount, method, and time.

[0076] The "AI score" is information that indicates the likelihood that a financial transaction corresponding to the transaction content information input into the trained model for detection is fraudulent or genuine.

[0077] The specific content of this AI score is arbitrary, but for example, we will explain using numerical information ranging from "0" to "1" indicating the likelihood of a fraudulent transaction from lowest to highest. In detail, for example, the larger the AI ​​score value, the higher the likelihood of a fraudulent transaction, i.e., the lower the likelihood of a genuine transaction. Also, for example, the smaller the AI ​​score value, the lower the likelihood of a fraudulent transaction, i.e., the higher the likelihood of a genuine transaction.

[0078] For example, if there is a financial transaction with an "AI score" of "0.3" ("First Financial Transaction"), a financial transaction with an "AI score" of "0.5" ("Second Financial Transaction"), and a financial transaction with an "AI score" of "0.9" ("Third Financial Transaction"), the third financial transaction is most likely to be fraudulent, the second financial transaction is second most likely to be fraudulent, and the first financial transaction is least likely to be fraudulent.

[0079] In this embodiment, a detection method is described in which a threshold value (a predetermined number between "0" and "1") is set to be compared with the AI ​​score, the AI ​​score is compared with the threshold value, and fraudulent transactions are detected based on the magnitude relationship of the AI ​​score relative to the threshold value. Specifically, the detection method is set so that if the AI ​​score is below the threshold value, the transaction is considered to be genuine and no fraudulent transaction is detected, and if the AI ​​score is above the threshold value, the transaction is considered to be fraudulent and a fraudulent transaction is detected.

[0080] In this detection method, if the threshold is, for example, "0.85," the AI ​​scores of the first and second financial transactions mentioned above are "0.3" and "0.5," respectively, which are below the threshold of "0.85," and therefore these first and second financial transactions will not be detected as fraudulent transactions. On the other hand, the AI ​​score of the third financial transaction mentioned above is "0.9," which is above the threshold of "0.85," and therefore this third financial transaction will be detected as fraudulent.

[0081] The learning method for such a trained detection model is arbitrary, but it may be generated by learning using machine learning with teacher data, for example. Specifically, it may be generated by learning using machine learning using a large number of combinations of transaction content information of fraudulent transactions and the AI ​​score "1" corresponding to that transaction content information, and a large number of combinations of transaction content information of genuine transactions and the AI ​​score "0" corresponding to that transaction content information.

[0082] (Processing - Financial institution's fraud detection processing - Processing details) Next, the contents of the fraud detection process on the financial institution side will be explained, but since this process can be similar to known processes, only an outline will be explained.

[0083] For example, it is assumed that the learned detection model of Figure 4 is stored in the recording unit of the management terminal device 2 of Figure 1. Furthermore, for example, when an actual financial transaction is carried out, each transaction terminal device 1 of Figure 1 transmits executed financial transaction information indicating the content of the financial transaction (e.g., information including information corresponding to each item of Figures 2 and 3) to the management terminal device 2, and the content of the actually carried out financial transaction can be ascertained on the management terminal device 2 side based on the executed financial transaction information.

[0084] When a financial transaction is carried out, the control unit of the management terminal device 2 receives the executed financial information from the transaction terminal device 1 and uses the learned detection model recorded in the recording unit to detect fraudulent transactions using the detection method described above.

[0085] For example, let us consider a case where the threshold value is set to "0.85." In this case, the management terminal device 2 inputs the transaction content information contained in the executed financial information received from the transaction terminal device 1 into the trained model for detection and obtains the AI ​​score output from the trained model for detection.

[0086] Next, the acquired AI score is compared with the set threshold of "0.85," and if the acquired AI score is less than "0.85," the financial transaction corresponding to the transaction content information input into the trained model for detection is deemed to be genuine and is not detected as a fraudulent transaction. On the other hand, if the acquired AI score is "0.85" or higher, the financial transaction corresponding to the transaction content information input into the trained model for detection is deemed to be fraudulent and is detected as a fraudulent transaction.

[0087] Subsequent processing is optional, but for example, the control unit of the management terminal device 2 may record information indicating the detected fraudulent financial transaction in a recording unit, or may notify a manager or the like.

[0088] ===Interpretation of terms=== In addition, since the detection trained model and threshold (not shown) in Figure 4 can be interpreted as elements for detecting fraudulent transactions, both or one of these may be interpreted as corresponding to a "detection system."

[0089] (Processing - AML specialized F value specific processing) Next, the AML-specialized F-value identification process will be described. Fig. 5 is a flowchart of the AML-specialized F-value identification process (hereinafter, each step will be referred to as "S"). The AML-specialized F-value identification process is a process for identifying and evaluating an AML-specialized F-value, and is a process executed by, for example, the server device 3 in Fig. 1.

[0090] The timing for executing this AML-specialized F-value identification process is arbitrary, but for example, execution will begin when the administrator performs a specified operation to execute the process, and the explanation will begin from the point where execution begins (the same applies to the AML-specialized AUC identification process described below).

[0091] Also, here, for example, a case where an AML-specified F-measure is specified and evaluated for the first detection trained model and the second detection trained model will be described as an example. Note that the first detection trained model (also referred to as the "first model") and the second detection trained model (also referred to as the "second model") are the aforementioned "detection trained models" and are detection trained models generated using mutually different training data. In other words, the explanatory variables and objective variables of the first model and the second model are the transaction content information and AI score shown in Figure 4, which are mutually common, but the weights, biases, etc. are mutually different models.

[0092] ===AML specialized F value=== Fig. 6 is an explanatory diagram of the AML-specified F-value. The "AML-specified F-value" is a third evaluation index for evaluating a detection system, and specifically, is an evaluation index corresponding to the F-value based on precision and recall, and is a numerical value calculated, for example, by the calculation formula shown in Fig. 6. The AML-specified F-value will be described in detail later.

[0093] ===SA1=== 5, the control unit 33 of the server device 3 acquires target model-related information (information related to the trained model for detection to be evaluated). Specifically, although this is optional, for example, the target model-related information may be recorded in the recording unit 32, and the control unit 33 may be configured to acquire the recorded target model-related information, or the control unit 33 may be configured to acquire target model-related information input by an administrator.

[0094] Here, for example, information indicating the first and second models and the threshold values ​​used in each of the models is acquired as the target model related information.

[0095] ===SA2=== In SA2 of Figure 5, the control unit 33 of the server device 3 acquires fraudulent transaction related information and genuine transaction related information as information for evaluating the detection trained model indicated by the target model related information acquired in SA1.

[0096] Here, for example, the fraudulent transaction related information in FIG. 2 and the genuine transaction related information in FIG. 3 are acquired.

[0097] ===SA3=== 5, the control unit 33 of the server device 3 uses the fraudulent transaction-related information and genuine transaction-related information acquired in SA2 to identify the AML-specified F-value for the detection trained model indicated by the target model-related information acquired in SA1. Specifically, the following first to fourth steps are performed.

[0098] ==Step 1== In the first step, for each financial transaction indicated by each transaction-related information obtained in SA2, fraudulent transactions are detected using the aforementioned detection method (a method of detecting by comparing the AI ​​score with a threshold) using the detection-use trained model and threshold indicated by the target model-related information obtained in SA1.

[0099] Figure 7 is a diagram illustrating the results of fraudulent transaction detection. Figure 7 illustrates the results of fraudulent transaction detection performed using the first and second models for financial transactions indicated by the transaction-related information in Figures 2 and 3. The following explanation will be based on the results for the eight financial transactions illustrated in Figure 7.

[0100] The transaction IDs and account IDs in Figure 7 are the same as the information with the same names in Figures 2 and 3. The "First Model" column in Figure 7 shows examples of the detection results of fraudulent transactions performed using the first model, with "Fraud" indicating that the transaction was detected as fraudulent, or "Genuine" indicating that the transaction was not detected as fraudulent (i.e., the transaction was determined to be genuine). The "Second Model" column in Figure 7 shows examples of the detection results of fraudulent transactions performed using the second model, with the same information as in the case of the first model.

[0101] For example, when the transaction content information in the top row of Figure 2 is input to the first model indicated by the target model-related information acquired by SA1 for the financial transaction indicated by the information in the top row of Figure 2, the first model outputs an AI score of, for example, "0.95." If the threshold used in the first model indicated by the target model-related information is, for example, "0.85," the AI ​​score is equal to or greater than the threshold, and therefore a fraudulent transaction is detected (see the "First Model" column in the top row of Figure 7). Similar processing is then performed on the second model and other financial transactions to detect fraudulent transactions, as shown in the "First Model" and "Second Model" columns of Figure 7.

[0102] That is, for example, when the first model is used, five financial transactions with "transaction ID" = "F001," "F002," "F005," "C006," and "C007" are detected as fraudulent transactions, while the other three financial transactions are not detected as fraudulent transactions.

[0103] Furthermore, for example, when the second model is used, six financial transactions with "transaction ID" = "F001," "F002," "F003," "F004," "C006," and "C007" are detected as fraudulent transactions, while the other two financial transactions are not detected as fraudulent transactions.

[0104] ==Second Step== In the second step, a transaction unit matching rate is determined based on the detection results in the first step.

[0105] The "transaction-based relevance rate" is a first evaluation index based on detected fraudulent transactions, which are financial transactions detected as fraudulent by a detection system, and is the first evaluation index for evaluating a detection system. Specifically, the "transaction-based relevance rate" is an evaluation index corresponding to the relevance rate based on the fraudulent transactions detected by the detection system.

[0106] The content of the transaction-based conformance rate is arbitrary as long as it meets the above definition. For example, the transaction-based conformance rate may be the ratio of the number of financial transactions that were actually fraudulent to the number of financial transactions detected as fraudulent (detected fraudulent transactions).

[0107] In processing, the transaction-level matching rate is calculated as the result of dividing the number of financial transactions that were actually fraudulent among the detected fraudulent transactions by the number of financial transactions detected as fraudulent (detected fraudulent transactions).The "financial transactions that were actually fraudulent among the detected fraudulent transactions" correspond to the transaction IDs "F~~" in Figure 7 (i.e., financial transactions indicated by the fraudulent transaction-related information in Figure 2).

[0108] Figure 8 is a diagram showing an example of calculation of each index. Here, for example, the transaction unit conformance rate for the first model is determined as "3 (financial transactions of F001, F002, F005)" ÷ "5 (financial transactions of F001, F002, F005, C006, C007)" = 60% (see the "First model" and "Transaction unit conformance rate" columns in Figure 8).

[0109] Furthermore, for example, the transaction unit conformance rate for the second model is determined to be "4 (financial transactions of F001, F002, F003, F004)" ÷ "6 (financial transactions of F001, F002, F003, F004, C006, C007)" = 66.7% (see the "Second Model" and "Transaction Unit Conformance Rate" columns in Figure 8).

[0110] ==Third Step== In the third step, the account-level recall rate is determined based on the detection results in the first step.

[0111] The "account-based recall rate" is a second evaluation metric based on the detected fraudulent financial account, which is a financial account associated with a detected fraudulent transaction, and is a second evaluation metric for evaluating a detection system. Specifically, the "account-based recall rate" is an evaluation metric corresponding to the recall rate based on a financial account for the detection of fraudulent transactions by a detection system.

[0112] The content of the account-based recall rate is arbitrary as long as it meets the above definition, but for example, the account-based recall rate will be the ratio of the number of financial accounts in which financial transactions detected as fraudulent (detected fraudulent transactions) occurred to the number of financial accounts in which fraudulent transactions actually occurred. Note that duplicate financial accounts will be counted as one.

[0113] Regarding processing, the account-level recall rate is calculated as the result of the calculation: "the number of financial accounts (detected fraudulent financial accounts) in which financial transactions detected as fraudulent (detected fraudulent transactions) were made among the financial accounts in which fraudulent transactions actually occurred" divided by "the number of financial accounts in which fraudulent transactions actually occurred." Note that a financial account associated with at least one detected fraudulent transaction is defined as a "financial account (detected fraudulent financial account) in which a financial transaction detected as fraudulent (detected fraudulent transaction) was made," and a financial account associated with at least one fraudulent transaction is defined as a "financial account in which a fraudulent transaction actually occurred."

[0114] Here, for example, the account-level reproducibility for the first model is determined as "2 (both of the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" ÷ "2 (the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" = 100% (see the "First model" and "Account-level reproducibility" columns in Figure 8).

[0115] Furthermore, for example, the transaction-level accuracy rate for the second model is determined as "1 (only the financial account A001 among the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" ÷ "2 (the financial accounts A001 and A002 to which the financial transaction of "F~~" is associated)" = 50% (see the "Second model" and "Account-level recall" columns in Figure 8).

[0116] ==Fourth Step== In the fourth step, an AML-specified F-value is determined based on the processing results of the second and third steps.

[0117] The "AML-specific F-score" is a third evaluation metric based on the number of detected fraudulent transactions and fraudulent financial accounts, and is used to evaluate detection systems. Specifically, the "AML-specific F-score" is an evaluation metric corresponding to the F-score based on precision and recall.

[0118] The content of the AML-specialized F value is arbitrary as long as it satisfies the above definition, but for example, a case where the calculation result of the calculation formula shown in Figure 6 is used as the AML-specialized F value will be illustrated. Note that in Figure 6, "β" is a predetermined value greater than 0, but if emphasis is placed on the account-based recall rate, a value greater than "1" can be used. Note that the specific value of "β" is set by the administrator, etc., and for convenience, the following description will be given using as an example a case where "β" = "1".

[0119] Regarding the processing, the transaction-based matching rate determined in the second step and the account-based matching rate determined in the third step are applied to the calculation formula in Figure 6 to perform a calculation, and the calculation result is identified as the AML-specialized F value.

[0120] Here, for example, the AML-specialized F-value for the first model is specified as 75% (see the "First Model" and "AML-specialized F-value" columns in Figure 8), and the AML-specialized F-value for the second model is specified as 57% (see the "Second Model" and "AML-specialized F-value" columns in Figure 8).

[0121] ===SA4=== In SA4 of FIG. 5, the control unit 33 of the server device 3 evaluates the detection trained model and the like indicated by the target model related information based on the AML-specified F-measure identified in SA3.

[0122] The specific evaluation method is arbitrary and not limited, but for example, a case where a larger AML-specified F value is evaluated as being better will be described as an example.

[0123] Here, for example, as shown in the "AML-specialized F-value" column in Fig. 8, the first model may be evaluated as being superior because the AML-specialized F-value of the first model is larger than that of the second model. In this case, the combination of the first model and the thresholds used in the first model may be evaluated as being superior to the combination of the second model and the thresholds used in the second model.

[0124] Alternatively, for example, an F threshold (a predetermined numerical value) to be compared with the AML-specified F value may be determined in advance, and a model having an AML-specified F value greater than the F threshold may be evaluated as superior.

[0125] The subsequent processing is optional, but for example, the detection trained model evaluated as superior, or the combination of the detection trained model and a threshold, may be sent to the management terminal device 2 at the financial institution illustrated in Figure 1, and fraudulent transactions may be detected at the financial institution based on the information sent (the same applies to the AML-specialized AUC identification processing described below).

[0126] (Processing - AML-specific AUC specific processing) Next, the AML-specialized AUC identification process will be described. Fig. 9 is a flowchart of the AML-specialized AUC identification process. The AML-specialized AUC identification process is a process for identifying and evaluating the AML-specialized AUC, and is a process executed by, for example, the server device 3 in Fig. 1.

[0127] Also, here, for example, a case where the AML-specialized AUC is specified and evaluated for the first model and the second model will be described as an example.

[0128] ===AML specific AUC=== Fig. 10 is an explanatory diagram of the AML-specialized AUC. The "AML-specialized AUC" is a third evaluation index for evaluating a detection system, and specifically, is an evaluation index corresponding to the AUC based on the false positive rate and the true positive rate, such as the AUC (area under the ROC curve) shown in Fig. 10. The AML-specialized AUC will be described in detail later.

[0129] ===SB1=== In SB1 of FIG. 9, the control unit 33 of the server device 3 acquires target model related information (information related to the trained model for detection to be evaluated).

[0130] Here, for example, information indicating the first model and the second model described above is acquired as the target model related information.

[0131] ===SB2=== In SB2 in FIG. 9, the same processing as in SA2 in FIG. 5 is performed.

[0132] ===SB3=== 9, the control unit 33 of the server device 3 uses the fraudulent transaction-related information and genuine transaction-related information acquired in SB2 to identify the AML-specialized AUC for the detection trained model indicated by the target model-related information acquired in SB1. Specifically, the following first to fourth steps are performed.

[0133] ==Step 1== In the first step, for each financial transaction indicated by each transaction-related information acquired by SB2, the detection trained model indicated by the target model-related information acquired by SB1 is used to detect fraudulent transactions using the detection method described above (a method of detecting by comparing the AI ​​score with a threshold). Here, for example, detection is performed as shown in Figure 7.

[0134] ==Second Step== In the second step, the transaction-level false positive rate is determined based on the detection results in the first step.

[0135] The "transaction-based false positive rate" is a first evaluation metric based on detected fraudulent transactions, which are financial transactions detected as fraudulent by a detection system, and is the first evaluation metric for evaluating a detection system. Specifically, the "transaction-based false positive rate" is an evaluation metric corresponding to the false positive rate based on the fraudulent transactions detected by the detection system.

[0136] The content of the transaction-level false positive rate is arbitrary as long as it meets the above definition. For example, the transaction-level false positive rate may be the ratio of the number of financial transactions that were not actually fraudulent but were mistakenly detected as fraudulent to the number of financial transactions that were not actually fraudulent (financial transactions that were actually genuine).

[0137] Regarding processing, the transaction-level false positive rate is calculated as the result of dividing the number of financial transactions that were mistakenly detected as fraudulent despite not actually being fraudulent by the number of financial transactions that were not actually fraudulent (financial transactions that were actually genuine).

[0138] In the case of FIG. 7, the transaction-based false positive rate for the first model is determined to be "2 (financial transactions of C006 and C007)" ÷ "3 (financial transactions of C006, C007, and C008)" = 66.7%.

[0139] In the case of FIG. 7, the transaction-based false positive rate for the second model is also determined to be "2 (financial transactions of C006 and C007)" ÷ "3 (financial transactions of C006, C007, and C008)" = 66.7%.

[0140] ==Third Step== In the third step, the account-level true positive rate is determined based on the detection results in the first step.

[0141] The "account-level true positive rate" is a second evaluation metric based on detected fraudulent financial accounts, which are financial accounts associated with detected fraudulent transactions, and is a second evaluation metric for evaluating a detection system. Specifically, the "account-level true positive rate" is an evaluation metric corresponding to the true positive rate based on financial accounts regarding the detection of fraudulent transactions by a detection system.

[0142] The content of the account-level true positive rate is arbitrary as long as it meets the above definition, but for example, similar to the account-level recall rate described above, we will use the ratio of the number of financial accounts in which financial transactions detected as fraudulent (detected fraudulent transactions) occurred to the number of financial accounts in which fraudulent transactions actually occurred as the account-level true positive rate. Note that duplicate financial accounts will be counted as one.

[0143] The processing is the same as in the case of the account-based true recall described above.

[0144] ==Fourth Step== In the fourth step, the threshold (the threshold compared with the AI ​​score output from the detection trained model) is changed in multiple stages (for example, the threshold is increased from "0" to "1" in increments of "0.0001"), and steps 1 to 3 are repeated multiple times to identify multiple combinations of transaction-level false positive rates and account-level true positive rates. An ROC curve is then identified by connecting the coordinates consisting of combinations of transaction-level false positive rates and account-level true positive rates in the coordinate system shown in Figure 10, and the AUC corresponding to the area under the identified ROC curve is identified as the AML-specialized AUC.

[0145] Here, for example, although not specifically shown in FIG. 10, the AUC corresponding to the ROC curve for the first model is identified as the AML-specialized AUC of the first model, and the AUC corresponding to the ROC curve for the second model is identified as the AML-specialized AUC of the second model.

[0146] ===SB4=== In SB4 of FIG. 9, the control unit 33 of the server device 3 evaluates the detection trained model, etc. indicated by the target model related information, based on the AML-specialized AUC identified in SB3.

[0147] The specific evaluation method is arbitrary and not limited, but for example, a case where the larger the value of the AML-specialized AUC, the better the evaluation will be explained as an example.

[0148] Here, for example, if the AML-specialized AUC of the first model is greater than that of the second model, the first model may be evaluated as being superior.

[0149] Alternatively, for example, an AUC threshold to be compared with the AML-specialized AUC may be determined in advance, and a model whose AML-specialized AUC is greater than the AUC threshold may be evaluated as superior.

[0150] Subsequent processing is optional, but for example, the detection trained model evaluated as superior, or a combination of the detection trained model and a threshold, may be sent to the management terminal device 2 at the financial institution illustrated in Figure 1, and fraudulent transactions may be detected at the financial institution based on the information sent.

[0151] (Effects of the embodiment) Thus, according to this embodiment, by specifying the AML-specialized F-value and AML-specialized AUC, which are the third evaluation indices based on detected fraudulent transactions and detected fraudulent financial accounts, it is possible to use evaluation indices that take into account, for example, both financial transactions (detected fraudulent transactions) and financial accounts (detected fraudulent financial accounts), thereby making it possible to appropriately evaluate the detection system.

[0152] Furthermore, by specifying an AML-specific F-measure, which is an evaluation index corresponding to an F-measure based on the precision rate based on fraudulent transactions and the recall rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation index that takes into account, for example, both the precision rate for financial transactions and the recall rate for financial accounts, thereby making it possible to appropriately evaluate detection systems.

[0153] By specifying an AML-specific AUC, which is an evaluation metric that corresponds to an AUC based on the false positive rate based on fraudulent transactions and the true positive rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation metric that takes into account, for example, both the false positive rate for financial transactions and the true positive rate for financial accounts, thereby making it possible to appropriately evaluate detection systems.

[0154] Furthermore, by evaluating the detection system based on the AML-specific F-score and AML-specific AUC, which are the third evaluation indexes, it becomes possible to appropriately evaluate the detection system, for example.

[0155] [Modifications to the embodiment] Although the embodiments of the present invention have been described above, the specific configurations and means of the present invention can be modified and improved as desired within the scope of the technical concept of the present invention as set forth in the claims. Such modifications will be described below.

[0156] (About the problem to be solved and the effects of the invention) First, the problems that the invention aims to solve and the effects of the invention are not limited to those described above, and may vary depending on the implementation environment of the invention and the details of the configuration, and may solve only some of the problems described above or achieve only some of the effects described above.

[0157] (Regarding decentralization and integration) Furthermore, the electrical components described above are functional concepts and do not necessarily have to be physically configured as shown in the drawings. In other words, the specific form of distribution or integration of each part is not limited to that shown in the drawings, and all or part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, the term "device" in this application is not limited to a single device, but includes a device configured from multiple devices.

[0158] (shape, numbers, structure, time series) The components illustrated in the embodiments and drawings may be modified and improved as desired within the scope of the technical concept of the present invention in terms of shape, numerical value, or the structure or chronological relationship of multiple components.

[0159] (Evaluation (Part 1)) Furthermore, in the above embodiment, a case has been described in which a detection system is evaluated based on the AML-specified F-value identified in SA3 of Figure 5 or the AML-specified AUC identified in SB3 of Figure 9, but the detection system may be configured to be evaluated using only the AML-specified F-value, or the detection system may be configured to be evaluated using only the AML-specified AUC, or the detection system may be configured to be evaluated using both the AML-specialized F-value and the AML-specialized AUC.

[0160] (Evaluation (Part 2)) Alternatively, the "first model" described in the above embodiment may be a trained detection model currently used by the financial institution illustrated in FIG. 1, and the "second model" may be a trained detection model that is a candidate for new application to the financial institution. The server device 3 may then use the method described in the embodiment to evaluate which of the first model and the second model is superior. If the server device 3 determines that the second model is superior to the first model, it may transmit the second model to the management terminal device 2 and apply the second model to the financial institution. The management terminal device 2 then performs a process to detect fraudulent transactions using the second model. If the server device 3 determines that the first model is superior to the second model, it may not apply the second model.

[0161] (Evaluation (Part 3)) Furthermore, as the "detection trained model that is a candidate for new application to financial institutions" described as the "second model" in "(Regarding Evaluation (Part 2))" above, a detection trained model generated by re-training the "first model" (i.e., a tuned detection trained model) may be adopted, or a detection trained model newly generated separately from the "first model" may be adopted.

[0162] (Application examples) The AML-specialized AUC may also be configured to be used to early terminate machine learning of a trained detection model. Specifically, losses may be calculated for training training data and validation training data, and training may be terminated when the difference between the two calculated loss values ​​exceeds a predetermined threshold. The specific configuration of the training training data and validation training data is arbitrary as long as they are information related to financial transactions. For example, data with a configuration similar to that of the fraudulent transaction-related information in FIG. 2 and the genuine transaction-related information in FIG. 3 may be used. The loss may also be arbitrary as long as the AML-specialized AUC is used. For example, the calculation result of "w1 × ordinary cross entropy error (CrossEntropyLoss) + w2 × AML-specialized AUC" may be used as the loss. Note that w1 and w2 are predetermined values ​​set by an administrator.

[0163] (About output methods) Furthermore, the control unit 33 of the server device 3 may be configured to include an output means. The "output means" refers to a means for outputting information indicating the third evaluation index identified by the third identification means. The term "outputting information" includes, for example, displaying and outputting information, outputting information as sound or voice, outputting information in the form of a log file or a database table (i.e., saving information), printing and outputting information on an external medium such as paper, recording and outputting information on a recording medium such as an external memory, and transmitting information.

[0164] Here, for example, a case will be described in which the server device 3 displays and outputs information indicating the third evaluation index. Fig. 11 is a display example of the first display screen, and Fig. 12 is a display example of the second display screen.

[0165] ===1st display screen=== The "first display screen" in Fig. 11 is a screen that displays the AML-specialized F-value, and more specifically, is a concept that shows the change in the AML-specialized F-value over time. In this first display screen, the AML-specialized F-value for each date is displayed in the form of a line graph, with the horizontal axis representing the date and the vertical axis representing the AML-specialized F-value.

[0166] The display method for this first display screen is arbitrary, but for example, the server device 3 may be configured to store the detection trained model and thresholds used by the financial institution illustrated in Figure 1, and to periodically and repeatedly transmit or input information corresponding to each piece of transaction-related information shown in Figures 2 and 3 from the financial institution to the server device 3, thereby accumulating new information as each piece of transaction-related information shown in Figures 2 and 3. For example, the server device 3 may be configured to transmit or input information about actual fraudulent transactions and genuine transactions for a single day to the server device 3 at a predetermined time on that day, thereby accumulating new information as each piece of transaction-related information shown in Figures 2 and 3.

[0167] Then, on a specific date (for example, a date at seven-day intervals, such as June 1, 2024 or June 8, 2024), the control unit 33 of the server device 3 executes SA1 to SA3 of FIG. 5 using the transaction-related information of FIGS. 2 and 3 for the seven days immediately preceding that date to identify an AML-specialized F-value, generates screen information for displaying a first display screen showing the identified AML-specialized F-value, and sends the generated screen information to an arbitrary display device (for example, an administrator's terminal device (personal computer, smartphone, tablet terminal, etc.)), thereby displaying the first display screen on that display device. Here, for example, the first display screen of FIG. 11 is displayed.

[0168] The above process may be modified as desired. For example, the AML-specialized F value may be identified using the transaction-related information in Figures 2 and 3 for the most recent specified number of days for each specified number of days, so that the first display screen includes the change over time in the AML-specialized F value, including the latest AML-specialized F value (the same applies to the second display screen).

[0169] Furthermore, the display format of the AML-specialized F value is not limited to a line graph format, and may be configured to display in any other graph format, or may be configured to display a display format other than a graph format (for example, numerical text information, etc.) (the same applies to the second display screen).

[0170] ===Second display screen=== The "second display screen" in Figure 12 is a screen that displays the AML-specialized AUC, specifically, a concept showing the change in the AML-specialized AUC over time. In this second display screen, the AML-specialized AUC for each date is displayed in the form of a line graph, with the horizontal axis representing the date and the vertical axis representing the AML-specialized AUC.

[0171] The display method for this second display screen is the same as the display method for the first display screen described above.

[0172] That is, on a specific date (for example, a date at seven-day intervals, such as June 1, 2024 or June 8, 2024), the control unit 33 of the server device 3 executes SB1 to SB3 in Figure 9 using the transaction-related information in Figures 2 and 3 for the seven days immediately preceding that date to identify the AML-specialized AUC, generates screen information for displaying a second display screen showing the identified AML-specialized AUC, and transmits the generated screen information to an arbitrary display device, thereby displaying the second display screen on that display device. Here, for example, the second display screen in Figure 12 is displayed.

[0173] By configuring in this manner, it is possible to provide information useful for evaluating the detection system, for example, by outputting information indicating the third evaluation index, the AML-specific F-value and the AML-specific AUC.

[0174] (About AML-specialized F-value) In the above embodiment, the calculation formula for the AML-specialized F-value is described using the formula shown in Fig. 6, but the present invention is not limited to this. For example, any modification may be made, such as omitting "β" in Fig. 6 or multiplying by another weight coefficient (numerical value).

[0175] (About the pre-trained detection model) The trained detection model in Figure 4 may be modified as desired. For example, it may be configured so that the objective variable is information indicating whether the financial transaction corresponding to the transaction content information is fraudulent (for example, "fraudulent" indicating a fraudulent transaction or "genuine" indicating a genuine transaction).

[0176] (Interpretation of terms) In addition, since the transaction-level relevance rate and account-level relevance rate in Figure 6 are used to identify the AML-specific F value, they can also be interpreted as evaluation indicators for evaluating detection systems.

[0177] Additionally, because the transaction-level false positive rate and account-level true positive rate in Figure 10 are used to identify the AML-specific AUC, they can also be interpreted as evaluation metrics for evaluating detection systems.

[0178] (Regarding the processing entity) Furthermore, for processes where the processing entity is not specified, it may be interpreted that the control unit of the related device is the processing entity.

[0179] (About combinations) Furthermore, the features of the above-described embodiment and the features of the modified examples may be combined in any manner.

[0180] (Addendum) The evaluation system of Appendix 1 is an evaluation system for evaluating a detection system for detecting fraudulent transactions in financial transactions associated with a financial account, and comprises: a first identification means for identifying a first evaluation indicator based on a detected fraudulent transaction, which is the financial transaction detected as the fraudulent transaction by the detection system, and for evaluating the detection system; a second identification means for identifying a second evaluation indicator based on a detected fraudulent financial account, which is the financial account associated with the detected fraudulent transaction, and for evaluating the detection system; and a third identification means for identifying a third evaluation indicator based on the detected fraudulent transaction and the detected fraudulent financial account, based on the first evaluation indicator identified by the first identification means and the second evaluation indicator identified by the second identification means, and for evaluating the detection system.

[0181] The evaluation system of Appendix 2 is the evaluation system described in Appendix 1, wherein the first identification means identifies, as the first evaluation index, an evaluation index corresponding to a precision rate based on the fraudulent transaction regarding the detection of the fraudulent transaction by the detection system, the fraudulent transaction itself; the second identification means identifies, as the second evaluation index, an evaluation index corresponding to a recall rate based on the financial account regarding the detection of the fraudulent transaction by the detection system; and the third identification means identifies, as the third evaluation index, an evaluation index corresponding to an F value based on the precision rate and the recall rate.

[0182] The evaluation system of Appendix 3 is the evaluation system described in Appendix 1, wherein the first identification means identifies, as the first evaluation index, an evaluation index corresponding to a false positive rate based on the fraudulent transaction regarding the detection of the fraudulent transaction by the detection system, the second identification means identifies, as the second evaluation index, an evaluation index corresponding to a true positive rate based on the financial account regarding the detection of the fraudulent transaction by the detection system, and the third identification means identifies, as the third evaluation index, an evaluation index corresponding to an AUC based on the false positive rate and the true positive rate.

[0183] The evaluation system of Supplementary Note 4 is the evaluation system according to Supplementary Note 1, further comprising an output means for outputting information indicating the third evaluation index identified by the third identification means.

[0184] The evaluation system of Supplementary Note 5 is the evaluation system according to Supplementary Note 1, further comprising an evaluation means for evaluating the detection system based on the third evaluation index identified by the third identification means.

[0185] The evaluation program of Appendix 6 is an evaluation program for evaluating a detection system for detecting fraudulent transactions in financial transactions associated with a financial account, and causes a computer to function as a first identification means for identifying a first evaluation indicator based on a detected fraudulent transaction, which is the financial transaction detected as the fraudulent transaction by the detection system, and for evaluating the detection system; a second identification means for identifying a second evaluation indicator based on a detected fraudulent financial account, which is the financial account associated with the detected fraudulent transaction, and for evaluating the detection system; and a third identification means for identifying a third evaluation indicator based on the detected fraudulent transaction and the detected fraudulent financial account, and for evaluating the detection system, based on the first evaluation indicator identified by the first identification means and the second evaluation indicator identified by the second identification means.

[0186] The evaluation method of Supplementary Note 7 is an evaluation method for evaluating a detection system for detecting fraudulent transactions in financial transactions associated with a financial account, and includes a first identification step of identifying a first evaluation indicator based on a detected fraudulent transaction, which is the financial transaction detected as the fraudulent transaction by the detection system, and for evaluating the detection system; a second identification step of identifying a second evaluation indicator based on a detected fraudulent financial account, which is the financial account associated with the detected fraudulent transaction, and for evaluating the detection system; and a third identification step of identifying a third evaluation indicator based on the detected fraudulent transaction and the detected fraudulent financial account, and for evaluating the detection system, based on the first evaluation indicator identified in the first identification step and the second evaluation indicator identified in the second identification step.

[0187] (Effect of supplementary notes) According to the evaluation system described in Appendix 1, the evaluation program described in Appendix 6, and the evaluation method described in Appendix 7, by identifying a third evaluation indicator based on detected fraudulent transactions and detected fraudulent financial accounts, it is possible to use evaluation indicators that take into account both financial transactions (detected fraudulent transactions) and financial accounts (detected fraudulent financial accounts), thereby making it possible to appropriately evaluate the detection system.

[0188] According to the evaluation system described in Appendix 2, by specifying an evaluation index corresponding to an F-value based on the precision rate based on fraudulent transactions and the recall rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation index that takes into account both the precision rate for financial transactions and the recall rate for financial accounts, thereby making it possible to appropriately evaluate the detection system.

[0189] According to the evaluation system described in Appendix 3, by specifying an evaluation metric corresponding to the AUC based on the false positive rate based on fraudulent transactions and the true positive rate based on financial accounts for detecting fraudulent transactions, it is possible to use an evaluation metric that takes into account both the false positive rate for financial transactions and the true positive rate for financial accounts, thereby making it possible to appropriately evaluate detection systems.

[0190] According to the evaluation system described in Supplementary Note 4, by outputting information indicating the third evaluation index, it becomes possible to provide information useful for evaluating the detection system, for example.

[0191] According to the evaluation system described in Supplementary Note 5, by evaluating the detection system based on the third evaluation index, it becomes possible to appropriately evaluate the detection system, for example. [Explanation of symbols]

[0192] 1. Transaction terminal equipment 2. Management terminal 3. Server equipment 31 Communications Department 32 Recording section 33 Control Unit 100 Information Processing Systems 321 Fraudulent Transaction Related Information Database 322 Genuine Transaction Related Information DB

Claims

1. 1. An evaluation system for evaluating a detection system for detecting fraudulent transactions or genuine transactions in financial transactions associated with a financial account, the evaluation system comprising: First transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; a storage means for storing second transaction-related information including second transaction content information indicating the content of the authentic financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification means for identifying a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means; the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-unit conformance rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system that correspond to the financial transactions indicated by the first transaction content information among the financial transactions detected as fraudulent by the control means using the detection system, to the number of financial transactions detected as fraudulent by the control means using the detection system, and identifies the transaction-unit conformance rate indicating the calculation result of the first calculation as the first evaluation index; the second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-based reproducibility ratio, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-based reproducibility ratio indicating the calculation result of the second calculation as the second evaluation index; the third identification means performs division using the product of the transaction unit conformance rate and the account unit recall rate as a dividend and the sum of the transaction unit conformance rate and the account unit recall rate as a divisor, and identifies division result information indicating a result of the division as the third evaluation index; Rating system.

2. 1. An evaluation system for evaluating a detection system for detecting fraudulent transactions or genuine transactions in financial transactions associated with a financial account, the evaluation system comprising: First transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; a storage means for storing second transaction-related information including second transaction content information indicating the content of the authentic financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification means for identifying a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means; the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-based false positive rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system among the financial transactions indicated by the second transaction content information to the number of financial transactions indicated by the second transaction content information, and identifies the transaction-based false positive rate, which indicates the calculation result of the first calculation, as the first evaluation index; the second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-level true positive rate, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-level true positive rate, which indicates the calculation result of the second calculation, as the second evaluation index; the detection system has criteria for detecting the fraudulent transactions; the first identification means performs the first calculation to obtain a plurality of transaction unit false positive rates for each of a plurality of stages when the criteria of the detection system is changed into a plurality of stages, The second identification means performs the second calculation to obtain a plurality of account-based true positive rates for each of a plurality of stages when the criteria of the detection system is changed into a plurality of stages. the third identification means identifies an ROC curve generated based on coordinates consisting of a combination of each of the plurality of transaction-unit false positive rates that are the calculation results of the first calculation and each of the plurality of account-unit true positive rates that are the calculation results of the second calculation in a coordinate system with the transaction-unit false positive rate as the horizontal axis and the account-unit true positive rate as the vertical axis, and identifies an AUC corresponding to the area under the identified ROC curve as the third evaluation index; Rating system.

3. further comprising an output means for outputting information indicating the third evaluation index identified by the third identification means, The evaluation system according to claim 1 or 2.

4. and an evaluation unit that evaluates the detection system by comparing the third evaluation index identified by the third identification unit with a predetermined threshold value. The evaluation system according to claim 1 or 2.

5. 1. An evaluation program for evaluating a detection system for detecting fraudulent transactions or genuine transactions in financial transactions associated with a financial account, the evaluation program comprising: Computer, First transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; a storage means for storing second transaction-related information including second transaction content information indicating the content of the authentic financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification means for identifying a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means; the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-unit conformance rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system that correspond to the financial transactions indicated by the first transaction content information among the financial transactions detected as fraudulent by the control means using the detection system, to the number of financial transactions detected as fraudulent by the control means using the detection system, and identifies the transaction-unit conformance rate indicating the calculation result of the first calculation as the first evaluation index; the second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-based reproducibility ratio, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-based reproducibility ratio indicating the calculation result of the second calculation as the second evaluation index; the third identification means performs division using the product of the transaction unit conformance rate and the account unit recall rate as a dividend and the sum of the transaction unit conformance rate and the account unit recall rate as a divisor, and identifies division result information indicating a result of the division as the third evaluation index; Evaluation program.

6. 1. An evaluation program for evaluating a detection system for detecting fraudulent transactions or genuine transactions in financial transactions associated with a financial account, the evaluation program comprising: Computer, First transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; a storage means for storing second transaction-related information including second transaction content information indicating the content of the authentic financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; a control means for detecting fraudulent transactions from among the financial transactions indicated by the first transaction content information and the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification means for identifying a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a second identification means for identifying a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means; a third identification means for identifying a third evaluation index for evaluating the detection system, the third evaluation index being different from the first evaluation index and the second evaluation index, based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means; the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-based false positive rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system among the financial transactions indicated by the second transaction content information to the number of financial transactions indicated by the second transaction content information, and identifies the transaction-based false positive rate, which indicates the calculation result of the first calculation, as the first evaluation index; the second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-level true positive rate, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-level true positive rate, which indicates the calculation result of the second calculation, as the second evaluation index; the detection system has criteria for detecting the fraudulent transactions; the first identification means performs the first calculation to obtain a plurality of transaction unit false positive rates for each of a plurality of stages when the criteria of the detection system is changed into a plurality of stages, The second identification means performs the second calculation to obtain a plurality of account-based true positive rates for each of a plurality of stages when the criteria of the detection system is changed into a plurality of stages. the third identification means identifies an ROC curve generated based on coordinates consisting of a combination of each of the plurality of transaction-unit false positive rates that are the calculation results of the first calculation and each of the plurality of account-unit true positive rates that are the calculation results of the second calculation in a coordinate system with the transaction-unit false positive rate as the horizontal axis and the account-unit true positive rate as the vertical axis, and identifies an AUC corresponding to the area under the identified ROC curve as the third evaluation index; Evaluation program.

7. 1. A method for evaluating a detection system for detecting fraudulent or genuine financial transactions associated with a financial account, the method comprising: The evaluation system includes: First transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; a storage means for storing second transaction-related information including second transaction content information indicating the content of the authentic financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; The evaluation method includes: a detection step in which the control means of the evaluation system detects the fraudulent transaction from the financial transaction indicated by the first transaction content information and the financial transaction indicated by the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification step in which a first identification means of the evaluation system identifies a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transaction by the control means; a second identification step in which a second identification means of the evaluation system identifies a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transaction by the control means; a third identification step in which a third identification means of the evaluation system identifies a third evaluation index for evaluating the detection system based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means, the third evaluation index being different from the first evaluation index and the second evaluation index; the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-unit conformance rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system that correspond to the financial transactions indicated by the first transaction content information among the financial transactions detected as fraudulent by the control means using the detection system, to the number of financial transactions detected as fraudulent by the control means using the detection system, and identifies the transaction-unit conformance rate indicating the calculation result of the first calculation as the first evaluation index; the second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-based reproducibility ratio, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-based reproducibility ratio indicating the calculation result of the second calculation as the second evaluation index; the third identification means performs division using the product of the transaction unit conformance rate and the account unit recall rate as a dividend and the sum of the transaction unit conformance rate and the account unit recall rate as a divisor, and identifies division result information indicating a result of the division as the third evaluation index; Evaluation method.

8. 1. A method for evaluating a detection system for detecting fraudulent or genuine financial transactions associated with a financial account, the method comprising: The evaluation system includes: First transaction-related information including first transaction content information indicating the content of the fraudulent financial transaction and first account identification information identifying the financial account associated with the financial transaction indicated by the first transaction content information; a storage means for storing second transaction-related information including second transaction content information indicating the content of the authentic financial transaction and second account identification information identifying the financial account associated with the financial transaction indicated by the second transaction content information; The evaluation method includes: a detection step in which the control means of the evaluation system detects the fraudulent transaction from the financial transaction indicated by the first transaction content information and the financial transaction indicated by the second transaction content information using the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means; a first identification step in which a first identification means of the evaluation system identifies a first evaluation index for evaluating the detection system based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transaction by the control means; a second identification step in which a second identification means of the evaluation system identifies a second evaluation index for evaluating the detection system, the second evaluation index being different from the first evaluation index, based on the first transaction-related information and the second transaction-related information stored in the storage means and the detection result of the fraudulent transaction by the control means; a third identification step in which a third identification means of the evaluation system identifies a third evaluation index for evaluating the detection system based on the first evaluation index identified by the first identification means and the second evaluation index identified by the second identification means, the third evaluation index being different from the first evaluation index and the second evaluation index; the first identification means performs a first calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine a transaction-based false positive rate, which is the ratio of the number of financial transactions detected as fraudulent by the control means using the detection system among the financial transactions indicated by the second transaction content information to the number of financial transactions indicated by the second transaction content information, and identifies the transaction-based false positive rate, which indicates the calculation result of the first calculation, as the first evaluation index; the second identification means performs a second calculation based on the first transaction-related information and the second transaction-related information stored in the storage means and the result of the fraudulent transaction detection by the control means to determine an account-level true positive rate, which is the ratio of the number of financial accounts associated with the financial transactions detected as fraudulent transactions by the control means using the detection system among the financial accounts identified by the first account identification information to the number of financial accounts identified by the first account identification information, and identifies the account-level true positive rate, which indicates the calculation result of the second calculation, as the second evaluation index; the detection system has criteria for detecting the fraudulent transactions; the first identification means performs the first calculation to obtain a plurality of transaction unit false positive rates for each of a plurality of stages when the criteria of the detection system is changed into a plurality of stages, The second identification means performs the second calculation to obtain a plurality of account-based true positive rates for each of a plurality of stages when the criteria of the detection system is changed into a plurality of stages. the third identification means identifies an ROC curve generated based on coordinates consisting of a combination of each of the plurality of transaction-unit false positive rates that are the calculation results of the first calculation and each of the plurality of account-unit true positive rates that are the calculation results of the second calculation in a coordinate system with the transaction-unit false positive rate as the horizontal axis and the account-unit true positive rate as the vertical axis, and identifies an AUC corresponding to the area under the identified ROC curve as the third evaluation index; Evaluation method.

Images