Hellinger decision tree for fraud detection

Abstract

Hellinger decision tree for fraud detection [Solution] The Hellinger decision tree can detect fraudulent transactions in a dataset of financial transactions. The Hellinger distance is used to apply the Hellinger decision tree. The Hellinger decision tree can be part of a machine learning algorithm. In one example, the Hellinger decision tree is a positive-and-unbalanced Hellinger decision tree used with unbalanced positive examples and unlabeled data.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims priority to the provisional patent application filed on 13 March 2023 and granted U.S. Provisional Patent Application No. 63 / 451,907, the disclosures of which are incorporated herein by reference.

[0002] This disclosure relates to machine learning, and more specifically, to machine learning for fraud detection. [Background technology]

[0003] Despite the widespread adoption of supervised learning, many real-world applications, including fraud detection, text classification, and medical diagnosis, suffer from inaccurate or incomplete labeling. Furthermore, these applications are often characterized by high class imbalance. These applications relate to two areas of study: PU learning from positive and unlabeled data, and imbalanced learning. In these applications, the underrepresented class is the target class (i.e., the positive class). In fraud detection, fraudulent cases account for less than 1 percent of all transactions. Despite less severe class imbalance in medical diagnosis, false negatives (i.e., undetected tumors) are more significant than false positives. Underrepresentation of minority classes can be further exacerbated by the problem of incomplete labeling. In medical data, often only positive information (i.e., diagnosed diseases) is reported. In fraud detection, a very small number of fraudsters are identified, while most are overlooked. Therefore, several applications can benefit from the collaboration of PU learning and imbalanced learning. In PU learning, labeled examples are assumed to be positive examples, while unlabeled examples are assumed to belong to either the positive or negative example class. Imbalanced learning aims to propose a method for dealing with situations where the class distribution is significantly unequal. Improved systems and techniques are needed that focus on learning from imbalanced datasets where the negative example class and some positive examples are unlabeled.

[0004] As evidenced by the increasing development of new methods, PU learning has become increasingly popular in recent years. One method initially used in text classification identified reliable negative examples and learned from positive examples and the resulting reliable negative examples. Another method assumed all unlabeled examples were negative and applied a standard classifier. Later methods utilized class prior probabilities (i.e., the proportion of positive examples in the class) in existing algorithms to enable PU learning.

[0005] Non-standard settings in domain-applied, PU learning have been explored. For example, a common assumption in PU learning is that labeled examples are a random subset of positive examples. However, in practice, this assumption is often broken. While most modern PU methods work well on some benchmark datasets, it remains unclear how well they perform on highly imbalanced datasets.

[0006] Imbalanced PU classification presents new challenges that have not been adequately addressed. In this particular setting, the fact that only a small number of positive example instances are known to the learner creates a more severe class imbalance. A suitable PU method for an imbalanced setting must be able to leverage the small number of labeled positive examples while still being able to learn from unlabeled instances.

[0007] There is little research focusing on imbalanced PU classification. PU learning has been proposed by optimizing the adaptation of the area under the receiver operating characteristic curve (ROC-AUC) for semi-supervised settings. However, optimizing ROC-AUC does not guarantee the optimization of metrics more relevant to imbalanced classification, such as the area under the precision-recall curve. Cost-aware learning from positive examples and unlabeled samples (CSPU), which introduces the use of misclassification costs to address class imbalance, has also been proposed. The requirement of CSPU to make misclassification costs available is not easily met, as determining misclassification costs is difficult in some areas. Finally, imbalanced non-negative PU learning methods rely on oversampling to balance the PU data. Nevertheless, oversampling can lead to unnecessary overfitting, and the oversampling rate may need to be tuned as an extra hyperparameter. Therefore, there is a need for a technique that can work well on highly imbalanced PU data without requiring resampling or misclassification costs.

[0008] Fraud detection is a challenging task for machine learning algorithms due to severe class imbalance and incomplete labeling information. In many fraud detection applications, fraudulent instances account for less than 1% of all cases. Furthermore, inaccurate labeling can be problematic if fraudsters consistently succeed in hiding from current detection systems. For example, in an investigation into auto insurance fraud in Quebec, claims adjusters were only able to identify one-third of all potential fraudsters. This setup, where only a subset of fraudsters are found, is referred to as positive and unlabeled data.

[0009] PU learning, or learning from positive and unlabeled data, is a setup where a binary classifier can only be trained on positive and unlabeled instances, the latter including both positive and negative instances. Many PU applications (e.g., fraud detection) are also characterized by class imbalance, creating challenging situations. Not only are there fewer examples for minority classes compared to cases where all labels are known, but there are also a very small number of unlabeled observations that are actually positive. Despite the relevance of the topic, few studies have considered class imbalance settings in PU learning. Therefore, improved systems and techniques are needed, especially for fraud detection. [Overview of the Initiative]

[0010] This disclosure provides a technique called PU Hellinger Decision Tree (PU-HDT) that can directly handle imbalanced PU data. This disclosure utilizes class prior probabilities to estimate the number of positive and negative examples at every node in the tree. Furthermore, the Hellinger distance is used as an alternative to more conventional partitioning criteria, as it has been demonstrated to be unaffected by class imbalance. This adaptation enables PU-HDT to function well in highly imbalanced PU datasets.

[0011] This disclosure also provides a PU-Stratified Hellinger Random Forest (PU-SHRF) that uses PU-HDT as its base learner and integrates stratified bootstrap sampling. PU-SHRF has the potential to substantially outperform state-of-the-art PU learning methods on imbalanced datasets in most experimental settings.

[0012] The present disclosure also provides a tree-based technique called PU-HDT, which is designed to learn from imbalanced PU data. PU-HDT does not require correcting imbalanced data distributions. In this class-prior probability-based method, PU-HDT utilizes the class prior probability (i.e., the proportion of positive examples) to enable PU learning. At each node, instead of assuming all unlabeled instances as negative examples, it estimates the true positive examples from the unlabeled instances. PU-HDT uses the Hellinger distance instead of conventional splitting criteria (e.g., Gini and entropy) that have been demonstrated to be inferior for imbalanced datasets. The Hellinger distance has shown robustness to extreme class imbalance in prior studies. These two improvements enable PU-HDT to effectively process highly imbalanced datasets. The performance of PU-HDT can be further improved using an ensemble.

[0013] The present disclosure demonstrates that a modified random forest with PU-HDT as its base learner outperforms state-of-the-art PU learning methods in various experimental settings with class imbalance.

[0014] Furthermore, an ensemble method that uses PU-HDT as the base learner is also provided.

[0015] In one embodiment, the fraud detection method includes receiving a dataset of financial transactions in a processor. A Hellinger decision tree is applied using the processor to detect fraud in the dataset of financial transactions. The Hellinger distance is used as part of this application.

[0016] The Hellinger decision tree can be part of a machine learning algorithm.

[0017] The Hellinger decision tree can be configured to capture the divergence between the positive class distribution and the negative class distribution without being dominated by class imbalance.

[0018] The Hellinger decision tree can be configured to use class prior probabilities to estimate the number of positive and negative examples at each node.

[0019] The method further includes using a processor to limit the size of the Hellinger decision tree after the tree nodes reach the maximum height, thereby avoiding overfitting.

[0020] The Hellinger decision tree can be an imbalanced Hellinger decision tree with positive examples. The dataset of fraudulent transactions for this instance can be imbalanced positive examples and unlabeled data.

[0021] The method further includes receiving an estimated fraud occurrence rate in a processor before application.

[0022] The Hellinger decision tree can be used as a base learner in a modified random forest.

[0023] In one instance, the Hellinger decision tree is an imbalanced Hellinger decision tree with positive examples. In this instance, the dataset of fraudulent transactions can be imbalanced positive examples and unlabeled data. The Hellinger decision tree can be configured to consider random feature selection when initializing tree nodes. The Hellinger decision tree can also be configured to use the size of stratified bootstrap samples and class prior probabilities.

[0024] The Hellinger distance can be used to capture the divergence between the positive example class distribution and the negative example class distribution.

[0025] The method can further include receiving a dataset of positive examples and unlabeled data in a processor and training a Hellinger decision tree using the dataset of positive examples and unlabeled data. The dataset of positive examples and unlabeled data can be imbalanced.

[0026] Financial transactions may include credit card transactions or insurance transactions.

[0027] The system can be configured to perform the method of the first embodiment. The system can be a computer or a server.

[0028] A non-temporary computer-readable medium for storing a program can be configured to instruct a processor to perform the method of the first embodiment. [Brief explanation of the drawing]

[0029] To better understand the nature and purpose of this disclosure, please refer to the following detailed description in conjunction with the attached drawings.

[0030] [Figure 1] This decision tree illustrates the difference between the Gini exponent and the Hellinger distance in node partitioning.

[0031] [Figure 2(a)] As discussed in Examples 1 and 2, a dataset of class-imbalanced variants with a class prior probability α = 5% is presented. The upper half-moon corresponds to the negative example class, while the lower half-moon corresponds to the positive example class. [Figure 2(b)] As discussed in Examples 1 and 2, a dataset of class-imbalanced variants with a class prior probability α = 5% is presented. The upper half-moon corresponds to the negative example class, while the lower half-moon corresponds to the positive example class. [Figure 2(c)] As discussed in Examples 1 and 2, a dataset of class-imbalanced variants with a class prior probability α = 5% is presented. The upper half-moon corresponds to the negative example class, while the lower half-moon corresponds to the positive example class. [Figure 2(d)] As discussed in Examples 1 and 2, a dataset of class-imbalanced variants with a class prior probability α = 5% is presented. The upper half-moon corresponds to the negative example class, while the lower half-moon corresponds to the positive example class.

[0032] [Figure 3(a)] To highlight the relative performance between the methods tested in Example 2, the rank of each indicator is shown. [Figure 3(b)] To highlight the relative performance between the methods tested in Example 2, the rank of each indicator is shown. [Figure 3(c)] To highlight the relative performance between the methods tested in Example 2, the rank of each indicator is shown.

[0033] [Figure 4(a)] As discussed in Example 2, the F1 score rank is shown. [Figure 4(b)] As discussed in Example 2, the ROC-AUC rank is shown. [Figure 4(c)] As discussed in Example 2, the PR-AUC rank is shown.

[0034] [Figure 5] The results of the sensitivity analysis performed in Example 2 are shown. [Figure 6] The results of the sensitivity analysis performed in Example 2 are shown.

[0035] [Figure 7] The table showing the experimental results is provided below. [Figure 8] The table showing the experimental results is provided below. [Figure 9] The table showing the experimental results is provided below. [Figure 10] The table showing the experimental results is provided below. [Figure 11] The table showing the experimental results is provided below. [Figure 12] The table showing the experimental results is provided below. [Figure 13] The table showing the experimental results is provided below. [Modes for carrying out the invention]

[0036] While the claimed subject matter is described in relation to specific embodiments, other embodiments, including those that do not provide all of the advantages and features described herein, are also within the scope of this disclosure. Various structural, logical, process, and electronic modifications can be made without departing from the scope of this disclosure. Accordingly, the scope of this disclosure is defined solely by reference to the appended claims.

[0037] Machine learning can be used for fraud detection. Embodiments disclosed herein include models that can address highly imbalanced PU datasets that may occur in financial fraud detection, where there may be only a limited pool of positive examples among a large number of unlabeled data points. Embodiments disclosed herein can address this imbalance.

[0038] While exemplary features of the fraud detection method have been described, it should be understood that such configurations should not be interpreted as limiting the present invention to such features. The method can be implemented in software, firmware, hardware, or a combination thereof. For example, the method is implemented as software as an executable program and executed by one or more dedicated or general-purpose digital computers such as personal computers (PCs, IBM-compatible, Apple-compatible, or others), personal digital assistants, workstations, minicomputers, or mainframe computers. The steps of the method can be implemented by a server or computer on which a software module resides or partially resides.

[0039] In general, with respect to hardware architecture, as will be well understood by those skilled in the art, such a computer includes a processor, memory, and one or more input and / or output (I / O) devices (or peripherals) communicably coupled via a local interface. The local interface may be, for example, one or more buses or other wired or wireless connections, as is known in the art. The local interface may have additional elements such as controllers, buffers (caches), drivers, repeaters, and receivers to enable communication. Furthermore, the local interface may further include address, control, and / or data connections to enable proper communication between other computer components.

[0040] A processor can be programmed to perform the functions of a fraud detection method. A processor is a hardware device for executing software, particularly software stored in memory. A processor can be any custom-made or commercially available processor, a main processing unit (CPU), an auxiliary processor among several processors included with a computer, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, or any device for executing software instructions in general.

[0041] Memory is associated with the processor and may include one or a combination of volatile memory elements (e.g., random access memory (RAM such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, hard drives, tapes, CD-ROMs, etc.). Furthermore, memory may incorporate electronic, magnetic, optical, and / or other types of storage media. Memory may have a distributed architecture in which various components are located apart from each other but are still accessed by the processor.

[0042] The software in memory may include one or more separate programs. Each separate program includes an ordered list of executable instructions that implement logical functions to perform the functions of the module. In the examples described herein, the software in memory includes one or more components of the Method and is executable on a suitable operating system (O / S).

[0043] This disclosure may include components provided as source programs, executable programs (object code), scripts, or any other entities including a set of instructions to be executed. In the case of source programs, the program needs to be translated, whether or not it is contained in memory, by a compiler, assembler, interpreter, etc., in order to function properly in conjunction with the OS. Furthermore, methodologies implemented in accordance with this teaching may be expressed as (a) an object-oriented programming language having classes of data and methods, or (b) a procedural programming language having routines, subroutines, and / or functions, such as, but not limited to, C, C++, Pascal, Basic, Fortran, Cobol, Ped, Java, and Ada.

[0044] In one embodiment of the PU-HDT algorithm, the Hellinger decision tree (HDT) utilizes the Hellinger distance to improve the partitioning mechanism in imbalanced settings. The Hellinger distance is used to quantify the similarity between two probability distributions. The goal of the Hellinger distance is to capture the divergence between the positive and negative class distributions without being dominated by class imbalance. Equation (1) illustrates the robustness to class imbalance in detail. The Hellinger distance can be calculated at the parent node i as follows:

number

number

[0045] HDT can be adapted to PU learning by utilizing the class prior probability, represented as α. Under the assumption that the data is completely randomly selected (SCAR), the class prior probability allows for the estimation of the label frequency c, which is the proportion of labeled positive examples in the data. Equation (2) shows the preceding description.

number

number

number

number

[0046] Figure 1 shows the difference between the Gini exponent and the Hellinger distance in node partitioning. The tree on the left is partitioned based on the Gini exponent, while the tree on the right follows the Hellinger distance with PU adaptation. As shown in Figure 1, the Gini exponent is superior in the left tree (0.166) compared to the right tree (0.169), and the PU Hellinger distance is superior in the right tree (0.707) compared to the left tree (0.459).

[0047] Algorithm 1 demonstrates how PU-HDT is constructed based on Hellinger distance, along with estimates of the number of positive and negative instance types. PU-HDT follows the same binary tree structure as other decision trees known in the art, such as CART and HDT. When used as a standalone classifier, Algorithm 1 may limit the tree size to avoid overfitting. For small pools of labeled data, PU-HDT can undergo hyperparameter tuning regarding tree size.

[0048] In embodiments where PU-HDT is used in bagging ensembles or random forests, overfitting is no longer a problem, so the trees may be grown to full depth. Algorithm 1 first creates a tree node n. Since it has a recursive aspect, the algorithm stops when the tree node n reaches the maximum depth h. After creating the tree node n, optionally, a random selection of feature f sel can be obtained for the tree node n. However, the standard decision tree does not perform random feature selection, and FX and f sel are the same set. Then, an optimal split value x sel can be found through a search within the set of features f * fmax . The optimal split value is the one that maximizes the PU-Hellinger distance. Then, the training instances, denoted as the features and labels (X, L), are split into two subsets (X * fmax , L fmax ; X left , L left ; X right , L right ) according to the position of the instances at x * fmax with respect to the optimal split value x fmax . Finally, for each subset, new tree nodes (n.left and n.right) can be created and this procedure can be repeated again. Figure 1 shows a scenario demonstrating the superiority of the (PU-)Hellinger distance over the Gini index. As described above, the Gini index favors node splits (left tree) that concentrate the majority of the positive and negative example classes in one child node, and the (PU-)Hellinger distance shows a preference for node splits (right tree) that concentrate all positive examples in one child node.

Number

[0049] Ensemble learning generally improves the performance of a single learner by training several basic learners and combining their outputs. One ensemble method is the Random Forest (relevant parts are incorporated by reference, Breiman 2001), which extends bagging by incorporating randomized feature selection where the basic learners are decision trees. In one embodiment, PU-HDT may be used as the basic learner within the Random Forest. Furthermore, the ensemble algorithm may be further modified based on insights from PU Bagging (relevant parts are incorporated by reference, Mordelet & Vert 2014). PU Bagging is well-suited to imbalanced learning because each bootstrap sample includes all available labeled positive examples and subsamples of unlabeled data, thus achieving a balanced training set. While PU-HDT can handle imbalanced data, there are practical reasons to take bootstrap samples from unlabeled data.

[0050] The ability of PU-HDT to learn from PU data can be hindered by bootstrap samples that contain a sparser representation of the positive example distribution. For example, as shown in Figure 2(b), PU-HDT cannot learn the complete true positive example distribution (ideally a crescent shape) because there is a small region on the left that does not contain any labeled positive examples that could be used to estimate the true positive examples.

[0051] One embodiment of the present invention is a PU-stratified Hellinger random forest (PU-SHRF), which can ensure that all labeled positive examples are represented in each bootstrap sample. Algorithm 2 outlines how PU-SHRF is designed based on PU-HDT and stratified bootstrap sampling.

[0052] Algorithm 2 represents a random forest setup of PU-HDT. Unlike standalone decision trees, Algorithm 2 considers random feature selection when initializing tree nodes, where φ is the number of features |F|. X Corresponds to the square root of |. This is the default option in scikit-learn. Next, bootstrap samples (X') from unlabeled data. U ,L′ U ) is obtained. The size of the bootstrap sample is K U Defined by: To provide training data (X′,L′) to PU-HDT, labeled positive example instances (X′ lab ,L′ lab ) is a bootstrap sample (X' U ,L′ U ) is added. PU-HDT may be added to the initialized forest. Algorithm 2 stops when a number of trees t are fitted and added to the forest. Compared to a standard random forest, PU-SHRF has two additional hyperparameters, namely the size of the stratified bootstrap samples K. U This also requires a prior probability α for the class.

number

[0053] Fraud detection can be performed by applying HDT to a dataset of financial transactions (e.g., credit card transactions or insurance transactions) using embodiments disclosed herein. The use of the Hellinger distance is part of this application. The Hellinger distance is used to capture the divergence between the positive and negative class distributions. An estimated fraud rate can be received by the processor before applying HDT to the financial transaction dataset.

[0054] Based on the largest dataset, Fraud IEEE, training PU-HDT took an average of 15.6 seconds over 10 runs. The training set represents approximately 70% of the dataset. The number of cases in the training set is 413,378. The algorithm also took an average of 1.22 seconds to provide a score for each instance in the test set (177,162 instances). There are no limitations on the size of the dataset that PU-HDT can process.

[0055] A processor can be used for this fraud detection. The processor may be part of a computer or server. HDT may be part of a machine learning algorithm executed using the processor. Non-temporary computer-readable media can store instructions for performing the fraud detection embodiments disclosed herein.

[0056] HDT can be configured to capture the divergence between the positive and negative class distributions without being dominated by class imbalance. HDT can be configured to use class prior probabilities to estimate the number of positive and negative examples at each node.

[0057] The size of the HDT can be limited using a processor after the tree nodes reach their maximum height. This helps to avoid overfitting.

[0058] HDT can be a PU-HDT tree. Therefore, the dataset of fraudulent transactions may consist of imbalanced positive examples and unlabeled data.

[0059] In one example, HDT is used as a base learner in a modified random forest. In this case, HDT can be a PU-HDT tree. Thus, the fraud transaction dataset may consist of imbalanced positive examples and unlabeled data. HDT can be configured to consider random feature selection when initializing the tree nodes. HDT can be configured to use the size and class prior probabilities of stratified bootstrap samples.

[0060] To train HDT, positive and unlabeled datasets are used. These datasets may be imbalanced.

[0061] While this disclosure describes one or more specific embodiments, it will be understood that other embodiments of this disclosure can be constructed without departing from the scope of this disclosure.

[0062] The following embodiments are provided to illustrate the present disclosure; they are not intended to limit the scope in any way.

[0063] Example 1 The following is an example of the "Half-Moon" dataset, which provides a comparison of decision boundaries for decision trees, including the PU-Hellinger decision tree in Algorithm 1, to illustrate the challenge of learning from positive and unlabeled data in an imbalanced setting. It includes data points for negative examples, hidden positive examples, and positive example classes. Regions with darker shading indicate higher certainty regarding classification into either the positive or negative example class. Therefore, the lighter the shading, the higher the uncertainty of the classifier. Figure 2(a) shows the Half-Moon dataset (ground truth), Figure 2(b) shows the PU-Hellinger decision tree in the test set, Figure 2(c) shows the Hellinger decision tree in the test set, and Figure 2(d) shows CART in the test set.

[0064] This "half-moon" dataset consisted of two-dimensional points generated from two alternating semicircles. Figure 2(a) shows a dataset of class-imbalanced variants with a class prior probability α = 5%. The upper half-moon corresponds to the negative example class, while the lower half-moon corresponds to the positive example class. Furthermore, half of the positive examples were mislabeled (i.e., unlabeled in the PU setting). The only information about the positive example class was the labeled positive examples, while the rest remained unlabeled.

[0065] In this example, three techniques were compared: PU-HDT (Figure 2(b)), HDT (Figure 2(c)), and CART (Figure 2(d)). As shown in Figure 2(d), the CART decision tree did not learn from the PU data, so only a small number of unlabeled positive examples were classified into the darker region, and the distribution of positive examples was not well represented by the decision boundary of the CART decision tree. As shown in Figure 2(c), HDT performed better due to its built-in insensitivity to imbalanced settings. However, this algorithm did not learn from both positive and unlabeled data. In the region where unlabeled positive examples were numerous (bottom left), HDT provided predictions with high uncertainty. As shown in Figure 2(b), PU-HDT learned from the PU data and was suitable for imbalanced settings. This technique considered some positive examples to be unlabeled, so most of the unlabeled positive examples were classified into the darker region.

[0066] This embodiment provides a clearer representation of a positive example distribution following a crescent shape. In this embodiment, class prior probabilities and Hellinger distance are essential and complementary in the method for imbalanced PU classification. One complex factor in imbalanced classification is data complexity. In datasets exhibiting high complexity, a small number of positive examples may not adequately represent the minority (positive example) class. Techniques for handling imbalanced data are more robust to underrepresentation of minority classes. As shown in Figure 2, HDT offers advantages over CART. However, the PU setting introduces another complex factor that Hellinger distance alone cannot solve. Some unlabeled (i.e., negative example) instances are positive examples. Therefore, HDT cannot correctly classify most of such unlabeled positive examples as positive. Class prior probabilities provide a way to calculate the proportion of unlabeled positive examples that need to be considered in the decision tree. In PU-HDT, both Hellinger distance and class prior probabilities are successfully utilized to enable PU learning on imbalanced datasets.

[0067] Example 2 The following is an example demonstrating the performance of Algorithm 2 (PU-HDT) compared to prior articulations. This example shows the advantage of PU-HDT's classification performance when applied to imbalanced PU data, such as datasets where only a small percentage of positive observations exist, and where, if labels are known, only a small percentage of unlabeled observations actually become positive observations. In this test, PU-HDT, PU-HRF, and PU-SHRF were compared to 12 prior articulations. Eight of the articulations were derived from PU literature. This included Hellinger Decision Tree (HDT), Hellinger Random Forest (HRF), and Stratified Hellinger Random Forest (SHRF). This evaluation utilized 19 benchmark datasets. Two evaluation metrics commonly used in imbalanced learning were used. Specifically, these are the area under the precision-recall curve (PR-AUC) and the F1 score, which represents the harmonic mean between precision and recall. In contrast to PR-AUC, the F1 score depends on a threshold that can be disadvantageous to techniques that do not provide a calibrated score, such as tree-based methods. Therefore, the threshold was optimized according to the validation set to maximize the F1 score for all techniques in each experimental setting. Furthermore, hypothesis testing was applied to statistically verify the empirical results. First, the Iman-Davenport test was applied to determine whether all methods performed equally well, as indicated in the null hypothesis. Next, Holm's post-hoc test was used to compare the best-performing model with the other techniques.

[0068] We examined 19 datasets, summarized in Table 1, covering various application examples such as churn prediction, fraud detection, and image recognition. The churn dataset originates from the telecommunications industry. The credit card fraud dataset was provided on Kaggle by Wordline and ULB (MLG, 2018) and Fraud IEEE. The auto insurance fraud dataset was provided by Oracle (Oracle, 2015). Pizza Cutter 1 and Satellite are OpenML datasets (Vanschoren, van Rijn, Bischl, & Torgo, 2013). Forest Cover, Speech, MNIST, Shuttle, Pendigits, and Mammography were found in ODDS (Shebuti, 2016). The Poker 8 vs 6, Car Good, and KDD Cup land-vs-portsweep datasets were available in the KEEL repository (Alcal'a-Fernandez et al., 2011). The Thyroid dataset was found in the UCI repository (Dua et al., 2019). The dataset was selected with the aim of comparing techniques in a highly imbalanced setting. In previous studies, class imbalance in benchmark datasets was not necessarily extreme, as some datasets showed imbalance ratios of less than 10. However, this example utilizes an extreme setting by using the most imbalanced dataset available in the aforementioned public repository. [Table 1]

[0069] To create PU data, in each dataset, several positive examples were flipped to unlabeled observations completely randomly (i.e., SCAR). In the experiments, the number of labeled positive examples was determined by the flip ratio, which is the proportion of unlabeled positive examples. Three flip ratio values, 25%, 50%, and 75%, were considered. For each experimental setup, 20 iterations of holdout validation were performed, and the data was split into a training set (70%) and a test set (30%) using different random seeds. A large number of iterations were necessary because training from imbalanced PU data could lead to unstable performance. For datasets containing more than 10,000 cases, 10,000 observations were sampled without replacement and used in each iteration to limit the computation time of the experiments. In experiments where the class prior probability estimate was equal to ground truth, there were a total of 1140 setups (19 datasets × 20 iterations × 3 flip ratios). For sensitivity analysis, there are 1900 settings (19 datasets × 20 iterations × 5 prior probability estimates for each class).

[0070] We compared a PU-Hellinger-based technique with eight well-known PU learning techniques: imbalanced non-negative PU learning (imbalanced nnPU) (Su et al., 2021), non-negative PU learning (nnPU) (Kiryo et al., 2017), unbiased PU learning (uPU) (Du Plessis et al., 2015), PU Bagging (Mordelet & Vert, 2014), Rank Pruning (Northcutt, Wu & Chuang, 2017), PU-Weighted Logistic Regression (Lee & Liu, 2003), Elkan-Noto's method, i.e., a preprocessing method using label probabilities (Elkan & Noto, 2008), and Spy-EM (Liu et al., 2002). This study also included four non-PU baselines, namely, random forest (Breiman, 2001), HDT (Cieslak & Chawla, 2008), combined with ADASYN (He et al., 2008), as well as ensemble-based HRF and SHRF.

[0071] Table 2 summarizes the hyperparameter configuration of the method in the experimental setting. The PU learning method in the experimental setting required hyperparameters that needed to be specified by the end user. However, recommended values ​​exist for most of the hyperparameters that have shown good results in previous experiments. For PU-Hellinger-based techniques and unbiased PU learning and their extensions, we determined that class prior probabilities are essential hyperparameters because they enable PU learning by weighting labeled positive examples. In this example, we assume that class prior probabilities are known at training time. In practice, class prior probabilities can be estimated using several methods from . Another important hyperparameter for PU-SHRF is K U This is the number of unlabeled instances within each bootstrap. PU Bagging is also K U While using PU Bagging, PU Bagging subsamples unlabeled cases to balance the training data and avoid sample contamination by hidden positive cases. PU-SHRF can, of course, handle class imbalances and PU data, thus increasing the number of unlabeled cases in the bootstrap sample. Therefore, K U = U training The default value was selected, which resulted in a layered bootstrap sample for each PU Hellinger tree in the ensemble. For PU-HDT and HDT, the maximum tree depth was set to 5 to avoid overfitting. Rank Pruning, PU-Weighted Logistic Regression, Elkan-Noto's method, and Spy-EM followed the hyperparameters recommended by the authors. The remaining hyperparameters were set to the default values ​​in scikit-learn. [Table 2]

[0072] The mean and standard deviation of the F1 score and PR-AUC are shown in Tables 3 and 4. The mean rank is included in parentheses. The mean F1 score and PR-AUC for each dataset are reported in the appendix. Furthermore, the results for ROC-AUC are shown in Table A1 of Figure 7. Underlined bold values ​​indicate the best-performing model for a given flip ratio. Based on the mean rank of each technique's metric across all datasets, the Iman-Davenport test rejects the null hypothesis that all methods perform equally well (p<0.01). Furthermore, Holm's post-hoc test was applied to identify the causes of performance differences. The best-performing model was used as a control classifier in pairwise comparisons with all other models.

[0073] PU-SHRF and PU-HRF outperformed all other techniques in F1 scores at optimized thresholds. Furthermore, Holm's post-hoc test further validated the statistical superiority of PU-SHRF and PU-HRF over other techniques at a 5% significance level. uPU, nnPU, and imbalanced nnPU, considered state-of-the-art in the literature, generally perform better than previous PU methods. Additionally, advantages were observed in using imbalanced nnPU, designed for imbalanced datasets, over nnPU. Imbalanced nnPU was judged to perform better than nnPU at all flip ratios. PU Bagging remained a competitive alternative even at the highest flip ratios. A possible explanation is that PU Bagging naturally handles class imbalance because each bootstrap sample consists of a balanced subset of the training data. The fact that techniques for imbalanced classification, such as ADASYN+Random Forest and HRF, outperform most PU methods highlights the weaknesses of traditional PU methods for imbalanced classification.

[0074] Regarding PR-AUC, HRF stood out as the best technique across all flip ratios. However, PU-SHRF was not outperformed by a statistical significance of 5%. Similarly, PU-HRF was not outperformed by a statistical significance of 5% when the flip ratio was either 25% or 50%. Another competing PU method that was not statistically outperformed was imbalanced nnPU at 75%. PU Bagging performed poorly for PR-AUC despite good performance for F1 scores and ROC-AUC. This may suggest that balanced bootstrap improved recall at the expense of precision. Furthermore, using conventional data-level methods to handle class imbalances, along with standard classifiers (e.g., ADASYN + Random Forest), was a superior alternative to older PU techniques. Based on the results in Table 4, techniques that naturally handle class imbalances were observed to perform well with respect to PR-AUC.

[0075] The results for each dataset also provide more insight into the performance of the techniques. Tables A2, A4, and A6 (Figures 8, 10, and 12) show the average F1 scores for each dataset at different flip ratios. Tables A3, A5, and A7 (Figures 9, 11, and 13) include the average PR-AUC for each dataset at different flip ratios. The best-performing techniques were consistent at low and medium flip ratios for most datasets. On the PEN dataset, PU-SHRF outperformed all other techniques in F1 scores, except for settings with high flip ratios where PU bagging was superior. This may explain why PU bagging was a competitive alternative to the method of the present invention in the aggregated results at high flip ratios (Table 3). Furthermore, PU-HRF and PU-SHRF showed strong performance in both F1 scores and PR-AUC compared to other PU methods. In the FCC dataset, which is the most imbalanced dataset, PU-HRF and PU-SHRF substantially outperformed the PU technique.

[0076] Empirical analysis yielded several general insights highlighting the superiority of PU-SHRF and PU-HRF. PU classification under high class imbalance presents challenges for most PU methods. Despite the inability to learn from PU data, resampling strategies can outperform most PU methods. PU methods that perform well on imbalanced datasets integrate specialized mechanisms to reduce bias towards the majority class. Imbalanced nnPU incorporates oversampling for risk minimization, while PU Bagging utilizes balanced bootstrap sampling. However, each of these strategies achieves either better recall (i.e., ROC-AUC or F1 score) or better precision (i.e., PR-AUC). HRF and SHRF were also observed to be competing techniques in imbalanced learning, particularly in precision-related metrics such as PR-AUC. The PU methods of the present invention (PU-SHRF and PU-HRF) combine a better mechanism for searching for unlabeled positive examples with the robustness of Hellinger distance for unbalanced learning, thereby enabling state-of-the-art performance. Furthermore, tuned bootstrap sampling, which ensures that positive example instances are always considered, can help improve performance under high flip ratios. [Table 3] The best-performing model in each column Bold and underlined text This is shown. Values ​​in bold indicate that the best-performing model does not outperform the classifier in the Model column at a 5% significance level. [Table 4] The best-performing model in each column Bold and underlined text This is shown. Values ​​in bold indicate that the best-performing model does not outperform the classifier in the Model column at a 5% significance level.

[0077] Figure 3 shows a comparison of the ranks of F1 scores, PR-AUC, and ROC-AUC at optimal thresholds for various flip ratios for the top five imbalanced datasets. This analysis presents more detailed results for the datasets with the highest class imbalances. This analysis focused on scenarios where minority class underrepresentation is severe, such as when the scenario relates to application examples such as fraud detection and medical diagnosis. In addition to the PU methods disclosed herein, HRF, SHRF, ADASYN+RF, and imbalanced nnPU were considered in the performance-based analysis described above. Furthermore, the five datasets exhibiting the highest class imbalances, namely FCC, FCO, POK, SAT, and KDD, were selected. Figure 3 visualizes the rank of each metric to highlight the relative performance between the methods.

[0078] Figure 3(a) demonstrates that the PU method disclosed herein exhibits better median rank than other methods across all flip ratios. Furthermore, the PU method disclosed herein can be observed to exhibit less variability in F1 score rank compared to other competitors. In contrast to the aggregated results, RF+ADASYN appears to outperform imbalanced nnPU on high-dimensional datasets. In Figure 3(b), SHRF and HRF maintain a slight advantage over the PU method disclosed herein. PU-SHRF and PU-HRF exhibit better median rank than RF+ADASYN and imbalanced nnPU. The overall strong performance of Hellinger-based methods in PR-AUC on top-tier imbalanced datasets provides evidence consistent with previous studies on the effectiveness of Hellinger distances on extremely imbalanced datasets. Finally, in Figure 3(c), PU-SHRF, PU-HRF, and RF+ADASYN perform similarly and outperform the other methods. At low and moderate flip ratios, the PU methods disclosed herein are superior to other competitors in median rank, with the exception of RF+ADASYN.

[0079] Figure 4 shows a comparison of the ranks of F1 scores, PR-AUC, and ROC-AUC at optimal thresholds for various flop ratios for the top five high-dimensional datasets. This figure displays the results for the dataset with the most features. Given that the PU method disclosed herein is based on a random forest algorithm, a large number of irrelevant features can negatively impact the method's performance. The method described above was used for this analysis. Furthermore, the following datasets with the most features were selected: SPE, FIE, MNI, TVC, and CCH. Figure 4 also visualizes the ranks of F1 scores, ROC-AUC, and PR-AUC.

[0080] As shown in Figure 4(a), the PU method disclosed herein performs slightly better than imbalanced nnPU and RF+ADASYN. However, significant overlap was observed between the disclosed methods using SHRF and HRF in F1 scores and PR-AUC. Furthermore, PU-SHRF and HRF exhibit less rank variability than SHRF and HRF. As shown in Figure 4(b), SHRF and HRF maintain a slight advantage over our PU method. PU-SHRF shows better median rank than RF+ADASYN and imbalanced nnPU across flip ratios. However, RF+ADASYN achieves the smallest interquartile range (IQR), while SHRF and HRF suffer from the largest IQR. Therefore, as shown in Figure 4(c), PU-SHRF and PU-HRF are superior to SHRF and HRF. Moreover, PU-SHRF achieves the best median rank at low and medium flip ratios. Nevertheless, at high flip ratios, imbalanced nnPU and RF+ADASYN are superior. Despite using XGBoost as its classifier, imbalanced nnPU does not outperform the other methods. In positive-negative example settings, boosting is more robust to high-dimensional datasets than bagging-based techniques. However, under PU settings, bagging often shows a better ensemble strategy because it is more robust to overfitting.

[0081] In this embodiment, we estimated a sensitivity analysis for class prior probabilities. Accurate estimation of class prior probabilities is crucial for most state-of-the-art PU methods. The tree-based methods disclosed herein utilize class prior probabilities to incorporate unlabeled positive examples into node partitioning. On the one hand, underestimating class prior probabilities can lead to insufficient searching of unlabeled positive examples for the training task. On the other hand, overestimating class prior probabilities can consistently produce more false positives because an excessive number of unlabeled instances are considered positive. Several studies have proposed techniques for providing estimates of class prior probabilities under the SCAR assumption. In this embodiment, we chose a sensitivity analysis to measure the change in performance when the class prior probabilities are misestimated. The sensitivity analysis focuses on a SCAR setting with a 50% flip ratio to represent a scenario where half of the positive examples are mislabeled. As with other evaluations in this embodiment, imbalanced nnPU, PU-HDT, PU-HRF, and PU-SHRF were selected for the sensitivity analysis. We utilized class prior probability estimates of various values ​​based on the ground truth class prior probability α. Sensitivity analysis uses class prior probability estimates.

number

[0082] Figures 5 and 6 show the results of the sensitivity analysis. Of the selected PU methods, imbalanced nnPU appears to be the most robust to erroneous class prior probability estimates. The robustness of (imbalanced)nnPU in this experiment supports previous results. With a modified PU loss function that prevents negative empirical risk, nnPU offers stronger robustness compared to uPU.

[0083] Figure 5 shows the mean F1 score (%) at the optimal threshold for each dataset in the 50% flip ratio across various class prior probability estimates. Each experimental setup was repeated 20 times. As shown in Figure 5 with respect to the F1 score, for some datasets (e.g., FCC, MAM, PCU, CGO, YEA, POK, KDD), the algorithms presented herein exhibit a "hat" pattern with a performance peak when the estimated class prior probabilities are ground truth. This pattern is more extreme in PU-HDT compared to PU-HRF and PU-SHRF. Therefore, the ensemble-based methods disclosed herein should be preferred over PU-HDT due to their stronger robustness to erroneous class prior probability estimates. Despite the lower mean F1 score, imbalanced nnPU exhibits a smoother "hat" pattern associated with smaller performance degradation when class prior probabilities are misestimated. Overestimation of class prior probabilities may be more detrimental to the PU methods disclosed herein than to imbalanced nnPU.

[0084] Figure 6 shows the mean PR-AUC (%) per dataset at 50% flip ratios across various class prior probability estimates. Each experimental setup was repeated 20 times. As shown in Figure 6 with respect to PR-AUC, the PU methods disclosed herein exhibit a stronger weakness to overestimation than to underestimation in some datasets. This pattern is more pronounced in PCU, CKO, CCH, and THY. In the case of PU-HDT, the performance degradation when class prior probabilities are overestimated is greater than in ensemble-based PU methods. Intuitively, overestimating class prior probabilities leads to more false positives in method training.

[0085] The PU learning techniques disclosed herein have the ability to handle highly imbalanced datasets and PU Hellinger decision trees (PU-HDT). PU-HDT utilizes the Hellinger distance as a partitioning criterion that exhibits robustness to extreme class imbalances. Furthermore, PU-HDT can learn from PU data by estimating positive examples from unlabeled instances at each node of the tree. Unlike other PU methods for imbalanced learning, PU-HDT does not require additional misclassification costs or resampling strategies.

[0086] One aspect of the disclosed invention is a PU-Hellinger stratified random forest (PU-SHRF) using PU-HDT as the basic learner. Empirical analysis generally suggests that PU-SHRF outperforms all known PU methods under all experimental settings. Techniques for imbalanced learning can outperform state-of-the-art PU methods without adaptation to imbalanced datasets. Statistical hypothesis testing is applied to further validate the empirical findings. Furthermore, this embodiment considers sensitivity analysis with respect to class prior probability estimates. PU-SHRF and PU-HRF have been shown to be more robust than PU-HDT to erroneous class prior probability estimates. Nevertheless, class prior probabilities may need to be estimated sufficiently well.

[0087] We assumed that labeled positive examples represent a random subset of the positive example class. This scenario referenced the SCAR assumption. However, the SCAR assumption does not apply to most real-world applications. Therefore, this example can be extended to accommodate more realistic assumptions. We can also analyze imbalanced data streams in accordance with this scenario.

[0088] Example 3 The following is an example of a Hellinger decision tree for fraud detection that implements the machine learning algorithms disclosed herein.

[0089] Hellinger trees can add value to many fraud detection applications. The inventions disclosed herein include algorithmic properties that enable state-of-the-art performance in fraud detection.

[0090] This technique provides robustness to class imbalances. The Hellinger distance has demonstrated strong theoretical robustness to extreme class imbalances in previous studies. Figure 2, based on the half-moon dataset in scikit-learn, illustrates the advantages of the Hellinger distance under an imbalanced PU setting. For the algorithm, half of the positive examples are represented as negative examples (i.e., unlabeled). It can be observed that Hellinger-based decision trees can learn the actual half-moon pattern of the dataset better than standard CART.

[0091] This technique provides the capability for PU learning. As long as the end user can provide an estimated fraud rate, the Hellinger tree can be easily adapted for PU learning. Figure 2 shows a comparison of decision boundaries of decision trees, including a PU-Hellinger decision tree, to illustrate the challenge of learning from positive and unlabeled data in an imbalanced setting. Negative examples, hidden positive examples, and positive example classes are represented. Regions with darker shading indicate higher certainty regarding classification into either the positive or negative example class. Therefore, the brighter the shading, the higher the uncertainty of the classifier. Specifically, Figure 2 shows an improvement by adding information about the fraud rate to the Hellinger decision tree. Compared to a vanilla Hellinger decision tree, the PU-Hellinger decision tree correctly identifies the majority of hidden fraudsters shown in Figure 2(b) as fraudsters falling into the darker shaded region of the decision boundary.

[0092] This technique is robust against overfitting under PU settings. Hellinger trees can be used as a basic learner for random forest settings. One advantage of random forests compared to modern boosting-based algorithms (e.g., XGBoost) is that bagging-based methods are less likely to overfit in settings with inaccurate or incomplete label information. Algorithms that suffer from overfitting can rapidly lose predictive performance over time as fraudulent activities evolve to evade detection systems.

[0093] In this example, we experimented with open-source fraud detection datasets to evaluate the performance of PU Hellinger trees. Fraud detection is particularly difficult due to high class imbalance. Most fraud datasets contain less than 10% fraud cases. The first dataset (FCC) was provided on Kaggle by Wordline and ULB. This dataset contains credit card transactions made by European cardholders in September 2013. This dataset represents transactions that occurred over two days and contains 282,982 transactions, with 0.16% fraud cases. The second dataset (FIE) originates from the IEEE-CIS research competition sponsored by Vesta Corporation and held on Kaggle. This dataset contains 590,540 online transactions, with 5.98% fraud cases. The last dataset (FCI) originates from auto insurance fraud detection provided by Oracle. This dataset contains 15,420 auto insurance claims from an unnamed insurance company between 1994 and 1996. The datasets are shown in Table 5. [Table 5]

[0094] To create PU data, in each dataset, several positive examples were flipped to unlabeled observations completely randomly. In the experiment, the number of labeled positive examples was determined by the flip ratio, which is the proportion of unlabeled positive examples. Three flip ratio values ​​were considered: 25%, 50%, and 75%. For each dataset, 20 holdout validations were performed to split the data into a training set (70%) and a test set (30%). If the process required parameter tuning, the training set was split into two equal sets to obtain a validation set.

[0095] The results of this embodiment included comparisons with two prior techniques, namely imbalanced nnPU and ADASYN+Random Forest (ADASYN+RF). The imbalanced nnPU method exhibits state-of-the-art performance in classification of imbalanced PU. In this embodiment, imbalanced nnPU utilized XGBoost as a wrapper method. This embodiment also included a random forest trained on data rebalanced by ADASYN. Resampling methods are one of the most common solutions for imbalanced learning tasks. Furthermore, Hellinger tree-based algorithms are disclosed. PU-Hellinger Decision Tree (PU-HDT) is a standalone decision tree that uses Hellinger distance as the splitting criterion instead of Gini or entropy in scikit-learn. PU-Hellinger Random Forest and PU-Layered Hellinger Random Forest are two ensemble methods that use PU-HDT as the base learner. The difference between PU-SHRF and PU-HRF is that the former implements stratified bagging, meaning that each base learner's training always includes positive (incorrect) instances.

[0096] For the Fraud Credit Card (FCC) dataset, PU-HRF and PU-SHRF were observed to outperform state-of-the-art imbalanced nnPU and ADASYN+RF in most experimental settings. At moderate and high flip ratios, PU-SHRF benefits from stratified bagging, resulting in better performance than PU-HRF. Intuitively, stratified bagging becomes more relevant as there are fewer positive examples available for training. Higher flip ratios result in fewer available positive examples. ADASYN helps improve the performance of random forests, outperforming XGBoost-based imbalanced nnPU. [Table 6] The best-performing model in each column Bold and underlined text This is indicated by the symbol *. The symbol * indicates that this technique is an invention disclosed herein. [Table 7] The best-performing model in each column Bold and underlined text This is indicated by the symbol *. The symbol * indicates that this technique is an invention disclosed herein.

[0097] For the Fraud IEEE (FIE) dataset, PU-HRF and PU-SHRF were observed to outperform state-of-the-art imbalanced nnPU and ADASYN+RF in most experimental settings. However, an exception arose at high flip-ratio settings where ADASYN+RF achieved the best PR-AUC performance. At high flip-ratios, PU-SHRF benefited from stratified bagging, resulting in better performance than PU-HRF. As in the previous paragraph, ADASYN+RF outperformed XGBoost-based imbalanced nnPU. [Table 8] The best-performing model in each column Bold and underlined textThis is indicated by the symbol *. The symbol * indicates that this technique is an invention disclosed herein. [Table 9] The best-performing model in each column Bold and underlined text This is indicated by the symbol *. The symbol * indicates that this technique is an invention disclosed herein. [Table 10] The best-performing model in each column Bold and underlined text This is indicated by the symbol *. The symbol * indicates that this technique is an invention disclosed herein. [Table 11] The best-performing model in each column Bold and underlined text This is indicated by the symbol *. The symbol * indicates that this technique is an invention disclosed herein.

[0098] For the Fraud Car Insurance (FCI) dataset, PU-HRF and PU-SHRF outperform the state-of-the-art imbalanced nnPU and ADASYN+RF in most experimental settings. At high flip ratio settings, the three methods—PU-HDT, PU-SHRF, and ADASYN+RF—achieve identical F1 score performance. As shown herein, PU-SHRF excels in extreme settings with fewer fraud cases.

Claims

1. A method for detecting fraud, The processor receives a dataset of financial transactions, A method comprising using the processor to apply a Hellinger decision tree to detect fraudulent transactions in a dataset of financial transactions, wherein using a Hellinger distance is part of the application.

2. The method according to claim 1, wherein the Hellinger decision tree is part of a machine learning algorithm.

3. The method according to claim 1, wherein the Hellinger decision tree is configured to capture the divergence between the positive example class distribution and the negative example class distribution without being dominated by class imbalance.

4. The method according to claim 1, wherein the Hellinger decision tree is configured to use class prior probabilities to estimate the number of positive and negative examples at each node.

5. The method according to claim 1, further comprising using the processor to limit the size of the Hellinger decision tree after the tree nodes have reached their maximum height, thereby avoiding overfitting.

6. The method according to claim 1, wherein the Hellinger decision tree is a Hellinger decision tree with positive examples and an imbalance, and the dataset of fraudulent transactions is an imbalance with positive examples and unlabeled data.

7. The method according to claim 1, further comprising receiving an estimated fraud rate in the processor before applying the above.

8. The method according to claim 1, wherein the Hellinger decision tree is used as a basic learner in a modified random forest.

9. The method according to claim 8, wherein the Hellinger decision tree is a Hellinger decision tree with positive examples and an imbalance, and the dataset of fraudulent transactions is an imbalance with positive examples and unlabeled data.

10. The method according to claim 9, wherein the Hellinger decision tree is configured to consider random feature selection when initializing tree nodes.

11. The method according to claim 9, wherein the Hellinger decision tree is configured to use the size and class prior probability of a stratified bootstrap sample.

12. The method according to claim 1, wherein the Hellinger distance is used to capture the divergence between the positive example class distribution and the negative example class distribution.

13. The aforementioned processor receives positive examples and an unlabeled dataset, The Hellinger decision tree is trained using the aforementioned positive examples and unlabeled datasets, The method according to claim 1, further comprising:

14. The method according to claim 13, wherein the positive examples and the unlabeled dataset are imbalanced.

15. The method according to claim 1, wherein the financial transaction is a credit card transaction.

16. The method according to claim 1, wherein the financial transaction is an insurance transaction.

17. A system configured to perform the method described in claim 1, wherein the system is a computer or a server.

18. A non-temporary computer-readable medium for storing a program configured to instruct a processor to perform the method described in claim 1.

Images