Isolated forest abnormal data detection method and system based on clustering enhancement and medium

Abstract

The invention belongs to the technical field of financial risk control systems, and particularly relates to an isolated forest abnormal data detection method and system based on clustering enhancement and a medium. S2, a K-Means clustering algorithm is adopted to screen out data clusters needing to be further detected; s3, constructing an isolated forest model according to the screened data clusters, and performing model training through a training set; s4, predicting the model through the test set, and calculating an initial abnormal score; the class density is calculated to serve as a weight to weight the initial abnormal score, and a final abnormal score is obtained; s5, judging the final abnormal score according to a preset threshold value, and outputting an abnormal data detection result; and S6, evaluating the isolated forest model, adjusting model parameters according to an evaluation result, and continuously optimizing the model until a preset requirement is met. The problems that abnormal data detection is complex in calculation and low in efficiency are solved.

Description

Technical Field

[0001] This invention belongs to the technical field of financial risk control systems, and in particular relates to a method, system and medium for detecting isolated forest anomalies based on clustering enhancement. Background Technology

[0002] With the continuous expansion of banking operations and the increasing level of digitalization, massive amounts of business data are constantly accumulating in bank data warehouses. This data is crucial for many aspects of the bank's operational decisions, risk management, and customer service. However, various anomalies may exist within this data, such as fraudulent transactions, erroneous data entry, and system malfunctions. If such anomalies are not detected and handled in a timely manner, they could potentially lead to significant economic losses and incalculable risks for the bank. Therefore, close monitoring and in-depth analysis of the data in the data warehouse are necessary to achieve timely processing.

[0003] Traditional statistical methods, such as mean and standard deviation, struggle to effectively handle complex data distributions and nonlinear relationships. For example, in data with multimodal or long-tailed distributions, these methods may generate numerous false positives and false negatives. Rule-based detection methods, while addressing false positives and false negatives to some extent, often rely on human experience for rule formulation, making them ill-suited to dynamic data changes and new anomaly patterns. For instance, existing rules may become inapplicable when business logic alters. With the widespread adoption of machine learning, methods like decision trees and support vector machines are used in data detection; however, they suffer from high computational complexity and inefficiency when processing high-dimensional and large-scale data. Summary of the Invention

[0004] This invention provides a method, system, and medium for detecting anomalies in isolated forests based on clustering enhancement. It uses the K-Means clustering algorithm for preliminary anomaly judgment and then outputs the final anomaly detection results through the isolated forest model, thereby solving the problems of computational complexity and low efficiency in existing technologies.

[0005] The basic solution provided by this invention is a method for detecting anomaly data in isolated forests based on clustering enhancement, comprising the following steps: S1: Data preprocessing; S2: The K-Means clustering algorithm is used to cluster the preprocessed data to obtain at least one data cluster. The silhouette coefficient and density analysis are used to make a preliminary anomaly judgment on each data cluster and screen out the data clusters that need to be further detected. S3: Construct an isolated forest model based on the selected data clusters, and train the model using the training set; S4: Predict the model using the test set, calculate the path length of each data point in the isolated tree in the test set, and calculate the initial anomaly score based on the expected path length and the average path length; calculate the class density using the average internal distance of the data clusters, and use the class density as a weight to weight the initial anomaly score to obtain the final anomaly score. S5: Determine the final anomaly score based on the preset threshold and output the anomaly data detection result; S6: Evaluate the isolated forest model and adjust the model parameters based on the evaluation results, continuously optimizing the model until it meets the preset requirements.

[0006] Preferably, in S1, data preprocessing includes data cleaning, data standardization, and feature selection.

[0007] Preferably, S2 includes S2-1) The clustering effect under different K values ​​is evaluated by the silhouette coefficient to determine the optimal number of clusters K for K-Means clustering; S2-2) Randomly select K data points as initial cluster centers, calculate the distance from each data point to each cluster center, assign the data points to the cluster to which the nearest cluster center belongs, and recalculate the center of each cluster until the clustering criterion function converges and the update stops. S2-3) Calculate the overall profile coefficient of each cluster and combine it with density analysis to determine clusters that cannot be clearly classified as normal or abnormal.

[0008] Preferably, S3 includes S3-1) Divide the data clusters that need further testing in S2 into training and test sets according to a preset ratio; S3-2) Set the model parameters according to the size, complexity and performance requirements of the data cluster, and build the basic framework of the isolated forest model based on the model parameters, and determine the random feature selection method and split value generation rules for each isolated tree; S3-3) The isolated forest model is trained using the training set. By continuously and randomly selecting features and segmentation values, multiple isolated trees are repeatedly generated to form a forest.

[0009] More preferably, in S3-2), the model parameters include the number of trees, the size of sub-data points, and the anomaly ratio, wherein the number of trees is 100, the size of sub-data points is auto, and the anomaly ratio is 0.1.

[0010] Preferably, in S4, the initial anomaly score is calculated as follows:

[0011] Where e(h(x)) is the expected value of the path length, and c(n) is the average path length, which is the average of the path lengths of the data point in all isolated trees. The expected value of the average path length is related to the number of data points.

[0012] More preferably, in S4, the class density is calculated as follows: a. Define class density, specifically using average internal distance to measure class density; b. For each data cluster, calculate the distance from each data point within the cluster to the cluster center; c. Take the average of all calculated distances to obtain the class density.

[0013] Preferably, in S6, the evaluation metrics for the isolated forest model include accuracy, recall, F1 score, and precision.

[0014] This solution also provides an isolated forest anomaly detection system based on clustering enhancement, applied to the above-mentioned anomaly detection methods, including: The data input and preprocessing module is used to collect raw data and preprocess it. The preliminary anomaly detection module uses the K-Means clustering algorithm to perform preliminary anomaly detection on the preprocessed data and filter out data clusters that need further detection. The isolated forest model building module is used to construct an isolated forest model based on the selected data clusters, train the model using a training set, and predict the model using a test set. It calculates the path length of each data point in the isolated tree in the test set, calculates the initial anomaly score based on the expected path length and the average path length, calculates the class density using the average internal distance of the data clusters, and uses the class density as a weight to weight the initial anomaly score to obtain the final anomaly score. The anomaly detection and result output module uses a trained model to detect anomalies in the data and outputs the anomaly detection results. The model evaluation and optimization module evaluates the isolated forest model and adjusts the model parameters based on the evaluation results, continuously optimizing the model until it meets the preset requirements.

[0015] This solution also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of an isolated forest anomaly data detection method based on clustering enhancement.

[0016] The principles and advantages of this invention are as follows: 1. By rapidly partitioning data clusters using K-Means, Isolation Forest is applied only to low-density clusters (potential anomalies), reducing subsequent computation. Isolation Forest only processes local data (such as sparse clusters), avoiding global computation, reducing time complexity, and effectively improving detection efficiency. Compared to traditional methods, which require building isolated trees for all data, this solution narrows the detection scope through clustering, significantly improving efficiency and making it suitable for large-scale data processing in banking applications.

[0017] 2. By using class density as a weight to weight the outlier scores of isolated forests, the distribution characteristics of the data are fully considered, enabling the model to more accurately assess the degree of anomaly under different data distribution conditions, enhancing the model's generalization ability, and solving the problems of false positives and false negatives of outlier data.

[0018] Preferably, adaptive clustering dynamically determines the optimal K value through the silhouette coefficient, avoiding the subjectivity of manual pre-setting and improving the rationality of clustering; dynamically updates the cluster centers to ensure the representativeness of the mean of data points within the cluster; and combines the silhouette coefficient (to measure the quality of clustering) and density analysis (such as the average distance within the cluster) to accurately identify "clusters that need further detection".

[0019] Preferably, targeted modeling is used, with each suspicious cluster being independently constructed into an isolated forest to avoid redundant computations in global modeling; model parameters are configured according to the selected suspicious clusters, and training is automated: random feature segmentation ensures the diversity of each tree, and ensemble learning improves robustness. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a logic block diagram of the system of the present invention; Figure 3 This is a schematic diagram of the architecture of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] The following detailed description illustrates the specific implementation method: The specific implementation process is as follows: Example 1 See Figure 1 A method for detecting anomaly data in isolated forests based on clustering enhancement, comprising the following steps: S1: Collect raw data and preprocess the data.

[0022] In this embodiment, the method is illustrated using the collection of a credit application model table.

[0023] S1-1: Clean the raw data to be tested, specifically including handling missing values ​​and removing duplicate values; Handling missing values: Fill in missing values ​​in the original data. First, determine the data type of the original data, and then select the corresponding processing method to handle missing values ​​based on the data type.

[0024] For numeric fields, the mean, median, or -999 is used for padding based on data distribution and missing value rate. Specifically, if the data is approximately normally distributed and the missing value rate is low, the mean is used for padding; if the data distribution is right-skewed (including outliers with large debts) and the missing value rate is moderate, the median is used to avoid the mean being affected by outliers; if the missing value rate is high, -999 is used to mark missing features. In this embodiment, in the credit application model table, numeric fields include application amount, monthly income, and debt amount. If the data in the "monthly income" field is approximately normally distributed and the missing value rate is low (3%), the mean is used for padding.

[0025] For categorical fields, the mode or the special value "other" is used for filling based on the data distribution and missing rate. Specifically, if the proportion is 60% (mode) and the missing rate is low, the mode is used for filling; if the data is scattered (no obvious mode) and the missing rate is high, "other" is used for filling. In this embodiment, "marital status" ("married" accounts for 55% and the missing rate is 6%) is filled with the mode; "loan purpose" (the categories are scattered and the missing rate is 20%) is filled with "other".

[0026] Remove duplicate values: Identify and delete duplicate records in the data to ensure the uniqueness of each data entry. Specifically, identify duplicate application records by combining "ID number + application date" and delete duplicates to ensure data uniqueness.

[0027] S1-2: Z-score standardization is used to convert the data into a format with uniform dimensions and distribution, which facilitates subsequent analysis and calculation; S1-3: By using methods such as correlation analysis and variance analysis, features that have a significant impact on anomaly detection are selected, reducing data dimensionality and improving model efficiency.

[0028] S2: The K-Means clustering algorithm is used to cluster the preprocessed data to obtain at least one data cluster. The silhouette coefficient and density analysis are used to make a preliminary anomaly judgment on each data cluster and screen out the data clusters that need to be further detected. S2-1) The clustering effect under different K values ​​is evaluated by the silhouette coefficient to determine the optimal number of clusters K for K-Means clustering; a. Profile coefficient: For a single data point Calculate its average distance from its own cluster and from other clusters: The average distance between a data point and other points in the same cluster:

[0029] in, For data points The cluster, For clusters The number of data points in the dataset, where d(i,j) is the number of data points. with co-cluster data points The distance between them; Minimum average distance between data points and other clusters:

[0030] in, To remove clusters Other clusters, For clusters The number of data points in the dataset, where d(i,j) is the number of data points. with cluster Mid-data points The distance between them; Profile coefficient:

[0031] in, The density of data points within the same cluster. This represents the separation degree of the data point from other clusters.

[0032] The range is [-1, 1], s(i)≈1: Closely clustered with points in the same cluster and far from other clusters (reasonable clustering); s(i)≈-1: It may be assigned to an incorrect cluster, s(i)≈0. Located at the cluster boundary.

[0033] b. Select the optimal K: Plot the relationship curve between K and the profile coefficient, and select the K corresponding to the peak value.

[0034] S2-2) includes a. Randomly select K data points as the initial cluster centers. ; b. Calculate the distance from each data point to each cluster center using the following formula, and assign the data point to the cluster to which the nearest cluster center belongs;

[0035] in, Let i be the feature vector of the i-th data point. arg is the centroid of the k-th cluster To return the cluster index k that minimizes the distance (i.e., the nearest cluster).

[0036] c. And recalculate the center of each cluster until the clustering criterion function converges and the update stops; Termination condition: The change in centroid is less than the threshold or the maximum number of iterations is reached.

[0037] S2-3) Calculate the overall profile coefficient of each cluster, i.e., the mean profile coefficient of all data points within each cluster. In conjunction with density analysis, clusters that cannot be clearly classified as normal or abnormal are identified.

[0038] Density analysis includes: a. Calculate the average intra-cluster distance:

[0039] in, For the k-th cluster, For clusters The number of data points in the data. Let k be the centroid of cluster k. For data points With the center of mass Euclidean distance (L2 norm); b. Low-density cluster determination: If Clusters with a density greater than α × global average density (e.g., α = 1.5) are marked as candidate anomalous clusters. The global average density is the mean of the distances within all clusters.

[0040] The rules for cluster classification are as follows: Less than 0.2 or Clusters with excessively high values ​​are those that cannot be clearly categorized.

[0041] S3: Construct an isolated forest model based on the selected data clusters, and train the model using the training set; S3 includes S3-1) Divide the data clusters that need further testing in S2 into training set and test set according to a preset ratio. In this embodiment, the training set:test set = 7:3. S3-2) Set the model parameters according to the size, complexity and performance requirements of the data cluster, and build the basic framework of the isolated forest model based on the model parameters, and determine the random feature selection method and split value generation rules for each isolated tree; The model parameters include the number of trees, the size of sub-data points, and the anomaly ratio. Specifically, these parameters are adjusted according to the size of the cluster and the complexity of the data. Generally, the larger the cluster and the more complex the data, the greater the number of trees and the size of the sub-samples can be. In this embodiment, the number of trees is 100, the size of the sub-data points is auto (256), and the anomaly ratio is 0.1.

[0042] S3-3) The isolated forest model is trained using the training set. By continuously and randomly selecting features and segmentation values, multiple isolated trees that can effectively distinguish between normal and abnormal data are repeatedly constructed to form a forest.

[0043] S4: Predict the model using the test set, calculate the path length of each data point in the isolated tree in the test set, and calculate the initial anomaly score based on the expected path length and the average path length; calculate the class density using the average internal distance of the data clusters, and use the class density as a weight to weight the initial anomaly score to obtain the final anomaly score. In S4, the initial anomaly score is calculated as follows:

[0044] Where e(h(x)) is the expected value of the path length, and c(n) is the average path length, which is the average of the path lengths of the data point in all isolated trees. The expected value of the average path length is related to the number of data points.

[0045] In S4, the class density is calculated as follows: a. Define class density, specifically using average internal distance to measure class density; b. For each data cluster, calculate the distance from each data point within the cluster to the cluster center; c. Take the average of all calculated distances to obtain the class density.

[0046] The average value is normalized and used as the class density weight. The final outlier score is calculated as the product of the class density weight and the initial outlier score. In this embodiment, the class density weight is 0.8, and the initial outlier score is 0.6, so the weighted outlier score is 0.6 * 0.8 = 0.48.

[0047] S5: Determine the final anomaly score based on the preset threshold and output the anomaly data detection result; The preset threshold is determined based on experiments or business needs.

[0048] The judgment rule is that data points with an abnormal score higher than the threshold are judged as abnormal data, and the abnormal data identifier and related information, such as the cluster to which it belongs and the original data value, are output.

[0049] S6: Evaluate the isolated forest model and adjust the model parameters based on the evaluation results, continuously optimizing the model until it meets the preset requirements. In S6, the evaluation metrics for the isolated forest model include accuracy, recall, F1 score, and precision.

[0050] The evaluation of the model involves comparing the predicted abnormal data points with pre-determined abnormal data points based on the model's prediction results for the test set data, calculating model evaluation indicators, and measuring the model's performance.

[0051] Specifically, adjusting the model parameters involves adjusting the parameters based on the evaluation results if the model's performance does not meet the requirements. This includes increasing the number of trees, adjusting the anomaly ratio, etc., and then retraining and evaluating until satisfactory performance is obtained.

[0052] It also includes S7, which uses anomaly scoring to perform attribution analysis on data with high scores, identify the causes of anomalies, and quickly resolve them.

[0053] Example 2 See Figure 2 An isolated forest anomaly detection system based on clustering enhancement, used in the aforementioned anomaly detection method, includes: The data input and preprocessing module is used to collect raw data and preprocess it. The preliminary anomaly detection module uses the K-Means clustering algorithm to perform preliminary anomaly detection on the preprocessed data and filter out data clusters that need further detection. The isolated forest model building module is used to construct an isolated forest model based on the selected data clusters, train the model using a training set, and predict the model using a test set. It calculates the path length of each data point in the isolated tree in the test set, calculates the initial anomaly score based on the expected path length and the average path length, calculates the class density using the average internal distance of the data clusters, and uses the class density as a weight to weight the initial anomaly score to obtain the final anomaly score. The anomaly detection and result output module uses a trained model to detect anomalies in the data and outputs the anomaly detection results. The model evaluation and optimization module evaluates the isolated forest model and adjusts the model parameters based on the evaluation results, continuously optimizing the model until it meets the preset requirements.

[0054] Example 3 A storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described cluster-enhanced isolated forest anomaly data detection method.

[0055] Those skilled in the art will understand that implementing all or part of the processes in a cluster-enhanced isolated forest anomaly detection method can be accomplished by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of various embodiments of the cluster-enhanced isolated forest anomaly detection method. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0056] This application embodiment also provides an electronic device 500 that utilizes the aforementioned cluster-enhanced isolated forest anomaly data detection system. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the aforementioned cluster-enhanced isolated forest anomaly data detection method. In this application embodiment, the processor is the control center of the computer method and can be a physical machine processor or a virtual machine processor.

[0057] Reference Figure 3 The electronic device 500 includes at least one processor 501, at least one communication interface 502, at least one memory 503, and at least one bus 504. The bus 504 is used for communication between these components, the communication interface 502 is used for signaling or data communication with other node devices, and the memory 503 stores machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 communicates with the memory 503 via the bus 504. When the machine-readable instructions are invoked by the processor 501, they execute the steps of the described method for detecting anomalies in isolated forests based on clustering enhancement.

[0058] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for detecting anomaly data in isolated forests based on clustering enhancement, characterized by the following steps: include: S1: Data preprocessing; S2: The K-Means clustering algorithm is used to cluster the preprocessed data to obtain at least one data cluster. The silhouette coefficient and density analysis are used to make a preliminary anomaly judgment on each data cluster and screen out the data clusters that need to be further detected. S3: Construct an isolated forest model based on the selected data clusters, and train the model using the training set; S4: Predict the model using the test set, calculate the path length of each data point in the isolated tree in the test set, and calculate the initial anomaly score based on the expected path length and the average path length; calculate the class density using the average internal distance of the data clusters, and use the class density as a weight to weight the initial anomaly score to obtain the final anomaly score. S5: Determine the final anomaly score based on the preset threshold and output the anomaly data detection result; S6: Evaluate the isolated forest model and adjust the model parameters based on the evaluation results, continuously optimizing the model until it meets the preset requirements.

2. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 1, characterized in that: In S1, data preprocessing includes data cleaning, data standardization, and feature selection.

3. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 1, characterized in that: S2 includes S2-1) The clustering effect under different K values ​​is evaluated by the silhouette coefficient to determine the optimal number of clusters K for K-Means clustering; S2-2) Randomly select K data points as initial cluster centers, calculate the distance from each data point to each cluster center, assign the data points to the cluster to which the nearest cluster center belongs, and recalculate the center of each cluster until the clustering criterion function converges and the update stops. S2-3) Calculate the overall profile coefficient of each cluster and combine it with density analysis to determine clusters that cannot be clearly classified as normal or abnormal.

4. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 1, characterized in that: S3 includes S3-1) Divide the data clusters that need further testing in S2 into training and test sets according to a preset ratio; S3-2) Set the model parameters according to the size, complexity and performance requirements of the data cluster, and build the basic framework of the isolated forest model based on the model parameters, and determine the random feature selection method and split value generation rules for each isolated tree; S3-3) The isolated forest model is trained using the training set. By continuously and randomly selecting features and segmentation values, multiple isolated trees are repeatedly generated to form a forest.

5. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 4, characterized in that: In S3-2), the model parameters include the number of trees, the size of sub-data points, and the anomaly ratio. The number of trees is 100, the size of sub-data points is auto, and the anomaly ratio is 0.

1.

6. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 1, characterized in that: In S4, the initial anomaly score is calculated as follows: Where e(h(x)) is the expected value of the path length, and c(n) is the average path length, which is the average of the path lengths of the data point in all isolated trees. The expected value of the average path length is related to the number of data points.

7. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 6, characterized in that: In S4, the class density is calculated as follows: a. Define class density, specifically using average internal distance to measure class density; b. For each data cluster, calculate the distance from each data point within the cluster to the cluster center; c. Take the average of all calculated distances to obtain the class density.

8. The method for detecting anomaly data in isolated forests based on clustering enhancement according to claim 1, characterized in that: In S6, the evaluation metrics for the isolated forest model include accuracy, recall, F1 score, and precision.

9. A cluster-enhanced isolated forest anomaly data detection system, characterized in that, include: The data input and preprocessing module is used to collect raw data and preprocess it. The preliminary anomaly detection module uses the K-Means clustering algorithm to perform preliminary anomaly detection on the preprocessed data and filter out data clusters that need further detection. The isolated forest model building module is used to construct an isolated forest model based on the selected data clusters, train the model using a training set, and predict the model using a test set. It calculates the path length of each data point in the isolated tree in the test set, calculates the initial anomaly score based on the expected path length and the average path length, calculates the class density using the average internal distance of the data clusters, and uses the class density as a weight to weight the initial anomaly score to obtain the final anomaly score. The anomaly detection and result output module uses a trained model to detect anomalies in the data and outputs the anomaly detection results. The model evaluation and optimization module evaluates the isolated forest model and adjusts the model parameters based on the evaluation results, continuously optimizing the model until it meets the preset requirements.

10. A storage medium on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements the steps of the isolated forest anomaly data detection method based on clustering enhancement as described in any one of claims 1-8.

Images