Big data intelligent cleaning method and system based on machine learning

Abstract

The invention provides a big data intelligent cleaning method and system based on machine learning, and relates to the technical field of data processing, and the method comprises the steps: 1, collecting and carrying out the preprocessing and feature engineering processing of a user transaction behavior data flow, extracting a multi-dimensional feature vector, and obtaining a to-be-cleaned feature data set; step 2, constructing a data feature space based on the to-be-cleaned feature data set, and mapping the multi-dimensional feature vector into the data feature space; and inputting the multi-dimensional feature vectors into an anomaly detection model constructed based on an isolated forest algorithm, and calculating a real-time anomaly score corresponding to each feature vector. The feature vector abnormal score is calculated through the isolated forest algorithm, the dynamic judgment threshold is generated in combination with the constructed quadrilateral judgment domain and the high-dimensional confidence ellipsoid projection, the unreal transaction data is cleaned, the model is iteratively updated along with the data feature distribution change, and the accuracy, adaptability and data reliability of big data cleaning are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a big data intelligent cleaning method and system based on machine learning. Background Technology

[0002] In the context of big data, real-time cleaning of user transaction behavior data is a crucial step in ensuring the quality of subsequent analysis and applications. Currently, machine learning-based anomaly detection methods, such as the Isolation Forest algorithm, are often used to identify atypical patterns in transaction data streams due to their lack of prior labels. However, these methods may face some challenges in practical deployment. A common consideration is that their anomaly judgment process mostly relies on a preset or static threshold obtained through initial data statistics. In practical applications, the characteristic distribution of user transaction behavior is not static and may gradually evolve with business dynamics. This situation is particularly evident in complex e-commerce transaction scenarios.

[0003] For example, during "flash sale" promotions or major holiday events on the platform, the normal fluctuation range of key characteristics such as transaction frequency and average order value may be temporarily widened, making the previously set static thresholds no longer fully applicable. At this time, the original model may misjudge some normal high-concurrency transactions caused by concentrated promotions as abnormal, or when the data returns to normal after the event, it may not be sensitive enough to new and hidden fraudulent behaviors, which may affect the accuracy and adaptability of the data cleaning results. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a big data intelligent cleaning method and system based on machine learning, so as to accurately locate non-real transaction data and reduce the false judgment rate and false negative rate.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a big data intelligent cleaning method based on machine learning, the method comprising: Step 1: Collect and preprocess user transaction behavior data streams and perform feature engineering to extract multi-dimensional feature vectors to obtain the feature dataset to be cleaned; Step 2: Based on the feature dataset to be cleaned, construct a data feature space and map multi-dimensional feature vectors into the data feature space; input the multi-dimensional feature vectors into the anomaly detection model built based on the isolated forest algorithm, and calculate the real-time anomaly score corresponding to each feature vector; Step 3: Perform statistical analysis on the real-time anomaly scores of all feature vectors to determine the initial judgment threshold; Step 4: In the data feature space, determine the first and second feature dimensions to establish an anchor set, and connect the anchors in the anchor set to form a quadrilateral anomaly determination domain. Step 5: Construct a high-dimensional confidence elliptic based on the covariance matrix and mean vector of the feature vectors. Project the high-dimensional confidence elliptic onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse. Calculate the area ratio of the overlapping region between the quadrilateral anomaly judgment domain and the two-dimensional confidence ellipse to obtain the threshold adjustment factor. Weight the threshold adjustment factor with the initial judgment threshold to generate a dynamic judgment threshold. Step 6: Compare the real-time anomaly score with the dynamic judgment threshold to identify and clean up non-genuine transaction data to obtain a clean data stream; continuously monitor the feature distribution changes of the clean data stream, and trigger the update process of the anomaly detection model and the dynamic judgment threshold when the distribution change exceeds the predetermined threshold.

[0006] Secondly, a big data intelligent cleaning system based on machine learning includes: The data acquisition module is used to collect and preprocess user transaction behavior data streams and perform feature engineering to extract multi-dimensional feature vectors to obtain a feature dataset to be cleaned. The computation module is used to construct a data feature space for the feature dataset to be cleaned, and to map multi-dimensional feature vectors to the data feature space; the multi-dimensional feature vectors are input into the anomaly detection model built based on the isolated forest algorithm to calculate the real-time anomaly score corresponding to each feature vector; The determination module is used to perform statistical analysis on the real-time anomaly scores of all feature vectors and determine the initial judgment threshold. A module is established to determine the first and second feature dimensions in the data feature space to establish an anchor set, and to connect the anchors in the anchor set to form a quadrilateral anomaly determination domain. The weighting module is used to construct a high-dimensional confidence ellipse based on the covariance matrix and mean vector of the feature vectors, project the high-dimensional confidence ellipse onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse; calculate the area ratio of the overlapping region between the quadrilateral anomaly judgment domain and the two-dimensional confidence ellipse to obtain the threshold adjustment factor; and weight the threshold adjustment factor with the initial judgment threshold to generate a dynamic judgment threshold. The update module compares the real-time anomaly score with the dynamic judgment threshold to identify and clean up non-genuine transaction data, resulting in a clean data stream. It continuously monitors the changes in the feature distribution of the clean data stream and triggers the update process of the anomaly detection model and the dynamic judgment threshold when the distribution change exceeds a predetermined threshold.

[0007] Thirdly, a computing device includes: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0008] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0009] The above-described solution of the present invention has at least the following beneficial effects: Through a dynamic threshold generation mechanism, it can respond in real time to normal fluctuations in transaction characteristics during promotional activities and peak holiday periods, avoiding misjudging high-concurrency normal transactions as abnormal. Simultaneously, it maintains sensitivity to new types of fraudulent behavior when data returns to normal, improving the adaptability of data cleaning in complex business scenarios. It integrates the efficient unsupervised detection capabilities of the Isolation Forest algorithm with the statistical rigor of high-dimensional confidence elliptic bodies, combined with multi-dimensional feature extraction and the spatial constraints of the quadrilateral anomaly determination domain, to locate non-genuine transaction data, reducing false positive and false negative rates and ensuring the quality of clean data streams. From data preprocessing and feature engineering to anomaly detection, dynamic threshold adjustment, and automatic model updates, a closed-loop intelligent cleaning process is formed, reducing the cost of manual intervention. By analyzing feature contribution to select core feature dimensions and constructing a visualized quadrilateral anomaly determination domain based on quartiles, combined with projection analysis of the high-dimensional confidence elliptic body, the logic and boundaries of anomaly determination are quantifiable, making the anomaly determination logic easier to understand and verify. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a machine learning-based intelligent big data cleaning method provided in an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of a big data intelligent cleaning system based on machine learning, provided by an embodiment of the present invention. Detailed Implementation

[0012] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0013] like Figure 1 As shown in the figure, an embodiment of the present invention proposes a big data intelligent cleaning method based on machine learning, the method comprising the following steps: Step 1: Collect and preprocess user transaction behavior data streams and perform feature engineering to extract multi-dimensional feature vectors to obtain the feature dataset to be cleaned; Step 2: Based on the feature dataset to be cleaned, construct a data feature space and map multi-dimensional feature vectors into the data feature space; input the multi-dimensional feature vectors into the anomaly detection model built based on the isolated forest algorithm, and calculate the real-time anomaly score corresponding to each feature vector; Step 3: Perform statistical analysis on the real-time anomaly scores of all feature vectors to determine the initial judgment threshold; Step 4: In the data feature space, determine the first and second feature dimensions to establish an anchor set, and connect the anchors in the anchor set to form a quadrilateral anomaly determination domain. Step 5: Construct a high-dimensional confidence elliptic based on the covariance matrix and mean vector of the feature vectors. Project the high-dimensional confidence elliptic onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse. Calculate the area ratio of the overlapping region between the quadrilateral anomaly judgment domain and the two-dimensional confidence ellipse to obtain the threshold adjustment factor. Weight the threshold adjustment factor with the initial judgment threshold to generate a dynamic judgment threshold. Step 6: Compare the real-time anomaly score with the dynamic judgment threshold to identify and clean up non-genuine transaction data to obtain a clean data stream; continuously monitor the feature distribution changes of the clean data stream, and trigger the update process of the anomaly detection model and the dynamic judgment threshold when the distribution change exceeds the predetermined threshold.

[0014] In this embodiment of the invention, a dynamic threshold generation mechanism can respond in real time to normal fluctuations in transaction characteristics during promotional activities and peak holiday periods, avoiding misjudging high-concurrency normal transactions as abnormal. Simultaneously, it maintains sensitivity to new types of fraudulent behavior when data returns to normal, improving the adaptability of data cleaning in complex business scenarios. It integrates the efficient unsupervised detection capabilities of the Isolation Forest algorithm with the statistical rigor of high-dimensional confidence elliptic bodies, combined with multi-dimensional feature extraction and the spatial constraints of the quadrilateral anomaly determination domain, to locate non-genuine transaction data, reducing false positive and false negative rates and ensuring the quality of clean data streams. From data preprocessing and feature engineering to anomaly detection, dynamic threshold adjustment, and automatic model updates, a closed-loop intelligent cleaning process is formed, reducing the cost of manual intervention. Core feature dimensions are selected through feature contribution analysis, and a visualized quadrilateral anomaly determination domain is constructed based on quartiles. Combined with projection analysis of the high-dimensional confidence elliptic body, the logic and boundaries of anomaly determination are quantifiable, making the anomaly determination logic easier to understand and verify.

[0015] In a preferred embodiment of the present invention, step 1 includes: Step 100: Collect user transaction behavior data stream. This data stream includes transaction timestamps, transaction amounts, buyer identifiers, seller identifiers, logistics status information, and user operation behavior sequences. Specifically, it involves synchronously collecting user transaction behavior-related data from multiple sources, including the transaction system, logistics management platform, and user operation log database, forming a continuous data stream. The collected data stream contains six core types of information: the transaction timestamp records the transaction occurrence time accurate to the second; the transaction amount is the actual payment amount for the transaction; the buyer and seller identifiers are unique user identification information assigned by the platform; the logistics status information records the logistics flow nodes corresponding to the transaction; and the user operation behavior sequence records the entire process of the user's operation steps from logging into the platform, browsing products, submitting an order, to completing payment. Real-time streaming transmission is used during the collection process to ensure the timeliness of data collection. A data collection verification mechanism is also established to check the consistency of data from various data sources and avoid data loss or errors during transmission.

[0016] Step 101 involves data cleaning of the user transaction behavior data stream, including removing duplicate records, filling in missing values, and correcting format errors. Based on the cleaned user transaction behavior data stream, time-dimensional features, monetary-dimensional features, behavioral sequence-dimensional features, and relationship network-dimensional features are extracted to form a multi-dimensional feature vector. Specifically, this includes using transaction timestamps, buyer identifiers, seller identifiers, and transaction amounts as the core verification combination. If multiple records have completely identical information in this combination, they are determined to be duplicate records, and only the earliest generated record is retained. For numerical missing data, such as missing transaction amounts, the average transaction amount of the same type of goods within the same time period is calculated. The mean is used to impute missing values. The mean is calculated by dividing the sum of all non-missing transaction amounts for the same type of product within the same time period by the number of records with non-missing transaction amounts. For categorized missing data, such as missing logistics status information, the mode imputation method is used, which calculates the most frequent logistics status in the buyer's historical transactions as the imputation result. Transaction timestamps are uniformly converted to a standard time format, and redundant characters are removed. Transaction amounts are uniformly adjusted to a numerical format retaining two decimal places. For abnormal format data exceeding a reasonable amount range, correction is performed based on the price range of similar products; that is, non-numeric characters in the amount data are first filtered out. For characters such as currency symbols, notes, and spaces, convert the remaining valid characters into numeric types; for digits with more than two decimal places, round to two decimal places according to the rounding rules, and pad with ".00" if there are no decimal places; at the same time, establish a reasonable amount range determination mechanism, calculated as follows: collect all valid transaction amount data of this category that has undergone basic cleaning within the past 90 days, sort them in ascending order, count the total number of data entries n, calculate the first quartile Q1 and the third quartile Q3, the position of Q1 is (n+1)×0.25, if the position is an integer, take the value of the corresponding index, if it is not an integer, take the average of the two adjacent values; the position of Q3 is (n+1). ×0.75, the calculation rule is the same as Q1; the interquartile range (IQR) is obtained by subtracting Q1 from Q3. The lower limit of the reasonable amount range is Q1 - 1.5 × IQR, and the upper limit is Q3 + 1.5 × IQR. If the amount after format correction exceeds this reasonable range, it is judged as an outlier and replaced with the median of the effective transaction amount of the category in the past 90 days. The median is calculated as follows: the effective transaction amounts in the past 90 days are sorted in ascending order. If the total number of data is odd, the value of the middle position after sorting is taken; if it is even, the sum of the two middle values ​​is divided by 2. Format errors in buyer and seller identifiers are corrected according to the platform's unified coding rules.

[0017] When extracting time-dimensional features, the unit of measurement for transaction time is seconds. The transaction interval is calculated by subtracting the timestamp of the user's last transaction from the timestamp of the current transaction; the difference is the user's transaction interval feature. A 24-hour day is divided into 12 equal time periods (each period is 2 hours, i.e., 00:00 to 02:00, 02:00 to 04:00, ... 22:00 to 24:00). After determining the time period corresponding to the current transaction timestamp, each time period is encoded using integers from 0 to 11; the encoding result is the transaction time period feature. Using the current transaction time as a baseline, the user is traced back 30 days to count the total number of transaction records within that time period; this total number is the transaction frequency feature. When extracting amount-dimensional features, the amount data of all the user's historical transactions is first counted, and the average historical transaction amount is calculated by dividing the sum of all historical transaction amounts by... The total number of historical transactions is calculated by dividing the current single transaction amount by the historical average. The result is the relative proportion of the transaction amount. The user's 10 most recent transactions are extracted. The average of these 10 transactions is calculated, and then the difference between each transaction amount and the average is calculated. Each difference is squared and summed. This sum is divided by 10 to obtain the variance. The square root of the variance is used to obtain the standard deviation of the last 10 transactions. Finally, this standard deviation is divided by the average of the last 10 transactions. The resulting ratio is the transaction amount volatility characteristic. The reasonable amount range is determined according to the calculation rules in the aforementioned format correction step (Q1 to 1.5IQR to Q3+1.5IQR of the same category of transactions in the last 90 days). The current transaction amount after format correction is determined to be within this range. If it is within the range, it is recorded as a normal characteristic value; otherwise, it is recorded as an abnormal characteristic value, forming the amount range attribution characteristic.

[0018] When extracting features from the behavioral sequence dimension, the time unit for each step in the user's operational behavior sequence is unified to seconds. The time difference between each consecutive operational step is calculated, and the sum of the time differences of all consecutive steps is the total time to complete the operational steps feature. First, the key verification steps in the transaction process are identified, such as mobile phone number verification, payment password input, and identity information verification. The completion status of each of these key steps is checked one by one. Each incomplete key step is recorded as 1, and a completed one as 0. The records of all key steps are accumulated, and the sum is the number of missing key operations. The number of operational steps actually completed by the user is counted and divided by the number of preset standard operational steps in this transaction scenario, such as the 5-step standard process of login, browsing, adding to cart, placing an order, and payment. The resulting ratio is the operational sequence completeness feature. When extracting network-level features, the platform's transaction record database is queried to count the total number of all historical transaction records between the buyer and the current seller before the current transaction. This total number represents the transaction frequency feature between the buyer and seller. Based on the current transaction time, the data is traced back 60 days to count the total number of different sellers the buyer transacted with during that period. The total number of transactions by the buyer in the past 60 days is also counted. The ratio obtained by dividing the total number of different sellers by the total number of transactions is the buyer's cross-seller transaction ratio feature. The total transaction amount of all transactions by the seller in the past 30 days is also counted. The ratio obtained by dividing the current buyer's single transaction amount with the seller's total amount is the transaction amount ratio feature. Through the above extraction process, each cleaned transaction data is converted into a multi-dimensional feature vector containing four dimensions: time, amount, behavioral sequence, and relationship network.

[0019] Step 102 involves standardizing the multi-dimensional feature vectors to obtain a cleaned feature dataset. Specifically, this includes: using Z-score standardization to uniformly process all multi-dimensional feature vectors, eliminating dimensional differences between different features; for each feature dimension, such as transaction time interval, relative amount ratio, or operation sequence completeness, the standardized feature values ​​are calculated as follows: First, calculate the mean of all feature values ​​in that dimension by dividing the sum of all feature values ​​in that dimension by the total number of feature values ​​in that dimension; Second, calculate the standard deviation of all feature values ​​in that dimension by first calculating the difference between each feature value and the mean obtained in the first step, squaring each difference, summing the sum, dividing the sum by the total number of feature values ​​in that dimension, and then taking the square root of the result; Third, subtract the mean obtained in the first step from each original feature value in that dimension, and then divide the difference by the standard deviation obtained in the second step. The final result is the standardized value of that feature value. After all three steps are completed for the feature vectors of all dimensions, a cleaned feature dataset with a unified structure and consistent dimensions is formed.

[0020] This embodiment ensures the integrity and basic accuracy of the original data by collecting data from multiple sources and cleaning the basic data to remove redundancy, fill in missing information, and correct errors. Multi-dimensional feature extraction covers key information throughout the entire transaction chain, such as transaction time, amount, operation behavior, and the relationship between buyers and sellers, comprehensively capturing the core differences in user transaction behavior. Standardization processing eliminates the influence of different dimensions of features, avoids the situation where a single high-level feature dominates the subsequent detection results, and ensures that each dimension of feature has equal weight.

[0021] In a preferred embodiment of the present invention, step 2 includes: Step 200 involves constructing a data feature space based on the feature dataset to be cleaned, and mapping the feature vectors to their corresponding coordinate positions in the data feature space. Specifically, this includes: first, defining the feature dimensions of the feature dataset to be cleaned, with each feature dimension corresponding to a coordinate axis in the data feature space. For example, transaction time intervals in the time dimension and relative proportions of amounts in the amount dimension are each used as independent coordinate axes. Then, determining the value range of each coordinate axis: based on all standardized feature values ​​under that feature dimension, the minimum value of the feature value in that dimension is taken as the lower limit of the coordinate axis, and the maximum value as the upper limit, ensuring that the corresponding dimension values ​​of all feature vectors fall within the coordinate axis range. Next, feature vector mapping is performed. For each multi-dimensional feature vector in the feature dataset to be cleaned, each feature value it contains is mapped to the corresponding coordinate axis in the data feature space. The value of each feature value is the coordinate point on that coordinate axis. The coordinate points of all dimensions together determine the unique coordinate position of the feature vector in the data feature space. After completing the mapping of all feature vectors, a visualized multi-dimensional data distribution space is formed.

[0022] Step 201: Construct an anomaly detection model using the Isolation Forest algorithm. The construction process of the anomaly detection model is as follows: Generate a preset number of isolated trees in the data feature space. Each isolated tree recursively and randomly partitions feature vectors to independent nodes. Calculate the average path length of each feature vector in all isolated trees. Generate an anomaly score based on the ratio of the average path length to a preset path length benchmark. Specifically, this includes: First, setting the number of isolated trees to be generated, with a default number of 100. If the dataset to be cleaned is large, the number can be increased proportionally to 200 trees; if the dataset is small, the number can be adjusted to 50 trees. This setting ensures the stability and accuracy of the model detection. Then, each isolated tree is generated one by one according to the following process: Randomly extracting a portion of feature vectors from the data feature space as the current... The training samples for the former isolated tree are set to 50% of the total number of samples in the feature dataset to be cleaned, ensuring that the samples are representative and do not repeatedly cover all data. A dimension is randomly selected from all feature dimensions, and a split point between the minimum and maximum values ​​of that dimension is randomly determined within the coordinate range of that dimension. Feature vectors in the current training sample set whose feature values ​​are less than the split point are assigned to the left child node, and feature vectors whose feature values ​​are greater than or equal to the split point are assigned to the right child node. The above operation is repeated for each child node, i.e., the feature dimension and split point are randomly selected again for splitting, until a child node contains only one feature vector, or the number of splits for that child node reaches the preset maximum depth. At this point, the recursive splitting of that child node stops, and that child node becomes a leaf node of the isolated tree.

[0023] For each feature vector, starting from the root node of each isolated tree, trace the branch path containing the vector and record the number of branches traversed from the root node to the leaf node containing the vector. This number is the path length of the feature vector in that isolated tree. Sum the path lengths of the same feature vector in all isolated trees, and then divide the sum by the total number of isolated trees to obtain the average path length of the feature vector. Determine the number of training samples in the current isolated tree, denoted as the training sample number. Calculate the harmonic number corresponding to this number of samples. The harmonic number is calculated by starting from 1 and successively adding 1 and dividing by 1, 1 and dividing by 2, etc., until the accumulated natural number equals the training sample number minus one. The sum of all accumulated results is the harmonic number. The harmonic number corresponding to the number of samples is calculated. Then, a fixed constant of 0.5772 is added to this harmonic number to obtain the preset path length benchmark. The fixed constant 0.5772 is the Euler-Marschroni constant, used to correct the bias of the harmonic number when the number of samples is limited. The average path length of each feature vector is divided by the preset path length benchmark to obtain a ratio. An anomaly score is generated based on this ratio. A smaller ratio indicates that the feature vector is isolated faster and the degree of anomaly is higher; a larger ratio indicates that the feature vector is closer to the distribution characteristics of normal data and the degree of anomaly is lower. Through the above process, the anomaly detection model based on the isolated forest algorithm is constructed, realizing a quantitative evaluation of the anomaly degree of each feature vector.

[0024] This embodiment employs an unsupervised isolated forest algorithm to construct the model, eliminating the need for manually labeled anomaly samples, thus reducing data preparation requirements. It is well-suited for transaction data scenarios lacking prior anomaly information, improving the method's practicality. The construction of the data feature space spatially maps multi-dimensional feature vectors, making data distribution more intuitive and facilitating rapid isolation of anomaly data. The integrated calculation of multiple isolated trees and the statistical method of average path length reduce the randomness brought about by a single partitioning rule, making the calculation of anomaly scores more objective and stable, and improving the reliability of anomaly detection. The recursive random partitioning calculation logic is simple and efficient, requiring no complex matrix operations, and can quickly respond to anomaly detection requests from massive transaction data streams. Generating anomaly scores based on path length ratios quantifies the degree of anomaly in feature vectors, helping to reduce false positives for normal transactions and false negatives for anomaly transactions.

[0025] In a preferred embodiment of the present invention, step 3, statistically analyzing the real-time anomaly scores of all feature vectors to determine an initial judgment threshold, includes: sorting the anomaly scores according to their numerical values, and selecting the values ​​located at a predetermined quantile after sorting as the initial judgment threshold, specifically including: After obtaining the real-time anomaly scores for all feature vectors in the dataset to be cleaned using the isolated forest anomaly detection model, statistical analysis is performed as follows to determine the initial judgment threshold: First, the real-time anomaly scores of all feature vectors are collected to form a complete set of anomaly scores. Then, the anomaly scores in this set are sorted in ascending order, starting with the score with the lowest anomaly severity and proceeding sequentially to the score with the highest anomaly severity, ensuring that all scores are presented in an ordered sequence. Next, the predetermined quantile and corresponding quantile percentage are defined. Based on the business requirements of cleaning user transaction data, the predetermined quantile is set to the 95th quantile, corresponding to a quantile percentage of 95%. This ratio can cover the abnormal score distribution of most normal transaction data, and at the same time, it can initially screen out a small number of suspected abnormal data with high abnormal scores. The calculation process of the quantile is as follows: First, count the total number of abnormal scores in the set, and multiply the total number by 95% (i.e., 0.95) to obtain the corresponding position index. If the calculated position index is an integer, the abnormal score corresponding to the index is directly selected as the initial judgment threshold. If the position index is not an integer, the value is rounded up to obtain the nearest integer index, and the abnormal score corresponding to the integer index is selected as the initial judgment threshold. Through the above statistical analysis process of ordered sorting and 95th quantile screening, the final determined value is the initial judgment threshold.

[0026] In a preferred embodiment of the present invention, step 4 includes: Step 400: Analyze the frequency of selection and information gain of each feature vector dimension in the anomaly detection model during isolated tree node splitting. Calculate the anomaly discrimination contribution of each feature vector dimension. Select the two feature vector dimensions with the highest anomaly discrimination contribution, denoted as the first feature dimension and the second feature dimension, respectively. Specifically, this includes: calculating the selection frequency of feature dimensions, i.e., traversing all isolated tree node splitting records in the anomaly detection model, counting the total number of times each feature vector dimension is randomly selected in all splitting processes, and dividing this total number by the total number of node splitting processes in all isolated trees to obtain the selection frequency of each feature dimension. A higher frequency indicates that the dimension is more active during splitting. The more attention is paid to it; for each feature dimension, each time it is selected for node splitting, the information gain is calculated according to the following steps: First, the node entropy value before splitting is calculated. The entropy value is used to measure the disorder of the sample distribution within the node. The calculation formula is: Entropy value = -(sample percentage 1 × log2(sample percentage 1) + sample percentage 2 × log2(sample percentage 2) + ... + sample percentage k × log2(sample percentage k), where k is the number of clusters of samples within the node according to the current splitting dimension, and the sample percentage is the number of samples contained in each cluster divided by the total number of samples in the node; if a certain sample percentage is 0, the calculation result of the corresponding term is 0 to avoid the meaninglessness of logarithmic operations; Calculate the weighted entropy value of the child nodes after the split: First, calculate the entropy values ​​of the left and right child nodes separately (using the same calculation method as before the split); then, divide the number of samples of the left child node by the total number of samples of that node before the split to obtain the weight of the left child node, and divide the number of samples of the right child node by the total number of samples of that node before the split to obtain the weight of the right child node; multiply the entropy value of the left child node by the weight of the left child node, and multiply the entropy value of the right child node by the weight of the right child node, and add the two to obtain the weighted entropy value after the split; calculate the information gain for a single split: subtract the weighted entropy value after the split from the node entropy value before the split, and the result is the information gain for that split; calculate the information gain for all splits in each feature dimension. The information gain is summed and then divided by the number of splits for that dimension to obtain the average information gain for that dimension. A higher information gain indicates a more significant improvement in the classification purity of the samples after splitting in that dimension. The anomaly discrimination contribution is calculated by using a weighted summation method to calculate the contribution of each feature dimension, with the weights allocated as follows: selected frequency (50%) and average information gain (50%). Specifically, the anomaly discrimination contribution = (selected frequency × 0.5) + (average information gain × 0.5). All feature dimensions are sorted from highest to lowest according to their anomaly discrimination contribution, and the top two dimensions are selected and defined as the first and second feature dimensions, respectively. These two dimensions have the strongest ability to distinguish anomaly data.

[0027] Step 401: In the two-dimensional feature subspace spanned by the first and second feature dimensions, based on the projection distribution of all feature vectors in the feature dataset to be cleaned onto the two-dimensional feature subspace, an anchor point set is established. The anchor point set consists of four anchor points: the first anchor point is located at the intersection of the lower quartile of the first feature dimension and the lower quartile of the second feature dimension; the second anchor point is located at the intersection of the upper quartile of the first feature dimension and the lower quartile of the second feature dimension; the third anchor point is located at the intersection of the upper quartile of the first feature dimension and the upper quartile of the second feature dimension; and the fourth anchor point is located at the intersection of the lower quartile of the first feature dimension and the upper quartile of the second feature dimension. Specifically, this includes: Each multi-dimensional feature vector in the dataset to be cleaned is mapped to a two-dimensional feature subspace consisting of a first feature dimension and a second feature dimension. The projection method involves extracting the values ​​of the first and second feature dimensions corresponding to each feature vector to form two-dimensional coordinate points. All coordinate points together constitute the projection distribution in this subspace. For the first feature dimension values ​​of all projection points in the two-dimensional subspace, they are sorted in ascending order, and the total number of projection points is counted. The total number is multiplied by 0.25 to obtain the lower quartile position, and multiplied by 0.75 to obtain the upper quartile position. If the position is an integer, the value corresponding to that position is taken as the lower or upper quartile of the first feature dimension. If the position is not an integer, the average of the two values ​​before and after that position is taken as the corresponding quartile. The same calculation method is used to obtain the lower and upper quartiles of the second feature dimension.

[0028] The anchor set consists of four anchor points. The coordinates of each anchor point are determined by the intersection of the corresponding quartiles. The first anchor point has the x-coordinate of the lower quartile of the first feature dimension and the y-coordinate of the lower quartile of the second feature dimension. The second anchor point has the x-coordinate of the upper quartile of the first feature dimension and the y-coordinate of the lower quartile of the second feature dimension. The third anchor point has the x-coordinate of the upper quartile of the first feature dimension and the y-coordinate of the upper quartile of the second feature dimension. The fourth anchor point has the x-coordinate of the lower quartile of the first feature dimension and the y-coordinate of the upper quartile of the second feature dimension.

[0029] Step 402: Connect the four anchor points in spatial order to form a quadrilateral anomaly determination domain. Specifically, starting from the first anchor point, connect the second, third, and fourth anchor points in sequence, and finally return to the first anchor point to form a complete quadrilateral anomaly determination domain. This quadrilateral anomaly determination domain can cover the distribution range of most normal projection points in the two-dimensional feature subspace, and its outer region is the initially determined anomaly region.

[0030] This embodiment selects the two most critical core dimensions for anomaly detection by calculating the contribution of feature dimensions to anomaly differentiation, avoiding interference from irrelevant dimensions. An anchor set is established based on quartiles, fully aligning with the actual distribution characteristics of the data, making the boundary setting of the quadrilateral anomaly determination domain more reasonable and effectively defining the distribution range of normal data. The construction of the two-dimensional feature subspace and the formation of the quadrilateral determination domain transform anomaly determination of high-dimensional data into intuitive spatial range determination, reducing the complexity of anomaly identification and enhancing the interpretability of the determination logic. The quadrilateral anomaly determination domain provides a spatial constraint basis for subsequent dynamic threshold generation, enabling threshold adjustment to combine the spatial distribution characteristics of the data, further improving the accuracy of anomaly detection and reducing misjudgments of normal transactions and missed judgments of abnormal transactions.

[0031] In a preferred embodiment of the present invention, step 5 includes: Step 500: Based on the covariance matrix of all feature vectors in the feature dataset to be cleaned, perform eigenvalue decomposition to obtain covariance eigenvectors and eigenvalues. Specifically, this includes: first, collecting all feature vectors in the feature dataset to be cleaned, assuming the dataset contains M feature vectors, each feature vector contains N feature dimensions, such as time, amount, behavior sequence, etc.; for each feature dimension (from the 1st to the Nth), summing all M eigenvalues ​​under that dimension, and then dividing the sum by the total number of eigenvalues ​​M to obtain the mean of each feature dimension, finally forming a mean vector containing N means, which serves as the basis for subsequent calculations; constructing the covariance matrix, which is an N-row N-column square matrix, where the element in the i-th row and j-th column represents the covariance between the i-th feature dimension and the j-th feature dimension. Variance is used to measure the degree of linear correlation between two dimensions of data. The specific calculation steps are as follows: For the i-th and j-th feature dimensions, extract the eigenvalues ​​of each feature vector in these two dimensions in turn; subtract the mean of the i-th dimension from the eigenvalue of each feature vector in the i-th dimension to obtain the deviation value of the i-th dimension; similarly calculate the deviation value of the j-th dimension; multiply the two deviation values ​​corresponding to each feature vector to obtain the deviation product of the vector in these two dimensions; sum all the deviation products of the M feature vectors, and then divide the sum by M (the total number of feature vectors) to obtain the covariance between the i-th and j-th dimensions; calculate all combinations of i and j in turn (i from 1 to N, j from 1 to N) using the above method, and fill the results into the corresponding matrix positions to form a complete N×N covariance matrix.

[0032] The covariance matrix is ​​an N×N real symmetric matrix, where the element in the i-th row and j-th column is exactly the same as the element in the j-th row and i-th column. Therefore, it naturally possesses the properties of being diagonalizable, having non-negative real eigenvalues, and having orthogonal eigenvectors corresponding to different eigenvalues. Based on these properties, eigenvalue decomposition is performed according to the following sequential steps: First, confirm that the covariance matrix is ​​a real symmetric matrix, denoted as C; simultaneously, prepare an identity matrix I with dimensions identical to C. The diagonal elements of the identity matrix are all 1s, and all off-diagonal elements are 0. 0. Let the eigenvalue to be found be λ. By subtracting the product of the covariance matrix C and λ and the identity matrix I, we obtain a new matrix C-λI. Then, we calculate the determinant of the new matrix C-λI. The determinant is a numerical value that can characterize the core properties of the matrix. For an N×N matrix, it can be calculated by expanding by row or column. Finally, we will get a polynomial in λ. Set the result of the determinant to 0, forming the equation det(C-λI)=0. This equation is the characteristic equation, and its solution is the eigenvalue of the covariance matrix C.

[0033] The characteristic equation det(C-λI)=0 is a univariate polynomial equation of degree N in λ. Solving this equation yields N non-negative real roots, which are all the eigenvalues ​​of the covariance matrix C. If multiple identical real roots (i.e., repeated roots) appear during the solution process, for example, a certain real root... If it appears k times (k is an integer greater than 1), then Let be a k-fold eigenvalue, and each k-fold eigenvalue corresponds to k linearly independent eigenvectors. For each obtained eigenvalue (including the multiple eigenvalues), the corresponding eigenvectors are solved step by step according to the following process: first, the eigenvalue... Substituting into the homogeneous linear equation system (C- I) v = 0, where v is the N-dimensional eigenvector to be determined, and 0 is the N-dimensional zero vector with all elements equal to 0; then, the system of equations is simplified using Gaussian elimination, and the coefficient matrix (i.e., C-) of the system of equations is reduced through elementary row operations. I) Transforming the matrix into its row minimum form involves several transformations: swapping any two rows, multiplying a row by a non-zero constant, and adding multiples of one row to another. These transformations convert the coefficient matrix into its row minimum form, where the first non-zero element (pivot) of each row is 1, and all other elements in the column containing the pivot are 0. The pivots are shifted right from top to bottom. After simplification, the number of rows with all zeros in the row minimum form determines the number of free variables in the system of equations (free variables are variables corresponding to columns without pivots; they can take values ​​independently without affecting the compatibility of the system of equations). Since the zero vector cannot be used as an eigenvector, each free variable must be assigned a non-zero arbitrary value (e.g., 1 for a single free variable, or uncorrelated non-zero values ​​such as 1, 2, etc. for multiple free variables, ensuring the solution vector is non-zero and linearly independent). Substitute the assigned free variable values ​​into the simplified row econometric system of equations, and calculate the values ​​of the remaining principal variables (variables corresponding to columns with principal elements). The value of each principal variable can be directly solved from the row equations. Combine the calculated principal variable values ​​with the assigned free variable values ​​in the original variable order to form a complete N-dimensional non-zero vector. The vector That is, eigenvalues The corresponding covariance eigenvector; if the current eigenvalue If the eigenvalue is k-fold (i.e., an eigenvalue that appears k times), then the above operation is repeated k times. In each operation, the free variables are assigned different combinations of non-zero values ​​(ensuring that the solution vectors obtained each time are linearly independent). Finally, k linearly independent eigenvectors are obtained, and all of these vectors are the k-fold eigenvalues. The corresponding covariance eigenvector.

[0034] To ensure that all eigenvectors are orthogonal, the obtained eigenvectors are processed according to the following rules: If the eigenvectors correspond to different eigenvalues, based on the inherent properties of real symmetric matrices, these eigenvectors are already orthogonal and require no additional processing; if the eigenvectors come from the same k-fold eigenvalue (i.e., k linearly independent eigenvectors obtained from the same eigenvalue), the Schmitt orthogonalization method is used. Specifically, the first vector from these k vectors is selected as the reference vector, denoted as u1; the second vector v2 is taken, and the projection component of v2 onto the reference vector u1 is calculated by dividing the dot product of v2 and u1 by the dot product of u1 and u1, and then multiplying the result by u1. 1. Subtract the projection component from v2 to obtain a new vector u2, which is now orthogonal to u1. Take the third vector v3 and calculate its projection components onto u1 and u2 (using the same method as before). Subtract these two projection components from v3 to obtain a new vector u3, which is orthogonal to both u1 and u2. Repeat this process until the orthogonality of k vectors is achieved, resulting in k mutually orthogonal vectors. Through these complete and coherent steps, two core results are obtained: one set consists of N mutually orthogonal covariance eigenvectors, each N-dimensional and corresponding to a single eigenvalue; the other set consists of the corresponding N non-negative real eigenvalues. The larger the eigenvalue, the higher the dispersion of the data in the direction indicated by the corresponding eigenvector, and the better that direction reflects the main distribution characteristics of the data.

[0035] Step 501: Construct a high-dimensional confidence elliptic based on the covariance eigenvector and eigenvalues. The covariance eigenvector determines the principal axis direction of the high-dimensional confidence elliptic, and the product of the square root of the eigenvalue and the preset confidence coefficient determines the semi-axis length of each principal axis direction. Specifically, considering the data reliability requirements of the business scenario, the preset confidence coefficient is a core parameter for defining the coverage range of normal data. Its value is determined based on the standard normal distribution quantile table. The specific logic and process are as follows: First, clarify the definition of the standard normal distribution: it is a symmetrical probability distribution with a mean of 0 and a standard deviation of 1. Its probability density curve is bell-shaped, perfectly symmetrical on both sides, and the total area under the curve represents the probability of all data (i.e., 1). Then, explain the role of the quantile table. This tool is based on the standard normal distribution. Its core function is to establish the correspondence between cumulative probability and quantile values ​​(also known as critical values). Cumulative probability refers to the area under the standard normal distribution curve from negative infinity to a certain quantile value, directly reflecting the cumulative proportion of data to the left of that quantile value. Next, it clarifies the conversion logic between confidence level and cumulative probability. The confidence level represents the proportion of normal data to be covered. Since the data distribution is bilaterally symmetrical (normal data is concentrated in the middle area, and abnormal data is distributed on both sides), the confidence level must first be converted into the cumulative probability corresponding to the quantile table. The conversion formula is: Cumulative probability = (1 + confidence level) / 2. Finally, it completes the coefficient lookup. If the business requires a 95% confidence level (i.e., covering 95%), then... (For normal data), first calculate the cumulative probability using the formula: Cumulative Probability = (1 + Confidence Level) / 2. Substituting the values, we get the cumulative probability = (1 + 0.95) / 2 = 0.975. Next, use the standard normal distribution quantile table to find the corresponding quantile value. The first step is to understand the core structure of the quantile table. The left column of the table indicates the integer part and one decimal place of the quantile value, such as 1.9, 2.0, 2.5, etc., and the top row indicates the two decimal places of the quantile value, such as 0.00, 0.01, 0.02...0.09, etc. The value of each cell in the table corresponds exactly to the cumulative probability of the corresponding quantile value. The second step is to locate the cell with a value equal to 0.975 in the main area of ​​the table (this cell directly corresponds to the calculated cumulative probability of 0). (975); The third step is to track the position of the cell, find its corresponding left column value 1.9 and top row value 0.06, and concatenate these two values ​​to obtain the quantile value 1.96; If the business requires a 99% confidence level, follow the same logic step by step, first calculate the cumulative probability = (1 + 0.99) / 2 = 0.995; then refer to the structure of the quantile table, find the cell corresponding to the cumulative probability 0.995 in the table, and finally track the left column value 2.5 and top row value 0.08 corresponding to the cell, and concatenate them to obtain the quantile value 2.58. The preset confidence coefficient determined by this process can ensure that the elliptic accurately covers the vast majority of normal data at the corresponding confidence level, avoiding normal data being misjudged as abnormal.

[0036] The N covariance eigenvectors obtained from the eigenvalue decomposition in step 500 are directly used as the N principal axes of the high-dimensional confidence elliptic. Since all covariance eigenvectors are orthogonal to each other, the N principal axes of the elliptic are also perpendicular to each other, and the principal axis directions are precisely aligned with the main discrete directions of the data distribution. This ensures that the elliptic closely matches the actual distribution trend of the data and does not deviate from the data distribution. For each principal axis direction, the semi-axis length is calculated according to the following steps to ensure that the result reflects the discrete characteristics of the data in that direction. The first step is to extract the corresponding eigenvalue: extract the eigenvalue corresponding to the current principal axis. The magnitude of this value directly reflects the data... The first step is to determine the degree of dispersion along the principal axis. The second step is to calculate the standard deviation by taking the square root of the extracted eigenvalues. This result represents the standard deviation of the data along the principal axis, further quantifying the level of data dispersion. The third step is to calculate the semi-axis length by multiplying the standard deviation by a pre-set confidence coefficient. For example, if the square root (standard deviation) of the eigenvalue corresponding to a principal axis is 2 and the pre-set confidence coefficient is 1.96, then the semi-axis length of that principal axis = 2 × 1.96. The larger the eigenvalue, the higher the degree of data dispersion, the larger the corresponding standard deviation, and the longer the semi-axis length. The ellipse covers more of the data along this direction. The wider the range of normal data, the better. Using the mean vector calculated in step 500 as the center of the ellipse (i.e., the intersection of all principal axes, and also the center of the data distribution), and combining the directions of the N principal axes and their corresponding semi-axis lengths, a closed high-dimensional confidence ellipse is constructed in the N-dimensional data space. This ellipse is a symmetrical high-dimensional geometric figure, and its internal region clearly defines the distribution range of normal feature vectors corresponding to a preset confidence level (95% or 99%). When the preset confidence level is 95%, the internal region exactly covers 95% of the normal data, and all feature vectors located inside do not exceed a confidence coefficient of 1.96. The corresponding normal distribution boundary; when the preset confidence level is 99%, the internal region exactly covers 99% of the normal data, and all feature vectors located inside do not exceed the normal distribution boundary corresponding to the confidence coefficient of 2.58. The external region clearly defines the distribution range of abnormal feature vectors that deviate from the normal distribution. All feature vectors in this region exceed the normal data distribution boundary under the corresponding preset confidence level (95% or 99%) and confidence coefficient (1.96 or 2.58), and belong to the abnormal candidate objects that need to be investigated in detail. The boundary between the internal and external regions is based on the distribution boundary quantified by the corresponding confidence coefficient, which is clear and can be accurately determined.

[0037] Step 502: Project the high-dimensional confidence elliptic onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse. Specifically, the core purpose of the projection is to transform the high-dimensional elliptic into a two-dimensional planar graphic composed of the first and second feature dimensions, so as to perform geometric operations with the quadrilateral anomaly detection domain constructed in step 402. The first and second feature dimensions are the core dimensions with the highest anomaly discrimination contribution selected in step 400, and their projection results can accurately reflect the distribution pattern of normal data on key features. Check the N principal axes of the high-dimensional elliptic; if there exists a relationship with the first feature dimension, the second... For principal axes with completely consistent feature dimensions, directly extract the corresponding semi-axis lengths. If the principal axis direction does not directly coincide with the two core dimensions, the transformation is completed through vector projection operations. The specific steps are as follows: Construct unit vectors for the first feature dimension (vectors of length 1 with the same direction as that dimension) and the second feature dimension respectively; Perform a dot product between each high-dimensional principal axis vector and the first feature dimension unit vector to obtain the projection component of that principal axis on the first feature dimension; Calculate the projection component of each high-dimensional principal axis vector on the second feature dimension using the same method; Determine the weighted summation weights, with the weights set to... The normalized semi-axis lengths of each high-dimensional principal axis are calculated as follows: first, the sum of the semi-axis lengths of all high-dimensional principal axes is calculated; then, the semi-axis length of each principal axis is divided by this sum to obtain the weight of the corresponding principal axis (the weight ranges from 0 to 1, and the sum of all principal axis weights is 1), ensuring that the weight allocation matches the degree of influence of the principal axis on the data distribution; a weighted sum is performed on the projection components of the first feature dimension of all principal axes (the weight is the normalized semi-axis length of the corresponding principal axis), and a weighted sum is also performed on the projection components of the second feature dimension of all principal axes. These two sums together constitute the equivalent principal axis directions corresponding to the first and second feature dimensions; furthermore, each The equivalent semi-axis length on the two-dimensional plane is obtained by multiplying the semi-axis length of the principal axis by the absolute value of the corresponding projection component and summing the results, ensuring that the projection parameters can truly reflect the characteristics of the high-dimensional data. The center point of the ellipse is determined on the two-dimensional plane with the mean of the first feature dimension as the center of the horizontal coordinate and the mean of the second feature dimension as the center of the vertical coordinate. The orientation of the major and minor axes of the ellipse is adjusted according to the direction of the equivalent principal axis, and the specific size of the ellipse is determined by combining the equivalent semi-axis length. Finally, a closed two-dimensional confidence ellipse is drawn. This ellipse is a precise projection of the high-dimensional confidence elliptic onto the core two-dimensional features, truly reflecting the distribution range of normal data in the key dimensions.

[0038] Step 503: Calculate the area of ​​the overlapping region between the quadrilateral anomaly detection domain and the two-dimensional confidence ellipse on the two-dimensional plane, and calculate the ratio of the overlapping area to the total area of ​​the quadrilateral anomaly detection domain. Use this ratio as a threshold adjustment factor. Specifically, this includes: first, clarifying the core boundary parameters of the quadrilateral anomaly detection domain and the two-dimensional confidence ellipse to provide a basis for area calculation. The horizontal coordinate range of the quadrilateral anomaly detection domain boundary is from the lower quartile to the upper quartile of the first feature dimension, and the vertical coordinate range is from the lower quartile to the upper quartile of the second feature dimension. The coordinates of the four vertices have been defined in step 401; the center coordinate of the boundary parameter of the two-dimensional confidence ellipse is... The values ​​are the means of the two core dimensions, the lengths of the major and minor axes are the equivalent semi-axis lengths, and the direction of the principal axis is the equivalent principal axis direction. The total area of ​​the quadrilateral anomaly detection domain is calculated using the rectangle area formula, since a quadrilateral is a standard rectangle composed of quartiles. The first step is to calculate the horizontal side length: the upper quartile of the first feature dimension minus the lower quartile of the first feature dimension. The second step is to calculate the vertical side length: the upper quartile of the second feature dimension minus the lower quartile of the second feature dimension. The third step is to calculate the total area: horizontal side length × vertical side length. The result is the total area of ​​the quadrilateral anomaly detection domain.

[0039] To calculate the area of ​​the overlapping region, follow these steps for accurate calculation and to ensure the results are accurate: First, find the valid intersection points. For each of the four sides of the quadrilateral (top, bottom, left, and right), write the equation of the line for each side: the top side is the line whose ordinate equals the quartile of the second characteristic dimension; the bottom side is the line whose ordinate equals the lower quartile of the second characteristic dimension; the left side is the line whose x-coordinate equals the lower quartile of the first characteristic dimension; and the right side is the line whose x-coordinate equals the upper quartile of the first characteristic dimension. Next, solve the standard equation of the two-dimensional confidence ellipse. This equation, with the center of the ellipse as the origin and the equivalent principal axes as the coordinate axes, takes the form: Where x is the coordinate of any point on the ellipse in the first feature dimension, y is the coordinate of that point in the second feature dimension, a is the equivalent semi-major axis length of the two-dimensional confidence ellipse, and b is the equivalent semi-minor axis length of the two-dimensional confidence ellipse. The system of equations consisting of the equations of the line and the standard equation of the ellipse is solved using algebraic operations to obtain the coordinates of all solutions. Finally, solutions whose coordinate values ​​lie within the range of the current edge segment are selected; these are the valid intersection points of the ellipse and that edge. The intersection points of the four edges are solved sequentially using the above process. The second step is to segment the overlapping region. Based on the obtained valid intersection point positions, the overlapping region of the ellipse and quadrilateral is divided into multiple directly calculable geometric sub-shapes. The segmentation results are rectangular and elliptical sectors or polygonal and elliptical segments, ensuring that the area of ​​each sub-shape can be calculated using conventional methods. The geometric formulas are used for calculation. The third step is to calculate the area of ​​the sub-shapes, using corresponding formulas for different types of sub-shapes. The area of ​​a rectangle is length × width (length and width are determined by the difference in boundary coordinates of the sub-shape). The area of ​​an elliptical sector is (central angle ÷ 360°) × π × major semi-axis × minor semi-axis (the central angle is determined by the angle between the intersection point and the center of the ellipse, which can be calculated using the dot product formula). The area of ​​a polygon is calculated using the shoelace formula: list the coordinates of all vertices of the polygon, calculate the cross product of the coordinates of adjacent vertices in sequence, sum the results, and finally take the absolute value of the sum and divide it by 2 to obtain the polygon area. The fourth step is to sum the overlapping areas: add the areas of all sub-shapes together; the sum is the total area of ​​the overlapping region between the two-dimensional confidence ellipse and the quadrilateral anomaly detection domain.

[0040] The threshold adjustment factor is calculated by dividing the total area of ​​the overlapping region by the total area of ​​the quadrilateral anomaly detection domain, resulting in a ratio between 0 and 1. This ratio is the threshold adjustment factor. The closer the ratio is to 1, the more normal data is contained in the quadrilateral detection domain, and the higher the matching degree between the detection domain and the normal data distribution. The closer the ratio is to 0, the fewer normal data is contained in the quadrilateral detection domain, and the greater the deviation from the normal data distribution.

[0041] Step 504: The threshold adjustment factor and the initial judgment threshold are weighted and calculated using preset weight coefficients to generate a dynamic judgment threshold. Specifically, considering the dual requirements of threshold stability and dynamic adaptability in the business scenario, the weight allocation rule is as follows: the weight coefficient of the initial judgment threshold is set to 0.6, the weight coefficient of the threshold adjustment factor is set to 0.4, and the sum of the two coefficients is 1. This allocation method retains the reliability of the initial judgment threshold based on a large amount of historical data statistics, while also adapting to changes in real-time data distribution through the threshold adjustment factor, avoiding the limitations of a single threshold. The dynamic threshold is calculated step by step according to the following logic: First, calculate the weighted value of the initial threshold. Take the initial judgment threshold determined in step 3 and multiply it by the corresponding weight coefficient 0.6 to obtain... The first step is to calculate the initial threshold weighted value, which forms the basis of the dynamic threshold and ensures the stability of the judgment standard. The second step is to calculate the adjustment factor weighted value. The threshold adjustment factor obtained in step 503 is taken out and multiplied by the corresponding weight coefficient of 0.4 to obtain the adjustment factor weighted value. This value is the adaptation part of the dynamic threshold, used to fit the real-time data distribution characteristics. The third step is to sum and generate the dynamic threshold. The initial threshold weighted value and the adjustment factor weighted value are added together. The result is the final dynamic judgment threshold. For example, assuming that the initial judgment threshold determined in step 3 is 0.8 and the threshold adjustment factor obtained in step 503 is 0.9, then the dynamic judgment threshold = 0.8 × 0.6 + 0.9 × 0.4. This threshold will be adjusted in real time according to the changes in data distribution, taking into account both stability and adaptability.

[0042] This embodiment constructs a high-dimensional confidence elliptic by combining the statistical properties of the covariance matrix, making full use of the overall distribution information of the data and making the definition of the normal data range more statistically reasonable. By projecting the high-dimensional elliptic onto the core two-dimensional feature plane, the distribution characteristics of the high-dimensional data are transformed into an intuitive two-dimensional geometric figure, reducing the computational complexity of overlapping with the quadrilateral anomaly judgment domain, while focusing on the core feature dimension and improving computational efficiency. The threshold adjustment factor is generated based on the geometric overlap ratio, which can quantitatively reflect the degree of matching between the normal data distribution and the judgment domain, making the adjustment of the initial threshold more objective and avoiding the bias caused by subjective settings. The dynamic judgment threshold integrates the initial statistical standard and the real-time data distribution characteristics, effectively solving the problem that the static threshold cannot adapt to the dynamic changes in data distribution, and improving the flexibility and adaptability of the anomaly judgment standard. The whole process combines statistical calculation and geometric analysis, making the threshold adjustment both statistically rigorous and spatially intuitive, further improving the accuracy of anomaly detection and reducing the situation of misjudging normal transactions and missing anomaly transactions.

[0043] In a preferred embodiment of the present invention, step 6 includes: Step 600 involves comparing the real-time anomaly score with the dynamic judgment threshold in real time. User transaction behavior data with a real-time anomaly score greater than the dynamic judgment threshold is identified as non-genuine transaction data. Specifically, this includes: acquiring user transaction behavior data in real time; calculating the real-time anomaly score for each transaction data using the previously constructed isolated forest anomaly detection model (the calculation logic is consistent with the anomaly score generation in step 201, derived based on information such as the path length of the feature vector in the isolated tree); acquiring the dynamic judgment threshold generated in step 504, which integrates the initial statistical standard and real-time data distribution characteristics and has dynamic adaptability; comparing the real-time anomaly score of each transaction data with the dynamic judgment threshold one by one. If the real-time anomaly score of a certain transaction data is greater than the dynamic judgment threshold, it indicates that the degree of deviation of the transaction behavior from the normal distribution exceeds the acceptable range, and it is identified as non-genuine transaction data; if the real-time anomaly score is less than or equal to the dynamic judgment threshold, its normal transaction data identification is retained.

[0044] Step 601 involves removing records marked as non-genuine transaction data from the user transaction behavior data stream to obtain a clean user transaction behavior data stream. Specifically, this includes: traversing the user transaction behavior data stream, filtering out all records marked as non-genuine transaction data based on the marking results of step 600 to form a subset of non-genuine transaction data; completely removing the subset of non-genuine transaction data from the original user transaction behavior data stream, and re-integrating the remaining unmarked normal transaction data according to the chronological or logical order of the original data stream to form a clean user transaction behavior data stream with a complete structure and no abnormal interference.

[0045] Step 602: Calculate the difference between the feature distribution of the clean user transaction behavior data stream and the historical feature distribution. When the difference exceeds a preset change threshold, trigger the anomaly detection model update process. The update process includes recalculating the anomaly detection contribution of each feature dimension using newly collected user transaction behavior data, reselecting the first and second feature dimensions, and reconstructing the quadrilateral anomaly judgment domain, high-dimensional confidence elliptic, and dynamic judgment threshold. Specifically, this includes using the feature distribution of normal user transaction behavior data accumulated in historical periods as a benchmark (the historical feature distribution needs to be stored in advance, containing each feature dimension). The calculation uses core statistics such as mean and variance of the degree of difference. Dimensions consistent with historical feature distributions are selected for calculation, including the mean and variance of all feature dimensions to ensure consistency and comprehensiveness in the difference calculation. The difference is calculated step-by-step: for each feature dimension, first, the absolute difference between the mean of that dimension in the clean data stream and the mean of that dimension in the historical feature distribution is calculated. This absolute difference is then divided by the historical mean to obtain the proportion of mean difference for that dimension. Next, the absolute difference between the variance of that dimension in the clean data stream and the variance of that dimension in the historical feature distribution is calculated. This absolute difference is then divided by the historical variance to obtain the proportion of variance difference for that dimension. The proportion of mean difference is then calculated. The variance percentage is summed with the variance percentage and then divided by 2 to obtain the variance of a single feature dimension. A weighted summation method is used to calculate the overall variance of all feature dimensions, with the weight being the anomaly discrimination contribution of each feature dimension (consistent with the contribution calculated in step 400, ensuring the variance calculation focuses on core features). Specifically, the variance of each feature dimension is multiplied by its corresponding anomaly discrimination contribution, and then all products are summed to obtain the overall variance between the clean data stream feature distribution and the historical feature distribution. The preset change threshold is a critical value determined based on the historical variance statistics of the past 6 to 12 months, specifically set to [value missing]. The 95th percentile of the historical overall difference is set to 0.3 (this value can be fine-tuned according to the sensitivity of the business scenario; it can be reduced to 0.25 in sensitive scenarios and increased to 0.35 in lenient scenarios). This is used to clearly define the normal fluctuation range of the feature distribution. The calculated overall difference is compared with the preset change threshold (0.3). If the overall difference is less than or equal to 0.3, it means that the current data distribution has not changed significantly and is within the normal fluctuation range, and the anomaly detection model remains unchanged. If the overall difference is greater than 0.3, it means that the data distribution has deviated from the historical normal, the feature drift phenomenon is obvious, and the model update process is triggered.

[0046] The model update process involves recalculating the contribution of each feature dimension, collecting user transaction behavior data within the new cycle (one month), and, following the calculation logic in step 400, recalculating the selection frequency, information gain, and average information gain of each feature dimension during the isolated tree node splitting process. This results in a recalculation of the anomaly detection contribution of each feature dimension. The core feature dimensions are then re-selected, sorted from highest to lowest based on the updated anomaly detection contribution, and the top two dimensions are chosen as the new first and second feature dimensions to ensure that the core features always align with the anomaly detection requirements of the current data. The relevant decision-making components are then reconstructed, following steps 401 to 402, by recalculating quartiles, establishing anchor point sets, and constructing a quadrilateral anomaly decision domain based on the new core feature dimensions. Finally, following steps 500 to 504, the covariance matrix is ​​recalculated based on the user transaction data of the new cycle, eigenvalue decomposition is performed, and a high-dimensional confidence elliptic and two-dimensional projection are constructed. This ultimately generates a dynamic decision threshold adapted to the current data distribution, completing the full-process update of the anomaly detection model.

[0047] This embodiment utilizes a real-time comparison and identification mechanism to rapidly filter out non-genuine transaction data, ensuring the cleanliness of user transaction behavior data streams in real time and providing high-quality data support for subsequent business decisions and data analysis. The generation of clean data streams effectively eliminates interference from abnormal data, preventing non-genuine transaction data from misleading business analysis results and improving the reliability of data applications. The calculation of feature distribution difference combines the local differences of individual features with the weighting of core features, comprehensively reflecting the changing trends of data distribution. The model update mechanism can adapt to significant changes in data distribution in a timely manner, avoiding misjudgments and omissions that occur in static models after data feature drift, ensuring the long-term effectiveness and accuracy of the anomaly detection model.

[0048] like Figure 2 As shown, embodiments of the present invention also provide a big data intelligent cleaning system based on machine learning, comprising: The data acquisition module is used to collect and preprocess user transaction behavior data streams and perform feature engineering to extract multi-dimensional feature vectors to obtain a feature dataset to be cleaned. The computation module is used to construct a data feature space for the feature dataset to be cleaned, and to map multi-dimensional feature vectors to the data feature space; the multi-dimensional feature vectors are input into the anomaly detection model built based on the isolated forest algorithm to calculate the real-time anomaly score corresponding to each feature vector; The determination module is used to perform statistical analysis on the real-time anomaly scores of all feature vectors and determine the initial judgment threshold. A module is established to determine the first and second feature dimensions in the data feature space to establish an anchor set, and to connect the anchors in the anchor set to form a quadrilateral anomaly determination domain. The weighting module is used to construct a high-dimensional confidence ellipse based on the covariance matrix and mean vector of the feature vectors, project the high-dimensional confidence ellipse onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse; calculate the area ratio of the overlapping region between the quadrilateral anomaly judgment domain and the two-dimensional confidence ellipse to obtain the threshold adjustment factor; and weight the threshold adjustment factor with the initial judgment threshold to generate a dynamic judgment threshold. The update module compares the real-time anomaly score with the dynamic judgment threshold to identify and clean up non-genuine transaction data, resulting in a clean data stream. It continuously monitors the changes in the feature distribution of the clean data stream and triggers the update process of the anomaly detection model and the dynamic judgment threshold when the distribution change exceeds a predetermined threshold.

[0049] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0050] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0051] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0052] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A big data intelligent cleaning method based on machine learning, characterized in that, The method includes: Step 1: Collect and preprocess the user transaction behavior data stream and perform feature engineering to extract multi-dimensional feature vectors to obtain the feature dataset to be cleaned; Step 2: Based on the feature dataset to be cleaned, construct a data feature space and map multi-dimensional feature vectors into the data feature space; input the multi-dimensional feature vectors into the anomaly detection model built based on the isolated forest algorithm, and calculate the real-time anomaly score corresponding to each feature vector; Step 3: Perform statistical analysis on the real-time anomaly scores of all feature vectors to determine the initial judgment threshold; Step 4: In the data feature space, determine the first and second feature dimensions to establish an anchor set, and connect the anchors in the anchor set to form a quadrilateral anomaly determination domain. Step 5: Construct a high-dimensional confidence elliptic based on the covariance matrix and mean vector of the feature vectors. Project the high-dimensional confidence elliptic onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse. Calculate the area ratio of the overlapping region between the quadrilateral anomaly judgment domain and the two-dimensional confidence ellipse to obtain the threshold adjustment factor. Weight the threshold adjustment factor with the initial judgment threshold to generate a dynamic judgment threshold. Step 6: Compare the real-time anomaly score with the dynamic judgment threshold to identify and clean up non-genuine transaction data to obtain a clean data stream; continuously monitor the feature distribution changes of the clean data stream, and trigger the update process of the anomaly detection model and the dynamic judgment threshold when the distribution change exceeds the predetermined threshold.

2. The big data intelligent cleaning method based on machine learning according to claim 1, characterized in that, Step 1 includes: Collect user transaction behavior data stream, which includes transaction timestamp, transaction amount, buyer identifier, seller identifier, logistics status information, and user operation behavior sequence; Data cleaning is performed on the user transaction behavior data stream, including removing duplicate records, filling in missing values, and correcting format errors. Based on the cleaned user transaction behavior data stream, time dimension features, amount dimension features, behavior sequence dimension features, and relationship network dimension features are extracted to form a multi-dimensional feature vector. The multi-dimensional feature vectors are standardized to obtain the feature dataset to be cleaned.

3. The big data intelligent cleaning method based on machine learning according to claim 2, characterized in that, Step 2 includes: A data feature space is constructed based on the feature dataset to be cleaned, and the degree feature vector is mapped to the corresponding coordinate position in the data feature space; An anomaly detection model is constructed using the isolated forest algorithm. The construction process of the anomaly detection model is as follows: a preset number of isolated trees are generated in the data feature space. Each isolated tree isolates the feature vector to an independent node through recursive random partitioning. The average path length of each feature vector in all isolated trees is calculated. An anomaly score is generated based on the ratio of the average path length to a preset path length benchmark.

4. The machine learning-based intelligent big data cleaning method according to claim 3, characterized in that, The abnormal scores are sorted according to their numerical values, and the values ​​that are located at a predetermined quantile after sorting are selected as the initial judgment threshold.

5. The machine learning-based intelligent big data cleaning method according to claim 4, characterized in that, Step 4 includes: The frequency of selection and information gain of each feature vector dimension in the anomaly detection model during the splitting of isolated tree nodes are analyzed. The anomaly discrimination contribution of each feature vector dimension is calculated. The two feature vector dimensions with the highest anomaly discrimination contribution are selected and denoted as the first feature dimension and the second feature dimension, respectively. In a two-dimensional feature subspace spanned by the first and second feature dimensions, an anchor point set is established based on the projection distribution of all feature vectors in the feature dataset to be cleaned onto the two-dimensional feature subspace. The anchor point set consists of four anchor points: the first anchor point is located at the intersection of the lower quartile of the first feature dimension and the lower quartile of the second feature dimension; the second anchor point is located at the intersection of the upper quartile of the first feature dimension and the lower quartile of the second feature dimension; the third anchor point is located at the intersection of the upper quartile of the first feature dimension and the upper quartile of the second feature dimension; and the fourth anchor point is located at the intersection of the lower quartile of the first feature dimension and the upper quartile of the second feature dimension. Connect the four anchor points in spatial order to form a quadrilateral anomaly detection domain.

6. The machine learning-based intelligent big data cleaning method according to claim 5, characterized in that, Step 5 includes: Eigenvalue decomposition is performed on the covariance matrix of all feature vectors in the feature dataset to be cleaned to obtain covariance eigenvectors and eigenvalues; A high-dimensional confidence elliptic is constructed based on the covariance eigenvector and eigenvalue, wherein the covariance eigenvector determines the principal axis direction of the high-dimensional confidence elliptic, and the product of the square root of the eigenvalue and the preset confidence coefficient determines the semi-axis length of each principal axis direction. Projecting the high-dimensional confidence elliptic onto a two-dimensional plane composed of the first and second feature dimensions yields a two-dimensional confidence ellipse. Calculate the area of ​​the overlapping region between the quadrilateral anomaly detection domain and the two-dimensional confidence ellipse on the two-dimensional plane, and calculate the ratio of the overlapping region area to the total area of ​​the quadrilateral anomaly detection domain. Use the ratio as a threshold adjustment factor. The threshold adjustment factor and the initial judgment threshold are weighted by a preset weighting coefficient to generate a dynamic judgment threshold.

7. The machine learning-based intelligent big data cleaning method according to claim 6, characterized in that, Step 6 includes: The real-time anomaly score is compared with the dynamic judgment threshold in real time, and user transaction behavior data with a real-time anomaly score greater than the dynamic judgment threshold is marked as non-real transaction data. Records identified as non-genuine transaction data are removed from the user transaction behavior data stream to obtain a clean user transaction behavior data stream; The difference between the feature distribution of the clean user transaction behavior data stream and the historical feature distribution is calculated. When the difference exceeds a preset change threshold, the anomaly detection model is updated. The update process includes recalculating the anomaly detection contribution of each feature dimension using newly collected user transaction behavior data, reselecting the first and second feature dimensions, and reconstructing the quadrilateral anomaly judgment domain, the high-dimensional confidence elliptic, and the dynamic judgment threshold.

8. A big data intelligent cleaning system based on machine learning, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to collect and preprocess user transaction behavior data streams and perform feature engineering to extract multi-dimensional feature vectors to obtain a feature dataset to be cleaned. The computation module is used to construct the data feature space of the feature dataset to be cleaned, and to map multi-dimensional feature vectors to the data feature space; Multi-dimensional feature vectors are input into an anomaly detection model built on the isolated forest algorithm, and the real-time anomaly score corresponding to each feature vector is calculated. The determination module is used to perform statistical analysis on the real-time anomaly scores of all feature vectors and determine the initial judgment threshold. A module is established to determine the first and second feature dimensions in the data feature space to establish an anchor set, and to connect the anchors in the anchor set to form a quadrilateral anomaly determination domain. The weighting module is used to construct a high-dimensional confidence ellipse based on the covariance matrix and mean vector of the feature vectors, and project the high-dimensional confidence ellipse onto a two-dimensional plane composed of the first and second feature dimensions to obtain a two-dimensional confidence ellipse; calculate the area ratio of the overlapping region between the quadrilateral anomaly detection domain and the two-dimensional confidence ellipse to obtain the threshold adjustment factor. The threshold adjustment factor is weighted with the initial judgment threshold to generate a dynamic judgment threshold; The update module compares real-time anomaly scores with dynamic judgment thresholds to identify and clean up non-genuine transaction data, resulting in a clean data stream. Continuously monitor the characteristic distribution changes of the clean data stream, and trigger the update process of the anomaly detection model and dynamic judgment threshold when the distribution change exceeds the predetermined threshold.

9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images