Noisy Data Classification Methods and User Classification Methods Based on Random Sampling Clustering

Abstract

This invention discloses a noisy data classification method based on random sampling clustering, comprising: acquiring noisy data to be classified; performing random sampling processing on the acquired noisy data, selecting several processed data to construct a partial centroid set; performing iterative processing on the acquired noisy data, selecting several processed data to add to the constructed partial centroid set, constructing a centroid set; constructing weighted instances for the constructed centroid set; and selecting a weighted noise method to complete the classification of the noisy data. This invention also discloses a user classification method including the aforementioned noisy data classification method based on random sampling clustering. By processing the noisy data to be classified through random sampling, selecting an iterative processing method, and employing stratified sampling, the noisy data classification is achieved. Moreover, this invention has high classification accuracy, strong reliability, good practicality, and low algorithm complexity.

Description

Technical Field

[0001] This invention belongs to the field of data processing, specifically relating to a noisy data classification method and a user classification method based on random sampling clustering. Background Technology

[0002] Clustering, as one of the most fundamental problems in unsupervised learning, has wide applications in fields such as bioinformatics, financial analysis, and data mining. Given a set of data objects, cluster analysis can divide them into several groups based on their characteristic attributes, minimizing the similarity between data objects within a group while maximizing the similarity between data objects between groups. Commonly used clustering models include k-means clustering, k-median clustering, and k-centroid clustering. Among known clustering problems, NP-hard problems are relatively classic, meaning that unless NP = P is assumed, it is impossible to find the optimal solution in polynomial time; currently, approximate algorithms are often chosen to solve clustering analysis problems.

[0003] As one of the fundamental models for clustering problems, the k-means clustering model is based on the principle of finding k centroids in a given space and assigning data to the k clusters formed by the centroids, such that the sum of the Euclidean distances from a given data point to its nearest centroid is minimized.

[0004] The lower bound problem for the k-median clustering model is the approximation ratio. In practical applications, data clustering problems inevitably encounter noise interference. Furthermore, the commonly used k-means clustering model is extremely sensitive to noisy data; even small noise disturbances in some data points can cause changes in the clustering results. During data analysis, this noise interference often severely affects the final clustering results. Therefore, how to remove the influence of noisy points during the clustering process is the problem of noisy clustering.

[0005] Currently, although there are existing clustering methods for noisy data clustering, most of these methods suffer from problems such as poor clustering accuracy, high time complexity, and poor performance in practical applications.

[0006] Similarly, current user classification methods based on the aforementioned noisy data clustering methods, while capable of user classification, still suffer from poor clustering accuracy and low reliability of user classification results in practical applications. Summary of the Invention

[0007] One of the objectives of this invention is to provide a noisy data classification method based on random sampling clustering that has high accuracy, high reliability, good practicality, and low algorithm complexity.

[0008] The second objective of this invention is to provide a user classification method that includes the aforementioned noisy data classification method based on random sampling clustering.

[0009] The noisy data classification method based on random sampling clustering provided by this invention includes the following steps:

[0010] S1. Obtain the noisy data to be classified;

[0011] S2. Randomly sample the noisy data obtained in step S1, and select several processed data to construct a partial set of center points;

[0012] S3. Iteratively process the noisy data obtained in step S1, select several processed data points and add them to the partial center point set constructed in step S2 to construct the center point set;

[0013] S4. For the set of center points constructed in step S3, construct a weighted instance;

[0014] S5. Use the weighted noise method to classify noisy data.

[0015] Step S2, which involves random sampling of the noisy data obtained in step S1, specifically includes:

[0016] Choose the following formula as the objective function:

[0017]

[0018] Where X is a given dataset, Z is the set of noise points removed, and p is a data point in the given dataset X; the dataset X is divided into K clusters. c i Let d(p,c) be the center point of the selected i-th center point set; j Let c be the center point from data point p to the j-th center point set. j Distance; selected according to random sampling method Z data points are used as the center point, where ε and η are input parameters; the probability that there is at least one normal point that is not a noise point in the data after random sampling is 1-η; the objective function removes the influence of z noise points during the optimization process.

[0019] Step S3 involves iteratively processing the noisy data obtained in step S1, selecting several processed data points and adding them to the partial centroid set constructed in step S2 to build the centroid set. Specifically, this includes:

[0020] Employing enhanced D 2- The sampling distribution independently samples several data points as a candidate set, and during the iterative processing, the size of the sampled dataset is determined:

[0021] If the size of the sampled candidate set is between (1+ε)z and Between these, where ε and δ are given input parameters, and z is the given number of noise points, according to the principle of hierarchical sampling, for the data after random sampling processing in step S2, randomly select from... Each data point is added to the partial center point set constructed in step S2 to construct the center point set;

[0022] If the size of the sampled candidate set is not within (1+ε)z and If the sampling fails, the current sampling process is abandoned, and a new candidate set for sampling is selected. The hierarchical sampling method ensures that at least one non-noise data point is found after each sampling, guaranteeing a 4-approximation of the clustering result on the sub-database. During each iteration, the distance between two data points needs to be recalculated. The iterative process ensures that the selected center point in each iteration comes from data points in different optimal clusters with a high probability. This process is repeated O(k) times to obtain k normal data points. All obtained data points are then added to the partial center point set constructed in step S2 to build the center point set. Here, O(k) = βk, where β is a set constant.

[0023] Step S4, which involves constructing weighted instances from the set of center points built in step S3, specifically includes: selecting... One central point, β is a set constant; first, remove the farthest z center points, and then assign all remaining data points to the center points closest to them. The weight of each center point is the number of data points assigned to that center point.

[0024] Step S5 describes the selection of a weighted noise method to classify noisy data. Specifically, this includes: selecting k centroids using a reduction method to transform the noisy data problem into a penalized problem; and in each iteration, utilizing enhanced D... 2 The Sampling distribution selects a center point and repeats the process until k center points are selected.

[0025] This invention also provides a user classification method that includes the aforementioned noisy data classification method based on random sampling clustering, comprising: acquiring original basic user information and using it as noisy data to be classified; using the aforementioned noisy data classification method based on random sampling clustering to classify the data to be classified; classifying users according to the obtained classification results, wherein the user classification is based on the information differences between users, and Euclidean distance is selected to represent the information differences between users.

[0026] The noisy data classification method and user classification method based on random sampling clustering provided by this invention process the noisy data to be classified through random sampling, selects an iterative processing method, and adopts a hierarchical sampling method to sample the data to achieve classification of noisy data; moreover, the classification accuracy of this invention is high, the reliability is strong, the practicality is good, and the algorithm complexity is low. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0028] Figure 2 This is a schematic diagram of the runtime of the method of the present invention on the KDDFULL dataset.

[0029] Figure 3 This is a schematic diagram illustrating the approximation of the method of the present invention on the KDDFULL dataset.

[0030] Figure 4 This is a schematic diagram of the user classification method of the present invention. Detailed Implementation

[0031] like Figure 1 The diagram shown is a flowchart of the method of the present invention: The noisy data classification method based on random sampling clustering provided by the present invention includes the following steps:

[0032] S1. Obtain the noisy data to be classified;

[0033] S2. Randomly sample the noisy data obtained in step S1, and select several processed data points to construct a partial center point set; the random sampling process for the obtained noisy data specifically includes:

[0034] Choose the following formula as the objective function:

[0035]

[0036] Where X is a given dataset, Z is the set of noise points removed, and p is a data point in the given dataset X; the dataset X is divided into K clusters. c i Let d(p,c) be the center point of the selected i-th center point set; j Let c be the center point from data point p to the j-th center point set. j Distance; selected according to random sampling method Z data points are used as the center point, where ε and η are input parameters; the probability that there is at least one normal point that is not a noise point in the data after random sampling is 1-η; the objective function removes the influence of z noise points during the optimization process.

[0037] S3. Iteratively process the noisy data obtained in step S1, and select several processed data points to add to the partial centroid set constructed in step S2, thus constructing a centroid set; specifically including:

[0038] Employing enhanced D 2 - The sampling distribution independently samples several data points as a candidate set. During the iterative processing, the size of the sampled dataset is determined: if the size of the candidate set is between (1+ε)z and... Between, where ε and δ are given input parameters, and z is the given number of noise points, according to the principle of hierarchical sampling, for the data after random sampling processing in step S2, randomly select from... Each data point is added to the partial centroid set constructed in step S2 to construct the centroid set; if the size of the sampled candidate set is not greater than (1+ε)z... If the sampling fails, the current sampling process is abandoned, and a new candidate set for sampling is selected. Hierarchical sampling ensures that at least one non-noise data point is found after each sampling, guaranteeing a 4-approximation of the clustering result on the sub-database. During each iteration, the distance between two data points needs to be recalculated. Iterative processing ensures that the selected center point in each iteration comes from data points in different optimal clusters with a high probability. This process is repeated O(k) times to obtain k normal data points. All obtained data points are then added to the partial center point set constructed in step S2 to build the center point set. Here, O(k) = βk, where β is a set constant.

[0039] S4. For the set of center points constructed in step S3, construct a weighted instance;

[0040] Step S4, which involves constructing weighted instances from the set of center points built in step S3, specifically includes: selecting... One central point, First, remove the farthest z center points, and then assign all remaining data points to the center points closest to them. The weight of each center point is the number of data points assigned to it.

[0041] S5. Select a weighted noise method to classify noisy data; specifically, this includes: using a reduction method (where the reduction method can transform a noisy k-means instance into a penalized k-means instance for solving while only losing constant approximation) to select k centroids, transforming the noisy data problem into a penalized problem; in each iteration, using enhanced D... 2 The Sampling distribution selects a center point and repeats the process until k center points are selected.

[0042] By selecting linear time data points with time complexity of n and obtaining k center points, we can obtain an approximation ratio with a relatively high probability of 1-λ. The solution (where, γ is a constant that can be calculated, ε is an input parameter, and the number of noise points removed is at most (1+ε)z.

[0043] Using the real-world dataset test (SUSY) as an example, this dataset contains 5,000,000 Monte Carlo waveform data points, each with 18 features. The first 10 features (columns 2-22) are kinematic properties measured by particle detectors in the accelerator, and the last 8 features are the corresponding functional features of the first 10 features. In the experiment, the parameter η and the parameter ε are set to 0.1. Figure 2 The diagram shows the runtime of the method of the present invention on the KDDFULL dataset, which represents the running time of the method of the present invention under the condition of a fixed k value, when the number of noise points is 1% of the total data volume of the dataset, and the noise amplitude is 5 and 10 respectively. Figure 3 The diagram illustrates the approximation of the method of this invention on the KDDFULL dataset. It shows the approximation ratio of this invention under a fixed k value, when the number of noise points is 1% of the total dataset size, and the noise amplitudes are 5 and 10, respectively. Experiments show that the method of this invention has a certain degree of stability and can be completed very quickly.

[0044] Figure 4 The diagram shows a flowchart of the user classification method of the present invention: The present invention also provides a user classification method including the above-mentioned noisy data classification method based on random sampling clustering, including obtaining the original basic user information and using it as noisy data to be classified; using the above-mentioned noisy data classification method based on random sampling clustering to classify the data to be classified; classifying users according to the obtained classification results, and classifying users mainly based on the information differences between users, and choosing Euclidean distance to represent the information differences between users.

[0045] A qualification review agency needs to collect user information to confirm whether users can pass the qualification review and to determine the user's rating. During independent random sampling, the qualification review agency may encounter some abnormal information, i.e., noise. Noise can generally be categorized into two types: output errors (the same data point yields two different evaluation results); and input errors (errors occur due to the randomness of the data source during data collection, such as incorrect information filled in by customers). To address these issues, a user classification method, including the method described in this invention, is employed, specifically comprising the following steps:

[0046] (1) Collect the original basic user information, including age, gender, asset status, etc.; and treat the basic user information as noisy data to be classified. The noise comes from labeling errors (e.g., a customer should be labeled "passed" but is ultimately labeled "failed"; or two customers with the same data are labeled "passed" and "failed" respectively), or input errors (e.g., the basic user information on the mobile phone itself is incorrect).

[0047] (2) The method of the present invention is used to classify noisy data to be classified; during the classification process, noise points in the data are found and discarded; customers with similar situations are clustered into the same cluster, and the remaining outliers are discarded as noise points.

[0048] (3) Based on the classification results described in step (2), the user's basic information is classified.

[0049] After the above steps, the final user information classification results with noise points removed are obtained, which can help qualification reviewers better analyze the user structure.

[0050] The user classification method of this invention can also be applied to the anomaly detection of fraudulent activities. Fraudulent activities are incredibly fast, thus posing a significant threat to credit card transaction institutions and banks. Fraudulent activities include banking fraud, mobile phone fraud detection, and commercial fraud. Noise detection methods based on outliers can be used to detect noisy data presented in the original dataset, enabling the identification and prediction of fraudulent activities. The user classification method of this invention provides a fraudulent activity classification and prediction method based on outlier detection, which can ensure high-quality fraud diagnosis, thereby achieving better fraud detection.

[0051] The user classification method of the present invention provides a means of detecting fraudulent behavior anomalies, specifically including:

[0052] 1) Obtain transaction data provided by financial companies or banking institutions;

[0053] 2) Perform feature abstraction and feature extraction on the provided transaction data; use the transaction data information obtained in step 1) as noisy data to be classified;

[0054] 3) The user classification method for noisy data classification based on random sampling clustering of the present invention is used to classify the data to be classified; during the classification process, noise points in the data are found and discarded; data with similar conditions are clustered into the same cluster, and the remaining outliers are treated as noise points and classified into another category.

[0055] 4) Based on the classification results obtained in step 3), predict and determine the fraud of the noise data; finally, obtain the determination result of the noise data to help institutions and banks to detect fraud.

[0056] The above examples of qualification review agencies and fraud detection can be similarly applied to traditional industrial sectors. For instance, when data quality assessment is required in the industrial sector, referencing qualification review agency examples, suppliers with poor quality can be eliminated from the data clustering results of multiple suppliers. Similarly, when companies need to conduct fraud detection, referencing fraud detection examples, data can be categorized, noise points can be identified, and fraudulent behavior can be judged based on these noise points, thereby preventing data-related risks.

Claims

1. A noisy data classification method based on random sampling clustering, comprising the following steps: S1. Obtaining noisy data to be classified; S2. Randomly sampling the noisy data obtained in step S1, selecting a plurality of processed data to construct a partial center point set; specifically comprising: selecting the following formula as the objective function: where X is a given data set, Z is a set of removed noise points, p is a data point in the given data set X; the data set X is divided into K clusters, c i is the center point of the selected ith center point set; d(p, c j ) is the distance from the data point p to the center point c j of the jth center point set; according to the random sampling method, K data points are selected as the center points, where ε and η are input parameters; S3. Iteratively processing the noisy data obtained in step S1, selecting a plurality of processed data to join the partial center point set constructed in step S2 to construct a center point set; S4. Constructing a weighted instance based on the center point set constructed in step S3; S5. Classifying the noisy data using the weighted noise method. 2.The random sampling clustering based noisy data classification method of claim 1, wherein The iterative processing of the noisy data obtained in step S1 in step S3, selecting a plurality of processed data to join the partial center point set constructed in step S2 to construct a center point set, specifically comprising: Enhanced D 2 -Sampling distribution independently sampling several data points as a sampling candidate set, judging the set size of the sampled data set in the process of iterative processing: if the set size of the sampling candidate set is between (1+ε)z and , where ε and δ are given input parameters, and z is the given number of noise points, according to the principle of hierarchical sampling, randomly sampling the data processed in step S2, randomly selecting data points from it and adding them to the partial center point set constructed in step S2 to construct the center point set; if the set size of the sampling candidate set is not between (1+ε)z and , then discard the current sampling processing and reselect the sampling candidate set; repeat the iteration O(k) times to obtain k normal data points, and add all the obtained data points to the partial center point set constructed in step S2 to construct the center point set; wherein O(k)=βk, and β is a constant set. 3.The random sampling clustering based noisy data classification method of claim 2, wherein The center point set constructed in step S3 is subjected to the step S4, and a weighted instance is constructed, specifically including: selecting center points from the center point set constructed in step S3; first, removing the farthest z center points, and assigning all the remaining data points to the nearest center point, and the weight of each center point is the number of data points assigned to the center point. 4.The random sampling clustering based noisy data classification method of claim 3, wherein The selection of the weighted noise method in step S5 completes the classification of the noisy data, and specifically includes: selecting k center points in a reduction manner, and converting the noisy data problem into a penalty problem; in each iteration process, the enhanced D 2 The sampling distribution selects a center point, and the repeated processing is performed until k center points are selected.

5. A user classification method including the noise data classification method based on random sampling clustering according to any one of claims 1 to 4, characterized by comprising obtaining original user basic information and taking it as noisy data to be classified; using the noisy data classification method based on random sampling clustering of any one of claims 1-4 to classify the data to be classified; and classifying users according to the obtained classification result.

Images