Noisy data classification method and user classification method based on random sampling clustering
Through the clustering method based on random sampling and greedy algorithms, the problems of low clustering accuracy and high complexity of noise data in the existing technology are solved, and efficient and reliable data classification is achieved, which reduces the complexity of the algorithm and improves the clustering. Class effects.
Patent Information
- Application Number
- CN202210375631.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-11
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-11
AI Technical Summary
When processing clustering of noise data, the prior art has poor accuracy and high complexity, making it difficult to effectively remove the influence of noise points, resulting in poor clustering effect.
The clustering method based on random sampling and greedy algorithm is adopted to initialize the center point set through random sampling, and data points are gradually added during the iteration process, combining cost calculation and noise point judgment to realize the classification and noise denoising of data points.
It improves the accuracy and reliability of clustering, reduces the complexity of the algorithm, realizes efficient noise-free data classification, and shows high stability and fast running speed in practical applications.
Smart Images

Figure CN114742155B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data processing, and in particular relates to a noisy data classification method and a user classification method based on random sampling clustering. Background Art
[0002] Clustering is one of the most basic problems in data analysis. Cluster analysis is to group data objects based on the information found in the data that describes the objects and their relationships. The purpose of cluster analysis is to group objects, and the objects within the group are similar (related) to each other, while the objects in different groups are different (unrelated). The greater the similarity within the group, the greater the gap between the groups, which means the better the clustering effect; that is, the goal of clustering is to obtain a higher intra-cluster similarity and a lower inter-cluster similarity, so that the distance between clusters is as large as possible, and the distance between the samples within the cluster and the cluster center is as small as possible.
[0003] Clustering algorithms are widely used, so clustering models have always been the focus of research; for example, k-means, k-median and k-center. The clustering problem is a classic NP-hard problem. Unless NP=P, it is impossible to find the optimal solution to the problem in polynomial time. Approximate algorithms are one of the methods for dealing with NP-hard problems. If the output function value of the algorithm is not greater than α times (set value) of the optimal solution, then the algorithm is called an approximate algorithm, and α is called the approximation ratio of the algorithm.
[0004] The k-center problem has always been a popular model in clustering. In the past few decades, the k-center model has attracted widespread attention, mainly studying approximate algorithms in Euclidean space and metric space. The k-center model requires finding k center positions, each of which forms a radius of R. i circles, which can cover all data points, and the objective function is to minimize the maximum radius R i At present, it has been proved that the approximation ratio of k-center problem in metric space is 2, which is the lower bound of the problem. In Euclidean space, the approximation ratio of the best k-center algorithm is 1.822.
[0005] However, in practical applications, the data set needs to be subject to certain restrictions during the clustering process, which is the restricted clustering problem. The restricted clustering problem can be regarded as a generalization of the general clustering problem, and the restricted clustering problem is more challenging. Common restricted clustering problems include: fair clustering problem, clustering problem with capacity restriction, clustering problem with noise, etc. Moreover, in real life, the data set is often noisy and contains some abnormal data points. In data analysis, outliers in the data set often seriously affect the final analysis results. How to remove the influence of noise points in the clustering process is the clustering problem with noise.
[0006] At present, although there are corresponding clustering methods in the industry for the problem of noisy data clustering, the existing methods have poor clustering accuracy and high complexity, and the effect is poor in practical application. Summary of the invention
[0007] One of the purposes of the present invention is to provide a noisy data classification method based on random sampling clustering with high accuracy, high reliability and good practicability.
[0008] A second objective of the present invention is to provide a user classification method including the noisy data classification method based on random sampling clustering.
[0009] The noisy data classification method based on random sampling clustering provided by the present invention comprises the following steps:
[0010] S1. Obtain a noisy data set to be classified;
[0011] S2. Randomly sample the data set obtained in step S1, and add the sampled data to the center point set to complete the initialization of the center point set;
[0012] S3. Based on the greedy algorithm, the data set is iteratively processed: in each iteration, several data points farthest from the center point set are obtained, and part of the data is sampled from the obtained data points and added to the center point set;
[0013] S4. Enumerate the data in the center point set and obtain several data points with the minimum cost as the center points of the cluster;
[0014] S5. Assign each data point in the data set to the center point closest to itself to complete the classification of noisy data based on random sampling clustering.
[0015] The random sampling of the data set obtained in step S1 described in step S2 specifically includes the following steps:
[0016] The following formula is used as the objective function of the random sampling process
[0017] max p∈X (min 1≤j≤K d(p,c j ))
[0018] Where X is the subset of noise points removed from the data set obtained in step S1, and Z is the number of noise points removed, N is the number of all data in the data set obtained in step S1; p is the data point in set X; set X is divided into K clusters, c i is the center point of the selected i-th center point set; d(p,c j) is the center point c from data point p to the jth center point set j The distance of the randomly sampled data points is Where log() represents the logarithm with base 2, η is the set parameter, γ is the probability of sampling the noise point in the data point set, and z is the default number of noise points, and n is the data set size.
[0019] The step S3 specifically includes the following steps:
[0020] Based on the idea of greedy algorithm, each time select (1+ε)z data points farthest from the center point set; ε is the set parameter, and z is the default number of noise points;
[0021] Then randomly select from (1+ε)z data points data points and add them to the center point set;
[0022] Make judgments during the iteration process:
[0023] If the condition d(Q j ,E)≤2r opt , then the (1+ε)z data points selected in the current round are directly regarded as noise points and discarded directly; where Q j is the selected noise point, E is the candidate center set, r opt is the optimal radius of the k-center, d(Q j ,E) is the distance from the selected noise point to the candidate center set, which is expressed as the shortest distance between the two data sets;
[0024] If the condition d(Q j ,E)≤2r opt , then continue to the next round of iteration.
[0025] The cost described in step S4 is specifically obtained by adopting the following steps:
[0026] Take the k candidate center points in the candidate center set as the center points, and assign all data points to the candidate center points closest to them; at this time, the data set is divided into k clusters, and the radius of the largest cluster is used as the cost value.
[0027] The present invention also provides a user classification method including the noisy data classification method based on random sampling clustering, which specifically includes the following steps:
[0028] A. Obtain the original basic information of the user;
[0029] B. Use the basic user information obtained in step A as a noisy data set to be classified;
[0030] C. Using the above-mentioned random sampling clustering-based noisy data classification method to classify the data set to be classified;
[0031] D. Classify users according to the classification results obtained in step C; wherein, user classification is performed according to the information differences between users, and the differences are represented by Euclidean distance.
[0032] The noisy data classification method and user classification method based on random sampling clustering provided by the present invention not only realize the classification of noisy data through innovative classification steps and calculation methods, but also have high classification accuracy, high reliability, good practicality and low algorithm complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 The figure is a flow chart of the classification method of the present invention.
[0034] Figure 2 This is a schematic diagram of the runtime of the classification method of the present invention under the Minst data set.
[0035] Figure 3 This is an Approximation diagram of the classification method of the present invention under the Minst data set.
[0036] Figure 4 The figure is a flowchart of the user classification method of the present invention. DETAILED DESCRIPTION
[0037] like Figure 1 The method flow chart of the classification method of the present invention is shown as follows: The noisy data classification method based on random sampling clustering provided by the present invention comprises the following steps:
[0038] S1. Obtain a noisy data set to be classified;
[0039] S2. Randomly sample the data set obtained in step S1, and add the sampled data to the center point set to complete the initialization of the center point set; random sampling specifically includes the following steps:
[0040] The following formula is used as the objective function of the random sampling process
[0041] max p∈X (min 1≤j≤K d(p,c j ))
[0042] Where X is the subset of noise points removed from the data set obtained in step S1, and Z is the number of noise points removed, N is the number of all data in the data set obtained in step S1; p is the data point in set X; set X is divided into K clusters, c i is the center point of the selected i-th center point set; d(p,c j ) is the center point c from data point p to the jth center point set j The distance of the randomly sampled data points is Where log() represents the logarithm with base 2, η is the set parameter, γ is the probability of sampling the noise point in the data point set, and z is the default number of noise points, and n is the data set size;
[0043] Through random sampling in this step, the probability that there is at least one normal point that is not a noise point is 1-η; at the same time, the influence of z noise points is removed in the process of optimizing the objective function;
[0044] S3. Based on the greedy algorithm, the data set is iteratively processed: in each iteration, several data points farthest from the center point set are obtained, and some data are sampled from the obtained data points and added to the center point set; specifically, the steps are as follows:
[0045] The greedy idea is based on Graham's idea. Each time, the data point farthest from the selected center point is found and opened. The clustering result can guarantee 2approximation. Each iteration needs to recalculate the distance between two data points.
[0046] Therefore, this step of the present invention is based on the idea of a greedy algorithm, and each time selects (1+ε)z data points that are farthest from the central point set; ε is a set parameter and the default value is 0.1, and z is the number of default noise points; (1+ε)z here is a relaxation condition for the noise problem, and the condition of just removing z noise points is relaxed to the condition of removing (1+ε)z noise points;
[0047] Then randomly select from (1+ε)z data points data points and add them to the center point set;
[0048] Make judgments during the iteration process:
[0049] If the condition d(Q j ,E)≤2r opt , then the (1+ε)z data points selected in the current round are directly regarded as noise points and discarded directly; where Q j is the selected noise point, E is the candidate center set, r opt is the optimal radius of the k-center, d(Q j,E) is the distance from the selected noise point to the candidate center set, which is expressed as the shortest distance between the two data sets;
[0050] If the condition d(Q j ,E)≤2r opt , then continue to the next round of iteration; that is, if the previous situation does not occur during the iteration, it can be guaranteed that there are data points from different optimal clusters in the candidate center points selected in each iteration, and the loop iteration O(k) times can make the candidate center set have k normal points;
[0051] S4. Enumerate the data in the center point set to obtain several data points with the minimum cost as the center point of the cluster; the cost is specifically obtained by the following steps:
[0052] Take the k candidate center points in the candidate center set as the center points, and assign all data points to the candidate center point closest to them; at this time, the data set is divided into k clusters, and the radius of the largest cluster is used as the cost value;
[0053] S5. Assign each data point in the data set to the center point closest to itself to complete the classification of noisy data based on random sampling clustering.
[0054] The time complexity of the method of the present invention is linear with respect to n, and k center points are accurately opened. When the value of k is fixed, a solution with an approximation ratio of 3 can often be obtained with a high probability (1-λ).
[0055] The effect of the method of the present invention is further described below in conjunction with an embodiment:
[0056] The real data set (Minst) is used for testing. The data set contains 60,000 pictures. Each picture consists of 28×28 pixels and can be expanded into a one-dimensional vector with a dimension of 784. In the experiment, the parameters η and ε are set to 0.1 and 0.1, respectively. Figure 2 It represents the running time of the method of the present invention under different k values. Figure 3 It represents the approximation ratio of the method of the present invention under different k value conditions. Experiments show that the method of the present invention has a certain stability and can be completed at a very fast speed when the k value is small.
[0057] like Figure 4 The user classification method of the present invention is shown as a flow chart of the method: the user classification method provided by the present invention, including the above-mentioned random sampling clustering-based noise data classification method, specifically includes the following steps:
[0058] A. Obtain the original basic information of the user;
[0059] B. Use the basic user information obtained in step A as a noisy data set to be classified;
[0060] C. Using the above-mentioned random sampling clustering-based noisy data classification method to classify the data set to be classified;
[0061] D. Classify users according to the classification results obtained in step C; wherein, user classification is based on information differences between users, and the differences can be represented by Euclidean distance.
[0062] For example, when a bank issues a credit card, it needs to collect user information to confirm whether the user meets the requirements for issuing the card and determine the credit limit. However, when the bank conducts independent random sampling, some abnormal information, namely noise, will appear. Noise can generally be attributed to the following two types.
[0063] Output error:
[0064] 1. The same data has two different evaluation results;
[0065] 2. Under the same evaluation results, different subsequent processing processes occurred.
[0066] Input error:
[0067] When collecting data, errors may occur due to the randomness of the data source (for example, errors made by customers when filling in information).
[0068] Then, for this type of situation, the user classification method of the present invention specifically includes the following steps:
[0069] a. Collect original basic user information;
[0070] User basic information includes age, gender, annual income, etc.
[0071] b. Use the basic user information obtained in step a as a noisy data set to be classified;
[0072] The noise comes from labeling errors (for example, a customer should be marked as "not issued" but is marked as "issued", or two customers with the same data are marked as "issued" and the other as "not issued"), or input errors (for example, the basic information of the mobile phone user itself is wrong);
[0073] c. Use the above-mentioned random sampling clustering-based noisy data classification method to classify the data set to be classified; during the classification process, the noise points in the data can be found and discarded; customers with similar situations are clustered into the same cluster, and outliers outside each cluster are discarded as noise points
[0074] d. Classify users according to the classification results obtained in step C;
[0075] After the above steps, the user classification result with noise points removed is finally obtained, which can help bank personnel better analyze the user structure.
Claims
1. A noisy data classification method based on random sampling clustering, comprising the following steps: S1. Obtain a noisy data set to be classified; S2. Randomly sample the data set obtained in step S1, and add the sampled data to the center point set to complete the initialization of the center point set; The random sampling of the data set obtained in step S1 specifically includes the following steps: The following formula is used as the objective function of the random sampling process max p∈X (min 1≤j≤K d(p,c j )) Where X is the subset of noise points removed from the data set obtained in step S1, and Z is the number of noise points removed, N is the number of all data in the data set obtained in step S1; p is the data point in set X; set X is divided into K clusters, c i is the center point of the selected i-th center point set; d(p,c j ) is the center point c from data point p to the jth center point set j The distance of the randomly sampled data points is Where log() represents the logarithm with base 2, η is the set parameter, γ is the probability of sampling the noise point in the data point set, and z is the default number of noise points, and n is the data set size; S3. Based on the greedy algorithm, the data set is iteratively processed: in each iteration, several data points farthest from the center point set are obtained, and some data are sampled from the obtained data points and added to the center point set; specifically, the steps are as follows: Based on the idea of greedy algorithm, each time select (1+ε)z data points farthest from the center point set; ε is the input parameter with a default value of 0.1, and z is the default number of noise points; Then randomly select from (1+ε)z data points data points and add them to the center point set; Make judgments during the iteration process: If the condition d(Q j ,E)≤2r opt , then the (1+ε)z data points selected in the current round are directly regarded as noise points and discarded directly; where Q j is the selected noise point, E is the candidate center set, r opt is the optimal radius of the k-center, d(Q j ,E) is the distance from the selected noise point to the candidate center set, which is expressed as the shortest distance between the two data sets; If the condition d(Q j ,E)≤2r opt , then continue to the next round of iteration; S4. Enumerate the data in the center point set and obtain several data points with the minimum cost as the center points of the cluster; S5. Assign each data point in the data set to the center point closest to itself to complete the classification of noisy data based on random sampling clustering.
2. The noisy data classification method based on random sampling clustering according to claim 1 is characterized in that The cost described in step S4 is specifically obtained by adopting the following steps: Take the k candidate center points in the candidate center set as the center points, and assign all data points to the candidate center points closest to them; at this time, the data set is divided into k clusters, and the radius of the largest cluster is used as the cost value.
3. A user classification method including the noisy data classification method based on random sampling clustering according to claim 1 or 2, specifically comprising the following steps: A. Obtain the original basic information of the user; B. Use the basic user information obtained in step A as a noisy data set to be classified; C. Using the noisy data classification method based on random sampling clustering as described in claim 1 or 2 to classify the data set to be classified; D. Classify users according to the classification results obtained in step C; User classification is based on the information differences between users, and the differences are expressed using Euclidean distance.
