Data processing method, device, electronic device and storage medium

Abstract

The present application provides a data processing method, device, electronic device, and storage medium, the method comprising: clustering data samples in a first data set according to a first cluster center set to obtain cluster clusters corresponding to each cluster center in the first cluster center set; obtaining distance information between each cluster center and the data samples in the cluster cluster corresponding to each cluster center; determining the next cluster center of the first data set according to the distance information to obtain a second cluster center set, the second cluster center set including the first cluster center set and the next cluster center; when the second cluster center set does not meet a preset condition, updating the first cluster center set to the second cluster center set; when the second cluster center set meets the preset condition, clustering the data samples in the first data set according to the second cluster center set to obtain a clustering result for the first data set. Based on the data processing method provided by the present application, the efficiency of cluster analysis and the accuracy of clustering results are improved.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] During data storage, human input errors or different abbreviations for the primary key of the same data sample can lead to multiple duplicate records in the database. These duplicate records have different primary keys in the database, but the properties associated with the primary keys correspond to the same physical object. For example, the primary keys "male" and "M" both correspond to physical objects representing males, but duplicate records of the object "male" in the database have different primary keys.

[0003] In related technologies, data cleaning of duplicate data is mainly based on cluster analysis of data samples combined with comparison algorithms. Among them, cluster analysis is a statistical analysis technique that divides research objects into relatively homogeneous groups (clusters). For example, the k-means clustering method, based on clustering, clusters similar data samples into the same class, and disperses different data samples into different classes, and clusters multiple data samples corresponding to duplicate records into the same cluster cluster, thereby effectively identifying multiple data samples corresponding to duplicate records. Since the data samples in the cluster cluster are not all duplicate records, it is necessary to combine a comparison algorithm to identify duplicate records. The comparison algorithms mainly include: Sorted Neighborhood Method (SNM), Priority Queue Algorithm, and Smith-Waterman Algorithm.

[0004] The nearest neighbor sorting algorithm uses a specified keyword to sort a data set. It then moves a fixed-size window across the sorted data set, moving one data sample downward at a time. It then compares that record with the records within the window to determine if there are any duplicates. Improper keyword selection can affect the accuracy of the comparison results. Furthermore, the window size is difficult to accurately control. A small window can miss duplicate records, while a large window increases the number of comparisons.

[0005] The priority queue algorithm scans all data samples one by one. If a data sample is determined to belong to an existing priority queue, it is clustered into that priority queue. If a data sample is determined not to belong to an existing priority queue, it is identified as a representative record to form a priority queue. At the same time, the representative record is given the highest priority; when the priority queue reaches its upper limit, the lowest priority queue is deleted. Because representative records are based on subjective judgment, the accuracy of the priority queue algorithm is affected. The Smith-Waterman algorithm uses a dynamic programming method to set gaps and thresholds to calculate the degree of match between two strings. By using gaps at specified positions, it can identify string abbreviations, but it cannot solve the problem of reversed string order.

[0006] In related technologies, when performing cluster analysis on data samples, it is necessary to manually set the number of clusters (k). When the k value is too large, there are too many clusters, and repeated data samples are clustered into different clusters. Duplicate records cannot be effectively identified, resulting in low cluster analysis efficiency. When the k value is too small, the number of clusters is small, causing non-duplicate data samples to be clustered together, failing to achieve the purpose of cluster analysis and resulting in low accuracy of clustering results. Therefore, how to improve the efficiency of cluster analysis and the accuracy of clustering results has become an important issue that needs to be addressed urgently. Summary of the Invention

[0007] The present application provides a data processing method, device, electronic device and storage medium, which can improve the efficiency of cluster analysis and the accuracy of clustering results.

[0008] This application provides a data processing method, including:

[0009] Clustering the data samples in the first data set according to the first cluster center set to obtain a cluster cluster corresponding to each cluster center in the first cluster center set;

[0010] Obtaining distance information between each cluster center and data samples in the clusters corresponding to each cluster center;

[0011] determining a next cluster center of the first data set according to the distance information, to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center;

[0012] When the second cluster center set does not meet the preset condition, updating the first cluster center set to the second cluster center set;

[0013] When the second cluster center set meets a preset condition, the data samples in the first data set are clustered according to the second cluster center set to obtain a clustering result of the first data set.

[0014] In one implementation, determining the next cluster center of the first data set according to the distance information includes:

[0015] Obtaining maximum value information of distance values ​​from the distance information corresponding to the plurality of cluster centers, determining a maximum value of the distance value from the maximum value information of the distance values ​​corresponding to the plurality of cluster centers, and obtaining a data sample corresponding to the maximum value of the distance value;

[0016] The data sample corresponding to the maximum value of the distance value is determined as the next cluster center of the first data set.

[0017] In one implementation, the preset condition is that the distance between each cluster center in the second cluster center set and the data sample in the cluster corresponding to the cluster center is less than a first threshold.

[0018] In one implementation, the first cluster center set includes at least two cluster centers, and clustering the data samples in the first data set according to the first cluster center set includes:

[0019] Determine the cluster center to which the data samples in the first data set belong according to the first distance information and the second distance information;

[0020] The first distance information is the distance information between the cluster centers in the first cluster center set; the second distance information is the distance information between the data samples in the first data set and the cluster centers in the first cluster center set;

[0021] The data samples in the first data set are clustered according to the cluster centers to which the data samples in the first data set belong.

[0022] In one implementation, the clustering result of the first data set includes a plurality of clusters, and the method further includes:

[0023] Sorting the plurality of data samples in the cluster according to preset keywords;

[0024] The sorted multiple data samples are traversed in the sliding window, and duplicate data samples with similarity greater than a preset value in the multiple data samples are deleted to obtain a second data set.

[0025] The present application provides a data processing device, comprising:

[0026] A first processing module is configured to cluster the data samples in the first data set according to the first cluster center set to obtain a cluster cluster corresponding to each cluster center in the first cluster center set;

[0027] A second processing module is used to obtain distance information between each cluster center and the data samples in the clusters corresponding to each cluster center;

[0028] determining a next cluster center of the first data set according to the distance information, to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center;

[0029] a third processing module, configured to update the first cluster center set to the second cluster center set when the second cluster center set does not meet a preset condition;

[0030] When the second cluster center set meets a preset condition, the data samples in the first data set are clustered according to the second cluster center set to obtain a clustering result of the first data set.

[0031] In one implementation, the second processing module is configured to determine a next cluster center of the first data set according to the distance information, including:

[0032] Obtaining maximum value information of distance values ​​from the distance information corresponding to the plurality of cluster centers, determining a maximum value of the distance value from the maximum value information of the distance values ​​corresponding to the plurality of cluster centers, and obtaining a data sample corresponding to the maximum value of the distance value;

[0033] The data sample corresponding to the maximum value of the distance value is determined as the next cluster center of the first data set.

[0034] In one implementation, the preset condition is that the distance between each cluster center in the second cluster center set and the data sample in the cluster corresponding to the cluster center is less than a first threshold.

[0035] In one implementation, the first cluster center set includes at least two cluster centers, and the first processing module is configured to cluster the data samples in the first data set according to the first cluster center set, including:

[0036] Determine the cluster center to which the data samples in the first data set belong according to the first distance information and the second distance information;

[0037] The first distance information is the distance information between the cluster centers in the first cluster center set; the second distance information is the distance information between the data samples in the first data set and the cluster centers in the first cluster center set;

[0038] The data samples in the first data set are clustered according to the cluster centers to which the data samples in the first data set belong.

[0039] In one implementation, the clustering result of the first data set includes a plurality of clusters, and the third processing module is further configured to:

[0040] Sorting the plurality of data samples in the cluster according to preset keywords;

[0041] The sorted multiple data samples are traversed in the sliding window, and duplicate data samples with similarity greater than a preset value in the multiple data samples are deleted to obtain a second data set.

[0042] The present application provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, any of the above-mentioned data processing methods is implemented.

[0043] The present application provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned data processing methods is implemented.

[0044] The data processing method provided by the present application clusters the data samples in the first data set according to the first cluster center set to obtain the cluster clusters corresponding to each cluster center in the first cluster center set; obtains the distance information between each cluster center and the data samples in the cluster cluster corresponding to each cluster center; determines the next cluster center of the first data set according to the distance information to obtain the second cluster center set, and the second cluster center set includes the first cluster center set and the next cluster center; when the second cluster center set does not meet the preset conditions, updates the first cluster center set to the second cluster center set; when the second cluster center set meets the preset conditions, clusters the data samples in the first data set according to the second cluster center set to obtain the clustering results of the first data set. Based on the above data processing method, clustering is stopped when the cluster center set corresponding to the data set meets the preset conditions, the number of cluster clusters corresponding to the data set is automatically determined, and the clustering results of the data set are dynamically updated according to the number of cluster clusters, thereby improving the efficiency of cluster analysis and the accuracy of clustering results.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of an application scenario of a data processing method provided in an embodiment of the present application;

[0047] Figure 2 A schematic flow chart of a data processing method provided in an embodiment of the present application;

[0048] Figure 3A schematic diagram of a method for determining the cluster to which a data sample belongs provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a process for determining the next cluster center of a first data set in an embodiment of the present application;

[0050] Figure 5 Schematic diagram of a process for clustering data samples of a first data set in an embodiment of the present application;

[0051] Figure 6 A schematic flow chart of another data processing method provided in an embodiment of the present application;

[0052] Figure 7 A schematic flow chart of another data processing method provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0054] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] If similar descriptions of "first / second" appear in the invention document, the following explanation is added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] Figure 1 An application scenario diagram of a data processing method provided in an embodiment of the present application is shown.

[0060] See also Figure 1 The first data set D1 is the data processing object. The first data set D1 contains N data samples, D1 = {x1, x2, ..., x N}, where x1 is the first sample data of the first data set D1, x N is the Nth sample data of the first data set D1. In practical applications, the N data samples may include data samples with duplicate data records. The data processing method of the present application can improve the efficiency of cluster analysis and the accuracy of clustering results. On this basis, the efficiency and accuracy of duplicate data cleaning can be improved.

[0061] In one implementation, the dataset D0 to be processed is preprocessed to obtain the first dataset D1.

[0062] Here, the dataset D0 to be processed contains N0 sample data, N0≥N, and among the N0 sample data, there are data samples with misspellings of primary keys, inconsistent abbreviations of primary keys, or incorrect data formats.

[0063] In one implementation, preprocessing the dataset D0 to be processed to obtain the first dataset D1 includes:

[0064] The data samples in the dataset D0 to be processed are normalized to obtain the first dataset D1.

[0065] For example, in the dataset D0 to be processed, the primary key fields corresponding to the data samples are normalized to unify the primary key of the N0 sample data. That is, multiple different descriptions representing the same attribute are normalized, for example, "male" is unified to represent male, or the abbreviation "M" is unified to represent male.

[0066] Exemplarily, in the data set D0 to be processed, the data formats corresponding to the data samples are normalized to unify the data formats of N0 sample data.

[0067] Exemplarily, in the data set D0 to be processed, dirty data or abnormal data is removed; here, the numerical range of the dirty data or abnormal data does not conform to the preset data range.

[0068] For example, the N0 data samples in the processed data set D0 correspond to the age data of a person, and the age data of a person has a certain numerical range. When the value of a data sample is greater than a preset value or is a negative number, the corresponding data sample is determined to be abnormal data.

[0069] Based on the above preprocessing process, multiple data samples in the data set are normalized to avoid the problem that multiple duplicate records corresponding to the same entity object cannot be identified due to different primary key fields, thereby improving the accuracy of data sample clustering.

[0070] Figure 2 A flow chart of a data processing method provided in an embodiment of the present application is shown.

[0071] See also Figure 2 The data processing method provided in the embodiment of the present application includes the following steps:

[0072] Step A201: clustering the data samples in the first data set according to the first cluster center set to obtain a cluster corresponding to each cluster center in the first cluster center set.

[0073] Here, the first cluster center set can be a set composed of multiple initial cluster centers of the first data set. The initial cluster centers can be set randomly or according to preset rules. The preset rule can be that the distance between any two cluster centers among the multiple initial cluster centers is greater than a preset threshold.

[0074] It should be understood that the distance between data samples, or the distance between cluster centers, or the distance between a data sample and a cluster center can be calculated by the similarity of a certain key-value attribute of the data samples in the database.

[0075] For example, the first cluster center set C1 includes M cluster centers, M≥2, C={C1, C2, ..., C m}. See Figure 3 , the first cluster center set C1 includes the first cluster center c1 and the second cluster center c2; the first cluster center c1 and the second cluster center c2 are the data samples with the largest distance between the two data samples in the first data set D1, x i is a data sample in the dataset D, x i ∈D1.

[0076] For example, the distance d(c1, c2) between the cluster center c1 and the cluster center c2, the data sample x in the first data set D1, and the distance d(c1, c2) between the cluster center c1 and the cluster center c2 are obtained. i The distance d(x i ,c1), compare d(c1,c2) ​​and d(x i ,c1), when d(c1,c2), d(x i ,c1) conforms to the inequality relationship "2d(x i ,c1)<d(c1,c2)”, it can be seen that d(x i ,c1)<d(x i ,c2), that is, the cluster center c1 and the data sample xi The similarity is greater than that between the cluster center c2 and the data sample x i The similarity of the data sample x is determined. i The cluster center is c1.

[0077] Referring to the above clustering process, we take 1 to N for i in turn, and cluster the N data samples x in the first data set D1 according to the first cluster center set C1. i Perform clustering to obtain M clusters corresponding to the M cluster centers in the first cluster center set C1.

[0078] Step A202: Obtain distance information between each cluster center and data samples in the clusters corresponding to each cluster center.

[0079] For example, for any cluster K among the M clusters j (1≤j≤M), from cluster K j Contains multiple sample data, and obtains the maximum distance d between the data sample and the cluster center max _j=max{d(x i ,c j )|x i ∈K j}, where K j is the cluster center c j The corresponding clusters.

[0080] Referring to the above acquisition process, j is taken from 1 to M in sequence to obtain the distance information between each cluster center and the data samples in the cluster corresponding to each cluster center.

[0081] For example, the distance information includes d max _1, d max _2, d max _j, ..., d max _M corresponds to data samples x i M distance values ​​from the M cluster centers.

[0082] Step A203: Determine the next cluster center of the first data set according to the distance information to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center.

[0083] For example, from d max _1, d max _2, d max _j, ..., d max The maximum distance value dmax between the cluster center and the data sample is determined from the M distance values ​​corresponding to _M, dmax=max{d max_j|1≤j≤M}, obtain the data sample corresponding to the maximum distance dmax between the cluster center and the data sample.

[0084] Here, the maximum distance value dmax between the cluster center and the data sample reflects the maximum distance value in the distance information corresponding to the multiple cluster centers.

[0085] In one implementation, the data sample corresponding to the maximum distance value dmax between the cluster center and the data sample is determined as the next cluster center of the first data set D1, that is, the M+1th cluster center.

[0086] In one implementation, when dmax satisfies the inequality relationship "dmax>h*d m ", the data sample corresponding to dmax is used as the next cluster center of the first data set D1. Among them, h is the reference coefficient, 0<h≤1, d m It is the average distance between the two cluster centers in the M cluster centers contained in the first cluster center set C1.

[0087] It should be understood that when dmax satisfies the inequality relationship "dmax>h*d m "When the distance between the data sample corresponding to dmax and the cluster center to which the data sample corresponding to dmax belongs is greater than h*d m , indicating that the similarity between the data sample corresponding to dmax and the cluster center to which the data sample corresponding to dmax belongs is small, which does not meet the preset conditions for cluster convergence. m "When the distance between the data sample corresponding to dmax and the cluster center to which the data sample corresponding to dmax belongs is less than h*d m , indicating that the data sample corresponding to dmax and the cluster center to which the data sample corresponding to dmax belongs have a high similarity, which meets the preset conditions to be met for clustering convergence.

[0088] For example, the reference coefficient h ranges from 0.5 ≤ h ≤ 1. Increasing the value of h decreases the number of iterations and the number of clusters, while decreasing it increases the number of iterations and the number of clusters. In practical applications, the reference coefficient h can be adjusted based on actual conditions to determine the number of clusters that meet preset conditions.

[0089] Exemplarily, the data samples in the first data set D1 are clustered according to the updated cluster center set to obtain the clustering result of the first data set D1, and the above clustering update process is repeated until the second cluster center set C2 meets the preset conditions.

[0090] Exemplarily, the set consisting of the first cluster center set C1 and the M+1th cluster center is used as the second cluster center set C2 of the first data set. The sample data in the first data set D1 is clustered according to the second cluster center set C2, thereby dynamically updating the clustering results of the first data set D1.

[0091] Step A204: When the second cluster center set does not meet the preset condition, the first cluster center set is updated to the second cluster center set.

[0092] For example, the preset condition is that each cluster center c in the second cluster center set j Relative cluster center c j The corresponding cluster K j The data sample x in i The distance d(x i ,c j ) is less than the first threshold d0, which meets the inequality relationship "d(x i ,c j )<d0”, where x i ∈K j , K j is the cluster center c j The corresponding clusters.

[0093] For example, from d max _1, d max _2, d max _j, ..., d max The maximum distance value dmax between the cluster center and the data sample is determined from the M distance values ​​corresponding to _M, and the size relationship between d0 and dmax is compared. When the first thresholds d0 and dmax meet the inequality relationship "dmax≥d0", it is determined that the second cluster center set C2 does not meet the preset conditions.

[0094] It should be understood that when the first thresholds d0 and dmax satisfy the inequality relationship "dmax ≥ d0," the cluster corresponding to the maximum distance dmax0 between the cluster center and the data sample does not meet the preset conditions required for cluster convergence. In this case, the clustering results of the first dataset D1 include data samples with low similarity. These data samples with low similarity have large differences, indicating that the current clustering results are inaccurate.

[0095] Exemplarily, when it is determined that the second cluster center set C2 does not meet the preset conditions, the first cluster center set C1 is updated to the second cluster center set C2, and the sample data in the first data set D1 is clustered based on the second cluster center set C2 to dynamically update the clustering results of the first data set D1.

[0096] Step A205: When the second cluster center set meets a preset condition, cluster the data samples in the first data set according to the second cluster center set to obtain a clustering result of the first data set.

[0097] For example, from d max _1, d max _2, d max _j, ..., d max The maximum distance value dmax between the cluster center and the data sample is determined from the M distance values ​​corresponding to _M, and the size relationship between d0 and dmax is compared. When the first thresholds d0 and dmax meet the inequality relationship "dmax<d0", it is determined that the second cluster center set C2 meets the preset conditions.

[0098] Exemplarily, when it is determined that the second cluster center set C2 meets the preset condition, the clustering result of the first data set D1 based on the second cluster center set C2 is determined as the clustering result of the first data set D1.

[0099] It should be understood that when the first thresholds d0 and dmax meet the inequality relationship "dmax<d0", the cluster corresponding to the maximum distance dmax between the cluster center and the data sample meets the preset conditions to be met when the cluster converges. At this time, for any cluster K in the clustering result of the first data set D1 j , cluster K j The maximum distance between the multiple data samples in is less than d0, indicating that the clustering result of the first data set D1 meets the similarity condition corresponding to d0, and the similarity of the multiple data samples in each cluster is high.

[0100] In related technologies, when clustering a data set, the number of clusters must be manually set. This makes it difficult to estimate the number of clusters when the data volume is large, and it can easily lead to local optimality, affecting the efficiency of cluster analysis and the accuracy of clustering results. In an embodiment of the present application, clustering is stopped when the set of cluster centers corresponding to a data set meets a preset condition, the number of clusters corresponding to the data set is automatically determined, and the clustering results of the data set are dynamically updated based on the number of clusters. This improves the efficiency of cluster analysis and the accuracy of clustering results.

[0101] In practical applications, the above steps A201 to A205 can all be implemented by a processor, and the above processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0102] In one implementation, in the above step A203, the next cluster center of the first data set is determined according to the distance information, see Figure 4 , including the following steps:

[0103] Step A2031: Obtain maximum distance value information from the distance information corresponding to multiple cluster centers, determine the maximum distance value from the maximum distance value information, and obtain a data sample corresponding to the maximum distance value.

[0104] For example, the maximum values ​​of the distance values ​​in the distance information corresponding to the M cluster centers are d max _1, d max _2, d max _j, ..., d max _M, from d max _1, d max _2, d max _j, ..., d max The maximum distance value dmax between the cluster center and the data sample is determined from the M distance values ​​corresponding to _M, dmax=max{d max _j|1≤j≤M}, obtain the data sample corresponding to the maximum distance value dmax.

[0105] Step A2032: Determine the data sample corresponding to the maximum distance value as the next cluster center of the first data set.

[0106] Exemplarily, the data sample corresponding to the maximum distance value dmax is determined as the next cluster center of the first data set D1, that is, the M+1th cluster center.

[0107] In one implementation, the preset condition is that the distance between each cluster center in the second cluster center set and the data sample in the cluster corresponding to the cluster center is less than a first threshold.

[0108] It should be understood that the distance between each cluster center in the second cluster center set and the data samples in the cluster cluster corresponding to the cluster center is less than the first threshold, which is equivalent to the maximum distance between each cluster center in the second cluster center set and the data samples in the cluster cluster corresponding to the cluster center is less than the first threshold.

[0109] Here, the first threshold may be a fixed value corresponding to a similarity threshold information; the first threshold may also be dynamically adjusted based on the average distance between two cluster centers in a plurality of cluster centers included in the cluster center set.

[0110] For example, the preset condition is that each cluster center c in the second cluster center set j Relative cluster center c j The corresponding cluster K j The data sample x in i The distance d(x i ,c j ) is less than the first threshold d0, which meets the inequality relationship "d(x i ,c j )<d0”, where x i ∈K j , K j is the cluster center c j The corresponding clusters.

[0111] For example, the first threshold d0=h*d m , where h is the reference coefficient, 0<h≤1, d m is the average distance between the two cluster centers in the M cluster centers contained in the first cluster center set C1, d m =average{d(c i ,c j )|c i ,c j ∈C1}. At this time, with the dynamic update of the clustering results of the first data set, d0 gradually decreases from d0=h*d(c1,c2), indicating that the similarity of data samples in the same cluster gradually increases.

[0112] It should be understood that for the first threshold value "d0=h*d m ”, where d m It reflects the average similarity between the two cluster centers in the M cluster centers contained in the first cluster center set C1. The h value can be flexibly set in the range of 0<h≤1. When the h value increases, the number of iterations and the number of clusters will be reduced. When the h value decreases, the number of iterations and the number of clusters will be increased.

[0113] In one implementation, in the above step A205, the first cluster center set includes at least two cluster centers, and the data samples in the first data set are clustered according to the first cluster center set. Figure 5 , including the following steps:

[0114] Step A2051: Determine the cluster center to which the data samples in the first data set belong according to the first distance information and the second distance information.

[0115] The first distance information is the distance information between the cluster centers in the first cluster center set; the second distance information is the distance information between the data samples in the first data set and the cluster centers in the first cluster center set;

[0116] For example, the distance between cluster center c1 and cluster center c2 is d(c1, c2), and a data sample x in the first data set D1 i The distance from the cluster center c1 is d(x i ,c1), in d(c1,c2), d(x i ,c1) conforms to the inequality relationship "2d(x i ,c1)<d(c1,c2)”, determine the data sample x i The cluster center is c1.

[0117] Step A2052: Clustering the data samples in the first data set according to the cluster centers to which the data samples in the first data set belong.

[0118] For example, i is sequentially taken from 1 to N, and the N data samples x in the first data set D1 are sorted according to the first cluster center set C1. i Perform clustering to obtain M clusters corresponding to the M cluster centers in the first cluster center set C1.

[0119] In one implementation, the clustering result of the first data set includes multiple clusters. After the above step A205, see Figure 6 , the above data processing method further includes the following steps:

[0120] Step A206: Sort the multiple data samples in the cluster according to the preset keywords.

[0121] For example, a representative keyword is selected to sort the data sets in the cluster.

[0122] Step A207: traverse the sorted multiple data samples in the sliding window, delete duplicate data samples with similarity greater than a preset value in the multiple data samples, and obtain a second data set.

[0123] Exemplarily, a sliding window of size n is set on the sorted data set, multiple data samples in the sliding window are traversed and checked, and data samples with high similarity in the sliding window are removed.

[0124] Exemplarily, the sliding window moves downward in the sorting direction. The data sample entering the sliding window at the current moment is compared with the n-1 data samples entering the window at the previous moment to see if they are duplicate records. The above data cleaning steps are repeated to obtain a second data set, which is a data set that does not contain duplicate records.

[0125] In an embodiment of the present application, a clustering algorithm is used to merge duplicate records into one cluster, and a nearest neighbor sorting algorithm is used to remove duplicate records in the same cluster, thereby reducing the number of comparisons between data records and improving the accuracy of duplicate record identification and data cleaning efficiency.

[0126] In an embodiment of the present application, based on clustering the data samples in the first data set D1, the clustering results of the first data set are obtained. The clustering results of the first data set include multiple clusters. The repeatedly recorded data samples all exist in the same cluster, and the similarity of the data samples in the same cluster is relatively high.

[0127] Figure 7 A flow chart of another data processing method provided in an embodiment of the present application is shown.

[0128] Based on the same technical concept as the above embodiment, see Figure 7 The data processing method provided in the embodiment of the present application includes the following steps:

[0129] Step A301: Obtain a first cluster center set of a first data set.

[0130] For example, two data samples x corresponding to the maximum distance between data samples are selected from the first data set D1. a and x b As the initial cluster center, the data sample x a and data sample x b The distance between them is denoted as d(x a ,x b ),d(x a ,x b ) satisfies d(x a ,x b )=max{d(x i ,x j ), i, j∈1,2,…,n}, data sample x a As the first cluster center, denoted as c1; data sample x bAs the second cluster center, it is recorded as c2; cluster centers c1 and c2 form the first cluster center set ( Figure 3 ).

[0131] Step A302: clustering the data samples in the first data set according to the first cluster center set.

[0132] For the detailed process of step A302, please refer to the above step A201 and will not be repeated here.

[0133] Step A303: Obtain distance information between each cluster center and the data samples in the clusters corresponding to each cluster center.

[0134] For the detailed process of step A303, please refer to the above step A202 and will not be repeated here.

[0135] Step A304: Determine the next cluster center of the first data set according to the distance information to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center.

[0136] For the detailed process of step A304, please refer to the above step A203 and will not be repeated here.

[0137] Step A305: Determine whether the second cluster center set meets a preset condition.

[0138] When the second cluster center set does not meet the preset condition, the following step A306 is performed; when the second cluster center set meets the preset condition, the following step A307 is performed.

[0139] Step A306: Update the first cluster center set to the second cluster center set, and continue to execute the above step A302.

[0140] For the detailed process of step A306, please refer to the above step A204 and will not be repeated here.

[0141] Step A307: Clustering the data samples in the first data set according to the second cluster center set to obtain a clustering result of the first data set.

[0142] For the detailed process of step A307, please refer to the above step A205 and will not be repeated here.

[0143] Step A308: Sort the multiple data samples in the cluster according to the preset keywords.

[0144] For the detailed process of step S308, please refer to the above step S206, which will not be described here in detail.

[0145] Step A309: traverse the sorted multiple data samples in the sliding window, delete duplicate data samples with similarity greater than a preset value in the multiple data samples, and obtain a second data set.

[0146] For the detailed process of step S309, please refer to the above step S207 and will not be described in detail here.

[0147] In related technologies, the data processing time of the clustering process is mainly used to calculate the distance between the data sample and the cluster center. For large data volumes, directly determining the cluster center to which the data sample belongs based on the distance information between the data sample and each cluster center is inefficient. In the embodiments of the present application, the cluster center to which the data sample belongs is determined by the triangle inequality principle, avoiding unnecessary distance calculations and comparisons, reducing the amount of computation in the clustering process, and improving the efficiency of cluster analysis.

[0148] Figure 8 A structural diagram of a data processing device provided in an embodiment of the present application is shown.

[0149] Based on the same technical concept as the above embodiment, see Figure 8 , an embodiment of the present application provides a data processing device, comprising:

[0150] A first processing module 810 is configured to cluster the data samples in the first data set according to the first cluster center set to obtain a cluster corresponding to each cluster center in the first cluster center set;

[0151] The second processing module 820 is used to obtain distance information between each cluster center and the data samples in the clusters corresponding to each cluster center;

[0152] Determine the next cluster center of the first data set according to the distance information to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center;

[0153] A third processing module 830 is configured to update the first cluster center set to the second cluster center set when the second cluster center set does not meet a preset condition;

[0154] When the second cluster center set meets the preset condition, the data samples in the first data set are clustered according to the second cluster center set to obtain a clustering result of the first data set.

[0155] In related technologies, when clustering a data set, the number of clusters must be manually set. This makes it difficult to estimate the number of clusters when the data volume is large, and it can easily lead to local optimality, affecting the efficiency of cluster analysis and the accuracy of clustering results. In an embodiment of the present application, clustering is stopped when the set of cluster centers corresponding to a data set meets a preset condition, the number of clusters corresponding to the data set is automatically determined, and the clustering results of the data set are dynamically updated based on the number of clusters. This improves the efficiency of cluster analysis and the accuracy of clustering results.

[0156] In one implementation, the second processing module 820 is configured to determine the next cluster center of the first data set according to the distance information, including:

[0157] Obtaining maximum value information of distance values ​​from the distance information corresponding to the multiple cluster centers, determining the maximum value of the distance value from the maximum value information of the distance values ​​corresponding to the multiple cluster centers, and obtaining a data sample corresponding to the maximum value of the distance value;

[0158] The data sample corresponding to the maximum distance value is determined as the next cluster center of the first data set.

[0159] In one implementation, the preset condition is that the distance between each cluster center in the second cluster center set and the data sample in the cluster corresponding to the cluster center is less than a first threshold.

[0160] In one implementation, the first cluster center set includes at least two cluster centers, and the first processing module 810 is configured to cluster the data samples in the first data set according to the first cluster center set, including:

[0161] Determine the cluster center to which the data samples in the first data set belong according to the first distance information and the second distance information;

[0162] The first distance information is the distance information between the cluster centers in the first cluster center set; the second distance information is the distance information between the data samples in the first data set and the cluster centers in the first cluster center set;

[0163] The data samples in the first data set are clustered according to the cluster centers to which the data samples in the first data set belong.

[0164] In one implementation, the clustering result of the first data set includes multiple clusters, and the third processing module 830 is further configured to:

[0165] Sort multiple data samples in the cluster according to preset keywords;

[0166] The sorted multiple data samples are traversed in the sliding window, and duplicate data samples with a similarity greater than a preset value among the multiple data samples are deleted to obtain a second data set.

[0167] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated on here.

[0168] Figure 9 A functional structure diagram of an electronic device provided in an embodiment of the present application is shown.

[0169] Based on the same technical concept as the above embodiment, see Figure 9 , an electronic device 900 provided in an embodiment of the present application includes: a memory 901 and a processor 902; wherein,

[0170] Memory 901, used to store computer programs and data;

[0171] The processor 902 is configured to execute a computer program stored in the memory to implement any one of the data processing methods in the above embodiments.

[0172] In practical applications, the memory 901 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 902.

[0173] The processor 902 may be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor. It is understood that for different augmented reality cloud platforms, the electronic device used to implement the above processor functions may also be other, and this embodiment of the application does not specifically limit this.

[0174] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0175] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0176] It should be understood that "one embodiment" or "an embodiment" mentioned in the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0177] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0180] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0181] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0182] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (for example, a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0183] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: include: Clustering the data samples in the first data set according to the first cluster center set to obtain a cluster cluster corresponding to each cluster center in the first cluster center set; The data samples in the first data set are data samples in a database; Obtaining distance information between each cluster center and data samples in the clusters corresponding to each cluster center; Determine the next cluster center of the first data set according to the distance information to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center; the next cluster center is a data sample in the first data set; When the second cluster center set does not meet the preset condition, updating the first cluster center set to the second cluster center set; When the second cluster center set meets a preset condition, clustering the data samples in the first data set according to the second cluster center set to obtain a clustering result of the first data set; Duplicate data of different primary keys in the database are identified according to the clustering result.

2. The method according to claim 1, characterized in that The determining a next cluster center of the first data set according to the distance information includes: Obtaining maximum value information of distance values ​​from the distance information corresponding to the plurality of cluster centers, determining a maximum value of the distance value from the maximum value information of the distance values ​​corresponding to the plurality of cluster centers, and obtaining a data sample corresponding to the maximum value of the distance value; The data sample corresponding to the maximum value of the distance value is determined as the next cluster center of the first data set.

3. The method according to claim 1, characterized in that The preset condition is that the distance between each cluster center in the second cluster center set and the data sample in the cluster corresponding to the cluster center is less than a first threshold.

4. The method according to claim 1, wherein The first cluster center set includes at least two cluster centers, and clustering the data samples in the first data set according to the first cluster center set includes: Determine the cluster centers to which the data samples in the first data set belong based on first distance information and second distance information; wherein the first distance information is distance information between cluster centers in the first cluster center set; and the second distance information is distance information between the data samples in the first data set and the cluster centers in the first cluster center set; The data samples in the first data set are clustered according to the cluster centers to which the data samples in the first data set belong.

5. The method according to any one of claims 1 to 4, characterized in that The clustering result of the first data set includes a plurality of clusters, and the method further includes: Sorting the plurality of data samples in the cluster according to preset keywords; The sorted multiple data samples are traversed in the sliding window, and duplicate data samples with similarity greater than a preset value in the multiple data samples are deleted to obtain a second data set.

6. A data processing device, characterized in that: include: A first processing module is configured to cluster the data samples in the first data set according to the first cluster center set to obtain a cluster cluster corresponding to each cluster center in the first cluster center set; The data samples in the first data set are data samples in a database; A second processing module is used to obtain distance information between each cluster center and the data samples in the clusters corresponding to each cluster center; Determine the next cluster center of the first data set according to the distance information to obtain a second cluster center set, where the second cluster center set includes the first cluster center set and the next cluster center; the next cluster center is a data sample in the first data set; a third processing module, configured to update the first cluster center set to the second cluster center set when the second cluster center set does not meet a preset condition; When the second cluster center set meets a preset condition, clustering the data samples in the first data set according to the second cluster center set to obtain a clustering result of the first data set; Duplicate data of different primary keys in the database are identified according to the clustering result.

7. The device according to claim 6, characterized in that The second processing module is configured to determine a next cluster center of the first data set according to the distance information, including: Obtaining maximum value information of distance values ​​from the distance information corresponding to the plurality of cluster centers, determining a maximum value of the distance value from the maximum value information of the distance values ​​corresponding to the plurality of cluster centers, and obtaining a data sample corresponding to the maximum value of the distance value; The data sample corresponding to the maximum value of the distance value is determined as the next cluster center of the first data set.

8. The device according to claim 6, characterized in that The preset condition is that the distance between each cluster center in the second cluster center set and the data sample in the cluster corresponding to the cluster center is less than a first threshold.

9. An electronic device, characterized in that: include: At least one processor, at least one memory, and computer program instructions stored in the memory, which implement the method according to any one of claims 1 to 5 when the computer program instructions are executed by the processor.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images