Clustering method and clustering equipment

Abstract

The present application provides a clustering method and clustering device. The clustering method can be used in the field of big data. The clustering method includes: dividing N sample data into M data blocks including the same number of sample data; then clustering each data block in the M data blocks to obtain K first clusters and K locally optimal first centroids; finally, based on the similarity between the K locally optimal first centroids and K second centroids in each data block in the M data blocks, M*K first centroids are divided into K second clusters, and for each second cluster, based on the value of each first centroid among the j first centroids included in the second cluster and the number of sample data included in the first cluster to which each first centroid belongs, different first clusters are merged to obtain K target centroids and cluster the N sample data based on the K target centroids. The clustering method can improve the clustering effect.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a clustering method and a clustering device. Background Art

[0002] With the development of big data, how to analyze and utilize massive amounts of data has become a key focus. In the process of data analysis, it is often necessary to use classification methods to train large amounts of data to obtain classification models. Finally, the classification model is used to predict new data to determine the category of the new data.

[0003] K-means clustering is a commonly used classification method. The K-means clustering method proceeds as follows: First, the sample data to be clustered is input into a clustering device. The clustering device then randomly selects K sample data from the sample data to be clustered as centroids, where K represents the number of clusters. The device then calculates the distance between each sample data and each centroid, assigning each sample data to the centroid closest to it, ultimately forming K sample clusters. For each sample cluster, the clustering device then re-determines the centroid based on all the sample data included in the cluster, until the termination condition for K-means clustering is met, at which point the clustering device outputs the final K centroids. After the clustering device outputs the final K centroids, to determine the category of new sample data, the distance from the new sample data to each of the K centroids is calculated, and the category of the new sample data is then determined based on the K distances.

[0004] However, when clustering equipment uses the above-mentioned K-means clustering method to cluster sample data, the clustering results obtained are not ideal. Therefore, how clustering equipment can cluster sample data to improve the clustering effect has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides a clustering method and a clustering device, which can improve the clustering effect.

[0006] In a first aspect, an embodiment of the present application provides a clustering method, which is applied to a clustering device, including: obtaining N sample data, where N is a positive integer; dividing the N sample data into M data blocks, wherein any two data blocks in the M data blocks include the same number of sample data; clustering the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method, wherein K indicates a preset number of clusters, and i takes a value of M from 1; obtaining a first centroid corresponding to each of the K first clusters corresponding to the i-th data block and the number of sample data included in each first cluster; and obtaining a first centroid based on each first centroid of the M*K first centroids corresponding to the M data blocks and any second centroid of the K second centroids. The similarity of the first centroid among the M*K centroids is determined, and each first centroid among the M*K centroids is assigned to the second centroid with the largest similarity, so as to divide the M*K first centroids into K second clusters, and the K second centroids are K centroids re-determined by the M*K first centroids; based on the target mapping relationship, the target centroid corresponding to each second cluster in the K second clusters is determined, and the target mapping relationship includes the relationship between the target centroid, the number of sample data included in the first cluster to which each first centroid among the j first centroids included in each second cluster belongs, and the value of each first centroid, where j is a positive integer; the N sample data are clustered based on the K target centroids, and the K target centroids correspond one-to-one to the K second clusters.

[0007] In the clustering method provided in this embodiment, the clustering device first divides the received N sample data into M data blocks with the same number of sample data, and then clusters each data block using the mean value clustering algorithm to obtain K first centroids corresponding to each data block. It can be understood that for M data blocks, a total of M*K first centroids will be obtained. Afterwards, the clustering device re-determines K second centroids from the M*K first centroids, and regards the M*K first centroids as data to be clustered. Based on the similarity between each first centroid in the M*K first centroids and any second centroid in the K second centroids, the M*K first centroids are divided into K second clusters. The target centroid is then determined based on the number of sample data included in the first cluster to which each first centroid belongs among the j first centroids included in each second cluster and the value of each first centroid. In other words, it can be considered that the target centroid is determined by fusing the different first clusters included in the second cluster.

[0008] It is understandable that the method provided in this embodiment first divides N sample data into multiple data blocks, clusters each data block, and then fuses the clustering results of each data block. In other words, it can be considered that clustering is first performed based on local sample data, and then the local clustering results are fused to obtain a global clustering result. Therefore, this method can reduce the tendency of existing clustering algorithms to fall into local optimal solutions.

[0009] In combination with the first aspect, in one possible implementation, the value of the target centroid is equal to the ratio of the first target value to the second target value, the first target value is obtained by the product of the number of sample data included in the first cluster to which each first centroid belongs and the value of each first centroid, and the second target value is obtained by summing the number of sample data included in the j first clusters corresponding to the j first centroids.

[0010] As an example, the target mapping relationship satisfies the following relationship:

[0011]

[0012] Among them, C represents the target center of mass, C j represents the value of the jth first centroid, x j Indicates the number of sample data included in the first cluster to which the j-th first centroid belongs.

[0013] In combination with the first aspect, in a possible implementation, the sample data in the i-th data block of the M data blocks are clustered into K first clusters using the mean clustering method, including: determining, from the i-th data block, sample data whose similarity with the first sample data is greater than or equal to a preset threshold, the first sample data including any sample data randomly selected from the i-th data block; forming a first sample cluster with the first sample data and the sample data whose similarity with the first sample data is greater than or equal to a preset threshold; determining the center value of the first sample cluster as a first initial centroid; and clustering the sample data in the i-th data block into K first clusters based on the first initial centroid.

[0014] In this implementation, when determining K initial centroids for the i-th data block, one of the initial centroids is not selected randomly, but is selected by taking the center of the first sample cluster formed by the first sample data and the sample data whose similarity with the first sample data is greater than or equal to a preset threshold as the selected initial centroid. It can be understood that by taking the center of the first sample cluster as the selected initial centroid, the probability that the selected initial centroid is an isolated point or noise data can be reduced, thereby improving the convergence speed of the i-th data block when using the K-means clustering algorithm.

[0015] In combination with the first aspect, in a possible implementation manner, the center value of the first sample cluster is equal to the average value of all sample data included in the first sample cluster.

[0016] In combination with the first aspect, in a possible implementation manner, dividing the N sample data into M data blocks includes: dividing the N sample data into M data blocks based on a distributed file system.

[0017] In combination with the first aspect, in a possible implementation, obtaining N sample data includes: obtaining a first data set, the first data set including L sample data, where L is a positive integer greater than or equal to N; and determining the N sample data from the L sample data.

[0018] In a second aspect, the present application provides a clustering device, comprising: a receiving module for acquiring N sample data, where N is a positive integer; a processing module for dividing the N sample data into M data blocks, wherein any two data blocks in the M data blocks include the same number of sample data; the processing module is further used to cluster the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method, wherein K indicates a preset number of clusters, and i takes a value of M from 1; the receiving module is further used to obtain a first centroid corresponding to each of the K first clusters corresponding to the i-th data block and the number of sample data included in each first cluster; the processing module is further used to obtain a first centroid corresponding to each of the M*K first centroids corresponding to the M data blocks and a first centroid corresponding to the K second clusters based on the first centroid of each of the M*K first centroids corresponding to the M data blocks. The similarity of any second centroid among the centroids is used to assign each first centroid among the M*K centroids to the second centroid with the largest similarity, so as to divide the M*K first centroids into K second clusters, and the K second centroids are K centroids re-determined by the M*K first centroids; the processing module is also used to determine the target centroid corresponding to each second cluster in the K second clusters based on the target mapping relationship, and the target mapping relationship includes the relationship between the target centroid, the number of sample data included in the first cluster to which each first centroid among the j first centroids included in each second cluster belongs, and the value of each first centroid, where j is a positive integer; the processing module is also used to cluster the N sample data based on the K target centroids, and the K target centroids correspond one-to-one to the K second clusters.

[0019] In combination with the second aspect, in one possible implementation, the value of the target centroid is equal to the ratio of the first target value to the second target value, the first target value is obtained by the product of the number of sample data included in the first cluster to which each first centroid belongs and the value of each first centroid, and the second target value is obtained by summing the number of sample data included in the j first clusters corresponding to the j first centroids.

[0020] As an example, the target mapping relationship satisfies the following relationship:

[0021]

[0022] Among them, C represents the target center of mass, C j represents the value of the jth first centroid, x j Indicates the number of sample data included in the first cluster to which the j-th first centroid belongs.

[0023] In combination with the second aspect, in a possible implementation, the processing module is specifically used to: determine, from the i-th data block, sample data whose similarity with the first sample data is greater than or equal to a preset threshold, the first sample data including any sample data randomly selected from the i-th data block; form a first sample cluster with the first sample data and sample data whose similarity with the first sample data is greater than or equal to a preset threshold; determine the center value of the first sample cluster as a first initial centroid; and cluster the sample data in the i-th data block into K first clusters based on the first initial centroid.

[0024] In combination with the second aspect, in a possible implementation, the center value of the first sample cluster is equal to the average value of all sample data included in the first sample cluster.

[0025] In combination with the second aspect, in a possible implementation manner, the processing module is specifically configured to: divide the N sample data into M data blocks based on a distributed file system.

[0026] In combination with the second aspect, in a possible implementation, the receiving module is specifically used to: obtain a first data set, the first data set including L sample data, L is a positive integer greater than or equal to N; and determine the N sample data from the L sample data.

[0027] In a third aspect, the present application provides a clustering device comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions in the memory to execute the method described in the first aspect or any possible implementation thereof.

[0028] In a fourth aspect, the present application provides a chip comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation thereof.

[0029] In a fifth aspect, the present application provides a computer-readable medium, wherein the computer-readable medium stores program code for computer execution, wherein the program code includes instructions for executing the method described in the first aspect or any possible implementation thereof.

[0030] In a sixth aspect, the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the method described in the first aspect or any possible implementation method thereof.

[0031] Among them, the technical effects brought about by any implementation method of the second to sixth aspects can refer to the technical effects brought about by any possible implementation method of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 A structural diagram of an application scenario provided for an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the process of a clustering method provided in one embodiment of the present application;

[0035] Figure 3 A schematic diagram of a process of a clustering method provided in another embodiment of the present application;

[0036] Figure 4 A schematic structural diagram of a clustering device provided in one embodiment of the present application;

[0037] Figure 5 A schematic diagram of the process of a clustering device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0039] With the development of big data, how to analyze and utilize massive amounts of data has become a key focus. In the process of data analysis, it is often necessary to use classification methods to train large amounts of data to obtain classification models. Finally, the classification model is used to predict new data to determine the category of the new data.

[0040] For example, Figure 1 This is a structural diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, in this application scenario, a sample database 101 and a classification model 102 are included. The sample database 101 includes a large amount of sample data. A clustering device is used to train the large amount of sample data in the sample database 101 using a classification method to obtain a classification model 102. New sample data can then be predicted using the classification model 102 to obtain the category of the new sample data. It should be noted that the clustering device described in this application may also be referred to as a clustering apparatus, which does not constitute a limitation of this application.

[0041] Currently, K-means clustering is a commonly used classification method. The K-means clustering method operates as follows: First, the sample data to be clustered is input into a clustering device. The clustering device then randomly selects K sample data from the sample data to be clustered as centroids, where K represents the number of clusters. The device then calculates the distance between each sample data and each centroid, assigning each sample data to the centroid closest to it, ultimately forming K sample clusters. For each sample cluster, the clustering device then re-determines the centroid based on all the sample data included in the cluster, until the termination condition for K-means clustering is met, at which point the clustering device outputs the final K centroids. After the clustering device outputs the final K centroids, in order to determine the category of new sample data, the distance from the new sample data to each of the K centroids is calculated, and the category of the new sample data is then determined based on the K distances.

[0042] However, when clustering data using the K-means clustering method, clustering devices often fall into a local optimum. Therefore, when processing massive amounts of text data, the final clustering results may not represent the overall clustering, resulting in suboptimal clustering results. Therefore, improving clustering performance has become a pressing technical issue.

[0043] In view of this, an embodiment of the present application provides a clustering method, which divides N sample data into M data blocks with the same number of sample data; then clusters each data block in the M data blocks to obtain K first clusters and K locally optimal first centroids; finally, based on the similarity between the K locally optimal first centroids and the K second centroids in each data block in the M data blocks, the M*K first centroids are divided into K second clusters, and for each second cluster, based on the value of each first centroid in the j first centroids included in the second cluster, and the number of sample data included in the first cluster to which each first centroid belongs, different first clusters are fused, thereby obtaining K target centroids and clustering the N sample data based on the K target centroids.

[0044] Figure 2 This is a schematic diagram of the process of clustering method provided in one embodiment of the present application. Figure 2 As shown, the method of this embodiment may include S201, S202, S203, S204, S205, S206 and S207. The method may be executed by a clustering device.

[0045] S201, obtain N sample data, where N is a positive integer.

[0046] In this embodiment, the N sample data can be considered as sample data for training a classification model. It should be noted that obtaining N sample data in this embodiment is also referred to as receiving N sample data, which does not constitute a limitation of this application.

[0047] In one implementation, the N sample data received by the clustering device may be N sample data input by a user through an input device, or may be N sample data received by the clustering device from other devices, or may be N sample data received by a module in the clustering device from other modules.

[0048] It should be noted that this embodiment does not limit the specific attributes of the N sample data, and may be data of text attributes, image attributes, or data of other attributes.

[0049] S202: Divide the N sample data into M data blocks, wherein any two data blocks in the M data blocks include the same number of sample data.

[0050] In this embodiment, after obtaining N sample data, the clustering device divides the N sample data into M data blocks. Furthermore, any two of the M data blocks contain the same number of sample data. For example, if the clustering device obtains 100 sample data and divides the 100 sample data into 10 data blocks, each data block will contain 10 sample data.

[0051] In specific implementation, in order to obtain M data blocks including the same number of sample data, the clustering device can upload the N sample data to the Hadoop distributed file system (Hadoop distributed file system, HDFS) after receiving N sample data. At this time, the N sample data will be divided into M data blocks of the same size (that is, any two data blocks in the M data blocks include the same number of sample data).

[0052] S203 , clustering the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method, where K indicates a preset number of clusters, and i ranges from 1 to M.

[0053] In this embodiment, each of the M data blocks is clustered into K first clusters using the mean clustering algorithm. That is, for any data block, the clustering device will use the mean clustering algorithm to cluster the data block into K clusters.

[0054] It should be noted that this embodiment does not limit the method of selecting the initial centroid when the clustering device clusters each data block into K first clusters using the value clustering method.

[0055] Exemplarily, in one implementation, the clustering device clusters the sample data in the i-th data block among the M data blocks into K first clusters using a mean clustering method, including: determining, from the i-th data block, sample data whose similarity with the first sample data is greater than or equal to a preset threshold, the first sample data including any sample data randomly selected from the i-th data block; forming a first sample cluster with the first sample data and the sample data whose similarity with the first sample data is greater than or equal to a preset threshold; determining the center value of the first sample cluster as a first initial centroid; and clustering the sample data in the i-th data block into K first clusters based on the first initial centroid.

[0056] It should be understood that when the similarity between two sample data is greater than or equal to a preset threshold, it means that the two sample data are very similar. Therefore, in this implementation, if the similarity between a certain sample data in the data block and the first sample data is greater than or equal to the preset threshold, it means that the certain sample data and the first sample data are very similar, that is, there is a probability that they may be classified into the same category. Based on this, in this implementation, the first sample data and all sample data in the data block whose similarity with the first sample data is greater than or equal to the preset threshold are classified into one category (that is, forming a first sample cluster), and then the center value of the first sample cluster is determined as the first initial centroid. It can be understood that in this implementation, the first initial centroid among the K initial centroids selected by the clustering device is not randomly selected, but by using the center of the first sample cluster as the selected initial centroid, the probability that the selected initial centroid is an isolated point or noise data can be reduced.

[0057] For example, after determining the first initial centroid, the clustering device can select a sample data point from the data block that is farthest from the first initial centroid as a second initial centroid neighboring point, and simultaneously delete sample data points in the data block whose similarity to the second initial centroid neighboring point is greater than a set threshold, and use the center of these sample data as the second initial centroid. In this case, the clustering device can cluster the N sample data into K clusters based on the first initial centroid and the second initial centroid.

[0058] For example, after determining the first and second initial centroids, the clustering device can further identify the sample data point with the largest distance to the selected initial centroid as the neighboring point of the next initial centroid, and simultaneously delete the sample data points in the data block whose similarity to the neighboring point of the next initial centroid exceeds a set threshold. The above operation is repeated to ultimately obtain K initial centroids. The obtained K initial centroids are then used as the initial centroids to cluster the sample data in the block data. After multiple iterations, until the K centroids no longer change or the upper limit of the iteration number is reached, the clustering device completes clustering the sample data in the data block.

[0059] S204 , obtaining a first centroid corresponding to each of the K first clusters corresponding to the i-th data block and the number of sample data included in each first cluster.

[0060] It should be understood that after the clustering device clusters the sample data in the i-th data block among the M data blocks into K first clusters, it can obtain the optimal K centroids (i.e., K first centroids) corresponding to the i-th data block. At the same time, it can also obtain the number of sample data in the first cluster to which each first centroid among the K first centroids belongs.

[0061] S205, based on the similarity between each first centroid among the M*K first centroids corresponding to the M data blocks and any second centroid among the K second centroids, assign each first centroid among the M*K centroids to the second centroid with the greatest similarity, so as to divide the M*K first centroids into K second clusters, and the K second centroids are K centroids re-determined through the M*K first centroids.

[0062] It should be understood that after the sample data in each of the M data blocks are clustered into K first clusters, the optimal K first centroids corresponding to each data block are obtained, as well as the number of sample data in the first cluster to which each of the K first centroids corresponding to each data block belongs. In other words, for M data blocks, a total of M*K first centroids are obtained.

[0063] In this embodiment, the K second centroids are K centroids re-determined from the M*K first centroids. It should be noted that this embodiment does not limit how the clustering device re-selects the K second centroids from the M*K first centroids.

[0064] For example, in a possible implementation, the clustering device may randomly select K first centroids from M*K first centroids as K second centroids.

[0065] For example, in another possible implementation, the clustering device may first form an array of the M*K first centroids. It is understood that the M*K first centroids included in this array can be considered as M*K sample data. Then, the array of the M*K first centroids is formed and the following process is performed on the array to determine K second centroids: a sample data point is randomly selected from the array as the initial centroid neighboring point, and other sample data points in the array whose similarity to the sample data point exceeds a set threshold are deleted. The centers of these sample data points are used as the first second centroid. Then, a sample data point farthest from the first second centroid is selected from the array as the second centroid neighboring point. Sample data points in the array whose similarity to the second centroid neighboring point exceeds a set threshold are deleted, and the centers of these sample data points are used as the second second centroid. Then, the sample point with the largest distance to the selected second centroid is found as the next centroid neighboring point, and sample data points in the array whose similarity to the next centroid neighboring point exceeds a set threshold are deleted. The above process is repeated to ultimately obtain K second centroids.

[0066] In this embodiment, after determining the M*K first centroids, the similarity between each first centroid of the M*K first centroids and any second centroid of the K second centroids is calculated, and then each first centroid of the M*K first centroids is divided into K second clusters by assigning the first centroid of the M*K first centroids to the second centroid with the greatest similarity. That is, in this embodiment, after determining the M*K first centroids, K centroids (i.e., K second centroids) are first reselected from the M*K first centroids, and then each first centroid of the M*K first centroids is assigned to the nearest second centroid based on the similarity between each first centroid and each second centroid of the K second centroids. That is, in this embodiment, after determining the M*K first centroids, clustering is performed using the M*K first centroids as sample data to be clustered and the K second centroids as initial centroids, thereby obtaining K second clusters.

[0067] S206, based on the target mapping relationship, determine the target centroid corresponding to each of the K second clusters, the target mapping relationship includes the relationship between the target centroid, the number of sample data included in the first cluster to which each first centroid belongs among the j first centroids included in each second cluster, and the value of each first centroid, where j is a positive integer.

[0068] In this embodiment, after the clustering device obtains K second clusters, each second cluster includes at least one first centroid. Moreover, for any second cluster, the first centroid included therein has the same number of sample data as the first cluster to which the first centroid belongs.

[0069] It should be understood that in the process of clustering and updating the centroid, the method of adding and averaging is always used to obtain the new centroid. In the process of adding, each sample data in the cluster is an equal point, that is, a point with the same weight, so the average value obtained by adding them up and dividing by their total number is the new centroid of the cluster. If each sample point has a weight, it would not be so easy to obtain the true "center" by simply adding and averaging. Therefore, in one implementation, the target mapping relationship in this embodiment includes the relationship between the target centroid, the number of sample data included in the first cluster to which each first centroid belongs among the j first centroids included in each second cluster, and the value of each first centroid.

[0070] In specific implementation, the target mapping relationship in this embodiment satisfies the following relationship:

[0071]

[0072] Among them, C represents the target center of mass, C j represents the value of the jth first centroid, x jIndicates the number of sample data included in the first cluster to which the j-th first centroid belongs.

[0073] For example, assume that there are two first clusters, namely cluster x and cluster y, the number of sample data included in cluster x is X and the center point of cluster x is C x , the number of sample data included in cluster y is Y and the center point of cluster y is C y Then after fusing the x cluster and the y cluster, the new center point (i.e., the target centroid) corresponding to the fusion of the two clusters is:

[0074]

[0075] In this embodiment, for any second cluster, the target centroid corresponding to the second cluster can be re-determined based on the target mapping relationship. That is, it can be considered that the target centroid is determined by fusing the first clusters corresponding to the j first centroids included in the second cluster.

[0076] S207 , clustering the N sample data based on the K target centroids, where the K target centroids correspond one-to-one to the K second clusters.

[0077] It can be understood that the clustering device will determine a target centroid for each of the K second clusters. Therefore, a total of K target centroids will be determined for the K second clusters.

[0078] In this embodiment, after determining K target centroids, the clustering device may cluster the N sample data based on the K target centroids.

[0079] Specifically, after determining the K target centroids, the clustering device can calculate the distance from each sample data in the N sample data to each target centroid in the K target centroids, and then determine which category each sample data belongs to based on the distance from each sample data in the N sample data to each target centroid in the K target centroids.

[0080] It should be noted that S205 to S207 in this embodiment can also be executed iteratively. For example, after determining K target centroids for the first time, the above S205 to S207 can be repeatedly executed to obtain more accurate K target centroids, and then the N sample data can be clustered based on the more accurate K target centroids.

[0081] In the clustering method provided in this embodiment, the clustering device first divides the received N sample data into M data blocks containing an equal number of sample data. Then, each data block is clustered using the mean value clustering algorithm to obtain K first centroids corresponding to each data block. It can be understood that for M data blocks, a total of M*K first centroids are obtained. Thereafter, the clustering device re-determines K second centroids from the M*K first centroids, and considers these M*K first centroids as the data to be clustered. Based on the similarity between each first centroid in the M*K first centroids and any second centroid in the K second centroids, the M*K first centroids are divided into K second clusters. After M*K first centroids are divided into K second clusters, the target centroid is determined based on the number of sample data included in the first cluster to which each first centroid belongs among the j first centroids included in each second cluster and the value of each first centroid. That is, it can be considered that a more accurate target centroid is determined by fusing the sample data corresponding to the j first centroids included in the second cluster.

[0082] It is understandable that the method provided in this embodiment first divides N sample data into multiple data blocks, then clusters each data block, and then fuses the clustering results of each data block. In other words, it can be considered that clustering is first performed based on local sample data, and then the results of local clustering are fused based on weights to obtain a global clustering result. Therefore, this method can reduce the risk of clustering algorithms falling into local optimal solutions, thereby improving clustering effects.

[0083] As an optional embodiment, the clustering device may further output a clustering result after S207 .

[0084] As an optional embodiment, the clustering device divides N sample data into M data blocks, including: dividing the N sample data into M data blocks based on a distributed file system.

[0085] For example, N sample data are partitioned into M data blocks based on HDFS. HDFS is the distributed file system for Hadoop (a distributed system infrastructure). It is one of the core projects in the Hadoop ecosystem and the foundation for data storage and management in distributed computing. For a detailed description of HDFS and Hadoop concepts, please refer to the related art and will not be repeated here.

[0086] It should be noted that this application does not limit the framework within which the clustering method described above is implemented by the clustering device. For example, implementation can be achieved through the "MapReduce" framework. Specifically, the "MapReduce" framework includes a map phase and a reduce phase (also known as a reduce phase). For the specific concepts and detailed description of the "MapReduce" framework, please refer to the description in the related art and will not be repeated here.

[0087] The following combination Figure 3 , illustrating a feasible implementation scheme when a clustering device executes the clustering method described in this application through the "MapReduce" framework.

[0088] like Figure 3 As shown in FIG, after the clustering device receives N data samples, the clustering device begins to perform operations in the Map phase. Specifically, the operations in the Map phase include:

[0089] Divide N data samples into M data blocks, then for each data block, select K initial centroids, and cluster the sample data in the data block based on the K initial centroids to obtain K locally optimal first centroids and the number of sample data included in the first cluster to which the K locally optimal first centroids belong.

[0090] As an example, a specific implementation method may be: for each data block, all sample data is read from the data block and stored in an array list. This array list is also backed up. α×k sample data are randomly selected from this array list as the candidate centroid set, where α is the sample coefficient, typically set to 10 times the value of K. A sample data is randomly selected from the candidate centroid set as the first initial centroid neighboring point. The similarity between all sample data in the candidate centroid set and this neighboring point is then calculated. All sample data with similarities greater than a set threshold are removed from the candidate centroid set, and the centers of these sample data are used as the first initial centroid. Then, from the remaining candidate centroids, a sample data point that is farthest from the selected centroid is found and used as the second centroid neighboring point. Points that are closer to the newly selected centroid, that is, sample data with a similarity threshold greater than a set threshold, are removed from the candidate centroid set, and these centers are used as the second initial centroid. Then, by the principle of minimum and maximum, the sample data farthest from the selected centroid is calculated from the remaining candidate centroids and used as the neighboring point of the next centroid. The process is repeated until K centroids are selected or the candidate centroid set is deleted. The remaining required centroids are randomly extracted from the original array minus the candidate centroid set, and the initialization process is finally completed to obtain K initial centroids. Finally, the K initial centroids just obtained are used as the initial centroids, and the backup array list is the sample data to be clustered. The clustering iteration process is performed to finally obtain K first clusters. Furthermore, the centroid of each first cluster (i.e., the first centroid) in the K first clusters corresponding to each data block and the number of sample data in the first cluster to which each first centroid belongs can be obtained.

[0091] like Figure 3 As shown in the figure, after the cluster device completes the Map phase, it can perform the Reduce phase. Specifically, the Reduce phase includes:

[0092] The data received in the Reduce stage are the K locally optimal first centroids corresponding to each data block and the number of sample data included in the first cluster to which each first centroid belongs. It is understandable that the number of first centroids received in the Reduce stage is M*K. Afterwards, the clustering device will first reselect K centroids (i.e., K second centroids) from the M*K first centroids, and then based on the similarity between each first centroid in the M*K first centroids and any second centroid in the K second centroids, the M*K first centroids will be re-divided into K second clusters, and then for each of the K second clusters, in each second cluster, based on the description in the above embodiment of this application, the first clusters corresponding to the multiple first centroids included in the second cluster are fused to re-obtain K target centroids. Furthermore, it is also possible to continue to divide in the above-mentioned similarity-based manner and update the K target centroids according to the weighted fusion method to obtain K target centroids with higher accuracy.

[0093] As an example, a feasible implementation plan may be to put all the first centroid data and the number of sample data of the corresponding first cluster into two arrays respectively, one array stores the first centroid data (referred to as the centroid array), and the other array stores the number of sample data of the corresponding first cluster. The contents at the internal positions of the two arrays form a one-to-one correspondence, and the contents of the same subscripts in the two arrays are the numerical values of the first centroid and the number of sample data included in the corresponding first cluster. It should be understood that the centroid array stores K*M number of sample data. For these data, for example, the method of selecting the initial centroid in the Map stage can be used to select K centroids (i.e., the second centroids) again, and then the remaining sample data in the centroid array are divided into the nearest second centroids according to the similarity, and then K second clusters are obtained. In each second cluster, the centroid corresponding to the second cluster (i.e., the target centroid) is calculated according to the weight fusion method proposed in the present application. Furthermore, the division can continue in the above-mentioned similarity-based manner and the K target centroids can be updated according to the weight fusion method to obtain K target centroids with higher accuracy.

[0094] Figure 4 This is a schematic diagram of the structure of a clustering device provided by one embodiment of the present application. Figure 4 As shown, the device 400 includes: a receiving module 401, a processing module 402 and an output module 403.

[0095] The receiving module 401 is used to obtain N sample data, where N is a positive integer; the processing module 402 is used to divide the N sample data into M data blocks, where any two data blocks in the M data blocks include the same number of sample data; the processing module 402 is also used to cluster the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method, where K indicates a preset number of clusters and i takes a value of M from 1; the receiving module 401 is also used to obtain the first centroid corresponding to each of the K first clusters corresponding to the i-th data block and the number of sample data included in each first cluster; the processing module 402 is also used to calculate the similarity between each first centroid of the M*K first centroids corresponding to the M data blocks and any second centroid of the K second centroids. , assigning each first centroid among the M*K centroids to the second centroid with the largest similarity, so as to divide the M*K first centroids into K second clusters, and the K second centroids are K centroids re-determined by the M*K first centroids; the processing module 402 is also used to determine the target centroid corresponding to each second cluster in the K second clusters based on the target mapping relationship, and the target mapping relationship includes the relationship between the target centroid, the number of sample data included in the first cluster to which each first centroid among the j first centroids included in each second cluster belongs, and the value of each first centroid, where j is a positive integer; the output module 403 is used to output K target centroids, and the K target centroids correspond one-to-one to the K second clusters; the processing module 402 is also used to cluster the N sample data based on the K target centroids.

[0096] In one possible implementation, the value of the target centroid is equal to the ratio of the first target value to the second target value, where the first target value is obtained by the product of the number of sample data included in the first cluster to which each first centroid belongs and the value of each first centroid, and the second target value is obtained by summing the number of sample data included in the j first clusters corresponding to the j first centroids.

[0097] Exemplarily, the target mapping relationship satisfies the following relationship:

[0098]

[0099] Among them, C represents the target center of mass, C j represents the value of the jth first centroid, x j Indicates the number of sample data included in the first cluster to which the j-th first centroid belongs.

[0100] In one possible implementation, the processing module 402 is specifically used to: determine, from the i-th data block, sample data whose similarity with the first sample data is greater than or equal to a preset threshold, the first sample data including any sample data randomly selected from the i-th data block; form a first sample cluster with the first sample data and sample data whose similarity with the first sample data is greater than or equal to a preset threshold; determine the center value of the first sample cluster as a first initial centroid; and cluster the sample data in the i-th data block into K first clusters based on the first initial centroid.

[0101] In a possible implementation, the center value of the first sample cluster is equal to an average value of all sample data included in the first sample cluster.

[0102] In a possible implementation, the processing module 402 is specifically configured to: divide the N sample data into M data blocks based on a distributed file system.

[0103] In a possible implementation, the receiving module 401 is specifically configured to: acquire a first data set, where the first data set includes L sample data, where L is a positive integer greater than or equal to N; and determine the N sample data from the L sample data.

[0104] Figure 5 A structural diagram of a clustering device provided in another embodiment of the present application. Figure 5 The device shown can be used to execute the method described in any of the above embodiments.

[0105] like Figure 5 As shown, the apparatus 500 of this embodiment includes: a memory 501, a processor 502, a communication interface 503, and a bus 504. The memory 501, the processor 502, and the communication interface 503 are connected to each other via the bus 504.

[0106] The memory 501 can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 501 can store programs. When the program stored in the memory 501 is executed by the processor 502, the processor 502 is used to execute Figure 2 or Figure 3 The individual steps of the method are shown.

[0107] The processor 502 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for executing relevant programs to implement the present application. Figure 2 or Figure 3 The method shown.

[0108] The processor 502 may also be an integrated circuit chip with signal processing capabilities. Figure 2 or Figure 3 Each step of the method can be completed by an integrated logic circuit of hardware in the processor 502 or an instruction in the form of software.

[0109] The processor 502 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 502 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor.

[0110] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501 and, in combination with its hardware, completes the functions required to be executed by the units included in the device of the present application, for example, it can execute Figure 2 or Figure 3 The various steps / functions of the illustrated embodiment.

[0111] The communication interface 503 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the apparatus 500 and other devices or a communication network.

[0112] The bus 504 may include a path for transmitting information between various components of the device 500 (eg, the memory 501 , the processor 502 , and the communication interface 503 ).

[0113] It should be understood that the device 500 shown in the embodiment of the present application may be an electronic device, or may be a chip configured in the electronic device.

[0114] It should be noted that the mean clustering method and device in this application can be used in the field of big data, and can also be used in any field other than the field of big data. This application does not limit the application field of the mean clustering method and device.

[0115] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0116] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0117] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0118] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0119] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0122] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0124] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0125] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A clustering method, characterized in that: Applicable to cluster devices, including: Obtain N sample data, where N is a positive integer, and the sample data is image attribute data or text attribute data; Dividing the N sample data into M data blocks, wherein any two data blocks in the M data blocks include the same number of sample data; Clustering the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method, where K indicates a preset number of clusters and i ranges from 1 to M; Obtaining a first centroid corresponding to each of the K first clusters corresponding to the i-th data block and the number of sample data included in each first cluster; Based on the M corresponding to the M data blocks The similarity between each first centroid among the K first centroids and any second centroid among the K second centroids is calculated by Each of the K first centroids is assigned to the second centroid with the largest similarity, so as to divide the M The K first centroids are divided into K second clusters, and the K second centroids are obtained by the M The K centroids re-determined from the K first centroids; Determine a target centroid corresponding to each of the K second clusters based on a target mapping relationship, wherein the target mapping relationship includes a relationship between the target centroid, the number of sample data included in the first cluster to which each of the j first centroids included in each second cluster belongs, and the value of each first centroid, where j is a positive integer; The N sample data are clustered based on the K target centroids, where the K target centroids correspond one-to-one to the K second clusters.

2. The method according to claim 1, characterized in that The value of the target centroid is equal to the ratio of the first target value to the second target value. The first target value is obtained by the product of the number of sample data included in the first cluster to which each first centroid belongs among the j first centroids and the value of each first centroid. The second target value is obtained by summing the number of sample data included in the j first clusters corresponding to the j first centroids.

3. The method according to claim 1, characterized in that Clustering the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method includes: Determining, from the i-th data block, sample data having a similarity with first sample data greater than or equal to a preset threshold, the first sample data comprising any sample data randomly selected from the i-th data block; The first sample data and sample data having a similarity with the first sample data greater than or equal to a preset threshold form a first sample cluster; Determine the center value of the first sample cluster as a first initial centroid; Based on the first initial centroid, the sample data in the i-th data block is clustered into K first clusters.

4. The method according to claim 3, characterized in that The center value of the first sample cluster is equal to the average value of all sample data included in the first sample cluster.

5. The method according to claim 4, characterized in that The step of dividing the N sample data into M data blocks includes: The N sample data are divided into M data blocks based on a distributed file system.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of N sample data includes: Acquire a first data set, where the first data set includes L sample data, where L is a positive integer greater than or equal to N; The N sample data are determined from the L sample data.

7. A clustering device, characterized in that: include: A receiving module is used to obtain N sample data, where N is a positive integer, and the sample data is image attribute data or text attribute data; a processing module, configured to divide the N sample data into M data blocks, wherein any two of the M data blocks include the same number of sample data; The processing module is further configured to cluster the sample data in the i-th data block in the M data blocks into K first clusters using a mean clustering method, where K indicates a preset number of clusters and i ranges from 1 to M; The receiving module is further configured to obtain a first centroid corresponding to each of the K first clusters corresponding to the i-th data block and the number of sample data included in each first cluster; The processing module is further configured to: The similarity between each first centroid among the K first centroids and any second centroid among the K second centroids is calculated by Each first centroid among the K centroids is assigned to the second centroid with the largest similarity, so as to divide the M The K first centroids are divided into K second clusters, and the K second centroids are obtained by the M The K centroids re-determined from the K first centroids; The processing module is further configured to determine a target centroid corresponding to each of the K second clusters based on a target mapping relationship, wherein the target mapping relationship includes a relationship between the target centroid, the number of sample data included in the first cluster to which each of the j first centroids included in each second cluster belongs, and a value of each first centroid, where j is a positive integer; The processing module is further configured to cluster the N sample data based on K target centroids, where the K target centroids correspond one-to-one to the K second clusters.

8. A clustering device, characterized in that: include: memory and processor; The memory is used to store program instructions; The processor is configured to call program instructions in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable medium, characterized in that The computer-readable medium stores a program code for computer execution, the program code including instructions for executing the method according to any one of claims 1 to 6.

10. A computer program product comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images