MS-kmeans clustering method and device based on hadoop distributed computing

Abstract

The application relates to the technical field of data clustering, and is an MS-Kmeans clustering method and device based on Hadoop distributed computing, which comprises the following steps: building a Hadoop cluster; dividing a total data set into a plurality of sub-data sets; executing a Hadoop distributed file command, uploading the total data set and the sub-data sets to a Hadoop distributed file system, respectively clustering each sub-data set by using a Mean Shift algorithm to obtain a plurality of candidate center vectors; selecting K farthest candidate center vectors as center vectors; and clustering all data in the total data set by using a K-means clustering algorithm. The application can run the MS-Kmeans clustering algorithm in the Hadoop distributed file system, and combines the advantages of the Mean Shift algorithm and the K-means clustering algorithm, thereby improving the clustering performance and effectively reducing the clustering running time.

Description

Technical Field

[0001] This invention relates to the field of data clustering technology, specifically an MS-Kmeans clustering method and apparatus based on Hadoop distributed computing. Background Technology

[0002] With the rapid development of the internet and the advent of the information age, the daily increase in data volume has reached petabyte (PB) levels. Clustering can be used to determine the distribution of data, grouping highly similar data into the same cluster and less similar data into different clusters. Deeper analysis of these clusters can uncover valuable information from massive amounts of data. Clustering can be applied to data analysis in many computer science fields, such as text analysis, web search, pattern recognition, and artificial intelligence.

[0003] However, with the development of the internet and the widespread use of mobile devices, data is now growing exponentially. Faced with massive amounts of data, traditional clustering algorithms running on single machines can no longer meet people's needs. To improve the storage and computing capabilities of massive data, the Apache Software Foundation developed the Hadoop distributed system infrastructure. Hadoop features high reliability, high scalability, high fault tolerance, and low cost. The introduction of Hadoop has provided the conditions for processing and mining massive amounts of data. How to design a distributed system using MapReduce to implement traditional clustering algorithms running on a single machine and achieve distributed computing on the Hadoop platform is of great significance. Summary of the Invention

[0004] This invention provides an MS-Kmeans clustering method and apparatus based on Hadoop distributed computing, which overcomes the shortcomings of the prior art. It can effectively solve the problem that the existing technology cannot run traditional clustering algorithms in the Hadoop distributed system and reduce the clustering running time.

[0005] One of the technical solutions of this invention is achieved through the following measures: an MS-Kmeans clustering method based on Hadoop distributed computing, comprising:

[0006] Set up a Hadoop cluster and establish a Hadoop distributed file system;

[0007] Obtain the total dataset and divide it into several equal subsets.

[0008] Execute the Hadoop Distributed File command to upload the total dataset and sub-datasets to the Hadoop Distributed File System. Use the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors.

[0009] Select the K farthest candidate center vectors as center vectors, and use the K-means clustering algorithm to cluster all data in the total dataset;

[0010] Start the Map task to assign the data to the nearest cluster and output the clustering results.

[0011] The following are further optimizations and / or improvements to the above-mentioned technical solution:

[0012] The Mean Shift algorithm was used to cluster each subset of the dataset, resulting in multiple candidate center vectors, including:

[0013] Select M candidate center vectors from a subset of the dataset and write them line by line into a Hadoop distributed file;

[0014] For each candidate center vector X', find the data X within a high-dimensional space centered on the candidate center vector and with a radius of D. j ;

[0015] Determine the offset mean vector of the candidate center vector X', and shift the candidate center vector X' in the direction of the offset mean vector. The offset mean vector and the shifted candidate center vector are shown below:

[0016]

[0017] Where x'm is the m-th candidate center vector, M(x' m ) t The offset mean vector obtained at state t, where G is the mean vector of S. D The number of data vectors contained within the range;

[0018] (x' m ) t+1 =M(x' m ) t +(x' m ) t

[0019] Among them, (x' m ) t+1 It is the m-th candidate center vector at time t+1, that is, the new m-th candidate center vector after being moved in the direction of the offset mean vector;

[0020] Determine whether the distance between the new candidate center vector and the old candidate center vector is less than a given threshold. If so, end the Mean Shift phase of the candidate center vector and repeat the above steps until M candidate center vectors have been iterated. If not, repeat the above steps for the candidate center vector.

[0021] Repeat the above steps, iterating through each subset, to obtain Y×M candidate center vectors, where Y is the number of subsets.

[0022] For each candidate center vector X', the above process involves finding the data X within a high-dimensional space centered on the candidate center vector and with a radius of D. j ,include:

[0023] Start the first Map task to scan and read in the candidate center vectors of the current subset of data;

[0024] Initialize key-value pairs<key1,value1> The content of each line of data read in the above steps is used as value1, and the line number of the data is used as key1, and stored in a key-value pair.<key1,value1> middle;

[0025] Read the subset of data containing the current candidate center vector row by row. If the distance between a data point and one of the candidate center vectors is less than D, then move that data point to the next subset.<key2,value2> The output is in the form of key2, where key2 is equal to key1 of the candidate center vector to which the data belongs, and value2 is the data.

[0026] Start a Combine task. The Combine function combines key-value pairs with the same key. The input is the output of the Map function, and the output is...<key3,value3> Key-value pairs, where key3 is the candidate center vector identifier and value3 is the combination of all vectors with the same key3.

[0027] The above selection of K candidate center vectors that are farthest apart as center vectors includes:

[0028] Randomly select a candidate center vector x' from the candidate center vector set X'. m And add it to set C;

[0029] Calculate the distance from the candidate center vector in the candidate center vector set X' to all vectors in the set C;

[0030] If the calculated distance satisfies the center vector criterion, the candidate center vector is added to set C. The center vector criterion is as follows:

[0031] Candidate center vector x' m The maximum value that satisfies the following expression;

[0032] min(dist(x'1,c1),dist(x'1,c2),...,dist(x'1,c k ),...,min(dist(x'm ,c1),dist(x' m ,c2),...,dist(x' m ,c k ))where dist(x' m ,c k ) is x' m With c k The Euclidean distance between them;

[0033] Repeat the above steps until K vectors are selected as the initial center vectors of K clusters and written to the Hadoop distributed file.

[0034] The above-mentioned clustering of all data in the total dataset using the K-means clustering algorithm includes:

[0035] For each vector x in the dataset n Using the following formula, the Euclidean distance to the center vectors of the K clusters is calculated, and x is... n Assign to the nearest cluster;

[0036]

[0037] Start a Reduce task to recalculate the center vector of each cluster using the following formula;

[0038]

[0039] Repeat the above steps until the center vectors of the K clusters remain unchanged. Then the clustering has converged, and the final k center vectors are obtained.

[0040] The second technical solution of the present invention is achieved through the following measures: an MS-Kmeans clustering device based on Hadoop distributed computing, comprising:

[0041] Set up units, build Hadoop clusters, and establish Hadoop distributed file systems;

[0042] The dataset acquisition unit acquires the total dataset and divides it into several sub-datasets.

[0043] The first execution unit executes Hadoop distributed file commands, uploads the total dataset and sub-datasets to the Hadoop distributed file system, and uses the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors.

[0044] The second execution unit selects the K farthest candidate center vectors as center vectors and uses the K-means clustering algorithm to cluster all data in the total dataset.

[0045] The output unit initiates a Map task, assigns the data to the nearest cluster, and outputs the clustering results.

[0046] This invention discloses an MS-Kmeans clustering method based on Hadoop distributed computing. The dataset is divided into multiple subsets. For each subset, non-outlier points are selected as candidate centroids, and the Mean Shift algorithm is used to shift the candidate centroids to denser data regions. Then, K points farthest apart from the candidate centroids are selected as the initial centroids for the K-means algorithm, and K-means clustering is performed on the data to find the final cluster centroids. All data are then assigned to the nearest cluster. Therefore, this invention enables the MS-Kmeans clustering algorithm to run on the Hadoop Distributed File System, effectively reducing clustering time while improving clustering performance. Attached Figure Description

[0047] Appendix Figure 1 This is a flowchart of the method of the present invention.

[0048] Appendix Figure 2 This is a flowchart illustrating the method of clustering each subset of data using the Mean Shift algorithm in this invention.

[0049] Appendix Figure 3 This is a schematic diagram of the device structure of the present invention.

[0050] Appendix Figure 4 This is an acceleration comparison chart for Embodiment 5 of the present invention. Detailed Implementation

[0051] The present invention is not limited to the following embodiments, and the specific implementation can be determined according to the technical solution of the present invention and the actual situation.

[0052] The present invention will be further described below with reference to embodiments and accompanying drawings:

[0053] Example 1: As shown in the attached document Figure 1 As shown, this embodiment of the invention discloses an MS-Kmeans clustering method based on Hadoop distributed computing, including:

[0054] Step S101: Build a Hadoop cluster and establish a Hadoop distributed file system;

[0055] Step S102: Obtain the total dataset and divide it into several sub-datasets;

[0056] Step S103: Execute the Hadoop Distributed File command to upload the total dataset and sub-datasets to the Hadoop Distributed File System. Use the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors.

[0057] Step S104: Select the K farthest candidate center vectors as center vectors, and use the K-means clustering algorithm to cluster all data in the total dataset;

[0058] Step S105: Start the Map task, allocate the data to the nearest cluster, and output the clustering results.

[0059] This invention discloses an MS-Kmeans clustering method based on Hadoop distributed computing. The dataset is divided into multiple subsets. For each subset, non-outlier points are selected as candidate centroids, and the Mean Shift algorithm is used to shift the candidate centroids to denser data regions. Then, K points farthest apart from the candidate centroids are selected as the initial centroids for the K-means algorithm, and K-means clustering is performed on the data to find the final cluster centroids. All data are then assigned to the nearest cluster. Therefore, this invention enables the MS-Kmeans clustering algorithm to run on the Hadoop Distributed File System, effectively reducing clustering time while improving clustering performance.

[0060] Example 2: This embodiment of the invention discloses an MS-Kmeans clustering method based on Hadoop distributed computing, including:

[0061] Step S201: Build a Hadoop cluster and establish a Hadoop distributed file system.

[0062] The above-mentioned setup of a Hadoop cluster specifically includes:

[0063] (1) Prepare multiple computers and connect them to the same local area network, with each computer acting as a node;

[0064] (2) Install CentOS 7 operating system, Hadoop 3.1.3 and JDK on each node, and configure the environment variables of Hadoop and JDK. Then configure the core-site.xml, hdfs-site.xml, yarn-site.xml and mapred-site.xml files in the Hadoop 3.1.3 file of each node to deploy ResourceManager, NameNode, SecondaryNameNode and jobhistory, thereby completing the Hadoop cluster. Finally, start the cluster for testing.

[0065] (3) Combine the Hadoop cluster to establish a Hadoop distributed file system.

[0066] Step S202: Obtain the total dataset and divide it into several sub-datasets. Here, both the total dataset and each dataset are stored in a separate file (in the following process, the sub-datasets are designated as Y sub-datasets).

[0067] Step S203: Execute the Hadoop Distributed File command to upload the total dataset and sub-datasets to the Hadoop Distributed File System. Use the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors.

[0068] This step involves executing Hadoop Distributed File Commands to upload the total dataset and its subsets to the Hadoop Distributed File System, which means uploading several files containing the total dataset and its subsets to the Hadoop Distributed File System.

[0069] In this step, the Mean Shift algorithm is used to cluster each subset of the dataset, resulting in multiple candidate center vectors, as shown in the appendix. Figure 2 As shown, it specifically includes:

[0070] Step S2031: Select M candidate center vectors from a certain subset of the dataset and write them line by line into a Hadoop distributed file; the Hadoop distributed file here is the file ' / sub_Y / Mcenter.txt'.

[0071] Step S2032: For each candidate center vector X', find the data X in the high-dimensional space centered on the candidate center vector and with a radius of D. j Specifically, it includes:

[0072] (1) Start the first Map task, scan the candidate center vectors of the current subset and read them in;

[0073] (2) Initialize key-value pairs<key1,value1> The content of each line of data read in step (1) above is used as value1, and the line number of the data is used as key1, and stored in a key-value pair.<key1,value1> middle;

[0074] (3) Read the subset of data containing the current candidate center vector row by row. If the distance between a data point and one of the candidate center vectors is less than D, then move the data to the next subset.<key2,value2> The output is in the form of key2, where key2 is equal to key1 of the candidate center vector to which the data belongs, and value2 is the data.

[0075] (4) Start a Combine task. The Combine function combines key-value pairs with the same key to reduce the computation of the Reduce function. The input of the Combine function is the output of the Map function, and the output of the Combine function is...<key3,value3> Key-value pairs, where key3 is the candidate center vector identifier and value3 is the combination of all vectors with the same key3.

[0076] Step S2033: Determine the offset mean vector of the candidate center vector X', and move the candidate center vector X' in the direction of the offset mean vector. The offset mean vector and the moved candidate center vector are shown below:

[0077]

[0078] Where, x' m It is the m-th candidate center vector, M(x') m ) t The offset mean vector obtained at state t, where G is the mean vector of S. D The number of data vectors contained within the range;

[0079] (x' m ) t+1 =M(x' m ) t +(x' m ) t

[0080] Among them, (x' m ) t+1 It is the m-th candidate center vector at time t+1, that is, the new m-th candidate center vector after being moved in the direction of the offset mean vector;

[0081] The above step S2033 specifically includes:

[0082] (1) Start a Reduce task. The input of the Reduce task is the output of step S2032. For each candidate center vector, the offset vector of each candidate center vector can be calculated based on the data vector within its radius.

[0083] (2) By shifting the candidate center vector in the direction of the offset mean vector, a new candidate center vector can be obtained. Then, the new candidate center vector is...<key4,value4> The output is in the form of ' / sub_Y / M_new_center.txt' in the Hadoop Distributed File System. Here, key4 is equal to key3 when it was read in, and value4 is the new candidate center vector. If there is only one data vector within the radius D of the candidate center vector, that candidate center vector will be considered an outlier and discarded.

[0084] Step S2034: Determine whether the distance between the new candidate center vector and the old candidate center vector is less than a given threshold. If yes, end the Mean Shift phase of the candidate center vector and repeat the above steps until M candidate center vectors have been iterated. If no, repeat the above steps for the candidate center vector.

[0085] The above judgment determines whether the distance between the new candidate center vector in the file ' / sub_Y / M_new_center.txt' and the old candidate center vector in the file ' / sub_Y / Mcenter.txt' is less than a given threshold. If the distance is yes, the Mean Shift phase of the candidate center vector ends; if no, the data in the file ' / sub_Y / M_new_center.txt' is copied to the file ' / sub_Y / Mcenter.txt'.

[0086] Step S2035: Repeat the above steps, iterating through each subset to obtain Y×M candidate center vectors, where Y is the number of subsets. That is, merge ' / sub_Y / M_new_center.txt' into 'YMcenter.txt'.

[0087] Step S204: Select the K farthest candidate center vectors as center vectors, and use the K-means clustering algorithm to cluster all data in the total dataset.

[0088] In the above steps, selecting the K farthest candidate center vectors as the center vectors specifically includes:

[0089] (1) Read the candidate center vector set X' from 'YMcenter.txt', and randomly select a candidate center vector x' from the candidate center vector set X'. m And add it to set C;

[0090] (2) Calculate the distance from the candidate center vector in the candidate center vector set X' to all vectors in the set C;

[0091] (3) Determine whether the calculated distance satisfies the center vector judgment condition. In response, add the candidate center vector to set C. The center vector judgment condition is as follows:

[0092] Candidate center vector x' m The maximum value that satisfies the following expression;

[0093] min(dist(x'1,c1),dist(x'1,c2),...,dist(x'1,c k ),...,min(dist(x' m ,c1),dist(x' m ,c2),...,dist(x' m ,c k ))where dist(x' m ,c k ) is x' m With c k The Euclidean distance between them;

[0094] (4) Repeat steps (1) to (3) above until K vectors are selected as the initial center vectors of K clusters and written to the Hadoop distributed 'center.txt' file.

[0095] The above steps involve using the K-means clustering algorithm to cluster all data in the total dataset, including:

[0096] (1) For each vector x in the dataset n The vector x is calculated using the following formula. n The Euclidean distance to the center vectors of the K clusters, and x n Assign to the nearest cluster;

[0097]

[0098] Specifically:

[0099] A. Initialize key-value pairs<key5,value5> Read the 'center.txt' file, use the K initial center vectors as value5, and store the row number of the data (i.e., the center vector identifier) ​​as key5 in the key-value pairs.<key5,value5> middle;

[0100] B. Read all data sets row by row, calculate the distance between each data set and all center vectors, and assign the data set to the nearest vector.<key6,value6> The output is in the form of key6, where key6 is equal to key5 of the center vector to which the data belongs, and value6 is the data.

[0101] C. Start a Combine task. The Combine function combines key-value pairs with the same key to reduce the computation of the Reduce function. The input of the Combine function is the output of the Map function, and the output of the Combine function is...<key7,value7> Key-value pairs, where key7 is the candidate center vector identifier and value7 is the combination of all vectors with the same key7.

[0102] (2) Start a Reduce task and recalculate the center vector of each cluster using the following formula;

[0103]

[0104] (3) Repeat the above steps until the center vectors of the K clusters remain unchanged. Then the clustering has converged and the final k center vectors are obtained.

[0105] Step S205: Start the Map task, allocate the data to the nearest cluster, and output the clustering results.

[0106] The above steps specifically include:

[0107] by<key8,value8> K centroid vectors are read from the HDFS file in key-value pair form, and then...<key9,value9> The data to be clustered is read in line by line in key-value pair format, where key9 is the row containing the data to be read and value9 is the data. After reading a data, the distance between the data and all the center vectors is calculated, the value of the nearest key8 is assigned to key9, and key9 and value9 are output to the file '\clusterResult.txt' in the Hadoop Distributed File System.

[0108] Example 3: As shown in the attached document Figure 3 As shown, this embodiment of the invention discloses an MS-Kmeans clustering device based on Hadoop distributed computing, comprising:

[0109] Set up units, build Hadoop clusters, and establish Hadoop distributed file systems;

[0110] The dataset acquisition unit acquires the total dataset and divides it into several sub-datasets.

[0111] The first execution unit executes Hadoop distributed file commands, uploads the total dataset and sub-datasets to the Hadoop distributed file system, and uses the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors.

[0112] The second execution unit selects the K farthest candidate center vectors as center vectors and uses the K-means clustering algorithm to cluster all data in the total dataset.

[0113] The output unit initiates a Map task, assigns the data to the nearest cluster, and outputs the clustering results.

[0114] Example 4: Taking three publicly available datasets on UCI and Kaggle—Iris, EXP-nonscale, and KDD-Cup-1999—as examples, the MS-Kmeans algorithm, K-means algorithm, and K-means++ algorithm under the Hadoop distributed system were used to cluster the Iris, EXP-nonscale, and KDD-Cup-1999 datasets, respectively. Table 1 shows the DBI (Davies-Bouldin Index) evaluation metrics after clustering the three datasets using the three clustering algorithms. Table 1 shows that MS-Kmeans performs better than K-means and K-means++ in clustering.

[0115] Example 5: Taking three publicly available datasets on UCI and Kaggle—Iris, EXP-nonscale, and KDD-Cup-1999—as examples, the MS-Kmeans algorithm, K-means algorithm, and K-means++ algorithm under the Hadoop distributed system were used to cluster the Iris, EXP-nonscale, and KDD-Cup-1999 datasets, respectively. Iris contains 150 data points with 4 attributes, EXP-nonscale contains 600 data points with 10,000 attributes, and KDD-Cup-1999 contains 6,142,547 data points with 42 attributes. (See attached...) Figure 4 The speedup of the three algorithms on three datasets is shown. (From the appendix...) Figure 4As can be seen, the speedup ratio increases with the increase in data volume and number of nodes, with the fastest increase observed on the KDD-Cup-1999 dataset. This indicates that MS-Kmeans is more suitable for clustering large datasets on Hadoop distributed systems.

[0116] Example 6: This embodiment of the invention discloses a storage medium storing a computer program that can be read by a computer. The computer program is configured to execute the MS-Kmeans clustering method based on Hadoop distributed computing at runtime.

[0117] The aforementioned storage media may include, but are not limited to, USB flash drives, read-only memory, portable hard drives, magnetic disks, optical disks, and other media capable of storing computer programs.

[0118] Example 7: This embodiment of the invention discloses an electronic device, including a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the MS-Kmeans clustering method based on Hadoop distributed computing.

[0119] The processor described above can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. It can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The memory can include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, portable hard drives, magnetic disks, or optical disks.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] The above technical features constitute the preferred embodiment of the present invention, which has strong adaptability and optimal implementation effect. Unnecessary technical features can be added or removed according to actual needs to meet the requirements of different situations.

[0124] Table 1 Evaluation Indicators

[0125] K-means K-means++ MS-Kmeans Iris 0.666391 0.66395 0.663137 Exp-nonscale 2.625545769 2.637542088 2.57584623 KDD-Cup-1999 0.827655 0.708568 0.423758

Claims

1. An MS-Kmeans clustering method based on Hadoop distributed computing, characterized in that, include: Set up a Hadoop cluster and establish a Hadoop distributed file system; Obtain the total dataset and divide it into several equal subsets. Execute the Hadoop Distributed File command to upload the total dataset and sub-datasets to the Hadoop Distributed File System. Use the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors. Select the K farthest candidate center vectors as center vectors, and use the K-means clustering algorithm to cluster all data in the total dataset; Start the Map task to assign the data to the nearest cluster and output the clustering results.

2. The MS-Kmeans clustering method based on Hadoop distributed computing according to claim 1, characterized in that, The Mean Shift algorithm is used to cluster each subset of data to obtain multiple candidate center vectors, including: Select M candidate center vectors from a subset of the dataset and write them line by line into a Hadoop distributed file; For each candidate center vector Find the data within a high-dimensional space centered on the candidate center vector and with a radius of D. ; Determine the offset mean vector of the candidate center vector, and then... The center is moved in the direction of the offset mean vector, where the offset mean vector and the moved candidate center vector are shown below: in, It is the first m candidate center vectors, Is t The offset mean vector obtained under the state, G is The number of data vectors contained within the range; in, yes The first moment m The candidate center vector, i.e., the new _th_ vector after shifting in the direction of the offset mean vector. m One candidate center vector; Determine whether the distance between the new candidate center vector and the old candidate center vector is less than a given threshold. If so, end the Mean Shift phase of the candidate center vector and repeat the above steps until M candidate center vectors have been iterated. If not, repeat the above steps for the candidate center vector. Repeat the above steps, iterating through each subset, to obtain Y×M candidate center vectors, where Y is the number of subsets.

3. The MS-Kmeans clustering method based on Hadoop distributed computing according to claim 2, characterized in that, For each candidate center vector Find the data within a high-dimensional space centered on the candidate center vector and with a radius of D. ,include: Start the first Map task to scan and read in the candidate center vectors of the current subset of data; Initialize key-value pairs<key1,value1> The content of each line of data read in the above steps is used as value1, and the line number of the data is used as key1, and stored in a key-value pair.<key1,value1> middle; Read the subset of data containing the current candidate center vector row by row. If the distance between a data point and one of the candidate center vectors is less than D, then move that data point to the next subset.<key2,value2> The output is in the form of key2, where key2 is equal to key1 of the candidate center vector to which the data belongs, and value2 is the data. Start a Combine task. The Combine function combines key-value pairs with the same key. The input is the output of the Map function, and the output is...<key3,value3> Key-value pairs, where key3 is the candidate center vector identifier and value3 is the combination of all vectors with the same key3.

4. The MS-Kmeans clustering method based on Hadoop distributed computing according to claim 1, 2, or 3, characterized in that, The selection of K candidate center vectors that are farthest apart as center vectors includes: From candidate center vector set Randomly select a candidate center vector And add it to set C; Calculate the candidate center vector set The distance from the candidate center vector in set C to all vectors in set C; If the calculated distance satisfies the center vector criterion, the candidate center vector is added to set C. The center vector criterion is as follows: Candidate center vector The maximum value that satisfies the following expression; in, for and The Euclidean distance between them; Repeat the above steps until K vectors are selected as the initial center vectors of K clusters and written to the Hadoop distributed file.

5. The MS-Kmeans clustering method based on Hadoop distributed computing according to claim 1, 2, or 3, characterized in that, The process of clustering all data in the total dataset using the K-means clustering algorithm includes: For each vector in the dataset The vector is calculated using the following formula. The Euclidean distance between the center vectors of the K clusters, and the ... Assign to the nearest cluster; Start a Reduce task to recalculate the center vector of each cluster using the following formula. ; Repeat the above steps until the center vectors of the K clusters remain unchanged. Then the clustering has converged, and the final k center vectors are obtained.

6. The MS-Kmeans clustering method based on Hadoop distributed computing according to claim 4, characterized in that, The process of clustering all data in the total dataset using the K-means clustering algorithm includes: For each vector in the dataset The vector is calculated using the following formula. The Euclidean distance between the center vectors of the K clusters, and the ... Assign to the nearest cluster; Start a Reduce task to recalculate the center vector of each cluster using the following formula. ; Repeat the above steps until the center vectors of the K clusters remain unchanged. Then the clustering has converged, and the final k center vectors are obtained.

7. An MS-Kmeans clustering device based on Hadoop distributed computing that applies the method described in any one of claims 1 to 6, characterized in that, include: Set up units, build Hadoop clusters, and establish Hadoop distributed file systems; The dataset acquisition unit acquires the total dataset and divides it into several sub-datasets. The first execution unit executes Hadoop distributed file commands, uploads the total dataset and sub-datasets to the Hadoop distributed file system, and uses the Mean Shift algorithm to cluster each sub-dataset to obtain multiple candidate center vectors. The second execution unit selects the K farthest candidate center vectors as center vectors and uses the K-means clustering algorithm to cluster all data in the total dataset. The output unit initiates a Map task, assigns the data to the nearest cluster, and outputs the clustering results.

8. A storage medium, characterized in that, The storage medium stores a computer program that can be read by a computer, and the computer program is configured to execute the MS-Kmeans clustering method based on Hadoop distributed computing as described in any one of claims 1 to 6 when it runs.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the MS-Kmeans clustering method based on Hadoop distributed computing as described in any one of claims 1 to 6.

Images