MapReduce-based distributed particle swarm clustering algorithm

Abstract

The invention relates to the field of artificial intelligence big data analysis, in particular to a MapReduce-based distributed particle swarm clustering algorithm, which is characterized by comprising the following steps of: 1, updating the centroid of a particle swarm by adopting MapReduce operation; 2, adopting MapReduce operation to evaluate the adaptability of the population with the new particle centroid generated in the step 1, calculating a new fitness value of the updated population, carrying out fitness evaluation based on a fitness function, and measuring the distances between all data points and the particle centroid by obtaining the average distance between the particle centroids; 3, combining the fitness value calculated in the step 2 with the updating group generated in thestep 1, and updating the optimal individual centroid and the optimal global centroid at the same time; and returning to the step 1, and carrying out next iteration. According to the method, the clustering problem of a super-large-scale commercial data set is effectively solved, and high-quality clustering is achieved.

Description

technical field [0001] The invention relates to the field of artificial intelligence big data analysis, in particular to a distributed particle swarm clustering algorithm based on MapReduce. Background technique [0002] With the development of Internet technology, the data that needs to be stored, analyzed, and processed has shown explosive growth. In addition to the huge amount of data, the data created or collected is also becoming more and more complex. Addressing how to efficiently generate, manage and analyze data and derive resulting information requires a comprehensive, end-to-end approach covering all stages from initial data acquisition to final analysis. Clustering is a data mining technique used when analyzing data. The main goal of the clustering algorithm is to divide a group of unlabeled data objects into different clusters, so that the cluster members have common specifications and relatively similar membership. In order to obtain high-quality clustering, t...

AI Technical Summary

Problems solved by technology

At present, most sequential clustering algorithms are inversely proportional to the growth of data set size and scalability, and the high time complexity and space complexity aggravate the cost of clustering algorithms.

Images