A distributed traffic big data parallel clustering method suitable for a wide area network

Abstract

The invention discloses a distributed traffic big data parallel clustering method suitable for a wide area network, wherein the parallel clustering operation of the distributed big data is divided into a history total quantity stage and a plurality of cycle increment stages according to a time sequence cycle and continuously executed; firstly, the maximum and minimum distance method is used to optimize the k-Means clustering algorithm; Secondly, a distributed parallel clustering computing framework based on MapReduce is constructed, which is suitable for wide area network, In this framework, the improved clustering algorithm is optimized again to realize the distributed parallel execution in WAN, and then the algorithm is used to realize the distributed parallel clustering operation for the whole historical data, from which the feature groups and their behavior patterns are extracted. Finally, the clustering results of the historical data are corrected periodically by using the clustering operation of multi-period incremental large data, and the existing clusters are dynamically updated or new clusters are generated. Distributed computing avoids copying movement and repetitive clustering of large data in wide area network, which reduces the cost of data movement and improves computational efficiency.

Description

technical field [0001] The invention belongs to the technical field of data processing, in particular to a distributed traffic big data parallel clustering method suitable for wide area networks. Background technique [0002] Big data is stored in a distributed manner in a wide area network environment, and the data with complex structure and huge amount (reaching TB or even PB level) is moved and concentrated before performing traditional parallel clustering operations based on LAN. Due to time, money and equipment Such costs are not directly applicable. The use of sampling to reduce the size of the data and the use of dimensionality reduction to reduce the complexity of the data all have an impact on the accuracy of the clustering results. This urgently requires us to change the traditional clustering mining method of the local area network, so that the efficiency and accuracy of data clustering can be improved. [0003] In terms of practical application, the clustering ...

AI Technical Summary

Problems solved by technology

[0002] Big data is stored in a distributed manner in a wide area network environment, and the data with complex structure and huge amount (reaching TB or even PB level) is moved and concentrated before performing traditional parallel clustering operations based on LAN. Due to time, money and equipment other costs are not directly applicable

Images