Mass vector data partition method and system based on Hadoop

Abstract

The invention relates to a mass vector data partition method and system based on Hadoop. The method comprises the steps that space encoding is conducted on space data-concentrated space elements on the basis of a Hilbert space filling curve; key value assignment on the space elements is achieved through a Map function and a Reduce function, and a space data sample information set is generated; space data partition matrixes are generated according to the space data sample information set; the space elements are partitioned in corresponding storage data blocks according to the space data partition matrixes, and meanwhile every two adjacent data blocks are distributed in a same cluster node. According to the system, the Hilbert space filling curve is introduced into a data sampling and partitioning rule, the influence factors such as the space position relation of adjacent objects of the space data, the self size of the space objects and the space object number of same encoding blocks are fully taken into account, therefore, the space distribution characteristics of the sample information set are guaranteed, the space index efficiency of the mass vector data is improved, and meanwhile load balance based on HDFS data block storage is guaranteed.

Description

technical field [0001] The invention relates to the technical field of spatial big data, in particular to a method and system for dividing massive vector data based on Hadoop. Background technique [0002] With the advent of the era of big data, traditional data storage and processing methods are facing severe challenges, such as volume, variety, velocity and low value density in the era of big data. And other characteristics make traditional tools and processing methods sometimes only look at "data" and sigh. In the field of geospatial vector data management, the existing mature geographic information system (GIS, geographic information system) spatial data mostly relies on relational database storage, which has inherent limitations in massive data management, high concurrent access, and scalability. Therefore, it can no longer play its due role in the era of big data. [0003] The emergence of cloud computing technology provides an ideal solution for the storage and mana...

AI Technical Summary

Problems solved by technology

Due to the randomness of the sample, for the spatial index technology itself, on the one hand, it cannot guarantee the consistency of the spatial index results, and on the other hand, it will lose the spatial distribution characteristics of the spatial data, resulting in unsatisfactory results of the final data division.

For the Hadoop platform, due to the randomness of the samples, the established data partition rules cannot guarantee the balanced distribution of data, which in turn causes the uneven distribution of the Reduce load, which reduces the efficiency of the entire task execution; at the same time, the result will directly lead to Severe data skew (Data Skew); In addition, although the current parallel space partition algorithm can store adjacent elements in the same data block, it cannot guarantee that adjacent data blocks are stored on the same cluster node

Images