Distributed spatial data storage and calculation method based on SPARK

Abstract

The invention discloses a distributed spatial data storage and calculation method based on SPARK. On the basis that a traditional geographic information system software function is multiplexed, by applying cloud computing software, high performance and high availability are brought to storage and calculation of spatial data. An implementation mode comprises the steps of taking a variable-length GeoHash code considering a spatial range as a spatial index, and realizing a distributed spatial data storage table structure compatible with multiple spatial data types on HBase; converting a file forstoring the spatial data to be in a storage format HFile corresponding to the HBase by using a MapReduce method, and realizing rapid preprocessing and storage of distributed spatial data; performing spatial range coarse retrieval on the spatial data stored in the HBase by using the variable-length GeoHash code; mapping the spatial data obtained from the HBase into a distributed spatial elastic data set object in the SPARK; and realizing a distributed spatial data processing interface of a distributed spatial elastic data set by multiplexing the traditional geographic information system software function, and carrying out fine screening and distributed calculation on the spatial data set obtained by coarsely querying the spatial range.

Description

technical field [0001] The invention belongs to the field of geographic information systems, and relates to a method for storing and computing distributed spatial data. Background technique [0002] In recent years, with the rapid development of sensor technology and earth observation technology, geospatial big data has become an important part of big data. The amount of data managed and processed by spatial information systems has increased from TB to PB or even EB. The characteristics of spatial big data in terms of type, acquisition speed, value density, accuracy, and variability also increase the complexity of geospatial data management and processing, and also mean that traditional data management systems and computing power are difficult to meet these needs . Traditional geographic information system (GIS) spatial data storage management either directly depends on existing databases (such as PostGIS, Oracle Spatial), or builds spatial data engine middleware (such as ...

AI Technical Summary

Problems solved by technology

However, the existing work still lacks the reuse of existing GIS software functions, and solutions to build GIS functions from scratch are often costly, time-consuming, labor-intensive, and error-prone; secondly, current research based on SPARK mostly uses distributed files The system stores spatial data, and uses distributed databases such as HBase to store and connect to SPARK for calculation. Due to the distributed storage structure and index design of spatial data, the mapping of spatial data tables to SPARK distributed spatial elastic data sets, etc. Difficult, still to be achieved

Images