Spark platform supported spatial data management-based diagram calculation system and method

Abstract

The invention discloses a Spark platform supported spatial data management-based diagram calculation system and method. The method comprises the following steps of: dividing a spatial range of spatial data into a plurality of rectangle areas according to geographic position information through receiving the spatial data, distributing data in each rectangle to different subareas, dividing the subareas into grids, sorting the grids, carrying spatial mapping on the data, and establishing a quadtree index; receiving query request of a diagram, converting the query request into data query, and carrying out search in an index of a data storage layer in a range of the requested diagram; if a plurality of diagram data exists, carrying out spatial connection on multiple diagrams according to the returned query result, and distributing edges, in a set range, of distances in the query result into a same subarea on the basis of a diagram subarea strategy of positions so as to realize the construction of a local diagram. According to the system and method, direct spatial range query and spatial connection operation on the diagram are realized through extending the diagram query, and the requirements of numerous scenes are satisfied.

Description

technical field [0001] The invention relates to a graph computing system and method supporting spatial data management based on a Spark platform. Background technique [0002] With the popularization of information, a large amount of data containing location attributes will be generated every moment, and these data have gradually become an indispensable part of our digital life. Data that is attached to geographical location attributes like this is called spatial data, and there are many valuable data relationships hidden in these data, and graphs are the most intuitive and popular tools to show the relationships in the data. Therefore, more and more researchers have begun to pay attention to the research of graph computing and graph analysis of spatial data. Since there are many scenarios in real life, such as the investigation of the New York Big Bang, we only need to investigate the relevant data of a local area in a region. For the calculation and analysis of large-sca...

AI Technical Summary

Problems solved by technology

[0003] However, after more in-depth research, we found that the SpatialGraphx framework has the following two important defects. It is these two defects that limit its performance when dealing with spatial graphs.

[0004] The first point: SpatialGraphx does not take into account the load imbalance caused by the different sizes of the graphs maintained by each slave node when executing spatial graph queries

This situation is due to the fact that in practice, the amount of spatial data generated in different regions is different due to differences in the economic level and traffic development of different regions, and SpatialGraphx partitions the overall data evenly according to the region (such as figure 1 ), the size of the data in each region is different, which causes the size of the graph maintained by each slave node to be different, and when the user wants to query the subgraph of the left rectangular part shown in the figure, only the slave node under the SpatialGraphx cluster A is in the working state, so whether it is in terms of graph storage or graph query operations, it will cause serious load imbalance to the cluster, resulting in reduced query efficiency

[0005] The second point: the data reading interface of the graph does not meet the needs of real-world scenarios, such as the analysis of local data for the overall data and the cross analysis of the same area between two spatial data sets, etc.

This takes a long time when the data is large

In addition, SparkSQL does not support spatial data types and spatial operations, so when the data is spatial data, it will be treated like ordinary data and will not take advantage of its spatial properties

Images