Spark platform based space data parallel calculating system and method

A spatial data and parallel computing technology, applied in transmission systems, electrical components, etc., can solve the problems of not supporting spatial data types and spatial operations, large system communication costs and load, imbalance, etc. Speed, the effect of increasing speed

Inactive Publication Date: 2016-12-07
SHANDONG UNIV
View PDF4 Cites 18 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] (2) When GeoSpark and SpatialSpark process spatial join (Spatial Join) queries, the system will have huge communication costs and load imbalance problems
This takes a long time when the data is large
In addition, SparkSQL does not support spatial data types and spatial operations, so when the data is spatial data, it will be treated like ordinary data and will not take advantage of its spatial properties

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark platform based space data parallel calculating system and method
  • Spark platform based space data parallel calculating system and method
  • Spark platform based space data parallel calculating system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0056] figure 1 It is a schematic structural diagram of a parallel computing system for spatial data based on a spark platform in an embodiment of the present invention. As shown in the figure, the parallel computing system for spatial data based on a spark platform in this embodiment includes:

[0057] The indexing and storage layer is configured to read and store the spatial data set to be processed in the spark cluster, the spatial data stored in the spatial dat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a spark platform based space data parallel calculating system and method. The system is used for parallel calculation of mobile conversation data, and comprises an index and storage layer, a query and operation layer and an API layer; the index and storage layer is configured to read a space data set to be processed from a spark cluster and stores the space data set, space data stored in the space data set is the mobile conversation data, and a space index is established for the space data set to be processed; the query and operation layer is configured to receive a space operation request of the API layer, makes responses to the space operation request, realizes space operation of the concentrated mobile conversation data in the space data set to be processed, and back feeds a result after space operation to the API layer; and the API layer receives the input space operation request through a space operation interface, transmits the input space operation request to the query and operation layer, and receives and outputs the result after space operation of the query and operation layer.

Description

technical field [0001] The invention relates to the technical field of mobile communication data services, in particular to a parallel computing system for spatial data based on a spark platform and a method thereof. Background technique [0002] With the in-depth development of information technology, various devices such as mobile phones and vehicle networks continue to generate a large amount of spatial data. Spatial datasets are usually very large, far exceeding the computing power of a single machine. Therefore, we need a cloud computing framework to store and compute large-scale spatial data. Usually we use platforms such as Hadoop or spark to assist in processing large data sets. [0003] On the one hand, similar to SpatialHadoop and Hadoop-GIS, they support parallel processing of spatial data by extending Hadoop. However, due to the nature of Hadoop's disk-level computing, these systems perform poorly in handling complex and interactive jobs. [0004] On the othe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/10H04L67/1001
Inventor 杨伯宇王海林鲁宗飞郭山清许信顺
Owner SHANDONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap