Method and system for automatic parallel processing of spatial big data based on kettle and geotools

A spatial data and parallel processing technology, applied in database management systems, special data processing applications, structured data retrieval, etc., can solve problems such as difficult automatic processing of spatial data, and achieve the effect of easy use and saving configuration time

Active Publication Date: 2022-04-29
湖北地信科技集团股份有限公司
View PDF10 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The technical problem to be solved by the present invention is to provide a method and system for automatic parallel processing of spatial big data based on Kettle and GeoTools, which can define spatial data in a visual programming manner, aiming at the defect that the current large-scale multi-source spatial data is difficult to be automatically processed The processing flow, while relying on the ETL theory, realizes the automatic processing of spatial big data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for automatic parallel processing of spatial big data based on kettle and geotools
  • Method and system for automatic parallel processing of spatial big data based on kettle and geotools
  • Method and system for automatic parallel processing of spatial big data based on kettle and geotools

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0171] In order to illustrate the idea of ​​the present invention more clearly, the method for automatic parallel processing of spatial big data based on Kettle and GeoTools is further described below, please refer to image 3 , using the building vector data of Laohekou City, Hubei Province as an example data, which is stored in the building.shp file, the layer name is BUILD, it contains 46849 polygon features, and its coordinate system is GCS_WGS_1984. Select the Shapefile extraction tool, spatial coordinate conversion tool, conditional spatial data cleaning tool, and Gdb data loading tool developed based on GeoTools as the spatial data processing tool. The specific steps are as follows:

[0172] S1: Create a spatial data processing flowchart through the Web client 101 to form a preliminary spatial data processing flowchart:

[0173] Create a flowchart drawing project;

[0174] Call the insertVertex method of the js process drawing library mxgraph to add Shapefile extractio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and system for automatic parallel processing of spatial big data based on Kettle and GeoTools. The method includes the following steps: making a spatial data processing flow chart through a Web client; executing the flow chart related parameter configuration to form a parseable kettle file ; Verify the validity of the kettle file; save the kettle file to the database resource library; the spatial data processing platform requests and parses the kettle file from the database resource library; designs spatial data extraction, conversion, and loading methods based on GeoTools; automatically executes the spatial data processing process in parallel; The system includes: client, used to configure, edit, and save kettle files in the spatial data processing process; spatial data processing platform, used to receive and analyze kettle files, and process and transfer the original spatial data according to the data processing process designed by the client to the resulting data. The invention can complete the processing process of various spatial data from extraction, conversion, cleaning to loading, which is convenient, low in coupling degree and high in efficiency.

Description

technical field [0001] The invention belongs to the field of automatic processing of spatial data, and in particular relates to a method and system for realizing automatic extraction, conversion, cleaning and loading of large spatial data based on Kettle and GeoTools. Background technique [0002] Due to the characteristics of massive data storage, spatio-temporal dynamics, and multi-scale, spatial data has complex storage and calculation, and the existing spatial data processing methods are difficult to automatically process raw spatial data in batches and on a large scale. In the current multi-core cluster parallel computing environment, relying on the ETL data processing theory in the data warehouse, the processing flow can be pre-edited in the form of visual programming, and then the processing of complex spatial data can be quickly and automatically completed. Contents of the invention [0003] The technical problem to be solved by the present invention is to provide ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F8/20G06F8/71G06F16/25
CPCG06F8/24G06F8/71G06F16/254
Inventor 李兵郭明强王波曹威张敏
Owner 湖北地信科技集团股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap