Geographical raster data parallel reading-writing method based on request aggregation

Abstract

The invention provides a geographical raster data parallel reading-writing method based on request aggregation. According to the technical scheme, for all processes, a GDAL (geographical data abstract library) is called to read geographical raster data files to be processed; geographical raster metadata information is acquired from the files; all processes calculate partition size and offset of respective reading-required geographical raster data in the geographical raster data files by means of uniform data partitioning; any process is response of creating a GTIFF output file; after creating, the process broadcasts the status of creating completion to other processes; the other processes read the geographical raster data to be processed; each process completes its respective calculation task, and results of calculation task completion are written to output files by means of uniform data partitioning. The method has the advantages that various formats of data can be processed, a parallel processing mechanism is good, and overall input / output efficiency is improved.

Description

technical field [0001] The invention relates to a method for parallel reading and writing of geographic grid data files in a multi-node and multi-processor cluster environment, and the technical application field is the parallel processing of large-scale geographic grid data in geographic information systems. Background technique [0002] Geographic raster data is a very important data type in geographic information systems and spatial information applications. It is mainly used to describe and express various sampling and statistical information on the surface. Wide range of uses. Geographic raster data is arranged in rows and columns of grid cells, and is a data structure that represents the distribution of spatial objects or phenomena with an array of pixels (grid cells) that are equally sized and evenly distributed and closely connected. The size of the grid unit determines the accuracy of the geographic data within the surface area covered by it. The finer the grid uni...

AI Technical Summary

Problems solved by technology

The main problem with GDAL is that it only supports parallel reading and writing of geographic raster data divided by rows. When multiple processes use GDAL to read and write parallel column division or block division, on the one hand, the efficiency of reading and writing is very low; Data accuracy cannot be guaranteed

The second is a geographic raster data model library suitable for parallel reading and writing, such as: HDF5 (Hierarchical DataFormat Five, hierarchical data format version 5), NetCDF (Network Common Data Form, network common data format), etc. However, these data models There are relatively few applications to store geographic raster data, and other commonly used data formats need to be converted during processing, which increases the complexity of the application

However, this method requires the support of the underlying parallel file system. In a non-parallel file system, if the distribution of read and write requests is highly random, the I / O efficiency will be greatly reduced.

Images