Spark-based grid hydrological model distributed calculation method

Abstract

The invention discloses a Spark-based grid hydrological model distributed calculation method, which comprises the following steps of: firstly describing grid hydrological model calculation parameters by adopting a netCDF format, and then describing grid hydrological model components and a dependency relationship thereof on the basis of an XML (Extensible Markup Language) specification; and then scheduling and calculating components of the grid hydrological model by adopting a Spark distributed calculation model, and finally outputting a result generated by each component after scheduling calculation. According to the method, netCDF-based model parameters and a model calculation result description format are defined, and drainage basin characteristics, the model parameters and hydrological data on monitoring points and a drainage basin surface which are used during grid hydrological model calculation are described; XML-based grid hydrological model components and dependency description specifications among the components are defined; and according to the dependency relationship between grids during component calculation, a Hash partitioning method and a dynamic data partitioning method are used for performing task division on the grids, and different nodes are allocated for calculation, so that the data distribution amount is reduced, and the calculation speed is improved.

Description

technical field [0001] The invention belongs to the field of information technology, in particular to a spark-based grid hydrological model distributed computing method. Background technique [0002] The traditional lumped hydrological model uses average and simplistic parameters to describe the hydrological characteristics of the entire watershed, without considering the spatial heterogeneity of different regions of the watershed. distributions vary widely) cannot be modeled well. The distributed hydrological model divides the watershed into sub-basins of different sizes and irregular spaces according to the characteristics of the underlying surface of the watershed, in order to consider the spatial heterogeneity of the watershed, and improve the efficiency of model calculation through parallel computing. The grid hydrological model is a kind of distributed hydrological model, which divides the watershed into multiple grid units, and the grid of each rectangular area has i...

AI Technical Summary

Problems solved by technology

The main problems are that, on the one hand, the parallel calculation problem within the sub-basin is not solved, and the calculation inside the sub-basin still adopts the serial calculation method, especially the slope and river confluence module, resulting in high computational complexity; on the other hand, The parallel computing method of the existing distributed model performs task distribution and data aggregation operations based on the calculation order of sub-basin units. Multiple task distribution and data aggregation operations will cause a lot of waste of running time and cause memory overflow.

Therefore, the existing parallel computing methods cannot meet the computing needs of large watersheds. At the same time, they have high requirements for computer resources, and cannot meet the needs of hydrological forecasting and simulation research in a laboratory environment that only includes several PCs with limited performance.

Images