A power grid timing sequence large data parallel loading method

Abstract

The present invention discloses a power grid timing sequence large data parallel loading method, to solve the problem, i.e., waiting phenomena occurs due to failing in parallel loading when multiple clients parallel loads an abundant amount of historical timing sequence data. By partitioning an index mapping table, the present invention performs partitioning preprocessing on a historical timing sequence data storage table according to the size of the amount of data to be loaded, and performs the processing of maintaining data locality of the abundant amount of historical timing sequence data to be loaded according to the range of the partition of the historical timing sequence data storage table distributed on each data node. After the abovementioned processes, disk IO conflicts and the network communication overheads between different nodes of the cluster encountered by the multiple clients when reading the index mapping data file can be effectively reduced when parallel loading the abundant amount of historical timing sequence data, therefore, performance issues caused by overload when loading the abundant amount of historical timing sequence data by a single node. The present method can fully use the distributed parallel processing ability to greatly reduce the time for loading the abundant amount of historical timing sequence data.

Description

technical field [0001] The invention relates to a data parallel loading method, which belongs to the field of big data processing and distributed real-time databases, and is particularly suitable for the parallel loading method of massive historical time series data in smart grids and the Internet of Things. Background technique [0002] With the continuous development of industrialization and informatization, large-scale process industry enterprises generate more and more massive historical time series data in the process of production informatization. Taking the power system as an example, on the one hand, the scale of measuring points is getting bigger and bigger, and it is expected to reach tens of millions or even more than one hundred million; This puts forward higher requirements on the processing scale and processing speed of the real-time database. [0003] Constrained by its traditional software architecture, traditional real-time databases can no longer meet the ...

AI Technical Summary

Problems solved by technology

Loading a large amount of historical time series data from a single client cannot exert the ability of distributed concurrent processing, and it takes a lot of time to complete, while the general multi-client parallel loading will encounter multiple clients reading and writing the index mapping table at the same time during data loading The file generates a large number of disk IO conflicts and the network communication overhead between different nodes of the cluster can not be loaded in parallel and the resulting waiting phenomenon is caused; after a preliminary search, no technical solutions to solve the above technical problems have been found.

Images