A Parallel Loading Method of Power Grid Time Series Big Data

Abstract

The invention discloses a method for parallel loading of time-series large data of a power grid to solve the problem that parallel loading of massive historical time-series data cannot be performed by multiple clients and the waiting phenomenon occurs. The present invention performs partition processing on the index mapping table, performs partition preprocessing on the historical time series data storage table according to the size of the data volume to be loaded, and according to the range of partitions of the historical time series data storage table allocated on each data node, the data to be loaded Massive historical time-series data is processed to maintain data locality. After the above-mentioned processing, it can effectively reduce the disk IO conflicts encountered by multiple clients when reading and writing index mapping table files and the network communication overhead between different nodes of the cluster when loading massive historical time-series data in parallel. , to avoid performance problems caused by heavy load when a single node loads massive historical time series data. This method can make full use of the distributed parallel processing capability, and greatly reduce the loading time of massive historical time series data.

Description

technical field [0001] The invention relates to a data parallel loading method, which belongs to the field of big data processing and distributed real-time databases, and is particularly suitable for the parallel loading method of massive historical time series data in smart grids and the Internet of Things. Background technique [0002] With the continuous development of industrialization and informatization, large-scale process industry enterprises generate more and more massive historical time series data in the process of production informatization. Taking the power system as an example, on the one hand, the scale of measuring points is getting bigger and bigger, and it is expected to reach tens of millions or even more than one hundred million; This puts forward higher requirements on the processing scale and processing speed of the real-time database. [0003] Constrained by its traditional software architecture, traditional real-time databases can no longer meet the ...

AI Technical Summary

Problems solved by technology

Loading a large amount of historical time series data from a single client cannot exert the ability of distributed concurrent processing, and it takes a lot of time to complete, while the general multi-client parallel loading will encounter multiple clients reading and writing the index mapping table at the same time during data loading The file generates a large number of disk IO conflicts and the network communication overhead between different nodes of the cluster can not be loaded in parallel and the resulting waiting phenomenon is caused; after a preliminary search, no technical solutions to solve the above technical problems have been found.

Images