Method and system for migrating electric power measured class data to HBase

Abstract

The invention relates to a method and system for migrating electric power measured class data to HBase. The system comprises a query module, a configuration module, a log module, a conversion module,a writing-in module and a management module, wherein the query module queries data from a relational database through JDBC; the configuration module is used for configuring connection information, tables, the query condition, the result output format, the writing-in mode and the like; the log module records the states of each time of migration of all the tables; the conversion module converts theresult data queried by the query module into the format required by the writing-in module according to a corresponding HBase structure; the writing-in module writes the data into the HBase through anAPI according to writing-in parameters of the configuration module; and the management module starts the migration task in a timing mode and uploads the data not successfully migrated in a supplementing mode according to log information recorded by the log module. Compared with the prior art, the method and system have the advantages of automatic migration, supporting for incremental delay synchronization, automatic uploading supplementing, high efficiency and the like.

Description

technical field [0001] The invention relates to the field of power data measurement, in particular to a method and system for migrating power measurement data to HBase. Background technique [0002] In order to improve the operation level of enterprises and further explore the potential value of data, it is necessary to migrate massive power measurement data from traditional relational databases to big data platforms. Through various data mining tools or query engines based on big data platforms, fast query, Advanced applications such as load forecasting and anomaly analysis. [0003] At present, it is possible to import relational databases into HBase through tools such as Sqoop, but it needs to be used in conjunction with other scheduling tools (such as Oozie) to achieve scheduled tasks, and there is no automatic supplementary transfer function, and it is impossible to automatically supplement data that fails to migrate. pass. Contents of the invention [0004] The pur...

AI Technical Summary

Problems solved by technology

[0003] At present, relational databases can be imported into HBase through tools such as Sqoop, but it needs to be used in conjunction with other scheduling tools (such as Oozie) to achieve scheduled tasks, and there is no automatic supplementary transfer function, and it is impossible to automatically supplement data that fails to migrate. pass

Images