Method and system for migrating electric power measured class data to HBase

A technology of power and data, applied in electrical digital data processing, data processing applications, special data processing applications, etc., can solve the problems of inability to realize automatic retransmission of failed data, and no automatic retransmission, so as to reduce the difficulty of execution, Increase flexibility and efficiency

Active Publication Date: 2018-11-13
SHANGHAI SHINEENERGY INFORMATION TECH DEV CO LTD
View PDF12 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, relational databases can be imported into HBase through tools such as Sqoop, but it needs to be used in conjunction with other scheduling tools (such as Oozie) to achieve scheduled tasks, and there is no automatic supplementary transfer function, and it is impossible to automatically supplement data that fails to migrate. pass

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for migrating electric power measured class data to HBase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0037] The invention can set the best writing mode according to the host hardware information and HBase cluster information, and is composed of six functional modules independently developed: query module, configuration module, log module, conversion module, writing module and management module.

[0038] The query module queries data from the relational database through JDBC;

[0039] The configuration module is used to configure connections, tables, query conditions, result output formats, writing methods, etc.;

[0040] The log module records the status of each migration of each table;

[0041] The conversion module converts the result data found by the query module into the format required by the write module according to the corresponding HBase structure;

[0042] The write module writes data into HBase through the API according to the configuration module write parameters;

[0043] The management module starts the migration task at regular intervals and retransmits the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a method and system for migrating electric power measured class data to HBase. The system comprises a query module, a configuration module, a log module, a conversion module,a writing-in module and a management module, wherein the query module queries data from a relational database through JDBC; the configuration module is used for configuring connection information, tables, the query condition, the result output format, the writing-in mode and the like; the log module records the states of each time of migration of all the tables; the conversion module converts theresult data queried by the query module into the format required by the writing-in module according to a corresponding HBase structure; the writing-in module writes the data into the HBase through anAPI according to writing-in parameters of the configuration module; and the management module starts the migration task in a timing mode and uploads the data not successfully migrated in a supplementing mode according to log information recorded by the log module. Compared with the prior art, the method and system have the advantages of automatic migration, supporting for incremental delay synchronization, automatic uploading supplementing, high efficiency and the like.

Description

technical field [0001] The invention relates to the field of power data measurement, in particular to a method and system for migrating power measurement data to HBase. Background technique [0002] In order to improve the operation level of enterprises and further explore the potential value of data, it is necessary to migrate massive power measurement data from traditional relational databases to big data platforms. Through various data mining tools or query engines based on big data platforms, fast query, Advanced applications such as load forecasting and anomaly analysis. [0003] At present, it is possible to import relational databases into HBase through tools such as Sqoop, but it needs to be used in conjunction with other scheduling tools (such as Oozie) to achieve scheduled tasks, and there is no automatic supplementary transfer function, and it is impossible to automatically supplement data that fails to migrate. pass. Contents of the invention [0004] The pur...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30G06Q50/06
CPCG06Q50/06
Inventor 苏鹏涛朱兵蔡晶吕山
Owner SHANGHAI SHINEENERGY INFORMATION TECH DEV CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products