A method and system for migrating power measurement data to hbase

A power and data technology, applied in data processing applications, structured data retrieval, database design/maintenance, etc., can solve the problems of unable to realize automatic retransmission of migration failure data, no automatic retransmission, etc., to reduce execution difficulty, Improved flexibility and high efficiency

Active Publication Date: 2021-08-31
SHANGHAI SHINEENERGY INFORMATION TECH DEV CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, relational databases can be imported into HBase through tools such as Sqoop, but it needs to be used in conjunction with other scheduling tools (such as Oozie) to achieve scheduled tasks, and there is no automatic supplementary transfer function, and it is impossible to automatically supplement data that fails to migrate. pass

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for migrating power measurement data to hbase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0037] The invention can set the best writing mode according to the host hardware information and HBase cluster information, and is composed of six functional modules independently developed: query module, configuration module, log module, conversion module, writing module and management module.

[0038] The query module queries data from the relational database through JDBC;

[0039] The configuration module is used to configure connections, tables, query conditions, result output formats, writing methods, etc.;

[0040] The log module records the status of each migration of each table;

[0041] The conversion module converts the result data found by the query module into the format required by the write module according to the corresponding HBase structure;

[0042] The write module writes data into HBase through the API according to the configuration module write parameters;

[0043] The management module starts the migration task at regular intervals and retransmits the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a method and system for migrating power measurement data to HBase. The system includes: a query module: querying data from a relational database through JDBC; a configuration module: used to configure connections, tables, query conditions, and result output formats , writing method, etc.; log module: record the status of each migration of each table; conversion module: convert the result data found by the query module into the format required by the writing module according to the corresponding HBase structure; writing module: follow the configuration module The write parameter writes data into HBase through the API; the management module: starts the migration task at regular intervals and retransmits the unsuccessfully migrated data according to the log information recorded by the log module. Compared with the prior art, the present invention has the advantages of automatic migration, support for incremental delay synchronization, automatic retransmission, and high efficiency.

Description

technical field [0001] The invention relates to the field of power data measurement, in particular to a method and system for migrating power measurement data to HBase. Background technique [0002] In order to improve the operation level of enterprises and further explore the potential value of data, it is necessary to migrate massive power measurement data from traditional relational databases to big data platforms. Through various data mining tools or query engines based on big data platforms, fast query, Advanced applications such as load forecasting and anomaly analysis. [0003] At present, it is possible to import relational databases into HBase through tools such as Sqoop, but it needs to be used in conjunction with other scheduling tools (such as Oozie) to achieve scheduled tasks, and there is no automatic supplementary transfer function, and it is impossible to automatically supplement data that fails to migrate. pass. Contents of the invention [0004] The pur...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/21G06Q50/06
CPCG06Q50/06
Inventor 苏鹏涛朱兵蔡晶吕山
Owner SHANGHAI SHINEENERGY INFORMATION TECH DEV CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap