Method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase

A multi-source heterogeneous and big data technology, applied in structured data retrieval, electronic digital data processing, relational database, etc., can solve problems such as multi-time overhead, inability to quickly migrate power data formats, and failure to consider space efficiency

Active Publication Date: 2017-03-22
ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER COMPANY +2
View PDF7 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, there is no reduction in the amount of transmitted data, and at the same time, because of JSON, it requires more time overhead; at the same time, this method does not take into account the problem of space efficiency
The data model is proposed in the patent [2], but this model is aimed at the ...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase
  • Method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase
  • Method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0048] Oracle database is a relational database developed by Oracle Corporation. It is mainly aimed at enterprises. It has high stability and supports complex queries and other operations. MySQL is an open source relational database owned by Oracle Corporation. It has the advantages of free, compact, easy to use, and high stability. A CSV file is a character-separated value file, which is often used to express a data table. txt file is a text file, which has the advantages of simple data structure and cross-platform. At the same time, HBase is a highly stable, highly reliable, column-oriented scalable distributed database based on the Hadoop platform. Using HBase technology, you can use cheap servers to build super-large-capacity storage clusters.

[0049] The fast migration method proposed by the present invention takes into account the problem of d...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and system for rapidly transferring multi-source heterogeneous power grid big data to HBase. The method includes the following steps that a power grid database is connected with an HBase data platform; each datum in the power grid database has a unique mark ID and is composed of the mark ID and data content; according to unique mark IDs, an ID mapping table is established; after a data source is selected, the ID mapping table is read and written into a memory for residence; codes in the ID mapping table are each compressed, so that data class ID row keys and data class internal ID row keys are formed; the data class ID row keys and the data class internal ID row keys are converted into a byte array; the data class ID row keys, the data class internal ID row keys and the row name qualifier of HBase are written into an HBase table in a byte form. Rapid transferring is achieved for a complex electric power data format through the method and system.

Description

technical field [0001] The invention relates to the field of power system information and technical communication, in particular to a method and system for rapidly migrating multi-source heterogeneous power grid big data to HBase. Background technique [0002] In the power grid system, power transmission and transformation equipment is the main equipment. The state of power transmission and transformation equipment involves the safety of the entire power grid system and the normal operation of the system. The data of the power grid system is mainly the state data of power transmission and transformation equipment. The data related to the status of power transmission and transformation equipment include runtime gas data QX, micro-meteorological data WQX, equipment ledger data, EMS data and GIS data, etc. At the same time, the status data of the equipment is time-series data, and the data volume will grow rapidly over time. Therefore, the power grid system data has the char...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/214G06F16/221G06F16/27G06F16/284
Inventor 林颖张方正苏建军李冬郭志红陈玉峰祝永新盛戈皞李勇吴观斌杨祎朱城铠刘斌王畅马艳李程启耿玉杰白德盟石鑫张雷
Owner ELECTRIC POWER RESEARCH INSTITUTE OF STATE GRID SHANDONG ELECTRIC POWER COMPANY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap