Hadoop-based panoramic big data distributed storage method

A distributed storage and big data technology, applied in relational databases, database models, data processing applications, etc., can solve problems such as low access efficiency, poor disaster recovery robustness, database storage redundancy, etc., and achieve high reliability disaster recovery security Improve performance, reduce system service dependence, and improve response speed

Inactive Publication Date: 2016-10-26
STATE GRID HENAN ELECTRIC POWER ELECTRIC POWER SCI RES INST +3
View PDF4 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a hadoop-based panoramic big data distributed storage method, which can provide a new type of big data distributed storage method for smart grid big data, and can effectively solve the storage redundancy and access problems of traditional relational databases. Low efficiency, poor disaster recovery robustness and other issues

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hadoop-based panoramic big data distributed storage method
  • Hadoop-based panoramic big data distributed storage method
  • Hadoop-based panoramic big data distributed storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] Such as figure 1 Shown, the present invention comprises the following steps:

[0028] The first step is to optimize the design of the Hbase table based on distributed data storage and access, as follows:

[0029] First of all, for the distributed database row key design, in the Hbase design, the row key design is the most critical part, which is directly related to the access performance of subsequent services. If the row key design is unreasonable, it will have a great impact on subsequent query services, and the efficiency will decrease exponentially. The following are the rules to follow in the design process:

[0030] (1) Avoid using monotonically increasing row keys; in the process of using Hbase, when performing a single-threaded full table scan, it can be found that all requests will be concentrated on a single Region, and only after completing all scans of the current Region, proceed to the next For a Region, if the stored Regions are concentrated on one node,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a hadoop-based panoramic big data distributed storage method, can provide a novel big data distributed storage method for big data of intelligent grids, and can effectively solve the problems of storage redundancy, low access efficiency, poor disaster recovery robustness and the like of the traditional relational database. The hadoop-based panoramic big data distributed storage method comprises the steps of firstly, optimizing the design of an Hbase table based on distributed data storage and access to minimize and fixedly set the row key length, then performing load balance, JVM (Java Virtual Machine) optimization and split-and-merge service optimization on Hbase system performance, and finally, optimizing large-scale small files of an HDFS (Hadoop Distributed File System) by adopting a labeling method. The method obviously improves the power big data distributed storage technology on storage redundancy optimization, fast access, high efficiency and high reliability and disaster recovery security.

Description

technical field [0001] The invention relates to the technical field of power data analysis, in particular to a hadoop-based distributed storage method for panoramic big data. Background technique [0002] In the existing smart grid business data mining, smart grid operation and equipment detection or monitoring data, power company marketing data and power company management data all have the following characteristics: 1) There are many types of data and a large amount of data; The collected data sent by numerous power equipment and monitoring instruments constitutes a large amount of real-time status data that needs to be continuously received and processed by the power information system. 2) The data format is not uniform and has poor versatility; for a long time, the communication rules between state monitoring devices and systems introduced by domestic and foreign power automation equipment manufacturers are not uniform, functions and interfaces are different, and differe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06Q50/06
CPCG06F16/284G06F16/2219G06Q50/06
Inventor 李强马建伟孙芊周凤珍杨磊王鹏王文博黄伟邹会权肖寒赵理
Owner STATE GRID HENAN ELECTRIC POWER ELECTRIC POWER SCI RES INST
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap