An HDFS storage system and a data storage method

A storage system and data storage technology, which is applied in the direction of electrical digital data processing, special data processing applications, redundancy in hardware for data error detection, etc., and can solve problems such as data loss and overloading

Inactive Publication Date: 2018-12-11
ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
View PDF3 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Since the logs are exported at a certain time interval, if the data is not synchronized to the slave NameNode before the failure of the master NameNode, it will cause data loss.
And only one NameNode provides external services at the same time, and there is a problem of overload

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • An HDFS storage system and a data storage method
  • An HDFS storage system and a data storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] The core of the present invention is to provide an HDFS storage system, which ensures the data consistency during the service switching process and does not lose the data to be stored during the switching process. The present invention also provides a data storage method, which has the above-mentioned embodiments.

[0027] The main NameNode of the existing HDFS storage system uses the method of reading mirror files to synchronize metadata. The main NameNode records the operations of the current system by writing log files, and writes the log information to the mirror files according to certain time rules. When it is found that the NameNode is switched, the slave NameNode will actively read the image file to obtain various states of the master NameNode, so as to achieve the data switching process. If the log file record is written before the image is written, the service is interrupted, which will cause data loss or data inconsistency.

[0028] In order to solve the sho...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an HDFS storage system, comprising: a plurality of metadata management nodes, a distributed system high-availability component connected with each metadata management node, a metadata storage pool. The metadata management nodes are used for receiving and processing a storage request of data to be stored. The distributed system high-availability component is used for transferring a storage request sent to the current metadata management node to another metadata management node when the current metadata management node corresponding to the current distributed system high-availability component is down. The metadata storage pool is used for storing the data to be stored, and the plurality of metadata management nodes establish communication links with the metadata storage pool. The HDFS storage system provided by the invention guarantees the data consistency in the service switching process and will not lose the data to be stored in the switching process. The invention also provides a data storage method, which has the beneficial effects.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to an HDFS storage system and a data storage method. Background technique [0002] HDFS is the storage component of Hadoop big data, which is responsible for the storage of overall data. NameNode is the metadata management module of HDFS. If there is a problem with NameNode, the overall HDFS storage system will be unavailable. For this reason, HDFS has pushed the high-speed system based on the active and standby mode. Available solutions. At the same time, the master NameNode is responsible for the data storage service of big data. If the master NameNode has a problem, the slave NameNode will take over the service to provide the overall big data storage service. [0003] In the architecture of the active and standby NameNodes in the traditional HDFS storage system, only the active NameNode is in the active state at the same time, and can receive data storage requests; the stand...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F11/20
CPCG06F11/20
Inventor 白学余海鑫高四辈
Owner ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap