A storage method and device for hadoop distributed file system
A technology of distributed files and survival time, which is applied in the storage field of Hadoop distributed file system, can solve the problems that the status information of data nodes cannot fully represent the status of data nodes, the factors of metadata nodes are limited, and the analysis of information reliability, etc., to achieve good User experience, increasing the reliability evaluation mechanism, and the effect of good reliability guarantee
Active Publication Date: 2019-07-02
CHINA MOBILE GRP GUANGDONG CO LTD
View PDF9 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0006] (1) A heartbeat mechanism is provided to detect whether the data node is faulty, but the reliability analysis of the collected information is not performed
[0007] (2) The state information of the data node cannot fully represent the state of the data node, and the factors considered by the metadata node in the load balancing and data storage strategy are limited, which may lead to uneven load
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0051] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0052] The embodiment of the present invention provides a kind of storage method of Hadoop distributed file system, such as figure 1 shown, including:
[0053] Step S100, the metadata node receives the node status information fed back by the data node, calculates the health status of the data node according to the node status information of each data node, and obtains the health evaluation value of each data node;
[0054] Step S200, calculate the reliability of each data node according to the number of downtimes and the survival time after recovery from each downtime, and obtain the reliability evaluation value of each data node, wherein the reliability evaluation value of the data node increases with the number of downtimes and decreased, and inc...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a storage method and device for a Hadoop distributed file system. The method comprises the steps of calculating a health assessment value of each data node by a metadata node according to received node state information fed back by the data nodes, and selecting out a first preset number of first groups of data nodes which rank high according to a high-low sequence of the health assessment values; calculating a reliability assessment value of each data node according to a downtime frequency and a survival time each time after recovery from downtime, and selecting out a second preset number of second groups of data nodes which rank high according to a high-low sequence of the reliability assessment values; selecting out a third preset number of third groups of data nodes according to a preset storage strategy; and obtaining N target data nodes for current data storage through screening according to the three groups of data nodes. According to the storage method and device provided by the invention, the storage selection strategy and the selection algorithm which are simple and easy to operate and are based on multiple factors are adopted, so the selection efficiency of the data nodes is improved, and the better user experience is brought.
Description
technical field [0001] The invention relates to the technical field of data processing control, in particular to a storage method and device of a Hadoop distributed file system. Background technique [0002] Hadoop has the characteristics of scalability, scalability, and fault tolerance, and has been widely used in recent years. How to efficiently, reliably and reasonably store massive amounts of data is particularly important. HDFS (Hadoop Distributed File System) is a distributed file system of Hadoop. HDFS cloud storage system, as the core sub-project of Hadoop, is responsible for data storage and management, and has become one of the hotspots of cloud storage research. [0003] HDFS adopts the master-slave structure model, and the HDFS cluster consists of a metadata node and several data nodes. Among them: the metadata node acts as the master server, managing the namespace of the file system and the client's access to files. Data nodes manage stored data, are responsi...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/182G06F16/11
CPCG06F16/122G06F16/182
Inventor 潘毅喻朝新张静娴朱定局
Owner CHINA MOBILE GRP GUANGDONG CO LTD