Test method and device for hdfs master node
A test method and master node technology, applied in the Internet field, can solve problems such as lower test efficiency, machine failure, and high machine resource overhead, and achieve high test efficiency and rapid construction
Active Publication Date: 2016-06-15
BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD
View PDF2 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] There are two problems in the stress test and performance test: 1. The machine resource overhead is large: with the expansion of the cluster size, the number of machines in the actual running HDFS is large, and the number of machines for testing generally cannot reach this level. kind of scale
In addition, this solution cannot avoid constructing a large amount of metadata in the master node, so it cannot solve the problem that this process is time-consuming and reduces test efficiency
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0022] Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention. On the contrary, the embodiments of the present invention include all changes, modifications and equivalents coming within the spirit and scope of the appended claims.
[0023] In the description of the present invention, it should be understood that the terms "first", "second" and so on are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance. In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "connected" and "connect...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a method and device for testing a primary node of an HDFS (Hadoop Distributed File System). The method comprises the steps of: forming multiple metadata files based on a preset configuration format; generating a metadata mirror image files in the primary node based on the multiple metadata files, wherein each metadata mirror image file comprises ID, number and size of multiple data blocks where each metadata file locates; using a simulator to simulate multiple secondary nodes of the HDF based on the ID, number and size of the multiple data blocks and establishing a mapping table of a relationship between multiple secondary nodes and multiple data blocks where each metadata file locates; establishing a thread between the mapping table and the primary node; transmitting the relationship between multiple primary nodes and multiple data blocks where each metadata file locates to a metadata mirror image file in the primary node through the thread; and using the thread to modify the mapping table so as to realize writing operation for the first file. According to the embodiment of the invention, the method has the advantages of saving machine resources and achieving high test rate.
Description
technical field [0001] The invention relates to the technical field of the Internet, in particular to a testing method and device for HDFS master nodes. Background technique [0002] HDFS (Hadoop Distributed File System) is a distributed file system, which is mainly used in Hadoop projects. It has the advantages of high fault tolerance and high data transmission rate, and is suitable for applications with very large data sets. [0003] There are mainly two types of nodes in the architecture of the HDFS file system: the master node (NameNode) and the slave node (DataNode). Among them, the master node is responsible for the management of file system metadata, including the name space of the file system, the mapping relationship between files and data blocks, and the mapping relationship between data blocks and slave nodes; the slave nodes are responsible for the storage of specific file content, and files are divided into multiple Data blocks are scattered and stored on diffe...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 潘瑾瑜
Owner BAIDU ONLINE NETWORK TECH (BEIJIBG) CO LTD