Unlock instant, AI-driven research and patent intelligence for your innovation.

Method for using lustre file system to replace HDFS file system of Hadoop

A file system and file technology, applied in the computer field, can solve the problems of not supporting the RDMA protocol, time-consuming, not large data transmission, etc., to achieve the effect of highlighting substantive features, reducing load, and filling gaps

Inactive Publication Date: 2017-07-07
ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
View PDF6 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

HTTP does not support the RDMA protocol (the RDMA protocol is commonly used and often required by the current distributed file system), so it is not a good choice for large data transmission
[0007] 3) Hadoop is used for Map / Reduce jobs, it is difficult to use HDFS as a normal file system extension
[0008] 4) For small files, using Hadoop is time-consuming

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for using lustre file system to replace HDFS file system of Hadoop

Examples

Experimental program
Comparison scheme
Effect test

specific Embodiment approach 1

[0042] figure 1 It is a method flowchart of the method for replacing the HDFS file system of Hadoop with the lustre file system according to the present invention. The method of replacing Hadoop's HDFS file system with lustre file system includes the following steps A-D:

[0043] Step A. Install and configure the lustre file system.

[0044] This step A completes the installation and configuration of the lustre file system through the following method steps:

[0045] s1. Initialization steps: turn off the firewall and selinux, add the node names of Hadoop to the host file, configure SSH to log in without password, edit the / etc / modprobe / lustre.conf file, and make the lustre module use the IB network;

[0046] s2, lustre server installation and configuration steps: install the lustre server software and the e2fs used with it on the mds server and oss ​​server respectively, configure the mdt server, mgs server, and oss ​​server, and then load the lustre module;

[0047] s3, lustre clien...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method for using a lustre file system to replace an HDFS file system of Hadoop. The method comprises the steps that A, the lustre file system is installed and configured; B, the lustre file system installed and configured in the step A is mounted; C, Hadoop is installed, and all configuration files corresponding to the installed Hadoop are configured; and D, a Hadoop-lustre-plugin-<HAL Version>.jar package is added under a Hadoop node class library directory corresponding to the Hadoop installed in the step C, and the corresponding configuration files of the Hadoop in the step C and relevant environment variables are modified so that the Hadoop installed in the step C can make direct access to the lustre file system mounted in the step B. Through the method, the performance of a Hadoop system can be improved, and expandability of the Hadoop system can be enhanced.

Description

Technical field [0001] The invention relates to the field of computer technology, in particular to a method for replacing the HDFS file system of Hadoop with a lustre file system. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. It implements a distributed file system (Hadoop Distributed File System, referred to as HDFS), allowing users to develop distributed programs without knowing the underlying details of distributed. [0003] In addition, HDFS has the characteristics of high fault tolerance, and is designed to be deployed on low-cost hardware; and it provides high throughput (high throughput) to access application data, especially suitable for processing large Data set (large data set) application. In addition, as a distributed data and computing framework, Hadoop is especially suitable for storing large amounts of semi-structured data sets, and it is very suitable for distributed computing-quickly processing lar...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/445G06F9/44
CPCG06F8/61G06F8/71G06F16/18
Inventor 魏桂宝
Owner ZHENGZHOU YUNHAI INFORMATION TECH CO LTD