Method for replacing Hadoop storage module by using PVFS (Parallel Virtual File System)

Abstract

A method for replacing HADOOP storage module by using a PVFS (Parallel Virtual File System) utilizes the parallel virtual file system PVFS to replace a Hadoop distributed type file system HDFS module. By virtue of the method, the connection from HADOOP to the PVFS is realized; and three modules including a PVFS program interface, a HADOOP-PVFS module and a JNI (Java Native Interface) connection module are provided. The method provided by the invention aims at selecting the more suitable distributed type file system as the storage module of the HADOOP to replace the HDFS, so that the expenditure of the HADOOP in the aspect of file operation is reduced, Map Reduce calculation of the HADOOP is improved, and particularly, the representation of data dense type calculation is improved.

Description

technical field [0001] The invention belongs to the field of computer technology, relates to distributed computing and distributed file systems, and in particular to the connection between distributed computing architecture and distributed file systems, specifically a method of connecting PVFS distributed file systems to Hadoop to replace Hadoop The implementation method of the original storage module HDFS. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed computing and storage. Hadoop is currently the most widely used distributed computing platform. It adopts the MapReduce distributed computing model and provides a series of interfaces and frameworks to help users efficiently utilize the computing resources of distributed clusters and improve the parallelism ...

AI Technical Summary

Problems solved by technology

In the early HDFS, there was only one NameNode, which may cause a serious single point of failure. When the NameNode crashes, all unsaved data will be lost; in later versions, HDFS added multiple secondary NameNodes, and when the primary NameNode crashes, the secondary NameNode continues to run, but there is still only one NameNode running at the same time, which is likely to cause network congestion when file access is frequent

[0006] 2. HDFS uses multiple copies to access files in parallel, and nodes can only perform parallel read and write operations in units of copies

Although this can improve the overall throughput of the file system, in order to meet the access requirements of different nodes for the same data file, HDFS must create enough copies, otherwise it will lose its parallelism

[0007] 3. HDFS stores data on the corresponding computing nodes, which saves network overhead, but with the development of computer network technology, the time spent on network communication has been one to two orders of magnitude lower than that of I / O operations. Almost negligible; the price HDFS pays for saving network overhead is the complete absence of intra-file parallelism

[0010] But as far as the current technology is concerned, in order to achieve higher performance, Hadoop is very closely integrated with HDFS. The virtual methods provided by the virtual classes related to the file system in Hadoop basically correspond to the upper layer interface of HDFS. However, this It brings inconvenience and difficulty to use other file systems instead of HDFS

Images