Method and system for reading and writing data for hadoop computing

Abstract

The invention provides a method and system for reading and writing-in of data used for HADOOP computation. The method is used for reading input data used for HADOOP computation from a non-HDFS, and is characterized by comprising the steps that data reading classes used for reading the data from the non-HDFS are defined, wherein the data reading classes inherit the Record Reader classes; a get Record Reader method and a get Splits method in Input Format classes in an HADOOP distributed type computation model are achieved, wherein in the achieved get Record Reader method, examples or objects of the defined data reading classes are created and are returned, and in the achieved get Splits method, the Record Reader examples or objects returned from the called get Record Reader method are determined as the examples or objects of the defined data reading classes. The strong dependence on an HDFS by the HADOOP is removed, the reading frequency of the data and the copying frequency of the data are decreased, storage space is saved, and processing time is shortened.

Description

technical field [0001] The present invention relates to a technology for distributed computing, in particular to a method and system for processing data used for distributed computing. Background technique [0002] The distributed computing platform is based on the distributed storage platform and is used to process the data in the distributed storage platform. The popular MapReduce computing model in recent years can use distributed computing power to process big data, and this convenience also stimulates data computing needs. [0003] However, the MapReduce computing model also has constraints such as strong binding between distributed computing platforms and storage platforms. Take HADOOP, which is a specific implementation of the MapReduce computing model, as an example. The HADOOP computing platform requires that the data it calculates must be stored on the storage platform (HDFS) that is strongly bound to it. This brings inconvenience to the specific application of H...

AI Technical Summary

Problems solved by technology

This brings huge consumption and wastes a lot of storage space (for example, more than 2 copies of data need to be stored), transmission bandwidth (a large amount of data transmission occupies bandwidth) and processing time (because there are many single points in the processing, resulting in overall time-consuming)

Images