Processing method and processing device of mass data

Abstract

The invention provides a processing method and a processing device of mass data. The method comprises the followings: a data file and an index file are set, wherein the data file comprises at least one file data block, and the length of each file data block is equal; the index file corresponds to the data file, comprising the address offset of each data object in the data file; when the data object is accessed, if the free space of the file data block currently appointed by a cursor is not enough to store the data object to be stored, the data file applies one or more new file data blocks to the system after the free space is filled with the data object to be stored, and the remaining data that is not accessed in the data object to be stored is stored in the new file data block, and the address offset of newly accessed data object is recorded in the index file; and the stored data object is read in the data file in a manner that the cursor points to the address offset of the data object to be read. The application of the technical proposal improves the access efficiency for mass data.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method and device for processing massive data. Background technique [0002] In the data processing industry, more and more data needs to be processed, and the data files are getting bigger and bigger. At present, for the access problem of massive data, it generally comes down to the following two methods: [0003] 1) Incorporate massive data into the database for management; [0004] 2) Create and serialize the data objects and store them in ordinary files. [0005] In the process of realizing the present invention, the inventor finds that there are at least the following problems in the prior art: [0006] Method 1) is a popular mass data management method. Although the database has certain advantages in managing large files and mass data, since this method uses a database for data management, the biggest bottleneck for mass data is efficiency. For systems with high efficie...

AI Technical Summary

Problems solved by technology

[0006] Method 1) is a popular mass data management method. Although the database has certain advantages in managing large files and mass data, since this method uses a database for data management, the biggest bottleneck for mass data is efficiency. For systems with relatively high efficiency requirements, the database cannot meet the high efficiency requirements;

[0007] And method 2) can only solve the problem of small data volume, and cannot meet the storage requirements of massive data

[0008] Therefore, fast access to large data volume files has increasingly become the technical bottleneck of the industry, and the traditional method of reading and writing files through I / O is difficult to meet the requirements of massive data and large files

Images