Access method and system for lots of small files

Abstract

The invention relates to an access method and system for lots of small files. The method comprises the following steps of compressing business files corresponding to obtained business data to obtain a file package; extracting a characteristic value of the business data according to a preset extraction rule; when the characteristic value is not stored in a file index database, encoding the characteristic value to obtain data coding according to a preset encoding rule, and storing a corresponding relation between the characteristic value and the data coding into the file index database; generating a corresponding directory structure according to the data coding and decompressing the file package to obtain the business files; storing the business files into a storage path of the directory structure; and reading the business files. The method and the system have the beneficial effects that lots of small files are effectively stored and read, the system access is simple, the efficiency of writing and reading mass data is high, and multidirectional elastic extension can be realized.

Description

technical field [0001] The invention relates to the field of data management, in particular to a method and system for accessing a large number of small files. Background technique [0002] When the existing data storage management system contains relational data and non-relational data, usually, network attached storage (Network Attached Storage, NAS) is used to store non-relational data, and relational database is used to store relational data. , and save the file path and file name on the NAS in the relational database. [0003] When the number of files is too large, the data access efficiency of this method will become slower and slower, and NAS expansion will create a new file system, which cannot be smoothly expanded, resulting in waste of storage space. If other systems need to access the file data on the storage, they must match the specific interface provided by this system. If the necessary parameters cannot be provided, the file resource cannot be used. [0004]...

AI Technical Summary

Problems solved by technology

[0003] When the number of files is too large, the data access efficiency of this method will become slower and slower, and NAS expansion will create a new file system, which cannot be expanded smoothly, resulting in waste of storage space

If other systems need to access the file data on the storage, they must match the specific interface provided by this system. If the necessary parameters cannot be provided, the file resource cannot be used

[0004] In addition, the number of files stored on the storage is generally the same as the data path and file name recorded in the database. When the amount of data reaches the order of 1 billion, the reading performance of the storage will drop significantly, which will greatly affect the efficiency of file access.

Images