Mass content storage system and method based on distributed big data blocks

Abstract

The invention provides a mass content storage system and method based on distributed big data blocks, and the system comprises a distributed storage module which integrates local storage disks on a whole distributed cluster node into a uniform distributed storage medium, formats the integrated distributed storage medium, and divides the integrated distributed storage medium into storage blocks with fixed sizes; a file block management module which is used for managing the formatted storage blocks and distributing the storage blocks to the file read-write module to read and write specific file contents; a file metadata module which is used for storing file metadata; a file reading and writing module which is responsible for reading and writing file contents; and a file block fragment arrangement module which is used for monitoring the fragment condition of the file storage block, arranging fragments of the file storage block, secondarily integrating effective storage into a complete storage block, and then releasing the original fragment storage block. According to the method, the small files are merged and stored by adopting the storage blocks, so that the number of the files is reduced, and the inode number limitation of a traditional file system is avoided.

Description

technical field [0001] The present invention relates to the technical field of data storage, in particular to a massive content storage system and method based on distributed large data blocks. Background technique [0002] Most of the existing technologies use content storage to realize enterprise content storage. Content storage generally stores file metadata in a relational database and stores files on a file system, such as IBM's CM and FileNet CE. However, the objective disadvantages of these content storages are: [0003] 1. A large number of small file metadata will occupy a large number of records in the relational database, resulting in the decline of relational database access performance, data backup, and inconvenient management, especially for files with a long storage period. [0004] 2. A large number of small files are stored on the file system, resulting in too many file system files and too many file directories. To solve this problem, most products use hash...

AI Technical Summary

Problems solved by technology

[0003] 1. A large number of small file metadata will occupy a large number of records in the relational database, resulting in a decline in the access performance of the relational database, data backup, and inconvenient management, especially for files with a long storage period

[0004] 2. A large number of small files are stored on the file system, resulting in too many file system files and too many file directories. To solve this problem, most products use hash to divide file directories, resulting in complex file structures and excessive management of file metadata by the operating system. many

[0005] 3. In order to ensure data reliability, SAN architecture storage is generally used to achieve unified storage and storage reliability, resulting in high cost, corresponding to massive data storage will become larger and larger, and long-term storage cannot be realized

Images