Index compression method and device, computer readable storage medium and electronic equipment

Abstract

The invention relates to an index compression method and device, a computer readable storage medium and electronic equipment. The method comprises the steps of dividing an original key of an index file into a high-order part and a low-order part; multiplexing the high-order part and the low-order part to represent a new offset; wherein when the offset is used as a new offset, the high-order part is used for representing an interval range of atom.dat in base.dat; the low-order part is used for representing an offset address relative to the atomic.dat; compressing the size of the original size into two bytes, and taking the two bytes as a new size; and forming a new index file based on the new key, the new offset, the new size and the original crc16. According to the method, the original key is segmented, only the low-order part is reserved and stored in the memory, the high-order part and the low-order part are reused as the offset, and the original size is compressed into two bytes to serve as the new size, so that the size of the index file is greatly reduced, and the size of the memory occupied by the index file is effectively reduced.

Description

technical field [0001] The invention relates to the technical field of storage indexing of massive distributed small files, in particular to an index compression method, device, computer-readable storage medium and electronic equipment. Background technique [0002] In the storage pressure scenario of a system that processes a large number of small files (such as files with a data size of a few KB or more than a dozen KB), especially the inode (index node) of the operating system will occupy part of the metadata, so the pressure on the kernel space is particularly large , when processing TB (Terabyte, terabyte) level data, there will be access delay phenomenon. Therefore, file aggregation technology is generally used now, and a large empty file is allocated on hdd (HardDiskDrive, hard disk drive) as a basic storage unit, named base.dat, and the size of each base.dat is set to 32GB. Each base.dat configures an index file named base.idx. Such as figure 1 As shown, base.idx ...

AI Technical Summary

Problems solved by technology

[0011] The invention provides an index compression method, device, computer-readable storage medium and electronic equipment, which can solve the technical problem that the index occupies too much memory in the prior art

Images