Index compression method and device, computer readable storage medium and electronic equipment

A technology of compression method and compression equipment, which is applied in computing, electrical digital data processing, digital data information retrieval, etc., can solve the problem of index occupying too much memory, and achieve the effect of reducing size and reducing size

Pending Publication Date: 2021-05-07
BEIJING UNISOUND INFORMATION TECH +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] The invention provides an index compression method, device, computer-readable storage medium and electr

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Index compression method and device, computer readable storage medium and electronic equipment
  • Index compression method and device, computer readable storage medium and electronic equipment
  • Index compression method and device, computer readable storage medium and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0042] The index compression method is applied to one or more electronic devices, and the electronic device is a device that can automatically perform numerical calculation and / or information processing according to preset or stored instructions, and its hardware includes but is not limited to Microprocessor, application specific integrated circuit (ApplicationSpecificIntegratedCircuit, ASIC), programmable gate array (Field-ProgrammableGateArray, FPGA), digital processor (DigitalSignalProcessor, DSP), embedded devices, etc.

[0043] The electronic device can be any electronic product capable of man-machine interaction with the user, for example, a personal computer, a tablet computer, a smart phone, a personal digita...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to an index compression method and device, a computer readable storage medium and electronic equipment. The method comprises the steps of dividing an original key of an index file into a high-order part and a low-order part; multiplexing the high-order part and the low-order part to represent a new offset; wherein when the offset is used as a new offset, the high-order part is used for representing an interval range of atom.dat in base.dat; the low-order part is used for representing an offset address relative to the atomic.dat; compressing the size of the original size into two bytes, and taking the two bytes as a new size; and forming a new index file based on the new key, the new offset, the new size and the original crc16. According to the method, the original key is segmented, only the low-order part is reserved and stored in the memory, the high-order part and the low-order part are reused as the offset, and the original size is compressed into two bytes to serve as the new size, so that the size of the index file is greatly reduced, and the size of the memory occupied by the index file is effectively reduced.

Description

technical field [0001] The invention relates to the technical field of storage indexing of massive distributed small files, in particular to an index compression method, device, computer-readable storage medium and electronic equipment. Background technique [0002] In the storage pressure scenario of a system that processes a large number of small files (such as files with a data size of a few KB or more than a dozen KB), especially the inode (index node) of the operating system will occupy part of the metadata, so the pressure on the kernel space is particularly large , when processing TB (Terabyte, terabyte) level data, there will be access delay phenomenon. Therefore, file aggregation technology is generally used now, and a large empty file is allocated on hdd (HardDiskDrive, hard disk drive) as a basic storage unit, named base.dat, and the size of each base.dat is set to 32GB. Each base.dat configures an index file named base.idx. Such as figure 1 As shown, base.idx ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/174G06F16/13
CPCG06F16/1744G06F16/13
Inventor 郭乃洪黄瑞祺陈吉胜
Owner BEIJING UNISOUND INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap