Method and device for document loading

Abstract

The invention provides a method and device for document loading. The method comprises the steps that a memory mapping document of a magnetic disk document is established by the document loading device; data is read successively from the memory mapping document, and pre-reading notices are sent to an operation system for multiple times; each time, the operation system is notified to pre-read the data at designated positions with a designated data size in the magnetic disk document to memory corresponding to the memory mapping document; and the document loading device stores the data read from the memory mapping document to a final data structure of the memory according to a storage format of the data. The document loading device comprises a data loading module, a document reading module and a memory management module. The method and device for the document loading provided by the application have the advantages that document loading speed can be increased; and a memory utilization amount during loading can be kept stable and controllable.

Description

technical field [0001] The present invention relates to the field of computers, and more specifically, to a file loading method and device. Background technique [0002] In a large-scale distributed file system, in order to meet the high-concurrency and low-latency access requirements of a large number of clients, the metadata server (Meta Server) usually stores all metadata in memory, and at the same time records the metadata operation log (operation log ) and periodically generate a metadata memory image file (checkpoint file) to achieve persistent storage of metadata. A metadata memory image file usually includes multiple data sections and a header. The data area records a large amount of metadata such as directory path tree, file table, and data copy location, and compresses large blocks of data (for example, more than 4KB) to save space; the file header records the start and end information of the data area, Compression parameters, etc. [0003] Fast loading of metad...

AI Technical Summary

Problems solved by technology

[0006] This method has at least the following two problems that affect the loading speed: 1. The overhead of the internal implementation of the file library: in order to maintain the internal buffer and ensure the safety of concurrency, the file library (for example, glibc file) needs to perform multiple operations for each operation. Item status value update and maintenance work

[0008] This method also has the following two problems: 1. The memory usage of the process exceeds the limit: after the memory-mapped file is accessed, the memory page (Page) containing the corresponding data is included in the resident memory of the process, resulting in it being used up data also consumes memory on the metadata server

Starting from the versatility of the operating system, the amount of pre-reading at one time is not large, which often cannot meet the needs of parallel processing, resulting in page fault interrupts, which is also not conducive to improving disk throughput.

Images