Single-machine-oriented mass small record efficient storage and management method

A technology for storage management and recording, which is applied in the direction of memory system, electrical digital data processing, memory address/allocation/relocation, etc., can solve the problems that cannot meet the needs of massive small record storage management, etc., and achieve enhanced processing of complex data types ability, reduce hardware performance requirements, and facilitate backup effects

Inactive Publication Date: 2015-04-22
BEIJING CSBIT TECH
View PDF4 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The main purpose of the present invention is to solve the problem that the existing storage methods cannot meet the storage management requirements of massive small records in a stand-alone environment, and propose a stand-alone-oriented high-efficiency storage management method for massive small records, so that it can efficiently receive and manage records on a single computer. Store small records at scale

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Single-machine-oriented mass small record efficient storage and management method
  • Single-machine-oriented mass small record efficient storage and management method
  • Single-machine-oriented mass small record efficient storage and management method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] This embodiment is a description of related conventions such as data structures and parameters in the present invention.

[0049] Without loss of generality, figure 1 Schematic diagram of the storage structure for the B-tree. Create a log storage directory / data under a certain directory / Root / ... on the device disk. The / data folder is equivalent to the root node of the B-tree. cFile is the block file to be written to, and the directory where cFile is located is the current working directory cPath .

[0050] The invention adopts the idea of ​​automatic block, and connects multiple small records into a block file and stores them in the disk. The block file mentioned in the present invention refers to the file saved in the disk storage. The present invention defines the order of the B-tree as M, that is, the maximum number of folders under a single folder on a node is M, and the maximum number of files is M. Define the maximum size of a single chunk file as fSize. R...

Embodiment 2

[0058] This embodiment is an implementation of the data caching mechanism, such as image 3 shown. Since the received data recording speed is very fast, and the records are relatively small. If the disk write operation is performed immediately every time a record is received, the disk I / O read and write will be too frequent, the seek time will be too long, and the storage efficiency will be low. In order to avoid frequent disk writing operations, the present invention introduces a data cache mechanism. The new records received are not stored directly in the library, but are stored in the cache area first, and only when the cache space is full, the write operation is performed. This method can reduce the number of disk writes as much as possible and improve storage efficiency.

[0059] In the data cache mechanism, the present invention chooses to allocate a memory area in the memory as the data cache area in advance, and the size of the cache area is configurable. Such as ...

Embodiment 3

[0062] This embodiment is the process of the data cache stage, such as Figure 4 shown, including the following steps:

[0063] Step 401: Receive a new record;

[0064] Step 402: Generate a record identifier for the new record;

[0065] Step 403: Determine whether the memory buffer area is full, that is, whether its used size plus the new record size exceeds the maximum value of the set memory buffer, if not, go to step 405, otherwise go to step 404;

[0066] Step 404: Obtain the current block file cFile, write the data in the memory cache area into the current block file, clear the memory cache area, and set the value of the pointer buffered to null;

[0067] Step 405: Temporarily store the record in the memory buffer area, and adjust the pointer buffered to point to the end address of the data record in the memory buffer area.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a single-machine-oriented mass small record efficient storage and management method, and belongs to the technical field of information storage. The method comprises the following steps: (1) data receiving and buffering: receiving small record data, and storing the received small record data in a memory buffer; (2) automatic segmentation: when the memory buffer is full, writing the data in the buffer into files, wherein each file is limited in size to prevent the influence on performance due to too large sizes of single files, and new files are automatically created when single files are up to segmentation sizes; (3) tree directory generation: limiting the total quantity of files under each file folder to prevent the influence on performance due to excessive files under a directory, and implementing tree organization of the file folders and segmented files according to a simplified tree management algorithm; and (4) identifier recording: adopting a physical address directly recorded by a special identifier in order to bring convenience to the access of records. Compared with the prior art, the method has the advantage that a higher capability of managing mass small record data can be obtained with a lower requirement on hardware performance in a single-machine condition.

Description

technical field [0001] The invention relates to a storage management method for efficiently storing a large number of small records on a single computer, and belongs to the related technical field of information storage. Background technique [0002] With the development of Internet technology, especially the advent of the era of big data, massive amounts of data have been generated in many fields of the information society, and the data is growing explosively. The demand for mass storage is facing a severe test, which makes data storage management technology more and more important in the information system. [0003] In many application systems, it is often necessary to manage large-scale log records in a stand-alone environment. A typical scenario is security audit records. This scenario has the following characteristics: [0004] 1) The single record of this kind of record is usually relatively small. For example, in the audit log records in information security, each...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F12/06G06F12/08G06F12/0804
Inventor 戴林边彬彬夏明双曹正德于中徐厉雅洁张浩陈永红
Owner BEIJING CSBIT TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap