Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and device for real-time storage of massive small files based on time series aggregation algorithm

A technology for massive small files and small files, applied in file systems, file access structures, computing, etc., can solve problems such as low storage efficiency, and achieve the effects of improving storage efficiency, reducing consumption, and reducing load pressure

Active Publication Date: 2020-03-27
HARBIN INST OF TECH AT WEIHAI +1
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The present invention aims to solve the technical problem of low storage efficiency when the existing distributed file system is used for massive small files, and provides a method and device for real-time storage of massive small files based on time series aggregation algorithm with high storage efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for real-time storage of massive small files based on time series aggregation algorithm
  • Method and device for real-time storage of massive small files based on time series aggregation algorithm
  • Method and device for real-time storage of massive small files based on time series aggregation algorithm

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] Before introducing specific embodiments of the present invention, some concepts are explained as follows:

[0020] Object-based Storage is a distributed storage architecture that manages data in the form of objects. Small file objects usually refer to files with a file size below 5MB. Aggregation space is a logical concept. When small file objects are aggregated, the files in the aggregation space will be aggregated and stored in the distributed file system in the form of one or more data files.

[0021] MD5 encryption algorithm: MD5 is Message-Digest Algorithm 5 (Information-Digest Algorithm 5), which is used to ensure the integrity and consistency of information transmission, and is one of the hash algorithms widely used by computers. The algorithm has the following characteristics: 1. Compressibility: For any length of data, the length of the calculated MD5 value is fixed. 2. Easy to calculate: It is easy to calculate the MD5 value from the original data. 3. Anti-...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention relates to a method and device for real-time storage of massive small files based on time series aggregation algorithm, which solves the technical problem of low storage efficiency existing in the existing distributed file system when used for massive small files, and adopts data aggregation strategy to The time characteristic defines the aggregation space, completes the combined storage of time series data, and converts random writing into sequential writing, and the invention can be widely applied to the storage of massive small files.

Description

technical field [0001] The invention relates to a file storage method and device, in particular to a real-time storage method and device for massive small files based on a time series aggregation algorithm. Background technique [0002] Existing distributed file systems, including the underlying local file system, are mainly used to process large files. For a large number of small files, the storage performance is greatly reduced in terms of metadata management, data layout, and cache management. Expressed as: [0003] (1) Metadata management is inefficient. Distributed file systems are designed with a focus on high aggregate bandwidth for large files. As far as the file system of the local disk is concerned, accessing a file requires at least three independent accesses, including directory entries, index nodes, and data. Concurrent access to small files brings a lot of inefficient random access. At the same time, due to the low efficiency of the metadata organization ab...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/182G06F16/172G06F16/13
CPCG06F16/13G06F16/172G06F16/182
Inventor 朱东杰张凯赵奇隆杜海文曲荣宁顾天凯逄志弘毛尉茜李亚彭暄
Owner HARBIN INST OF TECH AT WEIHAI