Mass small file storage performance optimization method and device based on real-time merging

A technology with massive small files and storage performance, applied in file systems, file access structures, special data processing applications, etc., can solve problems such as being unsuitable for large-scale file storage and low storage efficiency of small files on HDD disks

Inactive Publication Date: 2021-02-26
南京群顶科技股份有限公司
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, there is a need for a method that can solve the problems that the existing file storage directory tree is not suitable for large-scale file storage and the low storage efficiency of HDD disk small files.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass small file storage performance optimization method and device based on real-time merging
  • Mass small file storage performance optimization method and device based on real-time merging

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] see figure 1 , figure 1 Shown is a schematic flowchart of a method for optimizing storage performance of massive small files based on real-time merging provided by the embodiment of the present application. A method for optimizing the storage performance of massive small files based on real-time merging, including S1: the server receives the file data storage request sent by the client, and writes the files to be stored into SSD and HDD storage resource pools hierarchically through object storage;

[0031] In step S1, when the above-mentioned file to be stored is a large file, it is stored in the above-mentioned HDD storage resource pool, when the above-mentioned file to be stored is a small file, it is stored in the above-mentioned SSD storage resource pool, and the metadata information of the file is stored in the file index pool; according to The globally unique identifier key of the file generated by the above metadata information.

[0032] In detail, the server r...

Embodiment 2

[0058] see figure 2 , figure 2 Shown is a schematic diagram of a principle of a device for optimizing storage performance of massive small files based on real-time merging provided by the embodiment of the present application. A device for optimizing storage performance of massive small files based on real-time merging, including:

[0059] Data request receiving module: used to receive file data writing or access requests sent by the client;

[0060] File data hierarchical storage module: Generate a unique identity key according to the received metadata information of the file to be stored, and the file content data value value corresponding to the key; divide the file into large files and small files with 1M bytes as the dividing line For files, the value corresponding to large files is written into the HDD storage resource pool, and the value corresponding to small files is written into the SSD storage resource pool;

[0061] Small file data merging module: set the trig...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a mass small file storage performance optimization method and device based on real-time merging, and relates to the field of computer storage. The mass small file storage performance optimization method based on real-time merging comprises the steps that a file data storage request sent by a client is received, and files to be stored are written into SSD and HDD storage resource pools in a layered mode in an object storage mode; wherein when the to-be-stored file is a large file, the to-be-stored file is stored in the HDD storage resource pool, and when the to-be-storedfile is a small file, the to-be-stored file is stored in the SSD storage resource pool, and metadata information of the file is stored in a file index pool; and generating a globally unique identifierkey of the file according to the metadata information. According to the method, the writing speed of the small files can be increased, and the massive small files can be reasonably stored.

Description

technical field [0001] The invention relates to the field of computer storage, in particular to a method and device for optimizing storage performance of massive small files based on real-time merging. Background technique [0002] With the rapid development of the Internet, Internet of Things, cloud computing, big data and other industries, the number of various files such as audio and video, pictures, and logs has shown an exponential growth. Terminal devices need to upload a large number of files without interruption. Among them, small files below 1M The number often reaches millions, tens of millions, or even billions, and such files are defined as massive small files. Massive small files have higher requirements on the writing performance of terminal devices, but lower requirements on reading performance. How to properly store massive small files is of great significance to the sustainable development of the current big data era. [0003] Traditional file storage syste...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/174G06F16/13
CPCG06F16/13G06F16/174
Inventor 杨鹏杨波
Owner 南京群顶科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap