Unlock instant, AI-driven research and patent intelligence for your innovation.

Small file access system based on distributed storage technology and method thereof

A distributed storage and access system technology, applied in the field of data storage, can solve the problems of high network communication overhead, inefficient Cache management, complex I/O access mode, etc., and achieve the effect of improving performance, fast storage and reading

Active Publication Date: 2017-10-13
STATE GRID CORP OF CHINA +3
View PDF7 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to provide a small file access system and method based on distributed storage technology, which adopts a reasonable hardware structure and distributed storage technology to solve the problem of large network communication overhead and unreasonable data layout when accessing small files And the problem of complex I / O access mode and inefficient Cache management

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Small file access system based on distributed storage technology and method thereof
  • Small file access system based on distributed storage technology and method thereof

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0025] Such as figure 1 A described small file access system based on distributed storage technology includes a client server, a core server, a core layer network switch, and several storage node servers, and the core server includes a multi-core CPU, a solid state drive, a 10G network card or an InfiniBand network card , PCI multi-function parallel port card and 64G memory, solid-state hard drive, 10 Gigabit network card or InfiniBand network card, PCI multi-function parallel port card and 64G memory are all electrically connected to the multi-core CPU, and the core server is connected to the core layer network through 10 Gigabit network card or InfiniBand network card Switch communication, the client server communicates with the core layer network switch through the Internet, and each storage node server communicates with the core layer network switch through the Internet.

[0026] The multi-core CPU of the core server is provided with multi-path I / O channels.

[0027] Each...

Embodiment 2

[0033] A small file access method based on a distributed storage technology-based small file access system described in Embodiment 1: comprising the following steps:

[0034] Step 1: classify the size of the stored files into small files and large files, the core server judges the size of the file to be stored, if it is a small file, execute step 2, if it is a large file, execute step 2;

[0035] Step 2: The core server appends the tail identifier of BLOCK to the tail of the small file;

[0036] Step 3: the core server creates metadata for small files, and stores the metadata in the solid hard disk;

[0037] Step 4: The core server sends the small file and the metadata to the Cache memory of a storage node server through the core layer network switch. When the storage node server is full, the core server sends the small file to the In another storage node server Cache memory;

[0038] Step 5: the storage node server merges multiple small files in its own Cache memory into on...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a small file access system based on a distributed storage technology, and belongs to the field of data storage. The system comprises a client server, a core server, a core layer network switch, and a plurality of storage node servers. A rational hardware structure and a distributed storage technology are used, and problems that network communication overhead is high, data layout is not rational, I / O access mode is complex, and Cache management is low in efficiency in small file access are solved.

Description

technical field [0001] The invention belongs to the field of data storage. Background technique [0002] Current file systems, including local file systems, distributed file systems, and object storage systems, are mainly designed for large files, and they all focus on large files in terms of metadata management, data layout, stripe design, and cache management. , while the performance and storage efficiency of applications with a large number of small files will be greatly reduced, or even unable to work. With the widespread application of distributed systems and the appearance of massive small files, the problem of small files in the storage process of distributed file systems becomes more prominent. The problem of small files in the storage process is mainly manifested in the following aspects: inefficient metadata management, inefficient data layout, and complicated I / O access process. Contents of the invention [0003] The purpose of the present invention is to prov...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/172G06F16/182
Inventor 钱琳庞恒茂朱广新李东辉俞俊吴小志陈艳谢宾铭李蹊高方超
Owner STATE GRID CORP OF CHINA