Method for uploading mass small files in distributed storage system

A technology of distributed storage and massive small files, applied in special data processing applications, instruments, electrical digital data processing, etc., can solve the problems of increasing management data, waste of hardware resources, high complexity, etc., to optimize the creation rate and protect the hard disk , The effect of prolonging the service life of the hard disk

Active Publication Date: 2016-06-01
SUGON INFORMATION IND
View PDF4 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In order to solve the first problem, file systems such as GPFS adopt the method of multi-data server, which has the disadvantage of wasting hardware resources and relatively high cost; for the second problem, file systems such as GoogleFS adopt multiple small file aggregation The way to increase the IO bandwidth of the hard disk is to increase the IO bandwidth of the hard disk. The disadvantage is that additional management data needs to be added, and the complexity is high.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for uploading mass small files in distributed storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0030] like figure 1 As shown, the uploading method includes the following steps:

[0031] (1) The client searches for the target file with the creation request according to the standard POSIX (PortableOperatingSystemInterface indicates Portable Operating System Interface) semantics of the operating system;

[0032] Wherein, the client is the entrance of the distributed file system, according to the standard POSIX (PortableOperatingSystemInterface represents Portable Operating System Interface) semantics of the file system, the search and creation actions of the merged file; if it is detected that the intention of this search is to create, Then the creation action is completed on the server side, and the corresponding file metadata is brought back by the search request.

[0033] (2) The metadata server pre-creates the file and establishes a file pool;

[0...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a method for uploading mass small files in a distributed storage system. The method comprises the steps of looking for a target file with a creation request by a client according to standard POSIX semanteme of an operation system; pre-creating files by a metadata server and establishing a file pool; aggregating management file metadata by utilizing the metadata server; and sorting the files through a kernel module and synchronously uploading the files in batch. Therefore, the problems of long creation delay and small amount of the small files are solved, the uploading rate is greatly increased, the hardware resource waste is reduced, and the cost is reduced.

Description

technical field [0001] The invention relates to an uploading method, in particular to a method for uploading massive small files in a distributed storage system. Background technique [0002] In today's digital age, with the increasing amount of data that needs to be stored, it is difficult for a single storage hardware device to meet the storage needs of a large number of industries in terms of capacity and performance. In order to meet the storage needs of a large number of industries for unstructured data, a number of distributed file systems have emerged, such as PNFS, GPFS, Lustre, GoogleFS, HDFS and so on. These distributed file systems manage the hardware clusters in a unified way through software, and present a unified storage pool to the outside, so as to achieve the purpose of virtualizing and integrating hardware resources. [0003] For the current distributed structure data storage, according to its storage, it can be roughly divided into large file storage and ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 杨浩马照云王利虎苗艳超刘新春邵宗有
Owner SUGON INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap