Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Acceleration method for mass small document IO operation transmission in distribution type document system

A technology of distributed files and small files, applied in transmission systems, electrical components, special data processing applications, etc., can solve problems such as small file transmission bottlenecks, achieve the effect of reducing the number of network interactions and increasing transmission speed

Active Publication Date: 2014-04-09
上海浪潮云计算服务有限公司
View PDF4 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Due to the design principle of the distributed file system, it has inherent defects in solving the problem of massive small file transfer, which makes the existing cluster NAS system have a serious small file transfer bottleneck. figure 1 Shows the small file IO operation mode in the existing cluster NAS system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Acceleration method for mass small document IO operation transmission in distribution type document system
  • Acceleration method for mass small document IO operation transmission in distribution type document system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0013] Refer to the attached figure 1 Describe in detail the method for speeding up the transmission of massive small files in the distributed file system proposed by the present invention, the method involves the following main steps

[0014] 1. Client IO call interception. This step realizes the redirection of the IO operation of the client operating system by installing special software on the client, distinguishes between small files and large files by obtaining the file size, and uses the default IO path of the system for transmission of large files. Small files are intercepted.

[0015] 2. Merge traffic of massive small files. After the client intercepts the small file IO operation, it reads and encodes the file name, file length, file content, and file attributes of the small file, and writes it into the preset network path. At this time, the IO of a large number of small files is merged into the IO of a single large file. For the case where a large number of small ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an acceleration method for mass small document IO operation transmission in a distribution type document system. The method comprises the steps of buffering a large amount of small document operation at a sending end, executing flow aggregation, packing small documents into a single large document, transmitting the single large document to a receiving end in one time, analyzing the large document at the receiving end, obtaining the IO operation of each small document, and executing the writing operation of relevant document. The method greatly reduces the network interaction times and improves the transmission speed of mass small document IO operation.

Description

technical field [0001] The invention relates to the field of computer storage, in particular to a method for accelerating transmission of massive small files in cluster storage. Background technique [0002] With the rapid development of network applications, the amount of network information data is increasing, and PB-level mass data storage is becoming more and more important. The traditional file system can no longer meet the requirements of large capacity, high reliability, and high performance of existing applications. In order to meet these new requirements, the cluster NAS system developed based on the distributed file system has received extensive attention. [0003] The existing distributed file system research is mainly divided into separate management of metadata and actual data storage. Through the isolation of metadata and actual data, the dynamic increase and decrease of capacity and the current expansion of performance of distributed file system are realized. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08G06F17/30
Inventor 孟圣智
Owner 上海浪潮云计算服务有限公司
Features
  • Generate Ideas
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com