Distributed storage system-oriented erasure coding write buffer method with stream detection technology

A technology of distributed storage and buffering method, which is applied in the field of erasure code write buffering for distributed storage system with streaming detection technology, which can solve problems such as excessive read operations, reduce read operations, improve command response speed and write The effect of operating bandwidth

Active Publication Date: 2016-02-03
HUNAN ANCUN TECH CO LTD
View PDF4 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to provide an erasure code write buffer method for distributed storage system with stream detection technology, so as to solve the technical problem of too many read operations in the existing overwrite storage process

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed storage system-oriented erasure coding write buffer method with stream detection technology
  • Distributed storage system-oriented erasure coding write buffer method with stream detection technology
  • Distributed storage system-oriented erasure coding write buffer method with stream detection technology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] 101. Allocate memory with a size of B bytes on the client side as a buffer layer for IO write operations. B is usually set to 1M, the maximum not exceeding 1G;

[0049] 102. Establish an IO write operation queue, and all buffered IO write operations form a queue in a linked list structure. The queue head pointer points to the first incoming IO write operation, and the queue tail pointer points to the last incoming IO write operation. The queue is initially empty.

[0050] 103. Receive an application IO operation within a specified time, that is, receive an IO operation from a user program;

[0051] "Receiving application IO operations within a specified time" means that the client waits to receive the IO operation commands issued by the user operation application. If the IO operation is not received after the preset time, it will be processed as timeout.

[0052] 104. Determine whether the operation has timed out;

[0053] 105. In the case of no timeout, judge wheth...

Embodiment 2

[0056] 201. Allocate memory with a size of B bytes on the client as a buffer layer for IO write operations. B is usually set to 1M, the maximum not exceeding 1G;

[0057] 202. Establish an IO write operation queue, and all buffered IO write operations form a queue in a linked list structure. The queue head pointer points to the first incoming IO write operation, and the queue tail pointer points to the last incoming IO write operation. The queue is initially empty.

[0058] 203. Receive an IO operation from a user program;

[0059] 204. Determine whether the operation has timed out;

[0060] 205. In the case of no timeout, determine whether the IO operation is a write operation;

[0061] 206. If it is a write operation, determine whether the data volume (for example 128KB) of the write operation is greater than the remaining space of the buffer layer;

[0062] 207. If it is larger than the remaining space (for example, 256KB), determine whether there are other write opera...

Embodiment 3

[0065] 301. Allocate memory with a size of B bytes on the client as a buffer layer for IO write operations. B is usually set to 1M, the maximum not exceeding 1G;

[0066] 302. Establish an IO write operation queue, and all buffered IO write operations form a queue in a linked list structure. The queue head pointer points to the first incoming IO write operation, and the queue tail pointer points to the last incoming IO write operation. The queue is initially empty.

[0067] 303. Receive an IO operation from a user program;

[0068] 304. Determine whether the operation has timed out;

[0069] 305. In the case of no timeout, judge whether the IO operation is a write operation;

[0070] 306. If it is a write operation, determine whether the data volume (for example 128KB) of the write operation is greater than the remaining space of the buffer layer;

[0071] 307. If it is less than the remaining space (for example, 64KB), insert the write operation into the buffer queue acc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a distributed storage system-oriented erasure coding write buffer method with a stream detection technology. The method comprises the steps that: when overwriting operations are performed on original data, the length of the first-time overwriting operation is from an initial position of data in a buffer layer to a first alignment point, and the length of the last overwriting operation is from a last alignment point to a termination position of the data in the buffer layer; and the lengths of the overwriting operations between the first-time overwriting operation and the last overwriting operation are all equal to the length between the adjacent alignment points, and the positions are aligned. Multiple write operations are combined in the buffer layer, and the write operations of only the head and tail ends of the data in the buffer layer are non-aligned, so that the read operations caused by the non-aligned write operations under the overwriting condition in an erasure coding storage system are reduced, and the command response speed is increased.

Description

technical field [0001] The invention relates to the technical field of computer storage, in particular to an erasure code write buffering method oriented to a distributed storage system with stream detection technology. Background technique [0002] Erasure coding (EC) is a data protection method, which divides the original data into several pieces, expands and encodes the redundant data blocks, and stores them in different physical locations. Erasure coding can be used in any application or system that has large amounts of data and requires fault tolerance, such as disk array systems, data grids, distributed storage applications, object storage, or archival storage. The protection provided by erasure codes can be expressed by a simple formula: K=M+N. The variable "M" represents the value of the original data or symbol, the variable "N" represents the value of the extra or redundant symbols added to provide protection, and the variable "K" represents the total value of symb...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/10G06F3/06
Inventor 王全宝郭家勇黄芳张诗芳邵江王剑雄刘友苗
Owner HUNAN ANCUN TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap