Adaptive granularity erasure code encoding and decoding acceleration method and system based on load awareness
A load sensing, encoding and decoding technology, applied in the field of erasure coding encoding and decoding acceleration, can solve the problems of low encoding and decoding performance, reducing user experience, high read and write delay, etc., to improve computing bandwidth, reduce encoding and decoding delay, The effect of low latency
Active Publication Date: 2021-10-15
INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
In 2013, StreamScale released their erasure code implementation BigParity, but this implementation is not open source
In the current mainstream storage architecture, the client provides data reading and writing services in the form of multi-threads. The data encoding and calculation are completed in the data writing thread, and the data decoding and calculation are in the process of degraded reading of temporarily invalid data. Encoding and decoding performance leads to high read and write delays, which reduces user experience
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0064] Encoding and decoding optimization is aimed at accelerating the encoding and decoding process of erasure codes, and requires a load-aware acceleration mechanism for adaptive encoding granularity. Through the evaluation of the codec performance, it can be concluded that:
[0065] 1) Single-threaded encoding and decoding calculations cannot obtain the maximum encoding performance;
[0066] 2) The concurrency of encoding strips is different under different segmentation granularities. Under the optimal segmentation granularity, concurrent encoding can achieve peak performance;
[0067] 3) The optimal segmentation granularity corresponding to encoded data of different sizes is different;
[0068] 4) Encoded data with a large amount of data can obtain higher encoding bandwidth at the optimal segmentation granularity than encoded data with a small amount of data.
[0069] In the actual storage environment, the encoding and decoding calculation operations in the storage syste...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a self-adaptive granularity erasure code encoding and decoding acceleration method and system based on load awareness. The calculation pressure of the current node is judged according to the coding and decoding load of the current node, the current optimization direction is distinguished according to the calculation pressure, data segmentation and calculation are carried out according to the optimal segmentation granularity distribution diagram when facing the low load, and self-adaptive granularity aggregation is carried out according to the task pressure when facing the high load.
Description
technical field [0001] The invention relates to the fields of data storage and erasure codes, in particular to an erasure code encoding and decoding acceleration method and system based on load-aware adaptive granularity. Background technique [0002] In the era of big data, the storage of massive data is facing a contradiction between ensuring reliability through redundancy mechanisms and improving space utilization and reducing storage costs. At present, the storage fleet is mostly composed of commercial servers, and normal component failures threaten the reliability of the storage system. Disk arrays, replicas, and erasure coding are the mainstream mechanisms used to ensure data reliability. However, disk arrays are expensive and cost-effective. They mainly provide large-capacity storage services in a single machine and expand the storage capacity of a single machine. However, they cannot guarantee node-level data reliability in a distributed system and have poor scalabi...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06G06F9/50G06F11/10
CPCG06F3/067G06F11/1004G06F9/505Y02D10/00
Inventor 魏征邢晶谭光明孙凝晖
Owner INST OF COMPUTING TECH CHINESE ACAD OF SCI