Erasure pool creating method based on erasure codes and related device

Abstract

The invention discloses an erasure pool creating method and device based on erasure codes, a computer-readable storage medium and a server. The method includes the steps of modifying erasure strip size allocation information in a processing flow for creating the erasure pool into erasure strip unit allocation information; obtaining a creating instruction, and parsing the creating instruction to obtain erasure strip unit data, wherein the erasure strip unit data is set according to storage files; obtaining the erasure codes, and multiplying the data block values in the erasure codes with the erasure strip unit data to obtain strip size data; according to the strip size data, creating the erasure pool. By setting the strip size to prevent writing of excessive data, the overall read-write performance of a system is improved, an erasure algorithm can be better adapted through different erasure strip sizes, the running efficiency of the algorithm is improved, the consumption of a CPU is reduced, and since different erasure strip sizes can be applied to different file types, the whole IOPS is improved, and the system can show out the optimal performance.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a method for creating an erasure code-based erasure correction pool, a creation device, a computer-readable storage medium, and a server. Background technique [0002] With the development of computer technology, more and more data are stored in servers, and a single server cannot meet the requirements of storing data, so a distributed storage system has been developed. Among them, CEPH is currently a widely used distributed storage system. [0003] In the face of a large amount of data storage, the reliability of data storage has to be considered. The commonly used erasure code technology makes certain redundancy to the data to increase the reliability of the system, and divides the stored data into K data blocks. The algorithm calculates M check blocks, and any K data blocks among the K+M data blocks can be used to restore the original data. By setting the era...

AI Technical Summary

Problems solved by technology

[0004] However, in the process of calculating data blocks, in addition to the erasure code, the stripe size determines the number of data blocks. When data of different types and different file sizes are calculated based on the same stripe size, in order to make the length of the data equal to the stripe size If the length of a data block satisfies one data block, it will be filled with zero data, so its space utilization rate will be different. When the file size and the stripe size are very different, the space utilization rate will drop a lot. Therefore, you can modify the erasure The stripe size of the pool, but the modified stripe size needs to restart the system to make the modification take effect, which is very inconvenient

[0005] And the existing processing logic is to use a fixed stripe size to create an erasure pool. It is impossible to set different stripe sizes for different data, and it cannot adapt to different types of file storage, resulting in a decrease in space utilization.

The decline in space utilization will lead to read and write operations of redundant data, resulting in a decline in the overall read and write performance of the system

And it affects the running speed of the erasure correction algorithm and increases the CPU consumption

At the same time, using a fixed stripe size for different types of files will affect storage performance and reduce the IOPS of the system (the number of read and write operations per second)

Images