Encoding method based on erasure code, distributed system, equipment and storage medium

Abstract

One or more embodiments of the invention provide a coding method based on erasure codes, a distributed system, equipment and a storage medium. The method comprises the following steps: acquiring K data blocks of to-be-processed data; dividing the K data blocks into L groups of data groups; wherein each data group at least comprises K coding parameters respectively corresponding to the K data blocks; wherein K and L are integers greater than 1; for each data group, encoding by using at least part of data blocks in other data groups and encoding parameters corresponding to the at least part of data blocks in the data group to generate part of verification blocks of the data group; wherein the data group and the partial check block are stored in a data node of the distributed system. According to the embodiment of the invention, while the security of the user data is ensured, the amplification of the IOPS which needs to be subjected to data recovery when the data is lost or abnormal is reduced.

Description

technical field [0001] One or more embodiments of this specification relate to the field of computer data processing, and in particular to an encoding method based on erasure codes, a distributed system, a device, and a storage medium. Background technique [0002] The storage scale of distributed systems is becoming larger and larger. A distributed system is a loosely coupled system composed of multiple data nodes interconnected through communication lines. [0003] Erasure Code (Erasure Code) is a coding fault-tolerant technology. Its basic principle is to fragment the stored data, and generate k+m pieces of data through a certain verification calculation method for the k pieces of original data obtained by the pieces. And it can restore the original data through any k data in k+m data. In this way, even if some data is lost, the distributed system can still restore the original data. [0004] Erasure coding technology can minimize the storage overhead of the system on t...

AI Technical Summary

Problems solved by technology

[0005] However, based on the data stored by the existing erasure code technology, when some data is lost, the system needs to read enough data to recover the data required by the user. On the one hand, it will lead to IOPS (Input / Output PerSecond , the amount of input and output per second), on the other hand, it also consumes a lot of bandwidth during data restoration

Images