A method and system for erasure coding archiving based on task load perception

Abstract

The invention discloses an erasure code filing method and system based on task load awareness and belongs to the technical field of computer memory. The method comprises the steps as follows: firstly, acquiring the task number of each node of a filing strip in an erasure code cluster and the number of current filing strip data blocks contained in each node; then, selecting one memory node which has the minimum task number and contains more current filing strip data blocks as a coding node, and updating the task number of the coding node; sequentially selecting supply nodes of all the data blocks according to a task number minimization principle and a supply node selection rule, and updating the task numbers of the supply nodes; finally, allowing the coding node to perform coding calculation according to the data blocks provided by the supply nodes, generating check blocks, and finishing a filing task. The invention further realizes the erasure code filing system based on task load awareness. The problem of filing task imbalance due to the fact that a large number of filing tasks are centralized and distributed to certain data nodes in existing filing methods is solved.

Description

technical field [0001] The invention belongs to the technical field of computer storage, and more specifically relates to an erasure code archiving method and system based on task load perception. Background technique [0002] In order to ensure the reliability of cluster data and improve computing parallelism, most of the existing distributed storage clusters (HDFS, GFS) store data in the form of copies, that is, two or three copies of each data block are stored in the distributed cluster. Although data storage in the form of replicas can improve data reliability and parallelism, the storage overhead is large and the space utilization rate is not high. For a cluster environment where data is written once and read multiple times, erasure codes are used to archive infrequently accessed data, which can reduce data storage overhead and improve storage space utilization. During random layout archiving, load imbalance is an important factor affecting archiving performance. [0...

AI Technical Summary

Problems solved by technology

[0007] Aiming at the above defects or improvement needs of the prior art, the present invention provides an erasure code archiving method and system based on task load perception, the purpose of which is to sense the task load of each storage node in real time, and to Update the encoding module and supply module, complete the calculation and distribution of verification data, and finally complete the archiving of data, thereby solving the problems existing in the existing random layout archiving scheme: task load imbalance and task accumulation, further reducing the performance of disk random read , and the technical problem of network performance degradation due to uneven allocation of network resource bandwidth

Images