An Erasure Code Archiving Method Based on Spark Streaming Computing

Abstract

The invention discloses an erasure code archiving method based on SPARK stream computing, which belongs to the field of computer storage. The present invention selects the data blocks to be archived from the nodes of the cluster and constructs multiple RDDs under the SPARK framework, uses the RDD as the basic unit to perform erasure code archiving, and the nodes where the data blocks in the RDD are located undertake the intermediate verification points of the respective erasure codes. The calculation task of the block, and adopts the pipeline form, starting from the first node to send the intermediate verification block to the rear node, using the intermediate verification block and computing power of the rear node to update the issued intermediate verification block, until The tail node generates the final verification block using the delivered intermediate verification block and sends it to the verification nodes of the cluster. The method of the present invention adopts the Map / Reduce model to perform erasure code archiving under the SPARK big data processing framework, and the archiving process is implemented in an assembly line manner, and the encoding calculation process is distributed to multiple nodes to complete, thereby greatly improving the archiving performance.

Description

technical field [0001] The invention belongs to the field of computer storage, and more specifically relates to an erasure code archiving method based on SPARK flow calculation. Background technique [0002] With the development of big data and information storage, memory capacity is increasing day by day, new storage media and new technologies are constantly updated, and the amount of data processed by applications is also greatly increasing, which affects the data fault tolerance rate of memory, storage security and space utilization. put forward higher requirements. The data in the distributed storage cluster is usually stored using a three-copy redundancy mechanism, and with the node recovery strategy, it can be quickly repaired after failure; compared with the three-copy redundancy mechanism, erasure codes have higher storage efficiency and Fault tolerance can be customized on demand. [0003] Apache Spark is a fast and general-purpose computing engine designed for la...

AI Technical Summary

Problems solved by technology

[0004] At present, most researched archiving schemes in Hadoop clusters are performed on disks. With the expansion of memory in the cluster, and more and more applications are extended to the memory of the cluster for calculation, archiving on disks The program is far from meeting the needs

Images