Safety verification and repair method and device for HDFS storage platform

Abstract

The invention provides a security check and repair method and device for an HDFS storage platform. The device is arranged on an HDFS storage platform and comprises a name node and a data node, whereinthe name node is used for managing storage data node information of a file block, receiving a query request, sent by the data node, of a damaged file block, and returning backup storage data node information of the damaged file block to the data node; and the data nodes are used for storing the file blocks, periodically checking the stored file blocks, detecting whether the file blocks are damaged or not, downloading the data blocks from other data nodes according to backup storage data node information of the damaged file blocks returned by the name nodes, and completing the repair operationof the damaged file blocks. The file block check value is calculated by using the irreversible MD5 algorithm, the problem of file tampering possibly occurring in an original system is solved, the check reliability is ensured, the check interval is adjusted according to needs, and the check efficiency is improved.

Description

technical field [0001] The invention relates to the field of computer information technology, in particular to a method and device for safety verification and repair of an HDFS storage platform. Background technique [0002] Hadoop is a software framework that can perform distributed storage and massive information analysis on large amounts of data. HDFS (Hadoop Distributed File System, distributed file system) is one of the cores of Hadoop. It combines the disks of multiple servers into a distributed storage system for distributed storage of massive data. It simplifies the consistency model of files, provides high-throughput application data access functions through streaming data access, is suitable for applications with large data sets, and has high reliability, high performance, and scalability. [0003] HDFS is a distributed file system with a master-slave architecture, mainly composed of a name node NameNode and many data nodes DataNode. Data stored in HDFS is a file...

AI Technical Summary

Problems solved by technology

[0008] For method 1, HDFS calculates a checksum every fixed length (512 bytes by default), and saves the checksum with the file block. For a large file, it takes a long time to check after dividing it into small file blocks. Computational efficiency is low and storage resources are wasted

At the same time, the original HDFS uses cyclic redundancy check CRC32, but CRC32 is not out of data security considerations. It can only detect whether the file block is damaged according to whether the check value is the same, but cannot detect whether the file block has been tampered with. Attackers A fake file with the same check code can be generated through the original CRC32 check code. At this time, the file block is replaced but the CRC32 value is still the same, which cannot be detected and cannot be reliably verified for data integrity.

[0009] For method 2, HDFS scans every 504 hours (3 weeks). If a file block is damaged within this time period, the system cannot detect it and cannot guarantee the data integrity in the storage system.

[0010] To sum up, it can be seen that the security verification method of the existing HDFS storage platform has problems such as low verification efficiency, low verification frequency, and low verification accuracy. Therefore, a more effective HDFS storage platform is proposed. Platform security verification and repair methods are very necessary

Images