Method and device for security verification and repair of hdfs storage platform

Abstract

The invention provides a method and device for safety verification and repair of an HDFS storage platform. The device is set on the HDFS storage platform, including the name node and the data node; the name node is used to manage the storage data node information of the file block, receive the query request of the damaged file block sent by the data node, and return the damaged file block to the data node The backup storage data node information; the data node is used to store file blocks, regularly check the stored file blocks, detect whether the file block is damaged, and store the data node information from other data according to the backup of the damaged file block returned by the name node The node downloads the data block and completes the repair operation of the damaged file block. The invention solves the problem of file tampering that may occur in the original system by using the irreversible MD5 algorithm to calculate the check value of the file block, ensures the reliability of the check, and realizes the adjustment of the check interval according to the needs, and improves the check accuracy. test efficiency.

Description

technical field [0001] The invention relates to the field of computer information technology, in particular to a method and device for safety verification and repair of an HDFS storage platform. Background technique [0002] Hadoop is a software framework that can perform distributed storage and massive information analysis on large amounts of data. HDFS (Hadoop Distributed File System, distributed file system) is one of the cores of Hadoop. It combines the disks of multiple servers into a distributed storage system for distributed storage of massive data. It simplifies the consistency model of files, provides high-throughput application data access functions through streaming data access, is suitable for applications with large data sets, and has high reliability, high performance, and scalability. [0003] HDFS is a distributed file system with a master-slave architecture, mainly composed of a name node NameNode and many data nodes DataNode. Data stored in HDFS is a file...

AI Technical Summary

Problems solved by technology

[0008] For method 1, HDFS calculates a checksum every fixed length (512 bytes by default), and saves the checksum with the file block. For a large file, it takes a long time to check after dividing it into small file blocks. Computational efficiency is low and storage resources are wasted

At the same time, the original HDFS uses cyclic redundancy check CRC32, but CRC32 is not out of data security considerations. It can only detect whether the file block is damaged according to whether the check value is the same, but cannot detect whether the file block has been tampered with. Attackers A fake file with the same check code can be generated through the original CRC32 check code. At this time, the file block is replaced but the CRC32 value is still the same, which cannot be detected and cannot be reliably verified for data integrity.

[0009] For method 2, HDFS scans every 504 hours (3 weeks). If a file block is damaged within this time period, the system cannot detect it and cannot guarantee the data integrity in the storage system.

[0010] To sum up, it can be seen that the security verification method of the existing HDFS storage platform has problems such as low verification efficiency, low verification frequency, and low verification accuracy. Therefore, a more effective HDFS storage platform is proposed. Platform security verification and repair methods are very necessary

Images