Distributed storage system data storage method, apparatus, system, and storage medium

Abstract

The invention discloses a data storage method of a distributed storage system, which comprises the following steps: dividing a file to be stored into blocks to obtain a plurality of file blocks to bestored; comparing a file block to be stored with a file block stored in advance to judge whether there is a file block matched with the content of the file block to be stored in the system. If yes, obtaining a data storage location of a file block whose content matches in the system; indexing the matching file block to be stored according to the data storage location. By dividing the file to be stored into blocks and comparing the data to determine the redundant data, the method can improve the detection probability of the partial duplicate data in the file and realize the accurate data deletion. The invention also provides a distributed storage system data storage device, a system and a readable storage medium, which have the beneficial effects.

Description

technical field [0001] The invention relates to the field of computers, in particular to a data storage method, device, system and readable storage medium of a distributed storage system. Background technique [0002] With the rapid development of data information, various data interactions such as cloud computing, big data, and the Internet of Things have led to a rapid increase in stored data, and data management for storage devices has become a more critical issue. [0003] Data stored in storage devices generally has high redundancy, that is, there are many data stored repeatedly among different files, especially various backup storage systems and various operating systems. Reducing data redundancy can effectively improve the utilization of storage space and is an important research project in storage systems. [0004] At present, to reduce data duplication redundancy, it is usually a method of directly comparing files and deleting duplicate files. At present, most of ...

AI Technical Summary

Problems solved by technology

At present, most of the duplicate files are large files such as disaster recovery backups, and the content of the files is large. The current files rarely have complete duplication between files, mainly because there are many redundant data inside the files. By comparing the files as a whole There are fewer duplicate files that can be detected, and there are still many redundant data in the system, which has little effect on alleviating the excessive occupation of system memory

Images