Small file migration method, device and equipment and computer readable storage medium

Abstract

The invention discloses a small file migration method, device and equipment and a computer readable storage medium. The method includes: loading small files to be migrated; migrating the small files according to a preset migration strategy, judging whether the small files being migrated reach a migration upper limit or not, if yes, stopping loading the small files, and restoring to load the smallfiles when preset restoring conditions are met; if not, continuing loading the small files. By utilizing the method, when loading of the small files to be migrated reaches the migration upper limit, loading of the small files to be migrated is stopped, so that the problem of blockage and crash of a distributed storage system due to unlimited loading of the small files to be migrated is avoided, and utilization rate of distributed storage capacity and migration efficiency of massive small files are improved.

Description

technical field [0001] The invention relates to the technical field of distributed storage, in particular to a small file migration method. The invention also relates to a small file migration device, equipment and computer-readable storage medium. Background technique [0002] In the era of Internet big data, the transmission of massive large files requires a huge amount of bandwidth, the transmission of massive small files requires a huge IOPS (Input Output Operations Per Second, the number of read and write operations per second), and the large IOPS will bring unprecedented challenges to the distributed storage system, especially when a large number of small files are migrated between different distributed storage systems, it is particularly important to be able to fully utilize the distributed storage capabilities. [0003] The migration method of massive small files in the prior art will load the small files to be migrated without limit when migrating small files. When...

AI Technical Summary

Problems solved by technology

[0002] In the era of Internet big data, the transmission of massive large files requires a huge amount of bandwidth, the transmission of massive small files requires a huge IOPS (Input Output Operations Per Second, the number of read and write operations per second), and the large IOPS will bring unprecedented challenges to the distributed storage system, especially when a large number of small files are migrated between different distributed storage systems, it is particularly important to be able to fully utilize the distributed storage capabilities

[0003] The migration method of massive small files in the prior art will load the small files to be migrated without limit when migrating small files. When the number of loaded small files exceeds the maximum number of small files for a single migration of the distributed storage system, it will Distributed storage is blocked and paralyzed, reducing migration efficiency

Images