Method and device for realizing backup and recovery of CephFS massive small files

Abstract

The invention relates to the technical field of data storage, and particularly provides a method and device for achieving CephFS massive small file backup and recovery, the method comprises a backup process and a recovery process, and the backup process comprises the steps that S1.1, a backup task is accepted and started; s1.2, carrying out backup processing on the data and the metadata; s1.3, a task monitoring and fault-tolerant mechanism is backed up; the recovery process comprises the steps that S2.1, a recovery task is accepted and started; s2.2, recovering the backup data; and S2.3, recovering task monitoring. Compared with the prior art, data loss caused by local faults or disasters can be effectively prevented, and the secure storage level and the long-term storage reliability of the data are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and specifically provides a method and a device for implementing CephFS mass small file backup and recovery. Background Art

[0002] In the field of cloud computing, CephFS file system can provide unified, flexible and highly scalable storage services. CephFS file system occupies an important position in enterprise-level storage solutions due to its excellent scalability, high availability and high performance.

[0003] However, with the surge in information volume and users' pursuit of efficient access speed, how to build a system that can efficiently back up massive small files to ensure data security under the constraints of limited resources has become an inevitable choice. Summary of the invention

[0004] The present invention aims at overcoming the above-mentioned deficiencies in the prior art and provides a method for backing up and restoring a large number of small files in CephFS with strong practicability.

[0005] A further technical task of the present invention is to provide a device that is reasonably designed, safe and applicable for backing up and restoring massive small files of CephFS.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A method for backing up and restoring a large number of small files in CephFS is divided into a backup process and a recovery process. The backup process is as follows:

[0008] S1.1, Acceptance and initiation of backup tasks;

[0009] S1.2, data and metadata backup processing;

[0010] S1.3, backup task monitoring and fault tolerance mechanism;

[0011] The recovery process is as follows:

[0012] S2.1, recovery task acceptance and initiation;

[0013] S2.2, backup data recovery;

[0014] S2.3. Resume task monitoring.

[0015] Furthermore, in step S1.1, the backup process begins when the task platform receives a backup request. At this time, the task platform will immediately call the CephFS file system to create a snapshot to capture all file data in the CephFS file system at the time of the backup request. Subsequently, the platform will select the node with the most remaining resources based on the resource usage of the execution node and issue a backup start instruction, thus entering the substantive operation stage of the backup.

[0016] Furthermore, in step S1.2, data backup and metadata backup are included;

[0017] In the data backup, the system will determine whether incremental backup is required according to the issued instructions. If incremental backup is required, the create_time of each file in the CephFS file system snapshot is read and compared with the creation time of the last snapshot to determine whether it is data that needs to be backed up; if it is not incremental backup, it is directly recorded as data that needs to be backed up;

[0018] For the data that needs to be backed up, the number of backup groups is modulo operation based on the unique identifier inode of each file, and the data is divided into different backup groups according to the remainder; after the data is grouped, the Linux pipe technology and tar tool are combined for each group of data to package and compress the data, and the data is fragmented at a certain granularity according to the preset fragmentation rules, and each fragmented data is uploaded to the object storage system in the form of an object.

[0019] Furthermore, in the metadata backup, the system reads each file in the CephFS file system snapshot, compares the file path with the root path of the snapshot to calculate the hash value of the file relative path, and divides the file name into different metadata groups by bit operation to achieve efficient organization of metadata;

[0020] Different metadata groups also need to be fragmented. When the number of file names assigned to a metadata group reaches a certain level, the metadata file is directly uploaded to the object storage system, and a new metadata file is created locally to record the next file name information of the group.

[0021] Furthermore, in step S1.3, after the backup starts, the task platform continuously tracks the amount of uploaded data and metadata, and calculates the backup progress in real time. When it is found that the backup progress is stagnant for a long time or the task status becomes failed, the breakpoint resume mechanism should be automatically triggered;

[0022] Calculate the interruption point according to the uploaded data volume, and re-upload the backup in groups to ensure the success of the backup;

[0023] When the backup is completed, the system deletes the created temporary snapshot from the CephFS file system to free up unnecessary storage space.

[0024] Furthermore, in step S2.1, the recovery process begins when the task platform receives a recovery request. At this time, the task platform will immediately call the CephFS file system to create a snapshot to protect the existing data, and then modify the capacity quota of the CephFS file system to unlimited. When everything is ready, the recovery command is sent to the execution node to enter the substantive operation stage of the recovery.

[0025] Furthermore, in step S2.2, first, a full data list is queried from the object storage, and the overall amount of data recovered is calculated to provide basic data for progress monitoring;

[0026] Backup and recovery follows the principle of starting with the earliest full backup. Data files are downloaded from the object storage in sequence and written into the CephFS file system. Directory data files are downloaded first during the recovery process of each backup. When the directory data is restored, the data is downloaded from the object storage system to the local Linux data pipeline for temporary storage according to the number of groups at the backup time. When a certain amount of data is downloaded, the tar command is used to decompress the data in the pipeline and output it to the CephFS file system for storage.

[0027] Furthermore, when the last data is written to the CephFS file system, the metadata file of the last backup is downloaded from the object storage system to a local temporary directory. By reading each file in the CephFS file system, the file path is compared with the root path of the snapshot to calculate the hash value of the relative path, and the file name is grouped through bit operations. When the number of file names in a group reaches a certain level, the metadata files of the corresponding groups are opened in turn, and those that do not belong to this backup and should be cleaned up are determined by comparison, and the file names are recorded in temporary files. When the full data is traversed, the files and directories recorded in the temporary files are deleted from the CephFS file system in turn.

[0028] Furthermore, in step S2.3, when the recovery starts, the task platform queries the data size in the CephFS file system in real time to calculate the recovery progress. If the recovery fails, the temporary snapshot is used to overwrite the file system with the data at the time of initiating the recovery.

[0029] When the recovery is successful, the created temporary snapshot will be deleted from the CephFS file system and the capacity limit of the file system will be restored, freeing up storage space and restoring the system to normal operation.

[0030] A device for implementing CephFS mass small file backup and recovery, comprising: at least one memory and at least one processor;

[0031] The at least one memory is used to store a machine-readable program;

[0032] The at least one processor is used to call the machine-readable program to execute a method for implementing CephFS massive small file backup and recovery.

[0033] Compared with the prior art, the method and device for implementing CephFS mass small file backup and recovery of the present invention have the following outstanding beneficial effects:

[0034] (1) Improve data security and persistence: The present invention ensures data integrity and consistency through full backup and incremental backup strategies. Combined with the high availability of object storage, it effectively prevents data loss due to local failures or disaster events, greatly improving the data security storage level and long-term preservation reliability;

[0035] (2) Enhance system stability and fault tolerance: The introduction of a breakpoint-resume mechanism ensures that the backup process can be continued from the breakpoint even if there is an interruption, effectively maintaining task continuity and system stability. Real-time monitoring of backup and recovery progress further enhances the robustness of the system, reduces the need for human intervention, and reduces operation and maintenance costs;

[0036] (3) Flexible adaptation to diverse data needs: Metadata adopts hash grouping and fragmented upload strategies, which is particularly suitable for processing complex data sets containing a large number of small files, ensuring high performance under conditions of large data volumes and high concurrent access. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Attached Figure 1 It is a backup process sequence diagram in a method for realizing CephFS massive small file backup and recovery;

[0039] Attached Figure 2 The present invention is a recovery process sequence diagram in a method for implementing CephFS massive small file backup and recovery. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] A best embodiment is given below:

[0042] like Figure 1 As shown, a method for backing up and restoring a large number of small files in CephFS in this embodiment is divided into a backup process and a recovery process, wherein the backup process is:

[0043] S1.1, Acceptance and initiation of backup tasks;

[0044] The backup process begins when the task platform receives a backup request. At this point, the task platform will immediately call the CephFS file system to create a snapshot to capture all file data in the CephFS file system at the time of the backup request to ensure the integrity and consistency of the backup data. The platform will then select the node with the most remaining resources based on the resource usage of the execution node and issue the backup start instruction, entering the substantive operation phase of the backup.

[0045] S1.2, data and metadata backup processing;

[0046] Including data backup and metadata backup;

[0047] During the data backup process, the system will determine whether incremental backup is required based on the issued instructions. If incremental backup is required, the create_time of each file in the CephFS file system snapshot is read and compared with the creation time of the previous snapshot to determine whether it is data that needs to be backed up; if it is not an incremental backup, it is directly recorded as data that needs to be backed up.

[0048] For the data that needs to be backed up, the number of backup groups is modulo the unique identifier (inode) of each file, and the data is divided into different backup groups according to the remainder to reduce the burden of a single task and improve parallel processing capabilities.

[0049] After the data is grouped, each group of data is packaged and compressed through the combination of Linux pipe technology and tar tools, and then fragmented at a certain granularity according to the preset fragmentation rules, and each fragmented data is uploaded to the object storage system in the form of an object.

[0050] In metadata backup, the system reads each file in the CephFS file system snapshot, compares the file path with the root path of the snapshot, calculates the hash value of the file's relative path, and divides the file name into different metadata groups through bit operations to achieve efficient organization of metadata.

[0051] During backup and recovery, redundant data must be cleaned up based on the file names recorded in the metadata file. Memory overflow may occur in scenarios with massive small files. Therefore, sharding is also required for different metadata groups. When the number of file names assigned to a metadata group reaches a certain level, the metadata file is directly uploaded to the object storage system, and a new metadata file is created locally to record the next file name information for the group.

[0052] S1.3, backup task monitoring and fault tolerance mechanism;

[0053] When the backup starts, the task platform keeps tracking the amount of uploaded data and metadata, and calculates the backup progress in real time. When the backup progress is found to be stagnant for a long time or the task status becomes failed, the breakpoint resume mechanism should be automatically triggered, and the interruption point should be calculated according to the amount of uploaded data, and the backup should be re-uploaded according to the above grouping process to ensure the success of the backup. When the backup is completed, the system will delete the created temporary snapshot from the CephFS file system to release unnecessary storage space.

[0054] like Figure 2 As shown, the recovery process is:

[0055] S2.1, recovery task acceptance and initiation;

[0056] The recovery process begins when the task platform receives a recovery request. At this point, the task platform will immediately call the CephFS file system to create a snapshot to protect existing data and prevent the user's current data from being lost due to a recovery failure. The capacity quota of the CephFS file system will then be modified to be unlimited to prevent the recovery process from failing due to capacity exceeding the file system limit. When everything is ready, the recovery command is sent to the execution node, entering the substantive operation phase of the recovery.

[0057] S2.2, backup data recovery;

[0058] First, query the full data list from the object storage and calculate the overall amount of data restored to provide basic data for progress monitoring.

[0059] Backup and recovery follows the principle of starting with the earliest full backup, downloading data files from the object storage in sequence and writing them to the CephFS file system.

[0060] In the process of restoring each backup, the directory data files are downloaded first to avoid failures caused by directory structure problems during the restoration process. When the directory data is restored, the data is downloaded from the object storage system to the local Linux data pipeline for temporary storage according to the number of groups at the backup time. When a certain amount of data is downloaded, the tar command is used to decompress the data in the pipeline and output it to the CephFS file system for storage.

[0061] When the last data is written to the CephFS file system, the metadata file of the last backup is downloaded from the object storage system to a local temporary directory. By reading each file in the CephFS file system, the file path is compared with the root path of the snapshot to calculate the hash value of the relative path, and the file name is grouped through bit operations. When the number of file names in a group reaches a certain level, the metadata files of the corresponding groups are opened in turn, and the files that do not belong to this backup and should be cleaned up are determined by comparison, and the file names are recorded in temporary files. After the full amount of data is traversed, the files and directories recorded in the temporary files are deleted from the CephFS file system in turn to ensure the accuracy and consistency of the data.

[0062] S2.3, recovery task monitoring;

[0063] When the recovery starts, the task platform queries the data size in the CephFS file system in real time to calculate the recovery progress. If the recovery fails, the temporary snapshot is used to overwrite the file system with the data at the time of initiating the recovery to prevent data loss. When the recovery is successful, the created temporary snapshot is deleted from the CephFS file system and the capacity limit of the file system is restored, freeing up storage space and restoring the system to normal operation.

[0064] The above method realizes the backup function from CephFS file system to object storage system, taking advantage of the object storage system's advantages in long-term preservation, significantly improving data security and persistence, and effectively preventing data loss caused by local failures or disasters. Even in the face of massive small files, the memory usage can be reduced by grouping to avoid backup and recovery failures.

[0065] Based on the above method, a device for implementing CephFS massive small file backup and recovery in this embodiment includes: at least one memory and at least one processor;

[0066] The at least one memory is used to store a machine-readable program;

[0067] The at least one processor is used to call the machine-readable program to execute a method for implementing CephFS massive small file backup and recovery.

[0068] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0069] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for backing up and restoring a large number of small files in CephFS, characterized in that: It is divided into a backup process and a recovery process. The backup process is as follows: S1.1, Acceptance and initiation of backup tasks; S1.2, data and metadata backup processing; S1.3, backup task monitoring and fault tolerance mechanism; The recovery process is as follows: S2.1, recovery task acceptance and initiation; S2.2, backup data recovery; S2.

3. Resume task monitoring.

2. A method for backing up and restoring a large number of small files in CephFS according to claim 1, characterized in that: In step S1.1, the backup process begins when the task platform receives a backup request. At this time, the task platform will immediately call the CephFS file system to create a snapshot to capture all file data in the CephFS file system at the time of the backup request. The platform will then select the node with the most remaining resources based on the resource usage of the execution node and issue a backup start instruction, thus entering the substantive operation stage of the backup.

3. A method for implementing CephFS massive small file backup and recovery according to claim 2, characterized in that: In step S1.2, data backup and metadata backup are included; In the data backup, the system will determine whether incremental backup is required according to the issued instructions. If incremental backup is required, the create_time of each file in the CephFS file system snapshot is read and compared with the creation time of the last snapshot to determine whether it is data that needs to be backed up; if it is not incremental backup, it is directly recorded as data that needs to be backed up; For the data that needs to be backed up, the remainder operation is performed on the number of backup groups according to the unique identifier inode of each file, and the data is divided into different backup groups according to the remainder; After the data is grouped, each group of data is packaged and compressed through the combination of Linux pipe technology and tar tools, and then fragmented at a certain granularity according to the preset fragmentation rules, and each fragmented data is uploaded to the object storage system in the form of an object.

4. A method for implementing CephFS massive small file backup and recovery according to claim 3, characterized in that: In the metadata backup, the system reads each file in the CephFS file system snapshot, compares the file path with the root path of the snapshot to calculate the hash value of the file relative path, and divides the file name into different metadata groups by bit operation to achieve efficient organization of metadata; Different metadata groups also need to be fragmented. When the number of file names assigned to a metadata group reaches a certain level, the metadata file is directly uploaded to the object storage system, and a new metadata file is created locally to record the next file name information of the group.

5. A method for implementing CephFS massive small file backup and recovery according to claim 4, characterized in that: In step S1.3, when the backup starts, the task platform keeps tracking the amount of uploaded data and metadata and calculates the backup progress in real time. When it is found that the backup progress is stagnant for a long time or the task status becomes failed, the breakpoint resume mechanism should be automatically triggered; Calculate the interruption point according to the uploaded data volume, and re-upload the backup in groups to ensure the success of the backup; When the backup is completed, the system deletes the created temporary snapshot from the CephFS file system to free up unnecessary storage space.

6. A method for implementing CephFS massive small file backup and recovery according to claim 5, characterized in that: In step S2.1, the recovery process begins when the task platform receives a recovery request. At this time, the task platform will immediately call the CephFS file system to create a snapshot to protect the existing data, and then modify the capacity quota of the CephFS file system to unlimited. When everything is ready, it will send a recovery instruction to the execution node to enter the substantive operation stage of recovery.

7. A method for implementing CephFS massive small file backup and recovery according to claim 6, characterized in that: In step S2.2, first, a full data list is queried from the object storage, and the overall amount of data recovered is calculated to provide basic data for progress monitoring; Backup and recovery follows the principle of starting with the earliest full backup. Data files are downloaded from the object storage in sequence and written into the CephFS file system. Directory data files are downloaded first during the recovery process of each backup. When the directory data is restored, the data is downloaded from the object storage system to the local Linux data pipeline for temporary storage according to the number of groups at the backup time. When a certain amount of data is downloaded, the tar command is used to decompress the data in the pipeline and output it to the CephFS file system for storage.

8. A method for backing up and restoring a large number of small files in CephFS according to claim 7, characterized in that: When the last data is written to the CephFS file system, the metadata file of the last backup is downloaded from the object storage system to a local temporary directory. By reading each file in the CephFS file system, the file path is compared with the root path of the snapshot to calculate the hash value of the relative path, and the file name is grouped through bit operations. When the number of file names in a group reaches a certain level, the metadata files of the corresponding groups are opened in turn. By comparing, it is determined that the files that do not belong to this backup should be cleared, and the file names are recorded in temporary files. After the traversal of all data is completed, the files and directories recorded in the temporary files are deleted from the CephFS file system in turn.

9. A method for implementing CephFS massive small file backup and recovery according to claim 8, characterized in that: In step S2.3, when the recovery starts, the task platform queries the data size in the CephFS file system in real time to calculate the recovery progress. If the recovery fails, the temporary snapshot is used to overwrite the file system with the data at the time of initiating the recovery. When the recovery is successful, the created temporary snapshot will be deleted from the CephFS file system and the capacity limit of the file system will be restored, freeing up storage space and restoring the system to normal operation.

10. A device for backing up and restoring a large number of small files in CephFS, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images