Fragment archiving back migration storage method and system using ordering set mechanism

Abstract

The application relates to the technical field of data storage, in particular to a sharding archiving and back migration storage method and system using a sorting set mechanism, which comprises the following steps: a CLI receives a command to start various components, carries out configuration periodic archiving and file metadata scanning; interacts with XDFS, stores generated data into a database table, and stores sharding files into a cold storage device to keep a stub in the XDFS; when a gluster client reads a local unarchived file in a volume, sharding metadata {slice number: timestamp} is saved into a set cs-hot-{gfid} of Redis; when an archived file is read and written, corresponding sharding files are downloaded according to file extension attributes and a current read-write offset value. The application provides a low-cost expansion scheme for existing tight storage space; through sharding metadata design, network and CPU resources of the system are minimized; through the design of distributed archiving, maximum throughput performance is obtained.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a fragmented archive migration storage method and system utilizing a sorted set mechanism. Background Technology

[0002] With the increasing demand for storage, a niche market for storage has emerged, and the following summarizes three application scenarios:

[0003] Cloud storage for medical PACS image archiving: Hospitals generate a large number of PACS images daily. If only local storage devices are used, frequent expansions are required, leading to increasing storage and maintenance costs. PACS image data has a distinct characteristic: access frequency decreases over time, making it particularly suitable for storing these infrequently accessed images on lower-cost cloud storage. Furthermore, the real-time upload function allows cloud storage to serve as a backup system for PACS images. When local storage space is limited, a deletion policy can be implemented to delete the corresponding local files of files already saved to cloud storage, while retaining the stubs. When needed in the future, these files can be downloaded individually or in batches to the local storage.

[0004] Video surveillance in the cloud: With the rapid popularization of high-definition video, higher capacity and performance requirements are placed on the back-end storage system. However, by deploying a cloud archiving system and customizing a cooldown period strategy, video files that have been stored for more than a specified period can be uploaded to cloud storage without capacity limits, immediately freeing up local space or deleting the local version that has been uploaded to cloud storage by submitting a policy, which can greatly reduce the total storage cost.

[0005] iSCSI Volume Gateway: For video streaming data that only supports iSCSI volume storage, the cloud archiving system enables easy uploading of such data streams to the cloud, thereby expanding storage capacity.

[0006] These three typical application scenarios, as well as other requirements, all face the challenges of remote reading and writing of large files and the reading and writing experience. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention discloses a fragmented archiving and migration storage method and system utilizing a sorted set mechanism to solve the aforementioned problems.

[0008] This invention is achieved through the following technical solution:

[0009] In a first aspect, the present invention provides a fragmented archive migration storage method utilizing a sorted set mechanism, comprising the following steps:

[0010] The CLI receives commands to start various components, perform periodic configuration archiving, and scan file metadata.

[0011] Interact with XDFS, store the generated data in a database table, and store the fragmented files to a cold storage device while retaining stubs in XDFS;

[0012] When a Gluster client reads a local unarchived file from a volume, it saves the shard metadata {shard number: timestamp} to the Redis collection cs-hot-{gfid}.

[0013] When reading or writing archived files, download the corresponding file segments based on the file extension attributes and the current read / write offset.

[0014] Furthermore, in the method, commands are received via CLI to start various components, or the submitted strategies and configuration parameters are stored in the database, or the volume attributes of XDFS are set based on the submitted management parameters.

[0015] Furthermore, in this method, after configuring periodic archiving and file metadata scanning via CLI commands, it will interact with XDFS and store the generated data in a database table.

[0016] Furthermore, in the method, once there is a file in the archive task table that has entered the archive state, the archive task will read the file in XDFS and at the same time read the file's shard metadata in Redis. If necessary, the shard file will be stored in a cold storage device and a stub will be retained in XDFS.

[0017] Furthermore, in the method, after the metadata component generates data and stores it in the metadata table, the task generation component obtains the archiving task based on metadata and policy information, and the archiving management component archives the files that are in the pending archiving state.

[0018] Furthermore, in the method, the database includes a device table, a task table, and a strategy table;

[0019] The device table is used to store information about the refrigeration equipment and the volume information associated with that equipment;

[0020] The policy table is used to store user-submitted policies regarding cooldown time archiving or space-based threshold archiving, as well as immediate archiving;

[0021] The task table includes: file metadata fields file_name, dir_path, access_time, and file_type; archive status field: arch_status, including status values ​​0-4, representing pending archive, currently archived, archived, migrated back, and initial status, respectively; policy_name for querying; and volume-related fields.

[0022] Furthermore, in the method, the metadata is divided into two parts: the first part is the metadata used for sharding, which is stored in Redis; the second part is the extended attributes of the file.

[0023] Furthermore, the metadata of the shard includes:

[0024] cloudsync-local-{gfid}: When migrating a shard back to local storage, the shard number is inserted into the cloudsync-local-{gfid} list; when archiving a shard to a cold storage device and freeing up the shard space, the shard number is removed from this list and inserted into the cloudsync-remote-{gfid} list.

[0025] cloudsync-remote-{gfid}: After archiving a fragment and freeing up space, insert the fragment number into this list;

[0026] cloudsync-hot-{gfid}: When there is a file write operation, the slice number and the current timestamp are combined into a key-value pair {part_no:timestamp} and inserted into the sorted set, with the timestamp as the score of the element;

[0027] cloudsync-archiving-restore: Temporarily stores the gfid:part_no:timestamp currently being archived and the timestamp of the current archive. If the archiving is successful, the successfully archived member is immediately removed from the collection. If it fails, it waits for the exception handling task, which retrieves the timed-out fragments, re-inserts them into cloudsync-hot-{gfid}, and sets the files to the pending archiving state, thus completing the secondary archiving.

[0028] cloudsync-archiving-file: Stores the GFID value of the current archive file, which is used by archive tasks on different nodes to complete the archive of a file, so as to achieve the purpose of distributed archiving.

[0029] Furthermore, in the method, the extended attribute is a logical attribute used to set the remote, slice_size, and object_size attributes; the values ​​of the slice_size and object_size attributes are obtained by obtaining the volume feature features.slice_size and the file size to be read, respectively.

[0030] Secondly, the present invention provides a fragmented archive migration storage system utilizing a sorted set mechanism. The system supports the implementation of the fragmented archive migration storage method utilizing a sorted set mechanism as described in the first aspect, and includes an archive component and a migration component, wherein…

[0031] The archiving component is used to accomplish the following:

[0032] Handle CLI requests from the SDK;

[0033] File metadata scanning under volumes and directories;

[0034] Based on the analysis of file metadata and regular policies, archive tasks are generated;

[0035] Based on the analysis of file metadata and file system space threshold strategies, an archiving task is generated.

[0036] Files that have entered the archive-ready state will be archived in a distributed manner, and the corresponding data space will be released while the stubs will be retained.

[0037] Regularly check archives that show abnormalities and restore them to the pending archive state;

[0038] The migration component, which starts the xlator by enabling the features.cloudsync option, resides in Gluster and is responsible for migrating archived files. The cloudsyncxlator adds specific extended attributes for archived files, which the archiving service sets through cloudsyncxlator. The cloudsyncxlator retrieves these attribute values ​​to perform tasks such as migration. These attributes include slice_size, remote, object_size, upload_slice_end, and the virtual attribute upload_base. When a file is archived, after the archiving service sets the upload_slice_end attribute, cloudsyncxlator clears the space occupied by the file content and retains the stub information. When these files receive POSIX interface access, cloudsyncxlator performs immediate migration based on the value of the remote extended attribute and adds "downloaded" after the entire file content is downloaded. To handle sharded archiving and migration, when reading or writing files, the xlator generates the metadata required for sharding processing and stores it in Redis.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention designs a fragmented metadata management system based on sorted sets to solve the complex problems encountered in fragmented archiving / relocation. When a file is written to a block, the fragment number and the current timestamp are calculated and stored in a sorted set. During archiving and relocation, the file's corresponding sorted set and related algorithms are processed based on the timestamp. A relocation algorithm is designed to solve the data consistency problem in distributed systems. A low-cost expansion scheme is provided for existing limited storage space; network and CPU resources are minimized through fragmented metadata design; and maximum throughput performance is achieved through distributed archiving design. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the fragmented archiving migration storage method that utilizes the sorted set mechanism.

[0043] Figure 2 This is a state diagram of some embodiments of the present invention during archiving;

[0044] Figure 3 This is a diagram of the equipment according to an embodiment of the present invention;

[0045] Figure 4 This is a strategy table diagram of an embodiment of the present invention;

[0046] Figure 5 This is a task table diagram of an embodiment of the present invention;

[0047] Figure 6 This is a flowchart of the cooling cycle task generation process according to an embodiment of the present invention;

[0048] Figure 7 This is a flowchart of the immediate archiving process according to an embodiment of the present invention;

[0049] Figure 8 This is a diagram of the read / write backhaul algorithm in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0051] Reference Figure 1 As shown, this embodiment provides a fragmented archive migration storage method utilizing a sorted set mechanism, including the following steps:

[0052] The CLI receives commands to start various components, perform periodic configuration archiving, and scan file metadata.

[0053] Interact with XDFS, store the generated data in a database table, and store the fragmented files to a cold storage device while retaining stubs in XDFS;

[0054] When a Gluster client reads a local unarchived file from a volume, it saves the shard metadata {shard number: timestamp} to the Redis collection cs-hot-{gfid}.

[0055] When reading or writing archived files, download the corresponding file segments based on the file extension attributes and the current read / write offset.

[0056] In this embodiment, commands are received via CLI to start various components, or to store submitted policies and configuration parameters into the database, or to set XDFS volume attributes based on submitted management parameters.

[0057] In this embodiment, after configuring periodic archiving and file metadata scanning via CLI commands, it will interact with XDFS and store the generated data in a database table.

[0058] In this embodiment, once there is a file in the archive task table that has entered the archive state, the archive task will read the file in XDFS and at the same time read the file's shard metadata in Redis. If necessary, the shard file will be stored in a cold storage device and a stub will be kept in XDFS.

[0059] In this embodiment, the metadata component generates data and stores it in the metadata table. The task generation component obtains the archiving task based on metadata and policy information. The archiving management component archives the files that are in the pending archiving state. Example 2

[0060] Building upon Example 1, this example provides a sharded archive migration storage system utilizing a sorting set mechanism. For ease of distinction, cloud archiving is roughly divided into an "archiving component" and a "migration component." The reason for using "roughly" is that the archiving component, in addition to the main archiving tasks, also includes CLI, task generation, file metadata scanning, policy settings for archiving, and file attribute settings for migration. The migration component, on the other hand, is an xlator, commonly referred to as cloudsyncxlator, whose main function is migration; furthermore, the extended attributes required for file archiving and the collection of metadata for shards are also accomplished through cloudsyncxlator.

[0061] The archiving component in this embodiment performs the following tasks:

[0062] (1) Handle CLI requests from the SDK;

[0063] (2) Scanning file metadata under volumes and directories;

[0064] (3) Generate archiving tasks based on the analysis of file metadata and periodic strategies;

[0065] (4) Generate archiving tasks based on the analysis of file metadata and file system space threshold strategies;

[0066] (5) Distribute the archive of files that have entered the archive state, and release the corresponding data space and retain the stubs;

[0067] Regularly check archives that show abnormalities and restore them to the pending archive state.

[0068] In this embodiment, the migration component (cloudsyncxlator) is started by enabling the features.cloudsync option. It resides in Gluster and is responsible for migrating archived files. cloudsyncxlator adds specific extended attributes (xattr) for archived files. The archiving service completes the setting of these extended attributes through cloudsyncxlator, and cloudsyncxlator uses these attribute values ​​to perform tasks such as migration. There are slice_size (the specific name and meaning of the attribute will be introduced later), remote and object_size, upload_slice_end, and the virtual attribute upload_base. When a file is archived, after the archive service sets the upload_slice_end attribute, cloudsyncxlator will clear the space occupied by the file content and retain the stub information. When these files receive access from the POSIX interface, cloudsyncxlator will immediately perform a migration based on the value of the extended attribute remote, and add downloaded after the entire file content is downloaded. In order to handle sharded archiving and migration, when reading and writing files, the metadata required for sharding processing is generated in this xlator and stored in Redis.

[0069] In this embodiment, CLI is the entry point for the entire system's archiving components. CLI receives commands to start various components, or stores submitted policies and configuration parameters into the database, or sets XDFS volume attributes based on submitted management parameters. After periodic archiving and file metadata scanning are configured via CLI commands, the operation of the two components will interact with XDFS and store the generated data in the database table.

[0070] In this embodiment, once there is a file in the archive task table that has entered the archive state, the archive task will read the file in XDFS and at the same time read the file's shard metadata in Redis. If necessary, the shard file will be stored in a cold storage device and a stub will be kept in XDFS.

[0071] In this embodiment, the migration component, in addition to participating in the attribute settings in the above processing, saves the shard metadata {shard number: timestamp} to the Redis collection cs-hot-{gfid} when the Gluster client reads local unarchived files in the volume. When reading or writing archived files, it first downloads the corresponding shard file based on the file extended attributes and the current read / write offset value, and then adds the shard number to the Redis collection cs-local-{gfid} to avoid repeatedly downloading the same shard in a short period of time.

[0072] in addition, Figure 1The dashed lines in the diagram represent a basic process, not direct interaction between components. Specifically, the metadata component generates data and stores it in the metadata table. The task generation component then uses this metadata and policy information to generate archiving tasks. Finally, the archive management component archives the files that are in the pending archiving state.

[0073] This embodiment illustrates a 512MB file, partial.img, with a fragment size of 64MB, divided into 8 fragments numbered 0-7. When only fragments 1, 3, and 5 are archived, the temporary state comprised of XDFS, cloud archiving, and object storage is as follows: Figure 2 . Example 3

[0074] Based on Example 1, this example provides a database. This example designs three tables as needed, namely the device table (see...). Figure 3 ), task list (see Figure 4 ) and strategy table (see Figure 5 ).

[0075] In this embodiment, the device table is used to store information about the cold device and the volume information associated with the device. Currently, it is mainly designed based on the S3 storage device.

[0076] In this embodiment, the policy table stores user-submitted policies regarding cooldown-based archiving, space-based threshold archiving, and immediate archiving. The cooldown time refers to the distance between the file's modification time and the current time. Currently, days / hours / minutes (cd_days / cd_hours / cd_minutes) are provided. If cd_days=3, the file will enter a pending archiving state when the difference between the current time and the file access time is 3 days.

[0077] The task table in this embodiment has the most fields. It includes:

[0078] The file's metadata fields include file_name, dir_path, access_time, and file_type. Access_time serves as the basis for scheduled archiving.

[0079] The archive status field, arch_status, has 5 status values ​​(0-4): pending archive, archiving, archived, migrated back, and initial status.

[0080] The policy_name used for the query.

[0081] Volume-related fields:

[0082] The policy_name, volume_name, and dir_path fields may cause some redundancy in the data stored in the table, but this sacrifices some space for the sake of performance when querying the table. It may be possible to make it conform to the normalization requirements in the future.

[0083] In this embodiment, in order to avoid or reduce multi-table join queries, the table design does not fully follow the second normal form, resulting in some redundant columns, such as the volume name appearing in all three tables. Example 4

[0084] Based on the above embodiments, this embodiment provides a key / value metadata. The metadata in this embodiment is divided into two parts: the first part is metadata for sharding, which is stored in Redis; the second part is the extended attributes of the file.

[0085] Space occupancy estimation for the following situations:

[0086] 100 million files: over 300 GB

[0087] A 5TB file is divided into 64MB chunks: over 200MB.

[0088]

[0089] The fragment metadata for this embodiment is shown in the following table:

[0090] cloudsync-local-{gfid}: When migrating a shard back to local storage, the shard number is inserted into this list; when archiving a shard to a cold storage device and freeing up the shard space, the shard number is removed from this list and inserted into the cloudsync-remote-{gfid} list.

[0091] `cloudsync-remote-{gfid}`: This list is inserted after archiving a shard and freeing up space. (When a shard on a cold storage device is deleted, its shard number is removed from the list. This feature is currently unavailable because deletion is not allowed once archived.)

[0092] `cloudsync-hot-{gfid}`: When a file write operation occurs, a key-value pair `{part_no:timestamp}` is created using the segment number and the current timestamp and inserted into this sorted set. The timestamp serves as the element's score. During archiving, the segment number with the smallest timestamp can be found based on the score, easily enabling archiving from oldest to newest based on time. Additionally, during archiving, if the timestamp of the segment number is too close to the current time (e.g., within 1 minute, the time difference should be determined through discussion and theory), archiving should not be performed to avoid immediate rollback after archiving.

[0093] cloudsync-archiving-restore: Temporarily stores the currently archived gfid:part_no:timestamp and the current archive timestamp (as a fraction). If the archiving is successful, the successfully archived member is immediately removed from the collection. If it fails, it waits for the exception handling task, which retrieves the timed-out fragments (timeout set to 3 minutes), re-inserts them into cloudsync-hot-{gfid}, and sets the files to the pending archive state, thus completing the secondary archiving.

[0094] `cloudsync-archiving-file`: Stores the .g.fid value of the current archive file, used by archive tasks on different nodes to jointly complete the archiving of a single file, achieving distributed archiving. However, in extreme cases (for ease of description), if all files are smaller than the shard, only one file will be processed at any given time. Therefore, this key-value pair should be deleted as soon as possible. Once the archive shard is saved and it is determined that there are no more shards to archive, it should be deleted immediately to avoid sequential archiving.

[0095] This embodiment has the following extended attributes: To complete the migration after archiving, the remote (object storage) path and file space size of the archived files are required. Additionally, to handle shard archiving and migration, the shard size is also needed. Therefore, the following extended attributes have been added to achieve this.

[0096]

[0097] In this embodiment, the attribute `trusted.cloudsync.upload_base` is a logical attribute whose purpose is to set the `remote`, `slice_size`, and `object_size` attributes. The values ​​of `slice_size` and `object_size` are obtained by obtaining the volume feature `features.slice_size` and the read file size, respectively. Only `remote` needs to be set. Therefore, the path of the archive file on the remote end can be used as the parameter of `trusted.cloudsync.upload_base` to complete the setting of the three attributes. Since the actual attributes after setting are `remote`, `slice_size`, and `object_size`, they are also called logical attributes. After setting, they do not exist in the file's extended attributes.

[0098] The volume characteristics of this embodiment are as follows:

[0099] The following volume features have been added for cloud archiving.

[0100]

[0101] Example 5

[0102] Based on the above embodiments, this embodiment provides a method for generating archive tasks based on cooldown time, as detailed below.

[0103] Task generation accomplishes the following:

[0104] For files that are in the "initial state" and meet the archiving time requirement (i.e., timenow-timemodify≥Lengthcold), change them from the "initial state" to the "pending archiving state".

[0105] For files that are in the "archived" state and whose latest write time is earlier than timenow-Lengthcold, change them from the "initial state" to the "pending archive" state.

[0106] The main processing flow is as follows Figure 6 As shown, the task generation component, like the file metadata scan, uses crontab to manage its scheduled execution, such as running every night at 1 AM, or at fixed time intervals. Currently, it runs on only one node. Future modifications are needed to allow it to run collaboratively across multiple nodes, i.e., a distributed architecture, and to be started via CLI commands.

[0107] This embodiment enables immediate archiving. It accepts a volume or a volume and directory as parameters. After the command is completed, files that meet the immediate archiving criteria are set to a pending archiving state, and the archiving component will immediately archive the file. Additionally, all information from this command is stored in the `archive_policy` table for easy retrieval of the archiving status.

[0108] Reference Figure 7 As shown, the specific processing flow is as follows:

[0109] ① Select a file

[0110] ② Determine if the file is not a symbolic link. If it is, discard the process; otherwise, proceed to step 3.

[0111] ③ Based on the file name, path, and volume name, check if it already exists in the archive_task table. If it does, proceed to step 5; otherwise, proceed to step 4.

[0112] ④ Obtain information such as file size, access time, modification time, and path, and create a task record to be archived, store it in archive_task, and then execute step 1.

[0113] ⑤ Check the archiving status of the file record. If it is in the initial state, update it to the pending archiving state; otherwise, proceed to step 6.

[0114] ⑥ Check the archive status of the file. If it is already archived, check if there is a newly written fragment in cloudsync-hot-{gfid}. If so, update it to be pending archive. Otherwise, proceed to step 1.

[0115] The flowchart of the above steps is as follows: Figure 7 As shown.

[0116] Example 6

[0117] Based on the above embodiments, referring to Figure 8 As shown, this embodiment provides a read / write migration algorithm, specifically as follows: It uses local, remote, remote, downloading, downloaded, and repair to describe the state of a file from archiving to migration and the process thereafter. File changes are tracked by setting the extended attribute `trusted.cloudsync.status` to the above status values. When a file receives a request, the corresponding operation is performed based on the status value. For example, when a read request for an archived file is received, based on the file's remote (on the remote end) status, posixxlator (located in the brick service process) reads the file's `trusted.cloudsync.remote` and `trusted.cloudsync.object_size` extended attribute values ​​and stores them in the dictionary `xdata`. Then, clousyncxlator (located in the fuse mounting process) directly retrieves these two attribute values ​​from `xdata` to complete the download of the remote file.

[0118] The following discussion focuses on the key function `cs_readv()` in CloudSyncXLator for migrating back a single archive file. The key steps performed by this function are as follows:

[0119] ① Get the status from the file's inode or extended attributes.

[0120] ② If the file status is local or downloaded, proceed to step 6; otherwise, proceed to step 3.

[0121] ③ Lock the file (Q: Is it a blocking lock, i.e., if one user locks the file, another user will be blocked) and obtain the file's remote path (in object storage S3) and file size.

[0122] ④ Download the file and call the next level's writev() function to save the downloaded content into the current file.

[0123] ⑤ Unlock.

[0124] ⑥ Call the next level's readv() function.

[0125] In summary, this invention designs a fragmented metadata management system based on sorted sets to solve the complex problems encountered in fragmented archiving / relocation. When a file is written to a block, the fragment number and the current timestamp are calculated and stored in a sorted set. During archiving and relocation, the file's corresponding sorted set and related algorithms are processed based on the timestamp. A relocation algorithm is designed to solve the data consistency problem in distributed systems. A low-cost expansion scheme is provided for existing limited storage space; network and CPU resources are minimized through fragmented metadata design; and maximum throughput performance is achieved through distributed archiving design.

[0126] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for archiving and dearchiving storage using a sorted set mechanism, characterized in that, The method comprises the following steps: The CLI receives commands to start the components, configure periodic archiving and file metadata scanning; Interact with XDFS, store the generated data into the database table, and store the shard file to the cold storage device to keep the root in XDFS; When the gluster client reads the local unarchived file in the volume, save the shard metadata {piece number: timestamp} to the set cs-hot-{gfid} in Redis; When reading and writing the archived file, download the corresponding shard file according to the file extension attribute and the current read-write offset value.

2. The sharded archival rollback storage method using a sequencing set mechanism of claim 1, wherein, In the method, the CLI receives commands to start the components, or stores the submitted policies and configuration parameters into the database, or sets the volume attributes of XDFS according to the submitted management parameters.

3. The sharded archival rollback storage method using a sequencing set mechanism of claim 2, wherein, In the method, after the periodic archiving and file metadata scanning are configured through the CLI command, interact with XDFS, and store the generated data into the database table.

4. The sharded archival rollback storage method using a sequencing set mechanism of claim 3, wherein, In the method, once there is a file in the archiving task table that enters the state of being archived, the archiving task reads the file in XDFS, reads the shard metadata of the file in Redis, and stores the shard file to the cold storage device and keeps the root in XDFS according to the need.

5. The sharded archival rollback storage method using a sequencing set mechanism of claim 1, wherein, In the method, the metadata component generates data, stores it into the metadata table, and the task generation component obtains the archiving task based on the metadata and the policy, and the archiving management component archives the file in the state of being archived.

6. The sharded archival rollback storage method using a sequencing set mechanism of claim 1, wherein, The database is provided with a device table, a task table and a policy table; The device table is used to store information about the cold device and the volume information associated with the device. The policy table is used to store the policies submitted by the user about the cooling time archiving or the threshold archiving based on space, and immediate archiving. The task table comprises the metadata fields file_name, dir_path and access_time of the file, file_type, the archiving state field arch_status comprising state values 0-4, which are respectively the state of being archived, the state of being archived, the state of being archived, the state of being migrated back, and the initial state, the policy_name for query, and the volume-related field.

7. The sharded archival rollback storage method using a sequencing set mechanism of claim 1, wherein, In the method, the metadata is divided into two parts: the first part is the metadata for sharding, which is stored in Redis; The second part is the extension attribute of the file.

8. The sharded archival rollback storage method using a sequencing set mechanism of claim 7, wherein, The metadata for sharding comprises: cloudsync-local-{gfid}: when a shard is migrated back to the local, the piece number is inserted into the cloudsync-local-{gfid} list; when a shard is archived to the cold storage device and the piece space is released, the piece number is deleted from the list and inserted into the cloudsync-remote-{gfid} list; cloudsync-remote-{gfid}: when a shard is archived and the space is released, the piece number is inserted into the list; cloudsync-hot-{gfid}:When there is a write operation of a file, the part number and the current timestamp are combined into a key-value pair {part_no:timestamp}, which is inserted into the sorted set, and the timestamp is used as the score of the element; cloudsync-archiving-restore:Temporarily stores the gfid:part_no:timestamp of the file currently being archived and the timestamp of the current archiving, and if the archiving is successful, the member of the successful archiving is immediately removed from the set, and if it fails, an exception handling task is waited for, which obtains the expired shard, re-inserts it into cloudsync-hot-{gfid}, and sets the file to the state of being archived, and then completes the secondary archiving; cloudsync-archiving-file:Stores the gfid value of the current archiving file, which is used for different nodes to complete the archiving of a file together to achieve distributed archiving.

9. The sharded archival rollback storage method using a sequencing set mechanism of claim 7, wherein, In the method, the extended attribute is a logical attribute used to set the remote, slice_size, and object_size attributes; the attribute values of slice_size and object_size are obtained by acquiring the volume features features.silce_size and reading the file size, respectively.

10. A sharded archival rollback storage system utilizing a sequencing set mechanism, the system configured to support implementing a method of sharded archival rollback storage utilizing a sequencing set mechanism as claimed in any one of claims 1-9, characterized in that, The method comprises an archiving component and a restoration component, wherein The archiving component is used to complete the following things, including: Processing CLI from SDK requests; File metadata scanning under the volume and directory; Generating archiving tasks based on file metadata and periodic policy analysis; Generating archiving tasks based on file metadata and file system usage space threshold policy analysis; Distributed archiving of files in the state of being archived, and releasing the corresponding data space and retaining the root; Periodically checking the abnormal archiving and restoring it to the state of being archived; The migration component starts the xlator by opening the features.cloudsync option, exists in gluster, is responsible for archiving file migration, the cloudsync xlator adds an extension attribute for the archived file, the archiving service sets the extension attribute through the cloudysnc xlator, and the cloudsync xlator completes tasks such as migration by obtaining the values of the attributes; there are slice_size, remote and object_size, upload_slice_end, and a virtual attribute upload_base, when the file is archived, the archiving service sets the attribute upload_slice_end, and the cloudsync xlator will clear the space occupied by the file content and retain the stub information; when these files receive POSIX interface access, the cloudsync xlator will immediately migrate according to the value of the extension attribute remote, and add downloaded after the entire file content is downloaded; in order to handle slice archiving and migration, when the file is read and written, the metadata required for slice processing is generated in the xlator and saved in Redis.

Images