Method and apparatus for managing storage data life cycle based on priority

Abstract

The application provides a priority-based storage data life cycle management method and device. The method comprises the following steps: binding at least one migration policy to be executed to an inode in a file system, the inode corresponding to a directory or a file in a distributed file system; and executing the corresponding migration policy according to the bound inode. The priority-based storage data life cycle management method and device provided by the application can realize flexible binding of the policy and the directory.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a priority-based method and apparatus for managing the lifecycle of stored data. Background Technology

[0002] In the field of distributed file system data storage, files, as the basic unit of data storage, have a variety of metadata characteristics, such as (the user who owns the file, the file access time, etc.). Some files require a higher level of data redundancy (such as 3 copies).

[0003] Storage strategy: refers to setting a series of conditions based on certain characteristics of files (such as file owner, file access time, file extension, etc.), and classifying files that meet these conditions into a category and storing them in a specific storage area.

[0004] Storage area: refers to a collection of storage media with different levels of data redundancy and different performance, availability and reliability.

[0005] Data hierarchical storage refers to storing files in different regions according to storage strategies, with files of a certain type stored in the same region for easier management.

[0006] Data migration refers to the periodic migration of files to specific storage areas based on strategies that utilize changes to certain file attributes, such as ownership, modification time, and access time.

[0007] A single strategy often cannot meet the diverse data management needs of users in multiple scenarios, so multiple strategies are required. In order to control the execution order of multiple strategies, the concept of strategy priority is needed.

[0008] To meet the data management needs of different users in different scenarios, it is necessary to provide a multi-user, multi-strategy data management method and device.

[0009] Chinese patent application CN103605728A discloses a data hierarchical storage method and system. This technical solution mainly introduces: a data hierarchical storage method and system, consisting of preset placement and migration strategies, placement modules, and migration modules. The basic unit of data hierarchical storage is a file (i.e., an inode), and each inode is bound to a storage device level identifier. The placement and migration strategies are bound to the mount point of the hierarchical storage device. The mount point of the hierarchical storage device refers to the mount point (mount directory) of the hierarchical storage device. Through customer-defined placement and migration strategies and placement and migration modules, inodes are bound to storage devices to achieve hierarchical data storage.

[0010] This technical solution has the following disadvantages:

[0011] 1. The placement and migration strategies are bound to the tiered storage device mount point in the following order: policy -> tiered storage device mount point. This works when one policy corresponds to one tiered storage device mount point. However, when multiple placement and migration strategies correspond to one tiered storage device mount point, the priority issue of the policies cannot be resolved.

[0012] 2. Defining the placement strategy and migration strategy as two separate strategies is unreasonable, as they both perform the same function: migrating specific files to the target storage device. They should be uniformly defined as a single migration strategy, which then combines the functions of file initialization placement and file migration.

[0013] 3. The placement and migration strategies are tied to the mount points of the tiered storage devices, which is an unreasonable design.

[0014] 4. During the migration process, the handling of failures when migrating large files or migrating files has not been optimized. Summary of the Invention

[0015] The technical problem to be solved by the present invention is to provide a method and apparatus for managing the lifecycle of stored data based on priority, which can realize flexible binding of policies and directories.

[0016] To address the aforementioned technical problems, this invention provides a priority-based method for managing the lifecycle of stored data. The method includes: binding at least one migration strategy to be executed to an inode in the file system, where the inode corresponds to a directory or file in the distributed file system; and executing the corresponding migration strategy based on the bound inode.

[0017] In some implementations, a corresponding migration strategy is executed based on the inode bound to it, including: determining whether the file conforms to the migration strategy, and if so, performing the migration; creating a new empty file in the current directory and setting the storeid of its file inode to the storeid of the migration strategy; copying the content and metadata information of the original file to the new file; and renaming the new file to the name of the original file.

[0018] In some implementations, the corresponding migration strategy is executed based on the inode to which it is bound, and the following further includes: when the file migration fails, the name of the failed file and the offset of the failed data need to be recorded, and this data is recorded in a key-value database.

[0019] In some implementations, the method further includes: using the directory inode id with the policy set as the key and its directory migration status as the value to form a key-value pair, which is then stored in a key-value database for recording.

[0020] In some implementations, when multiple migration tasks need to access a key-value database, a mutex lock is set on the key-value database being accessed.

[0021] In some implementations, the placement functionality for the migration strategy is hidden and specifically implemented in the internal API of the distributed file system.

[0022] In some implementations, the placement function is implemented as follows: when a file is created in the open_file interface, the migration policy of the directory is obtained. If the parent directory does not have the policy, the parent-parent directory is searched until a directory containing the migration policy is found. Then, the file's metadata information is compared with the migration policy. If the file matches the policy, the file's storeid is set to the specified ID of the migration policy.

[0023] Furthermore, the present invention also provides a priority-based storage data lifecycle management device, the device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the priority-based storage data lifecycle management method described above.

[0024] With this design, the present invention has at least the following advantages:

[0025] 1. Enables flexible binding of policies and directories, and supports binding multiple migration policies to a single directory.

[0026] 2. Integrate the placement function into the migration strategy to achieve functional integration.

[0027] 3. Multi-process concurrent execution of migration strategies enables rapid migration and adds a mechanism for handling file migration failures. Attached Figure Description

[0028] The above is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Figure 1 This is a flowchart illustrating the method.

[0030] Figure 2 A flowchart illustrating the file read / write process in a distributed file system;

[0031] Figure 3 This is a flowchart illustrating the migration strategy settings module;

[0032] Figure 4 This is a flowchart illustrating the file placement function;

[0033] Figure 5 This is a flowchart illustrating the file migration function. Detailed Implementation

[0034] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0035] This invention provides a method and apparatus for managing the lifecycle of stored data based on priority. Figure 1 A flowchart of the method is shown. See also Figure 1 The method specifically includes the following steps:

[0036] S11 binds at least one migration strategy to be executed to an inode in the file system. An inode corresponds to a directory or file in the distributed file system.

[0037] S12, execute the corresponding migration strategy based on the inode it is bound to.

[0038] By executing the above method and process, this invention achieves flexible binding between policies and directories, and supports binding multiple migration policies to a single directory.

[0039] The device mainly includes: a distributed file system module, a storage area identification module, a migration strategy setting module, and a migration strategy execution module.

[0040] A distributed file system (FSS) is a software system that features a unified file system namespace. Data can be stored across multiple storage areas. These areas can support various data redundancy modes, such as erasure coding and multiple replicas, enhancing data protection. The number of storage devices in each area can vary, and the storage devices can be dynamically expanded. This distributed file system requires a metadata service and a data service. The metadata service primarily stores file metadata, while the data service primarily stores file data.

[0041] as follows Figure 2 This is a diagram showing the structure of client access to the distributed file system.

[0042] The distributed file system module consists of at least one metadata server and one data server. The client first needs to obtain the file's metadata information from the metadata service, and then it can read and write the file's data. The file's inode contains the storeid, which identifies the storage region.

[0043] A storage region is a collection of storage devices. These devices vary in performance, capacity, and reliability. These differences in characteristics allow storage regions to have different features. For example, multiple high-capacity storage devices forming a storage region will result in a high-capacity storage region. Conversely, multiple high-performance storage devices such as SSDs will create a storage region with high performance and low latency. Furthermore, custom data redundancy strategies can be defined for this storage region, such as erasure coding or replication. Erasure coding effectively improves the capacity utilization of storage devices, while replication offers high reliability.

[0044] Storage Area Identification Module: This module identifies storage areas by their characteristics, classifying them into high-speed and low-speed data access areas, high-redundancy and low-redundancy areas, etc. Users can also customize these identifiers based on specific business scenarios.

[0045] like Figure 3 The diagram shows the process of setting up the migration strategy module.

[0046] The migration strategy setting module primarily configures directories within the distributed file system. This module creates specific migration strategies, such as those based on file IDs (uid, gid, file access time, strategy priority, etc.). Multiple migration strategies can be set for the same directory. The execution order of strategies is based on their priority. This strategy information is bound to a specific directory and stored in the directory's extended attributes. Since the file system is composed of inodes, and directories are a special type of inode, strategy information can be retrieved using the directory as the reference. The module uses the directory's inode ID (with the set strategy) as the key and its migration status as the value, creating a key-value pair that is recorded in a key-value database.

[0047] like Figure 4 As shown: The migration strategy execution module mainly includes two functional modules: the file placement module and the file migration module. Both of these modules utilize migration strategies. The file placement module integrates the migration strategy selection module into the file creation process of the file system for newly created files. This allows the storage area (storeid) of the newly created file to be specified based on the migration strategy selection module during file creation, without requiring data movement.

[0048] The file migration module retrieves information from the key-value database about which directories have migration policies set. Based on the directory's inode, it traverses all files within that directory, including subdirectories, using a breadth-first search algorithm, iterating through the files layer by layer. Then, it selects a migration module based on the migration policy, determining if a file matches the policy. If so, it initiates the migration process, physically moving the file to the designated storage area.

[0049] Therefore, the actual migration strategy provides both file placement and file migration functions.

[0050] like Figure 4 This is a diagram illustrating the file placement process. Figure 5 This is a schematic diagram illustrating the file migration process.

[0051] The specific implementation steps are as follows:

[0052] 1. Select a directory level in the distributed file system and set a migration strategy for it. The conditions can be set as follows: specify the storage area ID based on the file's uid, gid, atime, or mtime, or other custom conditions.

[0053] 2. On the client server, multiple migration tasks are started simultaneously. These tasks can also be scheduled to start during a period of low business activity, minimizing disruption to business operations. These multiple migration tasks run concurrently; each task is a separate process that queries the key-value database for the configured migration policies and retrieves the corresponding policy directory and policy information.

[0054] 3. If there are multiple migration strategies in the key-value database, migration tasks need to be executed sequentially. Mutual exclusion locks are used to resolve the issue of multiple migration tasks accessing the key-value database. When migration task A is executing migration strategy 1, migration task B will sequentially execute migration task 2 and update the completion status of the migration tasks in the value field of the key-value database.

[0055] 4. When obtaining the migration policy, retrieve detailed information about the migration policy and the full path of the directory based on its inode ID. Begin traversing all files in the directory using a breadth-first search algorithm. Check the file's stat; if the storeid in the file's inode matches the one in the migration policy, skip that file. Continue determining if the file meets the migration criteria. If it does, migrate the file to the specified storage area. The migration process is as follows:

[0056] 1) Based on the migration strategy, determine whether the file conforms to the migration strategy. If it does, then proceed with the migration.

[0057] 2) Create a new empty file in the current directory and set its inode storeid to the storeid of the migration policy.

[0058] 3) Copy the contents and metadata of the original file to the new file.

[0059] 4) Rename the new file to the original file name.

[0060] To address file migration failures, when a file migration fails, the name of the failed file and its offset must be recorded in a key-value database. These failed files will then be processed first during the next migration task.

[0061] 5. The hidden placement function for migration strategies is implemented within the distributed file system's internal API, primarily in the `open_file` interface. When a file is created, the migration strategy for the directory is retrieved. If the parent directory does not have the strategy, it searches the parent-parent directory until a directory containing the migration strategy is found. Then, the file's metadata information is compared with the migration strategy. If it matches the strategy, the file's `storeid` is set to the specified ID of the migration strategy. This ensures that the file content is written to the designated storage area.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, or alterations made by those skilled in the art using the disclosed technical content shall fall within the protection scope of the present invention.

Claims

1. A priority-based method for managing the lifecycle of stored data, characterized in that, include: Bind at least one migration strategy to be executed to an inode in the file system. An inode corresponds to a directory or file in the distributed file system. Based on the inode it is bound to, execute the corresponding migration strategy, including: Based on the migration strategy, determine whether the file conforms to the migration strategy; if it does, then proceed with the migration. In the current directory, create a new empty file and set its inode storeid to the storeid of the migration policy; Copy the contents and metadata of the original file to a new file; Rename the new file to the original file name; When a file migration fails, the name of the failed file and the offset of the failed data need to be recorded, and this data should be recorded in a key-value database.

2. The priority-based storage data lifecycle management method according to claim 1, characterized in that, Also includes: The inode ID of the directory with the policy set is used as the key, and its directory migration status is used as the value to form a key-value pair, which is stored in a key-value database for recording.

3. The priority-based storage data lifecycle management method according to claim 2, characterized in that, When multiple migration tasks need to access a key-value database, set a mutex lock on the key-value database to be accessed.

4. The priority-based storage data lifecycle management method according to claim 1, characterized in that, The placement functionality hidden by the migration strategy is specifically implemented in the internal API of the distributed file system.

5. The priority-based storage data lifecycle management method according to claim 4, characterized in that, The implementation process of the placement function includes: In the open_file interface, when a file is created, the migration strategy of the directory is obtained. If it is not found in the parent directory, it will search the parent-parent directory until a directory containing the migration strategy is found. Then, the file's metadata information is compared with the migration strategy. If it matches the strategy, the file's storeid is set to the specified ID of the migration strategy.

6. A priority-based storage data lifecycle management device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the priority-based storage data lifecycle management method according to any one of claims 1 to 5.

Images