A life cycle processing method based on ordered incremental key values

Abstract

A life cycle processing method based on ordered incremental key value, in a distributed KV storage system, records operation logs containing global monotonically increasing clock for file metadata change operation; the background consumes the logs, and generates an ordered incremental key value index arranged in dictionary order for each file according to the clock; create and maintain an expiration time column enumeration cursor for recording the end position of each life cycle scan; during life cycle execution, calculate the scan end marker according to the current global clock and the minimum expiration time of the rule, take the start position recorded by the cursor and the end marker as the range, perform range query on the ordered incremental key value index, obtain the file set to be processed, and perform life cycle rule matching and processing operation on the file set. This method converts full scan into accurate range query, eliminates the problems of empty scan and repeated enumeration, and significantly improves the execution efficiency of life cycle processing in large-scale storage system.

Description

Technical Field

[0001] This invention relates to distributed storage technology, and in particular to a lifecycle management method based on ordered incrementing key values. Background Technology

[0002] File and object interoperability storage systems typically employ a metadata service cluster to manage file metadata. The metadata service cluster consists of multiple nodes, forming a distributed transactional key-value store system. It collaboratively maintains a global metadata view through a consistency protocol, ensuring data consistency under concurrent access. The metadata service cluster stores file metadata in key-value format, with each file corresponding to two key-value pairs: 1) fh_key (the file's unique identifier): / bfs / fs / fsid / pino / 0 / filename_prefix / ino, value: pointing to dirent_key; 2) dirent_key: / bfs / fs / fsid / pino / 0 / filename / 0, value: the file's metadata structure (such as creation time, size, data layout, etc.); 3) The distribution of keys within the metadata service cluster is as follows: Figure 1Example. The metadata service cluster maintains an ordered key-value table, where all key-value pairs are stored in lexicographical order of the keys. This ordered storage enables the system to efficiently perform range scans—given any start and end keys, all key-value pairs falling within the range can be located through a single ordered traversal, with a time complexity of O(log N + K), where N is the total number of keys and K is the actual number of keys within the range. This ordered storage characteristic is the key foundation for achieving efficient range scans in this invention. When a transaction is submitted to the metadata service cluster, each transaction is assigned a global clock (ts). This global clock is monotonically increasing and unique globally, used to accurately identify each transaction operation. When generating ordered keys, the global clock is used as the key to ensure that the lexicographical order of the index is strictly monotonically increasing, supporting the correctness of subsequent range scans. Each time a transaction is started to write or update file metadata, the metadata service cluster assigns a global clock (ts) to the transaction and synchronously writes an operation log (bilog). A global clock (ts) is recorded in the operation log (bilog) and is monotonically increasing and unique globally, used to precisely identify each transaction operation. When generating ordered keys, the global clock is used as the key to ensure the lexicographical order of the index is strictly monotonically increasing, supporting the correctness of subsequent range queries. The operation log (bilog) records complete information about metadata changes, including operation type (create / update / delete), file unique identifier, and the global clock (ts) corresponding to the transaction. The operation log (bilog) is strictly appended in the order of writes and cannot be modified; it is the data source for subsequently generating ordered keys based on the global clock. A typical file and object interoperable storage system architecture is as follows: Figure 2 As shown. The process of writing bilog files is synchronized during file creation, as follows: Figure 3 As shown.

[0003] Lifecycle management (LC) is one of the core functions of file and file-interoperable storage systems, used to automatically archive, delete, and tier data according to preset rules. Typical lifecycle strategies include: 1) Time-based deletion rules: automatically deleting files after a specified number of days since their creation; 2) Time-based archiving rules: transferring cold data from high-performance storage tiers (such as SSDs / HDDs) to tape; 3) Time-based tiered storage rules: moving data from standard storage (STANDARD) to low-frequency storage (STANDARD_IA).

[0004] Lifecycle rules include the following configuration items: 1) Rule ID, used to uniquely identify a lifecycle rule; 2) Rule status, used to indicate whether the rule participates in file matching; 3) File filtering conditions, including file prefix and file tag; 4) Lifecycle actions, including expired deletion and storage type conversion; 5) File expiration days, used to limit the time when a file expires.

[0005] The lifecycle management process typically includes the following steps: 1) Scanning the bucket's lifecycle rules; 2) Performing file enumeration according to the rules; 3) Rule matching and execution. Among these, file enumeration is the most time-consuming and resource-intensive step. The file metadata index is stored in lexicographical order on the metadata service cluster. Each time files are listed, the process starts from the root directory and recursively lists subdirectories until all files are listed. The final list result is returned in lexicographical order of the filenames. During lifecycle execution, concurrent listing is performed according to the rule. Because the list is returned in lexicographical order of filenames, expired files cannot be quickly listed, so a full list of all files in the entire bucket is required each time. Figure 4A and Figure 4B Then, based on prefix rules, modification time (mtime), etc., it is determined whether the data meets the requirements for expiration, transfer to IA, or archive to tape. In a single bucket of hundreds of billions of files, only tens of millions of files expire or need to be transferred each day. It is necessary to scan hundreds of billions of files to process the tens of millions of expired or transferable files. The whole process has many invalid lists and invalid matching rules, resulting in a slow overall lifecycle execution speed.

[0006] In existing technologies, a typical implementation of lifecycle management is as follows: File metadata indexes are stored lexicographically on a metadata service cluster. Each time files are listed, the process starts from the root directory and recursively lists subdirectories until all files are listed. The final list result is returned in lexicographical order of filenames. During lifecycle execution, concurrent listing is performed according to rules. Because the list is returned in lexicographical order of filenames, expired files cannot be quickly listed. Therefore, each time, a full list of all files in the entire bucket is required, and then the expiration, conversion to IA, or archiving to tape is determined based on prefix rules, modification time (mtime), etc.

[0007] The existing technology has the following main drawbacks: 1. High overhead of full list processing: Each execution requires traversing all file metadata, resulting in a "no-load scan" problem. In existing technical solutions, the lifecycle process needs to scan all files in the entire bucket each time it runs, checking each file one by one to see if it meets the lifecycle rules. This process suffers from a serious "no-load scan" problem: 1) Even if no file meets the expiration condition this time, the executor still needs to traverse all file metadata; 2) In large-scale storage systems, buckets contain hundreds of millions or even tens of billions of files, and each full scan consumes a large amount of computing resources and I / O bandwidth; 3) No-load scans lead to ineffective resource consumption and prolong the end-to-end latency of lifecycle processing. Figure 5 As shown: the first time, all files were listed, including those that had not expired, and then deleted or moved according to the matching rules; the second time LC was executed, all files were listed again, and those marked in red had been moved or had not yet expired, and the newly written files had not yet expired; in both of these listing processes, a full list was performed, listing those that had not expired and those that had expired and been moved, which caused the empty scan to consume a lot of CPU and I / O resources.

[0008] 2. Duplicate Enumeration Issue: When multiple rules have nested prefixes, the same file may be enumerated and evaluated multiple times. In scenarios with multiple lifecycle rules, if the prefixes of the rules are nested, the same file may be matched by multiple rules simultaneously, resulting in the file being repeatedly enumerated and processed. Specifically, when the prefix of rule A is a prefix of the prefix of rule B (i.e., the prefix of one rule is a superset of the prefix of another rule), the set of files corresponding to rule B is actually a subset of the set of files corresponding to rule A. For example, assuming the prefix of rule A is "logs / " and the prefix of rule B is "logs / archive / ", then all files matching the prefix "logs / archive / " (such as logs / archive / 2023 / data.txt) also match the prefix "logs / ", such as... Figure 6 As shown. In the existing full scan scheme, the same file will be listed twice to determine whether it meets the expiration conditions of each rule, resulting in: 1) Duplicate listing: The same file is listed multiple times, increasing the overhead of invalid metadata scanning; 2) Duplicate processing: Without an additional deduplication mechanism, the same file may be deleted or migrated multiple times, increasing the overhead of invalid processing.

[0009] 3. For files that do not match the rules, the check for matching the lifecycle rule must still be performed during the full list. This check is performed every day the lifecycle runs, resulting in a large number of repetitive and invalid operations. For example, the lifecycle rule is configured with prefix=aa, but a file named bb does not match the prefix, so the lifecycle rule does not need to be executed. However, during the full list, the file bb will be listed, and the check for matching the lifecycle rule for file bb must be performed every time the lifecycle rule is executed.

[0010] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0011] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a lifecycle processing method based on ordered incremental key values.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: A lifecycle management method based on ordered incrementing key values ​​includes: In a distributed key-value storage system, operation logs are recorded for changes to file metadata. These operation logs contain globally unique and monotonically increasing clock information. The background process consumes the operation logs and generates at least one ordered incremental key-value index for each file based on the clock information. The ordered incremental key-value indexes are arranged in lexicographical order according to the clock information in the KV system. Create and maintain an enumeration cursor, which is used to record the end position of each lifecycle scan and is configured with an expiration time; During lifecycle execution, a scan end marker is calculated based on the current global clock and the minimum expiration time in the lifecycle rules. Using the starting position of the enumerated cursor records and the scan end marker as the range, a range query is performed on the ordered incremental key-value index in the KV system to obtain the set of files to be processed. Perform lifecycle rule matching and processing operations on the files in the set of files to be processed.

[0013] A computer program product includes a computer program that, when executed by a processor, implements the lifecycle processing method based on ordered incrementing key values.

[0014] The present invention has the following beneficial effects: This invention addresses the problem of empty scans caused by the need for a full scan of all files within a bucket in existing lifecycle management systems, as well as the problem of repeated file enumeration and processing caused by nested rule prefixes. It proposes a lifecycle management method based on ordered, incremental key values. This method records an operation log containing a globally unique, monotonically increasing clock when file metadata changes. A background process consumes this operation log to generate ordered mtime and match_prefix key values ​​for each file, arranged lexicographically by clock. During lifecycle execution, only the scan end marker needs to be calculated based on the current global clock and the minimum expiration days in the rules. Combined with an enumeration cursor recording the scan progress, precise range queries are performed in the KV storage system, transforming the traditional full scan into a precise scan only of expired files matching the rule prefix. This fundamentally avoids the unnecessary overhead of traversing the entire metadata every time, significantly saving CPU and I / O resources and greatly improving the execution efficiency of lifecycle management in large-scale storage systems.

[0015] Furthermore, by prioritizing the enumeration of ordered key-value pairs (match_prefix) followed by ordered key-value pairs (mtime), and employing a bucket-level concurrent execution approach, the set of files to be processed obtained from a single range query can be used to match all lifecycle rules. This ensures that even if file prefixes of different rules have nested relationships, the same file will not be enumerated multiple times or processed repeatedly, eliminating conflicts and resource waste during concurrent execution of multiple rules. In addition, the enumeration cursor is configured with an expiration time, allowing it to be periodically cleared. This ensures that processed files are not scanned repeatedly while preventing long-term cursor remnants from causing missed scans. Taking ordered key-value pairs (mtime) as an example, during each lifecycle execution, the system only scans key-value pairs within the range from the start position of the cursor record to the end marker determined by subtracting the minimum expiration days from the current global clock, achieving precise location and incremental processing of expired files.

[0016] In summary, this invention effectively solves the problem of empty scans caused by full scans in lifecycle archiving, hierarchical, and deletion scenarios, as well as the problem of repeated enumeration caused by nested multi-rule prefixes. It transforms full scans into efficient range queries, significantly reduces system resource consumption, and shortens the end-to-end latency of lifecycle processing. It is especially suitable for ultra-large storage systems with hundreds of billions of files in a single bucket.

[0017] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the key distribution in a metadata service cluster in the existing technology.

[0019] Figure 2This is a typical architecture diagram of a file and object interoperable storage system in existing technologies.

[0020] Figure 3 A flowchart illustrating the synchronous write operation log (bilog) during file creation in existing technologies.

[0021] Figure 4A This is a schematic diagram of the lexicographical arrangement of metadata key-value pairs in the prior art.

[0022] Figure 4B A diagram illustrating the scope of the lifecycle in existing technologies.

[0023] Figure 5 This is a schematic diagram of the lifecycle air scan problem in existing technologies.

[0024] Figure 6 This is a diagram illustrating the repeated listing caused by nested rule prefixes in existing technologies.

[0025] Figure 7 This is a schematic diagram illustrating how ordered incremental key values ​​are stored in a KV system according to a global clock lexicographical order, as per an embodiment of the present invention.

[0026] Figure 8 This is an example diagram illustrating the effects of using mtime ordered key-value pairs in an embodiment of the present invention.

[0027] Figure 9 This is a flowchart illustrating the generation of operation logs when updating metadata in an embodiment of the present invention.

[0028] Figure 10 This is a flowchart illustrating the generation of an operation log when deleting metadata in an embodiment of the present invention.

[0029] Figure 11 This is a flowchart illustrating how a background process reads operation logs and generates ordered key-value pairs in an embodiment of the present invention.

[0030] Figure 12A and Figure 12B This invention provides a flowchart for listing expired files based on ordered key values ​​to complete their lifecycle.

[0031] Figure 13 This is a schematic diagram of the scanning effect using mtime ordered key values ​​as an example in an embodiment of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0033] In traditional lifecycle management methods, every time an archiving, hierarchical, or deletion operation is performed, all files within the entire bucket must be fully enumerated. Files that have already been archived or hierarchically managed, files that have not expired, and files whose lifecycle rule prefixes do not match do not require any processing; however, the full list method still scans them, leading to a severe empty scan problem and causing a huge waste of CPU and I / O resources. Furthermore, when file prefixes of multiple lifecycle rules are nested, the same file will be repeatedly enumerated by multiple rules, further exacerbating the inefficient consumption of resources. This invention aims to overcome the above-mentioned shortcomings of existing methods and provide a lifecycle processing method based on ordered incremental key values. By constructing an ordered key-value index, it transforms the full scan into a precise range query, processing only expired files, and employing a multi-rule merging enumeration method, a single traversal can cover all target files of all rules, thereby fundamentally eliminating the problem of the same file being repeatedly enumerated.

[0034] See Figures 7 to 13 This invention provides a lifecycle processing method based on ordered incrementing key values, comprising: In a distributed key-value storage system, operation logs are recorded for changes to file metadata. These operation logs contain globally unique and monotonically increasing clock information. The background process consumes the operation logs and generates at least one ordered incremental key-value index for each file based on the clock information. The ordered incremental key-value indexes are arranged in lexicographical order according to the clock information in the KV system. Create and maintain an enumeration cursor, which is used to record the end position of each lifecycle scan and is configured with an expiration time; During lifecycle execution, a scan end marker is calculated based on the current global clock and the minimum expiration time in the lifecycle rules. Using the starting position of the enumerated cursor records and the scan end marker as the range, a range query is performed on the ordered incremental key-value index in the KV system to obtain the set of files to be processed. Perform lifecycle rule matching and processing operations on the files in the set of files to be processed.

[0035] See Figure 7In some embodiments, generating at least one ordered incremental key-value index for each file specifically includes: generating an mtime ordered key-value, the key-value format of which includes a bucket identifier, an index type identifier, a string representation of the clock information, and a unique identifier of the file, such that files within the same bucket are stored consecutively in order of the size of the clock information; determining whether a file matches a file prefix in a lifecycle rule, and if so, generating a match_prefix ordered key-value, the key-value format of which also includes a bucket identifier, an index type identifier, a string representation of the clock information, and a unique identifier of the file.

[0036] See also Figure 7 In some embodiments, lifecycle execution is performed at the bucket level with concurrency granularity. A range query lists the set of files to be processed in a single query, which is used to match all lifecycle rules in order to avoid duplicate listing and processing caused by different rules due to nested file prefixes.

[0037] See Figure 8 In some embodiments, the expiration time of the enumerated cursor is used to periodically clear the position of the cursor record: when the current time exceeds the expiration time recorded in the enumerated cursor, the cursor is determined to be expired, and the next lifecycle scan will start from the root path of the ordered key-value index to prevent long-term residual cursors from causing files to be missed.

[0038] See Figure 11 In some embodiments, the ordered incremental key-value index is generated by consuming operation logs through a background process. Specifically, this includes: reading the operation log cursor and listing new operation log entries based on the cursor; for each operation log entry, performing a corresponding index generation or deletion operation based on its operation type; for creation operations, generating an mtime ordered key-value based on the file's unique identifier and the global clock, generating a match_prefix ordered key-value if the filename matches a rule prefix, and deleting the corresponding old ordered key-value based on the previous file's unique identifier and the previous global clock; for deletion operations, deleting the corresponding mtime key-value based on the file's unique identifier and the global clock. The metadata update operation generates new mtime ordered key-value pairs and match_prefix ordered key-value pairs based on the new file unique identifier and global clock. The old corresponding key-value pairs are deleted based on the old file unique identifier and the previous global clock. For renaming operations, if the new filename matches a rule prefix, a new mtime ordered key-value pair is generated based on the file unique identifier and the previous global clock. The generated or deleted ordered key-value pairs are submitted to the metadata cluster, and the operation log cursor is updated based on the clock information of the last processed operation log.

[0039] See Figures 12A to 12B In some embodiments, the step of calculating the scan end marker based on the current global clock and the minimum expiration time in the lifecycle rules during lifecycle execution specifically includes: reading the enumeration cursor and all lifecycle rules from the metadata cluster; traversing all lifecycle rules and calculating a minimum expiration day; obtaining the current global clock from the metadata cluster and subtracting the minimum expiration day from the current global clock to obtain the expiration time boundary; and constructing a first scan end marker for matching mtime ordered key values ​​and a second scan end marker for matching match_prefix ordered key values ​​based on the expiration time boundary.

[0040] Continue reading Figures 12A to 12B In some embodiments, the step of performing a range query on the ordered incremental key-value index in the KV system, using the starting position of the enumerated cursor record and the scan end marker as the range, to obtain a set of files to be processed, specifically includes: first, performing a range query on the ordered key-value pair `match_prefix`: determining whether the enumerated cursor has expired; if expired, the starting position is the storage root path of the ordered key-value pair `match_prefix`; otherwise, the starting position is the `match_prefix` marker recorded in the enumerated cursor; performing a range query using the starting position and the second scan end marker as the range; after completing the range query on the ordered key-value pair `match_prefix`, performing a range query on the ordered key-value pair `mtime`: determining whether the enumerated cursor has expired; if expired, the starting position is the storage root path of the ordered key-value pair `mtime`; otherwise, the starting position is the `mtime` marker recorded in the enumerated cursor; performing a range query using the starting position and the first scan end marker as the range.

[0041] See also Figures 12A to 12B In some embodiments, the lifecycle rule matching and processing operation performed on the files in the set of files to be processed specifically includes: extracting the unique identifier of the file from each key value parsed from the match_prefix ordered key value or the mtime ordered key value; reading the complete file metadata from the metadata cluster based on the file's unique identifier; traversing all lifecycle rules based on the file metadata to determine whether the file matches the current rule; if it matches the deletion rule, performing a file deletion operation; if it matches the archiving rule, performing a file archiving operation; and if it matches the storage tiering rule, performing a file storage type conversion operation.

[0042] See also Figure 8 and Figures 12A to 12BIn some embodiments, after performing a range query, the process further includes updating the enumeration cursor: when the ordered key values ​​of match_prefix within the range have not been fully enumerated, the match_prefix flag in the enumeration cursor is updated according to the clock information of the last enumerated ordered key value of match_prefix; when the ordered key values ​​of match_prefix within the range have been fully enumerated, the match_prefix flag is updated according to the last enumerated key value, and the process proceeds to enumerating ordered key values ​​of mtime; when the ordered key values ​​of mtime within the range have not been fully enumerated, the mtime flag in the enumeration cursor is updated according to the clock information of the last enumerated ordered key value of mtime, and the starting position is updated to continue enumeration; when the ordered key values ​​of mtime within the range have been fully enumerated, the mtime flag is updated according to the last enumerated key value; and the updated enumeration cursor is then permanently stored in the metadata cluster.

[0043] The proposed lifecycle processing method based on ordered incremental key values ​​transforms the full directory scan in lifecycle management into a precise range query by introducing an ordered incremental key value index based on a global clock. Each execution only processes expired files that match the rule prefix, eliminating the problem of empty scans at the root. At the same time, incremental scanning is achieved by using an enumeration cursor with expiration time to avoid repeated traversal of already processed files. Furthermore, concurrent execution at the bucket level and matching all rules in a single scan completely solves the problem of repeated file enumeration and processing caused by nested rule prefixes, thereby significantly reducing CPU and I / O resource consumption and improving the execution efficiency of lifecycle processing in large-scale storage systems.

[0044] The following further describes the implementation process and advantages of specific embodiments of the present invention.

[0045] This invention proposes a lifecycle processing method based on ordered incremental key values. The core idea is to utilize the global clock (ts) corresponding to transactions recorded in the operation log (bilog) generated when file metadata changes to generate two ordered incremental key value sequences for each file—mtime ordered key values ​​and match_prefix ordered key values—arranged in lexicographical order by the clock. This transforms the full scan in lifecycle processing into an efficient range query. The main methods can be summarized as follows: When file metadata creation, deletion, or update operations occur, the system synchronously records an operation log (bilog), which contains globally unique and monotonically increasing clock information (ts). A background process asynchronously consumes this operation log and generates two types of ordered, monotonically increasing key-value indices for each file based on the global clock (ts): an mtime ordered key-value and a match_prefix ordered key-value (for example, the format of the mtime ordered key-value is " / bfs / fs / fsid / lcmtime / ts_str_fh_key", where fsid is the file system ID, ts_str is the string representation of the global clock (with left-padding zeros in %011d format), and fh_key is the file's unique identifier; if the file matches a file prefix in the lifecycle rules, a match_prefix ordered key-value is further generated, in the format " / bfs / fs / fsid / lcprefix / ts_str_fh_key"). The generation of the match_prefix ordered key value involves matching all lifecycle rules. As long as a file matches any one of these rules, a match_prefix ordered key value will be generated for that file, regardless of the number of matching rules.

[0046] The system creates and maintains an enumeration cursor (lc_sort_key_list_marker) to record the end position of each enumeration operation performed on mtime-ordered keys and match_prefix-ordered keys. An expiration time is set for this cursor to periodically clear the recorded positions and prevent long-term remnants from causing missed scans. Specifically, the cursor's expiration time is dynamically adjusted based on system configuration: on the first execution, the expiration time of the enumeration cursor is set to the current time plus the system-configured number of days; on subsequent executions, it first checks if the current time has exceeded the expiration time. If it has, the mtime_marker and match_prefix_marker in the cursor are set to null, and the expiration time is updated to the current time plus the system-configured number of days, thus achieving periodic clearing and reset of the cursor.

[0047] During each lifecycle execution, the system only needs to scan the files corresponding to the key values ​​within the range from the start marker (start_marker) to the end marker (end_marker) to accurately list expired files that match the rule prefix. The start_marker is taken from the end marker at the end of the previous enumeration. If it's the first enumeration or there's no valid cursor, the starting position is the root path of the corresponding ordered key-value storage (e.g., " / bfs / fs / fsid / lcmtime / " or " / bfs / fs / fsid / lcprefix / "). The end_marker is calculated based on the current global clock and the minimum expiration day (expire_day) in the lifecycle rule (e.g., " / bfs / fs / fsid / lcmtime / current global clock - minimum expiration day"). This calculation logic is based on the fact that each change in the file modification time (mtime) is completed through a transaction updating metadata. Each update requests a global clock (ts) from the metadata service cluster; therefore, each mtime corresponds to a unique global clock. The original condition for determining file expiration is "the current system time minus the file's mtime is greater than the expiration day (expire_day)". By replacing mtime with its corresponding global clock, we can obtain the condition "when the current global clock minus the ts recorded in the operation log is greater than expire_day, it means the file has expired", thus supporting the calculation of expiration time boundaries using the global clock.

[0048] During each enumeration process, the system prioritizes enumerating the ordered key-value pairs `match_prefix`. Based on the parsed file unique identifier (`fh_key`), it reads the file metadata and performs lifecycle rule matching and processing operations (such as deletion, archiving, or storage tier conversion). Then, it enumerates the ordered key-value pairs `mtime`, parsing and processing `fh_key` in the same way. By prioritizing `match_prefix` over `mtime`, the system efficiently handles expired files matching the rule prefix and provides a fallback option by using the ordered key-value pairs `mtime` to handle files for which `match_prefix` indexes were not generated due to rule prefix changes.

[0049] The processing flow of the embodiments of the present invention is described in detail below.

[0050] The mtime ordered key and match_prefix ordered key are stored in the kv system in lexicographical order according to the global clock (ts), such as Figure 7 As shown.

[0051] The overall main processing flow is as follows: 1. Operation logs (bilog) generated when metadata is created, deleted, or updated; the operation logs (bilog) record the global clock (ts) corresponding to the transactions when metadata is created, deleted, or updated; 2. The background process reads the operation log (bilog) and, based on the global clock (ts) recorded in the operation log (bilog), generates or deletes two types of ordered, monotonically increasing key-value pairs: 1) The format of the ordered key-value pair for mtime is: “ / bfs / fs / fsid / lcmtime / ts_str_fh_key”, where fsid is the file system ID and ts_str is the global time (ts) string (aligned to a minimum width of 11, padded with 0s if necessary). This ensures that the ordered, incrementing keys corresponding to files within the same bucket are contiguous in the KV system and sorted in ascending order of ts value.

[0052] 2) `match_prefix` ordered key-value pairs: The filename is checked against the `prefix` in the lifecycle rule. If a match is found, an ordered `match_prefix` key-value pair is generated, in the format: `" / bfs / fs / fsid / lcprefix / ts_str _fh_key"`. Here, `fsid` is the file system ID, and `ts_str` is the global clock (TS) string (aligned to a minimum width of 11, padded with zeros if necessary). Files matching the lifecycle rule `prefix` within the same bucket will have corresponding ordered, incrementing keys, which are contiguous in the KV system and sorted in ascending order of `TS` value.

[0053] 3. The global clock (ts) is monotonically increasing and unique throughout the system, and the global clock (ts) is greater than the file's mtime. Therefore, if the current global clock - the global clock recorded in the operation log (ts) > expire_day (the expiration day configured in the LC rule), the file is also expired. Thus, end_ts = current global clock - expire_day (the expiration day configured in the LC rule). Files with ordered increasing key values ​​ts_str <= end_marker are all expired. The lifecycle execution flow is as follows: 1) First, scan the KV storage system for ordered, increasing key values ​​in the range [" / bfs / fs / fsid / lcprefix / ", " / bfs / fs / fsid / lcprefix / end_ts"] to obtain all expired files that match the lifecycle rule prefix; thus, the full scan is transformed into scanning only expired files that match the lifecycle rule. 2) To prevent situations where, after generating the ordered `match_prefix` key-value pair, the user modifies the lifecycle rule's prefix, resulting in not being able to list all files matching the lifecycle rule's prefix using the `match_prefix` ordered key-value pair, the `mtime` ordered key-value pair is used to list files matched after the lifecycle rule prefix is ​​modified. This involves scanning the KV storage system for ordered, increasing key-value pairs within the range [" / bfs / fs / fsid / lcmtime / ", " / bfs / fs / fsid / lcmtime / end_ts"]. This transforms the full scan into scanning only expired files, while still allowing lifecycle processing to be performed on files matching the new lifecycle rule. 3) Files listed using the ordered key values ​​of `mtime` and `match_prefix` may have already been deleted, moved, or archived. When the lifecycle is executed again, those moved or archived files will still be considered expired, but the move or archive operation will not need to be performed again. To address the issue of moved or archived files being scanned repeatedly, a list cursor is created to record the end position of each listing. The next listing will only start scanning from the position recorded by the cursor, thus avoiding repeated scanning of already processed files. Furthermore, moved or archived files will still need to be deleted later, so an expiration time is set for the list cursor, and the position of the list cursor record is periodically cleared to prevent long-term residue from causing missed scans.

[0054] struct lc_sort_key_list_marker { time_t expire_time; / / Cursor expiration time string mtime_marker; / / Starting position of the ordered key-value list of mtime string match_prefix_marker; / / Starting position of the ordered key-value list match_prefix } Taking mtime ordered keys as an example, the effects are shown below. Figure 8 Example shown: Specific processing procedures Operation logs generated when metadata is created, deleted, or updated The metadata service cluster synchronously writes an operation log (bilog) each time a transaction is initiated to write or update file metadata. The operation log (bilog) records complete information about this metadata change, including the operation type (create / update / delete), file unique identifier (fh_key), filename, the global clock corresponding to this transaction, and the global clock of the last metadata update. The operation log (bilog) is strictly appended in the order of writes and cannot be modified. The fields recorded in the operation log (bilog) are as follows: struct cos_bi_log_entry { time_t timestamp / / Global clock time_t pre_timestamp / / Global clock at the time of the last metadata update std::string file_name; / / filename int op; / / op type setxattr, delete, create, rename std::string fh_key; / / Unique ID of the file std::string pre_fh_key; / / Unique ID of the previous file when overwriting. bool mtime_change / / Whether mitime is changed } The steps to create metadata are as follows: 1) Obtain the global clock (ts) from the metadata cluster; 2) Before committing a transaction and creating file metadata, prepare the operation log (bilog); 3) Assign the global clock (ts) to the timestamp of the operation log (bilog), and save the global clock (ts) to the metadata of the file (to be used when updating the metadata); 4) Obtain fh_key and file_name from the file metadata and assign them to the operation log (bilog); 5) If it is an overwrite write, retrieve the old fh_key from the file metadata and assign it to the pre_fh_key in the operation log (bilog); 6) Package the operation log (bilog) and file metadata into a transaction and commit it to the metadata cluster.

[0055] The steps for updating (setxattr, rename) metadata are as follows (see...). Figure 9 ): 1) Obtain the global clock (ts) from the metadata cluster. 2) Prepare operation logs (bilog) 3) Assign the global clock (ts) to the timestamp of the operation log (bilog), and assign the global clock (ts) recorded from the file's metadata to the pre_timestamp of the operation log (bilog); 4) Package the operation log (bilog) and file metadata into a transaction and commit it to the metadata cluster.

[0056] The steps for deleting metadata are as follows (see...) Figure 10 ): 1) Prepare operation logs (bilog) 2) The global clock (ts) recorded in the file's metadata is assigned to the timestamp in the operation log (bilog); 3) Obtain fh_key and file_name from the file metadata and assign them to the operation log (bilog); 4) Package the operation log (bilog) and file metadata into a transaction and commit it to the metadata cluster.

[0057] Generate ordered key-value pairs based on the operation log (bilog). A background process reads the operation log (bilog) and, based on the global clock (ts) recorded in the operation log (bilog), generates or deletes two types of ordered, monotonically increasing key-value pairs, in the following format: 1) mtime ordered key-value format: " / bfs / fs / fsid / lcmtime / ts_str_fh_key where fsid is the file system ID and ts_str is the global clock (ts) string (aligned to a minimum width of 11, padded with 0s if necessary).

[0058] 2) The ordered key-value format of match_prefix is: " / bfs / fs / fsid / lcprefix / ts_str_fh_key, where fsid is the file system ID and ts_str is the global clock (ts) string (aligned to a minimum width of 11, padded with 0s if necessary).

[0059] The steps for generating two types of ordered, incremental key-value pairs from the background operation log (bilog) are as follows: 1) Read bilog_marker from metadata and list the operation logs (bilog) following bilog_marker.

[0060] 2) Process each operation log (bilog) in a loop based on the operation log (bilog) array returned by the metadata cluster.

[0061] 3) Generate asynchronous processes based on the op types in the operation log (bilog): setxattr, delete, create, and rename, and process them accordingly.

[0062] 4) When the op type is create, the steps are as follows: A. Obtain the file's fh_key and global clock (ts) from the operation log (bilog); B. Generate an ordered mtime key value based on the file's fh_key and global clock (ts) in the format: " / bfs / fs / fsid / lcmtime / ts_str _fh_key"; C. Read the LC rules and determine if the filename matches the rule's prefix; D. If the file matches the rule prefix, then generate an ordered key value for match_prefix based on the file's fh_key and global clock (ts) in the format: " / bfs / fs / fsid / lcprefix / ts_str_fh_key"; E. Check if the pre_fh_key in the operation log (bilog) is empty. If it is not empty, delete the ordered mtime key value of " / bfs / fs / fsid / lcmtime / ts_str_pre_fh_key" based on pre_fh_key and pre_timestamp; delete the ordered match_prefix key value of " / bfs / fs / fsid / lcprefix / ts_str_pre_fh_key". F. Submit the ordered key-value pairs created in steps B and D, and the ordered key-value pairs deleted in step E, to the metadata cluster.

[0063] 5) When the op type is delete, the steps are as follows: A. Obtain the file's fh_key and global clock (ts) from the operation log (bilog); B. In the operation log (bilog), delete the ordered mtime key value of " / bfs / fs / fsid / lcmtime / ts_str_fh_key" and the ordered match_prefix key value of " / bfs / fs / fsid / lcprefix / ts_str_fh_key". C. Submit the deletion of ordered key-value pairs from step B to the metadata cluster.

[0064] 6) When the op type is setxattr, the steps are as follows: A. Obtain the file's fh_key and global clock (ts) from the operation log (bilog); B. If mtime_change in the operation log (bilog) is true, then execute steps C, D, and E; C. Generate an ordered mtime key value based on the file's fh_key and global clock (ts) in the format: " / bfs / fs / fsid / lcmtime / ts_str_fh_key", and generate an ordered match_prefix key value in the format: " / bfs / fs / fsid / lcprefix / ts_str_fh_key"; D. Based on fh_key and pre_timestamp, delete the mtime ordered key value of " / bfs / fs / fsid / lcmtime / ts_str_pre_fh_key"; delete the match_prefix ordered key value of " / bfs / fs / fsid / lcprefix / ts_str_pre_fh_key"; E. Submit the ordered key-value pairs created in step C and the ordered key-value pairs deleted in step D to the metadata cluster.

[0065] 7) When the op type is rename, the steps are as follows: A. Obtain the file's fh_key and global clock (ts) from the operation log (bilog); B. Read the LC rule and determine if the filename matches the rule's prefix. If it matches the rule, proceed to steps C and D. C. Then, based on fh_key and pre_timestamp, generate an ordered mtime key value in the format: " / bfs / fs / fsid / lcmtime / ts_str_pre_fh_key"; D. Submit the ordered key-value pairs created in step C and the ordered key-value pairs deleted in step D to the metadata cluster.

[0066] 8) Update bilog_marker based on the ts of the last operation log (bilog).

[0067] The process of the background reading operation logs (bilog) to generate two types of ordered, incrementing key-value pairs is as follows: Figure 11 As shown.

[0068] List expired files by ordered key value and execute their lifecycle. The global clock (ts) is monotonically increasing and unique throughout the system, and it is greater than the file's mtime. Therefore, if the current global clock - the global clock recorded in the operation log (ts) > expire_day (the expiration day configured in the LC rule), the file is also expired. Thus, end_ts = current global clock - expire_day (the expiration day configured in the LC rule). Files with ordered, increasing key values ​​ts_str <= end_ts are expired. The lifecycle steps for expired files, based on the ordered key-value format of mtime and the ordered key-value format of match_prefix, are as follows: 1. Read all rules related to lc_sort_key_list_marker and its lifecycle from the metadata cluster; 2. Based on all the lifecycle rules, calculate the minimum expiration time min_expire_day among the rules; 3. First, execute `list match_prefix` to match ordered key-value pairs. The steps are as follows: 1) Check if the expire_time of lc_sort_key_list_marker is greater than the current time. If it is, the marker expires and start_marker = " / bfs / fs / fsid + / lcprefix / ". If the marker does not expire, start_marker = lc_sort_key_list_marker.match_prefix_marker. 2) Obtain the current global clock cur_timestamp from the metadata cluster, then end_marker=" / bfs / fs / fsid / lcprefix / (cur_timestamp -min_expire_day)"; 3) Based on start_marker and end_marker, first execute list match_prefix ordered key-value pairs, listing only the ordered, increasing key-value pairs within the range of [start_marker and end_marker]; 4) Based on the ordered key-value array returned by the metadata cluster, process each ordered key-value pair in a loop, as follows: A. Parse fh_key from the ordered key value " / bfs / fs / fsid / lcprefix / ts_str_fh_key"; B. Read file metadata from the metadata cluster based on fh_key; C. Based on the information recorded in the file's metadata, traverse all lifecycle rules and determine whether the file matches. If it matches the deletion rule, delete the file; if it matches the archiving rule, archive the file; if it matches the layering rule, layer the file.

[0069] 5) Check if all ordered key-value pairs in the range [start_marker and end_marker] have been listed. If not, update match_prefix_marker based on the ts_str of the last ordered key-value pair and update start_marker=" / bfs / fs / fsid / lcprefix / ts_str_fh_key". If all key-value pairs have been listed, update match_prefix_marker based on the ts_str of the last ordered key-value pair and proceed with the mtime ordered key-value listing process.

[0070] 4. To execute a list mtime ordered key-value pair, follow these steps: 1) Check if the expire_time of lc_sort_key_list_marker is greater than the current time. If it is, the marker expires and start_marker = " / bfs / fs / fsid / lcmtime / ". If the marker does not expire, start_marker = lc_sort_key_list_marker.mtime_marker. 2) Obtain the current global clock cur_timestamp from the metadata cluster, then end_marker=" / bfs / fs / fsid / lcmtime / (cur_timestamp -min_expire_day)"; 3) Based on start_marker and end_marker, first execute list lcmtime ordered key-value pairs, listing only the ordered, increasing key-value pairs within the range [start_marker and end_marker]; 4) Based on the ordered key-value array returned by the metadata cluster, process each ordered key-value pair in a loop, as follows: A. Parse the fh_key from the ordered key value " / bfs / fs / fsid / lcmtime / ts_str_fh_key". B. Read file metadata from the metadata cluster based on fh_key. C. Based on the information recorded in the file's metadata, traverse all lifecycle rules and determine whether the file matches. If it matches the deletion rule, delete the file; if it matches the archiving rule, archive the file; if it matches the layering rule, layer the file.

[0071] 5) Check if all the ordered key-value pairs of lcmtime within the range of [start_marker and end_marker] have been listed. If not, update mtime_marker based on ts_str of the last ordered key-value pair, and update start_marker = " / bfs / fs / fsid / lcmtime / ts_str_fh_key"; if all the key-value pairs have been listed, update mtime_marker based on ts_str of the last ordered key-value pair.

[0072] 5. Update lc_sort_key_list_marker: Based on the ordered key-value format of mtime and the ordered key-value format of match_prefix, the lifecycle process of expired files is listed as follows: Figure 12A and Figure 12B As shown.

[0073] In the method of this invention, based on the global clock (ts) recorded in the operation log (bilog), the background reads the operation log (bilog) to generate an mtime ordered key value for each file, and simultaneously determines whether the file matches the rule prefix, generating a match_prefix ordered key value. Utilizing the characteristic that the mtime ordered key values ​​and match_prefix ordered key values ​​are stored in the metadata service cluster in the order of the global clock (ts), each time a list is executed, it only needs to scan the files corresponding to the key values ​​within the range of [start_marker = end_marker at the end of the previous list or " / bfs / fs / fsid / lcmtime / " or " / bfs / fs / fsid / lcprefix / ", end_marker = " / bfs / fs / fsid / lcmtime / current global clock - expire_day" (the expiration day configured in the LC rule)], thereby accurately listing expired files and files matching the rule prefix. After listing the files, all rules are matched and processed. This solution addresses the issues of wasted CPU and I / O resources caused by the need for a full file list during each lifecycle archiving, hierarchical, and deletion process (including the problem of duplicate file lists when there are multiple rules). An example using mtime-ordered keys is provided; see [link to example]. Figure 13During each lifecycle execution, the system scans only the key values ​​within the range determined by subtracting the minimum expiration days from the start position of the cursor record, thus achieving precise location and incremental processing of expired files.

[0074] In summary, this invention proposes a lifecycle management method based on ordered incremental key values. It establishes a lifecycle management mechanism by recording operation logs containing a globally monotonically increasing clock for file metadata change operations, and using a background process to consume these logs to generate lexicographically ordered `mtime` and `match_prefix` key values. During lifecycle execution, the system only needs to calculate the scan end marker based on the current global clock and the minimum expiration days of the rule, and then perform a precise range query on the ordered key value index using an enumeration cursor with expiration times. This transforms the full directory scan in traditional solutions into an incremental scan targeting only expired files that match the rule prefix. Furthermore, by prioritizing the enumeration of `match_prefix` key values ​​before enumerating `mtime` key values, and by executing concurrently at the bucket level and scanning all rules at once, the problem of repeated file enumeration and processing caused by nested rule prefixes is effectively eliminated. This method significantly reduces the CPU and I / O resource consumption of lifecycle management in large-scale storage systems, improves processing efficiency, and is particularly suitable for ultra-large storage systems with hundreds of billions of files in a single bucket.

[0075] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0076] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0077] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0078] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0079] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0081] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0082] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0084] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0085] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0086] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0087] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A lifecycle processing method based on ordered incrementing key values, characterized in that, include: In a distributed key-value storage system, operation logs are recorded for changes to file metadata. These operation logs contain globally unique and monotonically increasing clock information. The background process consumes the operation logs and generates at least one ordered incremental key-value index for each file based on the clock information. The ordered incremental key-value indexes are arranged in lexicographical order according to the clock information in the KV system. Create and maintain an enumeration cursor, which is used to record the end position of each lifecycle scan and is configured with an expiration time; During lifecycle execution, a scan end marker is calculated based on the current global clock and the minimum expiration time in the lifecycle rules. Using the starting position of the enumerated cursor records and the scan end marker as the range, a range query is performed on the ordered incremental key value index in the KV system to obtain the set of files to be processed. Perform lifecycle rule matching and processing operations on the files in the set of files to be processed.

2. The method as described in claim 1, characterized in that, The step of generating at least one ordered, incremental key-value index for each file specifically includes: Generate an ordered mtime key value, the key value format of which includes a bucket identifier, an index type identifier, a string representation of the clock information, and a unique identifier of the file, so that files in the same bucket are stored consecutively in order of the size of the clock information. Determine whether the file matches the file prefix in the lifecycle rule. If it matches, generate an ordered key value named match_prefix. The key value format also includes the bucket identifier, the index type identifier, the string representation of the clock information, and the unique identifier of the file.

3. The method as described in claim 2, characterized in that, The ordered, incremental key-value index is generated by consuming operation logs through a background process, specifically including: Read the operation log cursor and list new operation log entries based on the operation log cursor; For each operation log entry, perform the corresponding index generation or deletion operation based on its operation type; For creation operations, an mtime ordered key value is generated based on the file's unique identifier and the global clock. If the filename matches a rule prefix, a match_prefix ordered key value is generated, and the corresponding old ordered key value is deleted based on the previous file's unique identifier and the previous global clock. For deletion operations, delete the corresponding mtime ordered key value and match_prefix ordered key value based on the file's unique identifier and global clock. For metadata update operations, if the file modification time changes, new mtime ordered key values ​​and match_prefix ordered key values ​​are generated based on the new file unique identifier and global clock, and the old corresponding key values ​​are deleted based on the old file unique identifier and the previous global clock. For renaming operations, if the new filename matches the rule prefix, a new mtime ordered key value is generated based on the file's unique identifier and the previous global clock. The generated or deleted ordered key-value operations are submitted to the metadata cluster, and the operation log cursor is updated according to the clock information of the last processed operation log.

4. The method as described in claim 1, characterized in that, During lifecycle execution, the process of calculating the scan end marker based on the current global clock and the minimum expiration time in the lifecycle rules specifically includes: Read the enumeration cursor and all lifecycle rules from the metadata cluster; Iterate through all lifecycle rules and calculate a minimum expiration period; Obtain the current global clock from the metadata cluster, and subtract the minimum expiration days from the current global clock to obtain the expiration time boundary; Based on the expiration time boundary, a first scan end marker for matching mtime ordered key values ​​and a second scan end marker for matching match_prefix ordered key values ​​are constructed respectively.

5. The method as described in claim 4, characterized in that, The step of performing a range query on the ordered incremental key-value index in the KV system, using the starting position of the listed cursor records and the scan end marker as the range, to obtain the set of files to be processed, specifically includes: First, a range query is performed on the ordered key value of match_prefix: it is determined whether the enumeration cursor has expired. If it has expired, the starting position is the storage root path of the ordered key value of match_prefix; otherwise, the starting position is the match_prefix marker recorded in the enumeration cursor. The range query is then performed using the starting position and the second scan end marker as the range. After completing the range query for the ordered key value of match_prefix, perform a range query for the ordered key value of mtime: determine whether the enumeration cursor has expired; if expired, the starting position is the storage root path of the ordered key value of mtime; otherwise, the starting position is the mtime mark recorded in the enumeration cursor; and perform a range query with the starting position and the first scan end mark as the range.

6. The method as described in claim 5, characterized in that, The process of performing lifecycle rule matching and processing on the files in the set of files to be processed specifically includes: For each key value parsed from the match_prefix ordered key value or the mtime ordered key value, extract the file's unique identifier; Read complete file metadata from the metadata cluster based on the file's unique identifier; Based on the file metadata, traverse all lifecycle rules to determine whether the file matches the current rule; If a deletion rule is matched, the file deletion operation is performed; if an archiving rule is matched, the file archiving operation is performed; if a storage tiering rule is matched, the file storage type conversion operation is performed.

7. The method as described in claim 5, characterized in that, After performing a range query, the process also includes updating the enumeration cursor: When the range of ordered match_prefix key values ​​is not fully listed, the match_prefix flag in the listing cursor is updated according to the clock information of the last listed ordered match_prefix key value. When all the ordered key values ​​of match_prefix within the range have been listed, update the match_prefix flag according to the last listed key value, and proceed to the process of listing ordered key values ​​of mtime. When the mtime ordered key values ​​within the range have not been completely listed, the mtime flag in the listing cursor is updated according to the clock information of the last listed mtime ordered key value, and the starting position is updated to continue listing. When all the ordered key values ​​of mtime within the range have been listed, update the mtime tag according to the last listed key value; The updated enumeration cursor is permanently stored in the metadata cluster.

8. The method as described in claim 1, characterized in that, The expiration time of the enumerated cursor is used to periodically clear the position of the cursor record: when the current time exceeds the expiration time of the record in the enumerated cursor, the cursor is determined to be expired, and the next life cycle scan will start from the root path of the ordered key value index to prevent long-term residual cursors from causing files to be missed.

9. The method as described in claim 1, characterized in that, Lifecycle execution is performed at the bucket level with concurrency granularity. A range query enumerates the set of files to be processed, which is used to match all lifecycle rules to avoid duplicate enumeration and processing caused by nested file prefixes.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the lifecycle processing method based on ordered incrementing key values ​​as described in any one of claims 1 to 9.

Images