Method and device for managing the life cycle of a protocol interworking storage system

Abstract

The application provides a life cycle management method and device of a protocol interworking storage system. In one example, the method comprises: in the case of detecting a creation instruction or a modification instruction for a target data entity, recording identification information and a write time stamp information of the target data entity to a corresponding life cycle data set according to an expiration time of the target data entity; scanning the life cycle data set to determine a data entity whose life cycle expires; and performing corresponding processing on the data entity according to a life cycle management strategy of the data entity. The method can implement life cycle management of a data entity of a NAS protocol or a HDFS protocol, and improve the efficiency of life cycle management of the data entity.

Description

Technical Field

[0001] This application relates to the field of computer storage applications, and in particular to a lifecycle management method and device for a protocol-interoperable storage system. Background Technology

[0002] A protocol-interoperable storage system refers to an unstructured data fusion storage system that supports NAS (Network Attached Storage) / object / HDFS (Hadoop Distributed File System) protocols, and files / objects written in any one protocol can support access, modification, retrieval, and other operations in other protocols.

[0003] The S3 (Simple Storage Service) protocol supports configuring object lifecycle management policies to determine data expiration deletion or data tiering, but NAS and HDFS protocols do not support lifecycle management.

[0004] Therefore, how to manage the lifecycle of interoperable storage systems has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, this application provides a lifecycle management method and device for a protocol-interoperable storage system.

[0006] According to a first aspect of the embodiments of this application, a lifecycle management method for a protocol-interoperable storage system is provided, comprising:

[0007] Upon detecting a creation or modification instruction for a target data entity, the identifier information and write timestamp information of the target data entity are recorded in the corresponding lifecycle data set based on the current expiration time of the target data entity. Data entities recorded in the same lifecycle data set have expiration times within the same time range. The expiration time of a data entity is determined based on its write timestamp information and the cycle duration. For data entities using NAS or distributed HDFS protocols, the write timestamp information is the modification time m_time.

[0008] Scan the lifecycle dataset to identify data entities whose lifecycles have expired;

[0009] For data entities whose lifecycles have expired, the data entities shall be processed accordingly in accordance with the lifecycle management strategy for that data entity.

[0010] According to a second aspect of the embodiments of this application, a lifecycle management device for a protocol-interoperable storage system is provided, comprising:

[0011] The recording unit is configured to, upon detecting a creation or modification instruction for a target data entity, record the identification information and write timestamp information of the target data entity to the corresponding lifecycle data set based on the current expiration time of the target data entity; wherein, the expiration times of data entities recorded in the same lifecycle data set are within the same time range; the expiration time of the data entity is determined based on the write timestamp information and the period duration; for data entities of NAS or HDFS protocols, the write timestamp information of the data entity is the time when the data entity last changed;

[0012] The scanning unit is configured to scan the lifecycle data set to identify data entities whose lifecycles have expired;

[0013] The processing unit is configured to process a data entity whose lifecycle has expired, according to the data entity's lifecycle management strategy.

[0014] According to a third aspect of the embodiments of this application, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0015] Memory, used to store computer programs;

[0016] The processor, when executing a program stored in memory, implements the method provided in the first aspect.

[0017] According to a fourth aspect of the embodiments of this application, a non-transitory computer-readable storage medium is provided, wherein a computer program is stored in the non-transitory computer-readable storage medium, and the computer program implements the method provided in the first aspect when executed by a processor.

[0018] The lifecycle management method for the protocol interoperable storage system in this application embodiment, upon detecting a creation or modification instruction for a target data entity, records the target data entity's identification information and write timestamp information into the corresponding lifecycle data set based on the target data entity's current expiration time. By scanning the lifecycle data set, expired data entities are identified. Then, for the identified expired data entities, appropriate processing is performed according to the data entity's lifecycle management strategy. By aggregating data entities with similar expiration times into the same lifecycle data set and identifying expired data entities by scanning the lifecycle data set, the efficiency of data entity lifecycle management is improved. Furthermore, for NAS or HDFS protocol data entities, the lifecycle expiration time is determined based on the time of the most recent change and the duration of the cycle, thus achieving lifecycle management for NAS or HDFS protocol data entities. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a lifecycle management method for a protocol-interoperable storage system provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the structure of a lifecycle index file provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of a minimum index unit provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the lifecycle management device for a protocol-interoperable storage system provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.

[0025] 1. Object Storage: Taking the S3 protocol as an example, the S3 protocol uses a two-tier data model of buckets and objects to store data. Buckets can be viewed as directories, and objects can be viewed as files. Based on the S3 protocol, users can use Uniform Resource Identifiers (URIs) to find relevant data, similar to how users search for relevant files in a directory.

[0026] 2. NAS file system: It is a network-based centralized file system that provides file sharing services to multiple clients through a network (usually Ethernet). Essentially, it connects the storage device as an independent node to the network, allowing clients to operate the remote storage as if it were a local file.

[0027] 3. HDFS File System: A distributed file system designed specifically for distributed computing. Developed by the Apache Hadoop project, it is used to store and manage massive amounts of data at the petabyte (PB) or even exabyte (EB) level. It is a core storage component of the big data ecosystem (such as MapReduce and Spark).

[0028] 4. c_time, a_time, m_time: In file systems such as NAS or HDFS, c_time, a_time, and m_time are key time attributes used to record file state changes. These attributes reflect the timestamps of files under different operations. Among them:

[0029] m_time (Modification Time): Used to record the timestamp of the last time the file content was modified; the initial value is the timestamp of the file creation.

[0030] c_time(Change Time): Used to record the timestamp of the last time the file's metadata was modified.

[0031] a_time (Access Time): Used to record the timestamp of the last time the file content was accessed.

[0032] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0033] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0034] Please see Figure 1 This is a flowchart illustrating a lifecycle management method for a protocol-interoperable storage system provided in an embodiment of this application. Figure 1 As shown, the lifecycle management method for this protocol-interoperable storage system may include the following steps:

[0035] Step S100: Upon detecting a creation or modification instruction for the target data entity, the identifier information and write timestamp information of the target data entity are recorded in the corresponding lifecycle data set according to the current expiration time of the target data entity; wherein, the expiration times of data entities recorded in the same lifecycle data set are within the same time range; the expiration time of the data entity is determined based on the write timestamp information and the period duration of the data entity; for data entities of NAS protocol or HDFS protocol, the write timestamp information of the data entity is the time when the data entity was last changed.

[0036] For example, for object storage protocols, a data entity may include an object; for NAS or HDFS protocols, a data entity may include a file.

[0037] In this embodiment of the application, in order to realize lifecycle management of data entities for NAS or HDFS protocols, the time when the data entity was last changed can be used as last_modify_time. Based on the time when the data entity was last changed and the period duration, the expiration time of the data entity is determined.

[0038] For example, for data entities using NAS or HDFS protocols, the time of the most recent change can be either c_time or m_time (the choice between c_time and m_time can be determined based on the actual configuration). The initial value for the most recent change time is the creation time of the data entity.

[0039] For example, taking the time when a data entity last changed as m_time, assuming the data entity's m_time is t0 and the period length is ΔT, then the expiration time of the data entity is t0+ΔT.

[0040] For example, a data entity in the NAS / HDFS protocol includes a data entity created in accordance with the NAS / HDFS protocol and not modified in accordance with the object storage protocol, or a data entity modified in accordance with the NAS / HDFS protocol (the data entity may be created in accordance with the object storage protocol, NAS protocol or HDFS protocol).

[0041] It should be noted that, in this embodiment of the application, the expiration time of the data entity of the object storage protocol can be determined according to the definition of the object S3 protocol.

[0042] For example, the lifecycle rules based on the S3 protocol define data into two categories: data uploaded in chunks (Multipart) and data uploaded without chunks (PUT). According to the standard protocol, for data uploaded in chunks, the lifecycle begins at the time the chunking task is initialized; for data uploaded without chunks, the lifecycle begins at the object's last_modify_time (initially set to the creation time).

[0043] Taking the lifecycle expiration rules as an example, the expiration time of a sharded object is (the time of initializing the sharding task + the period duration), while the expiration time of a non-sharded uploaded object is (last_modify_time + period duration).

[0044] In this embodiment of the application, in order to improve the efficiency of data entity lifecycle management, data entities with similar expiration times can be aggregated and their identification information can be recorded in the same data set.

[0045] For example, the aforementioned data set may include, but is not limited to, files or database tables.

[0046] For example, when the data entity is an object, the identification information of the data entity can be the object key; when the data entity is a file, the identification information of the data entity can be the file name.

[0047] Accordingly, in this embodiment of the application, the identification information and the timestamp information of the data entity can be recorded in the lifecycle data set according to the expiration time of the data entity.

[0048] For example, when the data entity is an object, the timestamp information to be written can be last_modify_time or the timestamp of the initialization of the sharding task; when the data entity is a file, the timestamp information to be written is the time when the data entity was last changed (such as m_time or c_time).

[0049] Among them, the expiration times of data entities recorded in the same lifecycle data set are within the same time range.

[0050] For example, the time range mentioned above can cover a duration of several hours, one day, or several days.

[0051] Taking a time range of 1 day as an example, the expiration times of data entities recorded in the same lifecycle data set are all on the same day.

[0052] In this embodiment, considering that for interoperable storage systems, data entities can be modified via NAS or HDFS protocols, for example, by modifying and writing data entities via NAS or appending data entities via HDFS. Modification of a data entity will trigger a change in its expiration time.

[0053] In one example, the lifecycle data collection also records the version number of the data entity.

[0054] For example, for any data entity, when the data entity is created, an initial version number of the data entity can be generated and recorded in the corresponding lifecycle data set.

[0055] For example, if the expiration time of any data entity changes, such as when the data entity is modified, the version number of the data entity can be updated, and the identification information, version number (the latest version number), and timestamp of the data entity can be written and recorded in the corresponding lifecycle data set based on the expiration time of the data entity after the change.

[0056] Accordingly, for any data entity (which can be called the target entity), when a creation instruction or modification instruction for the target data entity is detected, the identification information, version number, and timestamp information of the target data entity can be recorded in the corresponding lifecycle data set according to the current expiration time of the target data entity.

[0057] For example, for object storage protocol data entities, the write timestamp information may include the time point of initializing the sharding task (for data entities uploaded in shards) or last_modify_time (for data uploaded without sharding); for data entities of NAS protocol or HDFS protocol, the write timestamp information of the data entity is the m_time of the data entity.

[0058] In one example, upon detecting a creation instruction for a target data entity, the identifier information and write timestamp information of the target data entity are recorded into the corresponding lifecycle data set based on the target data entity's current expiration time. This may include:

[0059] Upon detecting a creation instruction for a target data entity, the current expiration time of the target data entity is determined based on its creation time. The identification information of the target data entity and the timestamp are then recorded in the lifecycle data set corresponding to the current expiration time of the target data entity.

[0060] In one example, upon detecting a modification instruction for a target data entity, the identifier information and write timestamp information of the target data entity are recorded into the corresponding lifecycle data set based on the target data entity's current expiration time. This may include:

[0061] Upon detecting a modification instruction for the target data entity, a new version number is generated, and the current expiration time of the target data entity is determined based on the modification time of the target data entity. The identification information of the target data entity and the timestamp are then recorded in the lifecycle data set corresponding to the current expiration time of the target data entity.

[0062] Step S110: Scan the lifecycle data set to identify data entities whose lifecycles have expired.

[0063] Step S120: For the data entity whose lifecycle has expired, process the data entity accordingly based on its lifecycle management strategy.

[0064] In this embodiment of the application, expired data entities can be identified by scanning the lifecycle data set.

[0065] For example, lifecycle data set scanning can be performed periodically, or when a scanning command is detected.

[0066] For example, the scanning frequency of the lifecycle dataset can be adjusted according to the granularity of the cycle duration (i.e., the cycle precision). When the granularity of the cycle duration is smaller (the higher the cycle precision), the scanning frequency can be correspondingly higher.

[0067] For example, the scanning frequency when the cycle duration is in days (calendar days) can be lower than the scanning frequency when the cycle duration is in hours or minutes. In this embodiment, for data entities whose lifecycles have expired, determined by scanning the lifecycle data set, the data entities can be processed accordingly based on the lifecycle management strategy of the data entity.

[0068] For example, the above processing may include, but is not limited to, deleting data entities, or performing data layering and flow on data entities, or a "batch processing task" of a set of objects with the same characteristics.

[0069] For example, tiered data flow can include transferring expired data from solid-state drives (SSDs) to hard disk drives (HDDs). For example, the aforementioned "batch processing task" can include, but is not limited to: archiving or uploading data to the cloud after M days (M being a positive integer), compressing data after M days, or recording data after M days as events / tasks and publishing them to subscribers for use.

[0070] It can be seen that, in Figure 1 In the illustrated method flow, upon detecting a creation or modification instruction for a target data entity, the identifier and timestamp information of the target data entity are recorded in the corresponding lifecycle data set based on its current expiration time. By scanning the lifecycle data set, expired data entities are identified. Then, for these expired data entities, appropriate processing is performed according to their lifecycle management strategy. By aggregating data entities with similar expiration times into the same lifecycle data set and identifying expired data entities by scanning the lifecycle data set, the efficiency of data entity lifecycle management is improved. Furthermore, for data entities using NAS or HDFS protocols, the lifecycle expiration time is determined based on the time of the most recent change and the duration of the cycle, thus achieving lifecycle management for data entities using NAS or HDFS protocols.

[0071] In some embodiments, the identification information and write timestamp of data entities under the same namespace, the same periodic rules, and the same write time range are recorded in the same lifecycle data set.

[0072] For example, for object storage protocols, a namespace may include a bucket; for NAS or HDFS protocols, a namespace may include a directory.

[0073] For example, data entities with the same periodicity rule have the same period duration.

[0074] Accordingly, by recording the identification information and write timestamps of data entities under the same namespace, the same period rule, and the same write time range into the same lifecycle data set, data entities whose lifecycles have expired under the same namespace and the same time range can be aggregated into the same lifecycle data set.

[0075] It should be noted that, in the embodiments of this application, considering the number of data entities recorded in a single lifecycle data set, and the fact that the file size of a single lifecycle data set is usually limited, for the same lifecycle data set, a new lifecycle data set can be created to record data entity information when the number of data entities recorded in the lifecycle data set reaches a preset threshold, or when the file size reaches a preset file size threshold. In addition, for any lifecycle data set, the lifecycle data set can be closed and a new lifecycle data set can be created to record data entity information when the file opening time reaches a preset maximum time threshold.

[0076] Accordingly, the aforementioned same lifecycle data set may include multiple lifecycle data sets. The directory information of these multiple lifecycle data sets may be the same. These multiple lifecycle data sets record relevant information of data entities under the same namespace, the same period rule, and the same write time range.

[0077] For example, the catalog information of a lifecycle data set may include a cycle rule identifier and a write time range. Based on the catalog information of the lifecycle data set, the expiration time range of the data entities recorded in the lifecycle data set can be determined.

[0078] In some embodiments, the above-described scanning of the lifecycle data set to determine data entities whose lifecycles have expired may include:

[0079] For any lifecycle data set, if the current system time is later than the lower limit of the expiration time range corresponding to the lifecycle data set, based on the directory information of the lifecycle data set and the current system time, the lifecycle data set is scanned to identify data entities whose lifecycles have expired; wherein, the expiration time range corresponding to the lifecycle data set is determined based on the cycle rule and write time range corresponding to the lifecycle data set.

[0080] If, based on the directory information of the lifecycle data set and the current system time, it is determined that the current system time is earlier than the lower limit of the expiration time range corresponding to the lifecycle data set, scanning of the lifecycle data set is skipped.

[0081] For example, during the scanning of a lifecycle data set, for any lifecycle data set, the corresponding cycle rule (including cycle duration) and write time range can be determined based on the directory information of the lifecycle data set. Based on the corresponding cycle rule and write time range, the corresponding lifecycle expiration time range (which can be simply referred to as the expiration time range) can be determined.

[0082] For example, assuming the above time range covers a duration of one day, for a lifecycle dataset, assuming the period rule indicates a period duration of 7 days, and the write time range is March 1, 2025 (March 1, 2025 00:00:00~March 1, 2025 23:59:59), then the expiration time range is March 8, 2025 (March 8, 2025 00:00:00~March 8, 2025 23:59:59).

[0083] For example, once the expiration time range corresponding to the lifecycle data set is determined, the current system time can be compared with the lower limit of the expiration time range (e.g., March 8, 2025, 00:00:00 in the example above).

[0084] For example, if the current system time is later than the lower limit of the expiration time range corresponding to the lifecycle data set, the lifecycle data set is scanned to identify data entities whose lifecycles have expired; if, based on the directory information of the lifecycle data set and the current system time, it is determined that the current system time is earlier than the lower limit of the expiration time range corresponding to the lifecycle data set, scanning of the lifecycle data set is skipped.

[0085] In some embodiments, for any lifecycle data set, if it is determined that all data entities recorded in the lifecycle data set are data entities whose lifecycles have expired, the lifecycle data set can be deleted.

[0086] As can be seen, by setting the lifecycle data set directory as described above, it is possible to determine whether the data entities recorded in the lifecycle data set have reached their expiration time. This can reduce invalid scanning of the lifecycle data set, improve scanning efficiency, and reduce the consumption of system resources.

[0087] In one example, the lifecycle data set also records the version number of the data entity, which includes the initial version number generated when the data entity is created, or the new version number generated when the data entity is modified.

[0088] The aforementioned scanning of the lifecycle data set to identify data entities whose lifecycles have expired may include:

[0089] For any data entity in the lifecycle data set, if the current system time exceeds the lifecycle expiration time of the data entity based on the cycle rule corresponding to the lifecycle data set and the write timestamp information of the data entity, and if the version number of the data entity recorded in the lifecycle data set is consistent with the latest version number of the data entity, then the data entity is determined to be a data entity whose lifecycle has expired.

[0090] Based on the periodic rules corresponding to the lifecycle data set and the write timestamp information of the data entity, it is determined that the current system time has not exceeded the lifecycle expiration time of the data entity, and / or, if the version number of the data entity recorded in the lifecycle data set is inconsistent with the latest version number of the data entity, it is determined that the lifecycle of the data entity has not expired.

[0091] For example, if the version number of a data entity is also recorded in the lifecycle data set, it is also necessary to compare the version number of the data entity recorded in the lifecycle data set with the latest version number of the data entity to determine whether the data entity has expired.

[0092] For any data entity in the lifecycle data set, on the one hand, the lifecycle expiration time of the data entity can be determined based on the periodic rules corresponding to the lifecycle data set and the write timestamp information of the data entity. On the other hand, the current system time can be compared with the lifecycle expiration time of the data entity to determine whether the current system time exceeds the lifecycle expiration time of the data entity.

[0093] On the other hand, the version number of the data entity recorded in the lifecycle data set can be compared with the latest version number of the data entity recorded by the system to determine whether the two are consistent.

[0094] In cases where the version number of the data entity recorded in the lifecycle data set is inconsistent with the latest version number of the data entity recorded by the system, it can be confirmed that the data entity has been modified.

[0095] For example, if it is determined that the current system time exceeds the lifecycle expiration time of the data entity, and the version number of the data entity recorded in the lifecycle data set is consistent with the latest version number of the data entity, then the data entity is determined to be an expired data entity.

[0096] For example, if it is determined that the current system time has not exceeded the lifecycle expiration time of the data entity, and / or the version number of the data entity recorded in the lifecycle data set is inconsistent with the latest version number of the data entity, it is determined that the lifecycle of the data entity has not expired.

[0097] In this embodiment of the application, data entities with the same periodic rule prefix under the same namespace adopt the same periodic rule; data entities with the same periodic rule prefix under the same namespace correspond to the same lifecycle data set; the directory information of the lifecycle data set includes a periodic rule identifier, and the periodic rule identifier corresponds to the periodic rule prefix.

[0098] In some embodiments, the lifecycle management method for the protocol interoperability storage system provided in this application may further include:

[0099] If the period duration of a periodic rule for a data entity changes or the periodic rule for a data entity is deleted, update the expiration time range corresponding to the lifecycle data set of that data entity.

[0100] For example, when the period duration of the period rule of a data entity changes or the period rule of a data entity is deleted (which can be understood as the period duration becoming infinitely long), since the expiration time range of each data entity recorded in the same lifecycle data set remains consistent, and the directory information of the lifecycle data set does not need to be modified, the expiration time range corresponding to the lifecycle data set recording that data entity can be adaptively updated.

[0101] In some embodiments, the lifecycle management method for the protocol interoperability storage system provided in this application may further include:

[0102] When the periodic rule prefix of a data entity changes, the lifecycle data set recording the data entity is scanned, and data entities with different periodic rule prefixes recorded in the lifecycle data set are differentiated and recorded into different lifecycle data sets.

[0103] For example, since the periodic rules of data entities with different periodic rule prefixes under the same namespace are usually different, and the periodic rule identifiers corresponding to data entities with different periodic rule prefixes are usually different, and the directory information of the lifecycle data set includes the periodic rule identifier, when the periodic rule prefix of a data entity changes, its expiration time will usually change, and the directory information of the lifecycle data set will also change.

[0104] Accordingly, when the periodic rule prefix of a data entity changes, the lifecycle data set recording the data entity is scanned, and data entities with different periodic rule prefixes recorded in the lifecycle data set are differentiated and recorded into different lifecycle data sets.

[0105] For example, suppose that in a certain namespace, the periodic rule prefix AAA / (corresponding to a period duration of 7 days) is modified to / AAAB (corresponding to a period duration of 4 days) and AAAC / (corresponding to a period duration of 8 days). Then, the lifecycle data set corresponding to the periodic rule prefix AAA / in that namespace can be scanned, and the data entities recorded therein can be modified according to the periodic rule prefix, and the data entities can be recorded into the lifecycle data sets corresponding to the periodic rule prefixes AAAB / and AAAC / respectively.

[0106] In some embodiments, the lifecycle management method for the protocol interoperability storage system provided in this application may further include:

[0107] When a new periodic rule exists for a data entity, determine the lifecycle data set corresponding to the data entity of the new periodic rule, and record the data entity of the new periodic rule into the lifecycle data set.

[0108] For example, for a data entity with a newly added periodic rule (i.e., a data entity that changes from having no periodic rule to having a periodic rule), the lifecycle data set corresponding to the data entity with the newly added periodic rule can be determined, and the data entity with the newly added periodic rule can be recorded in the lifecycle data set.

[0109] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0110] This embodiment provides a scheme for lifecycle management of objects (data entities in object storage protocols) / files (data entities in NAS or HDFS protocols) under an interoperable storage system.

[0111] The specific implementation details of the solution are explained below.

[0112] I. Definition of lifecycle expiration rules for objects / files under interoperable storage systems.

[0113] 1) The lifecycle rules based on the S3 object protocol define data into two categories: multipart data and non-multipart data (PUT). According to the standard protocol, for multipart data, the lifecycle begins at the time the multipart task is initialized; for non-multipart objects, the lifecycle begins at the object's `last_modify_time`. Taking the expiration rules as an example, the expiration time for multipart objects is (the time the multipart task is initialized + the period duration), while the expiration time for non-multipart objects is (`last_modify_time` + the period duration).

[0114] 2) For NAS or HDFS protocols, there is no way to upload in chunks, so their expiration rule is defined as (last_modify_time + period duration).

[0115] 3) For the NAS / HDFS file system, the m_time attribute in the file attribute is defined as last_modify_time. Therefore, the expiration time of a file in the NAS / HDFS protocol is defined as (m_time + period duration).

[0116] 4) Since the S3 protocol does not modify fragmented / PUT data after it has been written, its expiration time can generally be considered unchanged after the data is written, unless a specific locking period is considered. However, in an interoperable protocol system, objects uploaded via the S3 protocol can be modified by the NAS protocol or appended by the HDFS protocol. Therefore, when an object is written according to the NAS / HDFS protocol, its last_modify_time will be updated to m_time, and the expiration time will also be updated to (m_time + period duration).

[0117] 5) For objects uploaded in fragments, before the fragment merging is complete, any overwriting (modification) or appending of the object itself still follows the object S3 protocol specification, using (the time of initialization of the fragment task + the period duration) as the expiration time. After the fragment merging is complete, any modifications / appendices made according to the NAS / HDFS protocol require updating the expiration time to (m_time + period duration).

[0118] II. Implementation plan for object / file lifecycle management based on the above lifecycle expiration rules.

[0119] 1. Because the lifecycle of objects / files in this interoperable storage system may change randomly, conventional object lifecycle management tasks will no longer be applicable to the interoperable storage system.

[0120] In this embodiment, the following lifecycle management strategy is designed (taking objects as an example, the implementation of files can be obtained in the same way).

[0121] When an object is newly created or modified, the object key is stored in a disk file (in this embodiment, taking the aforementioned data set as an example, the aforementioned lifecycle data set can be called a lifecycle index file). Specifically, whenever an object's expiration time is updated, an object key is appended to it.

[0122] For example, the disk file File1 contains the following serialized data:

[0123] key1-version_id-last_modify_time, key2-version_id-last_modify_time,….

[0124] An exemplary diagram illustrating the structure of the lifecycle index file can be found here. Figure 2 ,like Figure 2 As shown, a lifecycle index file can include multiple index blocks (index blk).

[0125] For any given index block, the index block may include block header information (which may include magic words such as...). Figure 2 The index block contains 'I'N'D'E'X'B'L'K', 'K'L'B'X'E'D'N'I', version number (ver), block length (Blklen), total count (Total count), valid count (Valid count), and checksum (CheckSum), as well as the block index entry (obj index entry); wherein, an index block may contain one or more block index entries.

[0126] Among them, the block index entry is the smallest unit of the index, such as... Figure 3 As shown, a block index entry may include a cell header (which may include CheckSum, ver, offset, length (Len), and metadata length (Metadata_len)) and object-related information (which may include object key, version number, and lastmodify_time). Here, offset is the offset of the block index entry within the entire index file, Len is the length of the block index entry, and Metadata_len is the length of the object-related information (variable length).

[0127] During the execution of a lifecycle task, the corresponding list of object keys can be deserialized by reading the "lifecycle index file." The task can quickly identify object keys to be deleted (i.e., identify expired object keys based on the current lifecycle index file) by scanning the disk object key list. Through object data consistency checks, such as comparing the object keys with the metadata system to see if they have been modified (e.g., comparing the version number recorded in the lifecycle index file with the latest system version number), objects that pass the consistency check can be determined to have expired.

[0128] 2. To improve task scanning efficiency, rules for serializing object keys to different "lifecycle index files" can include:

[0129] Rule 1: For object keys of the same lifecycle rule under the same bucket, serialize them into the same "lifecycle index file" (i.e. record them in the same "lifecycle index file").

[0130] Rule 2: Provided that Rule 1 is satisfied, object keys written on the same day are serialized into the same "object lifecycle index file".

[0131] That is, object keys written in the same bucket, with the same cycle rule, and on the same day (taking a writing time range of 1 day as an example) are serialized to the same "lifecycle index file".

[0132] Rule 3: Provided that Rules 1 and 2 are satisfied, if a "lifecycle index file" reaches the threshold for the number of recorded object keys, the file size threshold, or the maximum time the file has been open, the file will be closed and no new object keys will be recorded. Subsequent new object keys will be written to a new "lifecycle index file".

[0133] Rule 4: Regardless of whether the object is newly created or modified, the object key is rewritten to the corresponding "lifecycle index file" according to the above rules 1 / 2 / 3.

[0134] The following example illustrates the above rules.

[0135] Taking a cycle duration of an integer number of natural days as an example, assuming bucket1 and bucket2 have been created, their cycle rules are as follows:

[0136] The cycle rule for bucket1 is prefix="", expiration=7 days (meaning the cycle number of all objects in the bucket is 7 days), and the corresponding cycle rule ID (i.e. the cycle rule identifier mentioned above) = B1_LIFECYCLE_ID1.

[0137] The cycle rule for bucket2 is prefix1="7 / ", expiration=7 days (meaning that objects with the prefix 7 / have a cycle of 7 days), and the corresponding cycle rule ID is B2_LIFECYCLE_ID1; prefix2="30 / ", expiration=30 days (meaning that objects with the prefix 30 / have a cycle of 30 days), and the corresponding cycle rule ID is B2_LIFECYCLE_ID2.

[0138] Assuming objects are written in sequence:

[0139] bucket1:

[0140] March 1, 2025: Write to obj1, obj2, obj3, and obj4.

[0141] March 2, 2025: Write to obj3, obj4, obj5, and obj6.

[0142] bucket2:

[0143] March 1, 2025: Write to 7 / obj1, 30 / obj1, 7 / obj2, 30 / obj2, 7 / obj3, 30 / obj3.

[0144] March 2, 2025: Write to 7 / obj1, 30 / obj1, 7 / obj4, 30 / obj4, 7 / obj5, 30 / obj5.

[0145] The final serialized "lifecycle index file" is as follows (the specific serialization content is omitted, only the object key is retained):

[0146] The file in / bucket1 / B1_LIFECYCLE_ID1 / 20250301 / file_seqNum contains the following names: obj1, obj2, obj3, obj4.

[0147] The files in / bucket1 / B1_LIFECYCLE_ID1 / 20250302 / file_seqNum are: obj3, obj4, obj5, obj6.

[0148] The file in / bucket2 / B1_LIFECYCLE_ID1 / 20250301 / file_seqNum contains: 7 / obj1, 7 / obj2, 7 / obj3

[0149] Files in / bucket2 / B1_LIFECYCLE_ID1 / 20250302 / file_seqNum: 7 / obj1, 7 / obj4, 7 / obj5

[0150] The file in / bucket2 / B1_LIFECYCLE_ID2 / 20250301 / file_seqNum contains: 30 / obj1, 30 / obj2, 30 / obj3

[0151] The file in / bucket2 / B1_LIFECYCLE_ID3 / 20250302 / file_seqNum contains: 30 / obj1, 30 / obj4, and 30 / obj5.

[0152] As can be seen, by serializing the object key to the "lifecycle sequence file" according to the above rules, object keys that expire on the same day under the same bucket are serialized to the same "lifecycle sequence file". Among them, for objects that have been modified, the lifecycle expiration time will change, and duplicate object keys can be repeatedly written to a new "lifecycle sequence file".

[0153] For example, obj3 and obj4 in " / bucket1 / B1_LIFECYCLE_ID1 / 20250301 / file_seqNum" and " / bucket1 / B1_LIFECYCLE_ID1 / 20250302 / file_seqNum"; where obj3 and obj4 recorded in the latter are newer versions.

[0154] The execution rules for lifecycle management tasks can be as follows:

[0155] According to the bucket + cycle rule, scan the / bucket / cycle rule ID directory, which contains files with multiple write dates. According to the rule, scan the date directories that meet the following conditions:

[0156] Table of Contents Time + Periodicity Rule Days > Current Date

[0157] For example, if the current date when the task is executed is March 8, 2025, then the task will scan the following directories that meet the above rules:

[0158] / bucket1 / B1_LIFECYCLE_ID1 / 20250301

[0159] / bucket2 / B1_LIFECYCLE_ID1 / 20250301

[0160] Once the "Lifecycle Index Files" in the above directory have been scanned and it is confirmed that all objects in the "Lifecycle Index Files" have expired, the above directory and the "Lifecycle Index Files" therein can be deleted.

[0161] For other directories that do not meet the rules, no scanning is necessary, as the objects within them have clearly not yet expired. Based on this rule, the lifecycle rule management task will significantly reduce invalid scans of "lifecycle index files" that have not yet expired, improving scanning efficiency and reducing system resource consumption.

[0162] 3. Object lifecycle rules can be modified. When lifecycle rules change, there are several scenarios:

[0163] A. The cycle duration has been modified.

[0164] B. The periodic rules were directly deleted.

[0165] C. The prefix of the periodic rule has been modified.

[0166] D. Objects without periodic rules have had periodic rules added.

[0167] For modifications to A or B, the lifecycle rules for object keys in the same "Lifecycle Index File" remain consistent. Therefore, it is not necessary to modify the "Lifecycle Index File" itself; simply adjust the task execution time. For example, if the lifecycle duration is extended, the task execution time of the "Lifecycle Index File" in that directory will be postponed. If the lifecycle rule is deleted, the "Lifecycle Index File" will not be placed in the task queue (i.e., the lifecycle duration is infinitely long, it will not expire, and there is no need to scan it).

[0168] For modifications to C, a scanning task for differentiating "lifecycle index files" can be started, which will record object keys with different lifecycle rule prefixes in the "lifecycle index files" into different "lifecycle index files".

[0169] Modifying D allows you to start a scan task for file metadata. Once the scan task is complete, a "lifecycle index file" for that type of object (with the newly added lifecycle rules) can be generated, and lifecycle management will continue to be performed in accordance with the aforementioned rules.

[0170] 4. To reduce costs, the "lifecycle index file" can be written to a fixed area, such as a specific area on an HDD (hard disk drive). Furthermore, to avoid the space issue of repeated overwriting of the "lifecycle index file," the "lifecycle index file" for the same storage pool (i.e., the aforementioned fixed area, such as the area on the HDD used to store the "lifecycle index file") will reuse the same disk area for modification and writing as much as possible, reducing disk fragmentation issues caused by small files.

[0171] It should be noted that, in this embodiment, the above lifecycle management can be applied to "object deletion" tasks, "object transformation" (such as data tiered flow), or "batch processing tasks" of a set of objects with the same characteristics. For example, the "batch processing task" may include, but is not limited to: archiving or uploading data to the cloud after M days, compressing data after M days, or recording data after M days as an event / task and publishing it to subscribers as an event for their use.

[0172] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0173] Please see Figure 4This is a schematic diagram of the lifecycle management device for a protocol interoperable storage system provided in an embodiment of this application, as shown below. Figure 4 As shown, the lifecycle management device of the protocol-interoperable storage system may include:

[0174] The recording unit is configured to, upon detecting a creation or modification instruction for a target data entity, record the target data entity's identification information and write timestamp information to the corresponding lifecycle data set based on the target data entity's current expiration time; wherein, the expiration times of data entities recorded in the same lifecycle data set are within the same time range; the expiration time of a data entity is determined based on the data entity's write timestamp information and the period duration; for data entities using NAS or HDFS protocols, the write timestamp information of the data entity is the time when the data entity last changed;

[0175] The scanning unit is configured to scan the lifecycle data set to identify data entities whose lifecycles have expired;

[0176] The processing unit is configured to process a data entity whose lifecycle has expired, according to the data entity's lifecycle management strategy.

[0177] The specific implementation process of lifecycle management of the protocol interoperable storage system by each unit in the lifecycle management device can be found in the relevant descriptions in the above method embodiments.

[0178] Please see Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504. The memory 503 stores a computer program; the processor 501 can execute the lifecycle management method of the protocol interoperable storage system described above by executing the program stored in the memory 503.

[0179] The memory 503 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, memory 503 can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof.

[0180] This application also provides a non-transitory machine-readable storage medium storing a computer program, such as... Figure 5 The memory 503 in the computer program can be used by Figure 5 The processor 501 in the electronic device shown executes to implement the lifecycle management method of the protocol interoperable storage system described above.

[0181] This application also provides a computer program stored on a non-transitory machine-readable storage medium, such as... Figure 5 The memory 503 in the memory, and when the processor executes the computer program, it causes the processor 501 to execute the lifecycle management method of the protocol interoperable memory system described above.

Claims

1. A lifecycle management method for a protocol-interoperable storage system, characterized in that, include: Upon detecting a creation or modification instruction for a target data entity, the identifier information and write timestamp information of the target data entity are recorded in the corresponding lifecycle data set based on the current expiration time of the target data entity. Data entities recorded in the same lifecycle data set have expiration times within the same time range. The expiration time of a data entity is determined based on its write timestamp information and the period duration. For data entities using the Network Attached Storage (NAS) protocol or the Distributed File System (HDFS) protocol, the write timestamp information represents the time when the data entity most recently changed. The lifecycle data set is scanned, and the data entities whose lifecycles have expired are identified based on the write timestamp information of the data entities recorded in the scanned lifecycle data set. For data entities whose lifecycles have expired, the data entities shall be processed accordingly in accordance with the lifecycle management strategy for that data entity.

2. The method according to claim 1, characterized in that, Upon detecting a creation instruction for a target data entity, based on the current expiration time of the target data entity, the identification information and write timestamp information of the target data entity are recorded into the corresponding lifecycle data set, including: Upon detecting a creation instruction for a target data entity, the current expiration time of the target data entity is determined based on its creation time, and the identification information and timestamp of the target data entity are recorded in the lifecycle data set corresponding to the current expiration time of the target data entity. And / or, Upon detecting a modification instruction targeting a target data entity, based on the target data entity's current expiration time, the identifier information and write timestamp information of the target data entity are recorded into the corresponding lifecycle data set, including: Upon detecting a modification instruction for a target data entity, the current expiration time of the target data entity is determined based on the modification time of the target data entity. The identification information of the target data entity and the timestamp are then recorded in the lifecycle data set corresponding to the current expiration time of the target data entity.

3. The method according to claim 1, characterized in that, The identification information and write timestamp of data entities under the same namespace, the same period rule, and the same write time range are recorded in the same lifecycle data set.

4. The method according to claim 1, characterized in that, The catalog information of the lifecycle data set includes the cycle rule identifier and the write time range. Different data sets correspond to different cycle rules and / or different write time ranges. The step of scanning the lifecycle data set to determine data entities whose lifecycles have expired includes: For any lifecycle data set, if the current system time is later than the lower limit of the expiration time range corresponding to the lifecycle data set, based on the directory information of the lifecycle data set and the current system time, the lifecycle data set is scanned to identify data entities whose lifecycles have expired; wherein, the expiration time range corresponding to the lifecycle data set is determined based on the cycle rule and write time range corresponding to the lifecycle data set. If, based on the directory information of the lifecycle data set and the current system time, it is determined that the current system time is earlier than the lower limit of the expiration time range corresponding to the lifecycle data set, scanning of the lifecycle data set is skipped.

5. The method according to claim 4, characterized in that, The lifecycle data set also records the version number of the data entity, which includes the initial version number generated when the data entity is created, or the new version number generated when the data entity is modified. The step of scanning the lifecycle data set to determine data entities whose lifecycles have expired includes: For any data entity in the lifecycle data set, if the current system time exceeds the lifecycle expiration time of the data entity based on the cycle rule corresponding to the lifecycle data set and the write timestamp information of the data entity, and if the version number of the data entity recorded in the lifecycle data set is consistent with the latest version number of the data entity, then the data entity is determined to be a data entity whose lifecycle has expired.

6. The method according to claim 5, characterized in that, The method further includes: For any data entity in the lifecycle data set, if the current system time does not exceed the lifecycle expiration time of the data entity based on the cycle rules corresponding to the lifecycle data set and the write timestamp information of the data entity, and / or the version number of the data entity recorded in the lifecycle data set is inconsistent with the latest version number of the data entity, then the lifecycle of the data entity is determined not to have expired.

7. The method according to claim 1, characterized in that, The method further includes: For any lifecycle data set, if it is determined that all data entities recorded in the lifecycle data set are data entities whose lifecycles have expired, then delete the lifecycle data set.

8. The method according to claim 1, characterized in that, Data entities with the same periodicity rule prefix within the same namespace adopt the same periodicity rule; data entities with the same periodicity rule prefix within the same namespace correspond to the same lifecycle data set; the directory information of the lifecycle data set includes the periodicity rule identifier, which corresponds to the periodicity rule prefix; The method further includes: If the period duration of a periodic rule for a data entity changes or the periodic rule for a data entity is deleted, update the expiration time range corresponding to the lifecycle data set of that data entity. And / or, When the periodic rule prefix of a data entity changes, the lifecycle data set recording the data entity is scanned, and data entities with different periodic rule prefixes recorded in the lifecycle data set are differentiated and recorded into different lifecycle data sets. And / or, When a new periodic rule exists for a data entity, determine the lifecycle data set corresponding to the data entity of the new periodic rule, and record the data entity of the new periodic rule into the lifecycle data set.

9. A lifecycle management device for a protocol-interoperable storage system, characterized in that, include: The recording unit is configured to, upon detecting a creation or modification instruction for a target data entity, record the identification information and write timestamp information of the target data entity to the corresponding lifecycle data set based on the current expiration time of the target data entity; wherein, the expiration times of data entities recorded in the same lifecycle data set are within the same time range; the expiration time of the data entity is determined based on the write timestamp information and the period duration; for data entities using Network Attached Storage (NAS) protocol or Distributed File System (HDFS) protocol, the write timestamp information of the data entity is the time when the data entity last changed; The scanning unit is configured to scan the lifecycle data set and determine the data entities whose lifecycles have expired in the scanned lifecycle data set based on the write timestamp information of the data entities recorded in the scanned lifecycle data set. The processing unit is configured to process a data entity whose lifecycle has expired, according to the data entity's lifecycle management strategy.

10. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method as described in any one of claims 1 to 8.

Images