Object data lifecycle management method, device, equipment and medium

Abstract

The present application relates to the field of data processing, in particular to an object data life cycle management method, device, equipment and medium, the object data life cycle management method is executed by electronic equipment, and the object data life cycle management method includes: obtaining the first object upload time of the expired object of target bucket, using the time index prefix in the index table, multiple objects related to the first object upload time can be efficiently located, the index information of target object is accurately determined by comparing the second object upload time of these objects with the first object upload time, and the corresponding object data is deleted accordingly, the overdue object can be directly and quickly found according to the object time index to delete, and full scanning is not necessary, not only greatly improve the processing speed and efficiency of life cycle expiration deletion, but also greatly reduce the metadata processing resource occupation, improve the overall read-write performance of storage system.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, apparatus, device and medium for object data lifecycle management. Background Art

[0002] Current distributed object storage systems can accommodate massive amounts of object data, but because storage space is capped and each bucket has specific specifications, exceeding these limits can cause problems, such as insufficient metadata and data space requiring expansion, or excessive metadata leading to poor reconstruction performance, which can severely impact customer usage.

[0003] Generally speaking, as user data grows over time, it becomes increasingly useless. This old data needs to be deleted. However, with tens of thousands of objects in a bucket, manual deletion is no longer possible. Objects under a specified prefix or directory need to be automatically deleted. Current distributed object storage systems already support automatic deletion upon lifecycle expiration. Users simply need to set lifecycle rules for the bucket, and a background thread will automatically delete expired objects based on the configured rules.

[0004] The current lifecycle background thread defaults to scanning the bucket shards of the bucket with the lifecycle rules set at dawn, judging whether the object has expired based on the object information recorded in the listed object index, and deleting the expired objects. However, since the object index is arranged in lexicographical order by object name, a full scan is required to find and delete all expired objects. Even if all objects in the entire bucket are not expired, a full scan is required for identification and processing, which is very inefficient. For some scenarios where data needs to be written quickly in a short period of time and deleted quickly, the current mechanism cannot meet the requirements, and a large number of scanning operations will greatly affect the metadata processing capabilities, thereby affecting the normal read and write business performance of the system. Summary of the Invention

[0005] The purpose of the present invention is to provide an object data lifecycle management method, device, equipment and medium, which can directly and quickly find expired objects and delete them according to the object time index without the need for a full scan. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces the metadata processing resource usage and improves the overall read and write performance of the storage system.

[0006] In a first aspect, a method for managing an object data lifecycle is provided, which is executed by an electronic device. The method for managing an object data lifecycle includes:

[0007] Obtain a first object upload time of an expired object in a target bucket, where the object upload time of the expired object is determined based on the current time and a preset deletion time period;

[0008] Determining, from an index table of the target bucket, a plurality of objects of a time index according to a time index prefix, wherein the index table includes an enumeration index and a time index;

[0009] Based on the second object upload time corresponding to each of the multiple objects, determining index information of a target object having the same upload time as the first object from the multiple objects indexed by the time;

[0010] Delete the object data corresponding to the target object according to the index information of the target object.

[0011] In a preferred example, the present invention may be further configured as follows: deleting corresponding object data according to the index information of the target object includes:

[0012] Determining a first data fingerprint according to the index information of the target object;

[0013] Obtaining object data corresponding to the target object according to the index information of the target object;

[0014] Determining whether the first data fingerprint is consistent with the second data fingerprint of the object data;

[0015] If so, the object data corresponding to the target object is deleted.

[0016] In a preferred example, the present invention can be further configured to: obtain the first object upload time of the expired object in the target bucket, including:

[0017] Determining log information of a target bucket according to the multiple lifecycle log shards, where the target bucket is any one of the multiple storage buckets, and the log information of the target bucket represents a periodic management rule of object storage of the target bucket;

[0018] According to the log information of the target bucket, the first object upload time of the expired object in the target bucket is obtained.

[0019] In a preferred embodiment, the present invention may be further configured to include:

[0020] Periodically obtaining the popularity level and priority of each object in the target bucket, wherein the popularity level is determined based on at least one of the access duration, access mode, and access frequency of the object, and the priority indicates the importance of the object;

[0021] Determining an adjustment time based on the heat level and priority;

[0022] Determine the dynamic object upload time based on the object's corresponding adjustment time and object upload time;

[0023] generating an adjustment request for the object that needs to be adjusted according to the upload time of the dynamic object, so that the user can confirm whether adjustment is required;

[0024] After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

[0025] In a preferred embodiment, the present invention may be further configured to include:

[0026] Get the object deletion request of the object to be deleted;

[0027] According to the object deletion request, the head stripe data, the enumeration index and the time index, and the tail stripe data corresponding to the object to be deleted are deleted in sequence.

[0028] In a preferred embodiment, the present invention may be further configured to include:

[0029] Get object upload request;

[0030] Parsing the object upload request to obtain the object's storage bucket information, user identity, object name, upload time, object version, and object data;

[0031] Assemble metadata based on bucket information, user identity, object name, upload time, and object version;

[0032] dividing the object data into a plurality of stripes of data; and writing a plurality of first stripes of data from the plurality of stripes of data into a tail stripe of data; the plurality of stripes of data including a plurality of first stripes of data and a second stripe of data;

[0033] According to the metadata, record the enumeration index and time index corresponding to the object upload request on the bucket shard;

[0034] Head stripe data is written based on the metadata and the second stripe data.

[0035] In a preferred example, the present invention may be further configured as follows: deleting the object data corresponding to the target object according to the index information of the target object includes:

[0036] Determine the number of target objects to be deleted for the current user;

[0037] If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object;

[0038] If the amount is greater than the preset amount, determining whether the remaining available resources corresponding to the current user are less than the preset amount;

[0039] If the target resource amount is not less than the preset resource amount, object data corresponding to the target object is deleted according to index information of the target object;

[0040] If the target resource amount is less than the preset resource amount, a plurality of optimal resource providing users are determined according to user information of a plurality of other users and resource borrowing amounts, the user information including resource remaining amounts and storage object importance, and the resource borrowing amounts being determined based on the number and the remaining available resource amount;

[0041] A borrowing request is sent to the plurality of optimal resource providing users;

[0042] When consent information of a target optimal resource providing user is received, object data corresponding to the target object is deleted according to index information of the target object in a target resource amount, the target optimal resource providing user being an optimal resource providing user that receives the earliest consent information, and the target resource amount being determined based on the resource borrowing amount and the remaining available resource amount;

[0043] After the deletion of the object data corresponding to the target object is completed, the borrowed resource of the target optimal resource providing user is returned.

[0044] In a second aspect, an object data lifecycle management apparatus is provided, and includes:

[0045] An acquisition module is configured to acquire a first object upload time of an expired object of a target bucket, the object upload time of the expired object being determined based on a current time and a preset deletion time period;

[0046] A first determination module is configured to determine, from an index table of the target bucket, a plurality of objects indexed by time according to a time index prefix, wherein the index table includes enumerated indexes and time indexes;

[0047] A second determination module is configured to determine, from the plurality of objects indexed by time, index information of a target object having a same first object upload time as the target object based on respective second object upload times of the plurality of objects;

[0048] A deletion module is configured to delete object data corresponding to the target object according to the index information of the target object.

[0049] In a third aspect, an electronic device is provided, and includes:

[0050] One or more processors;

[0051] A memory;

[0052] One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the object data lifecycle management method shown in any possible implementation of the first aspect.

[0053] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes the steps of the object data lifecycle management method shown in any possible implementation method of the first aspect.

[0054] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements operations corresponding to the object data lifecycle management method shown in any possible implementation manner in the first aspect.

[0055] In summary, the object data lifecycle management method provided by the present invention has the following beneficial technical effects:

[0056] The electronic device obtains the first object upload time of the expired object in the target bucket, and uses the time index prefix in the index table to efficiently locate multiple objects related to the first object upload time. By comparing the second object upload time of these objects with the first object upload time, the index information of the target object is accurately determined, and the corresponding object data is deleted accordingly. According to the object time index, the expired object can be directly and quickly found for deletion without the need for a full scan. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces the metadata processing resource usage, and improves the overall read and write performance of the storage system.

[0057] In addition, the present invention also provides an object data lifecycle management device, equipment and medium, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0059] Figure 1 This is a flowchart of an object data lifecycle management method provided by an embodiment of the present invention;

[0060] Figure 2This is a flowchart of an object deletion process provided by an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of an object storage process provided by an embodiment of the present invention;

[0062] Figure 4 This is a flowchart of an object data lifecycle management method provided by an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of the structure of an object data lifecycle management device provided by an embodiment of the present invention;

[0064] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] This specific embodiment is merely an explanation of the present invention and is not intended to limit the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as they are within the scope of the present invention, they are protected by patent law.

[0066] It should be noted that in the optional embodiments of the present invention, the object information and other related data involved, when the embodiments of the present invention are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present invention involve data related to the object, it needs to be obtained through the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0068] In addition, the term "and / or" herein is merely used to describe an associated relationship, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " herein, unless otherwise specified, generally represents an "or" relationship between the front and rear associated objects.

[0069] In order to better understand and illustrate the scheme of the embodiments of the present application, the following will briefly describe some technical terms involved in the embodiments of the present application.

[0070] Distributed object storage system: refers to a distributed storage system for unstructured data objects.

[0071] Bucket: a bucket in object storage is a logical container for storing objects;

[0072] User: before operating data, a user must be created first, then a bucket is created under the user, and finally the data in the bucket can be operated. Multiple users can be created in the object storage system, and the data under different users are isolated from each other.

[0073] RESTful API: a design style and development method of web application, which relies on stateless client-server communication and uses HTTP protocol methods (such as GET, PUT, POST, DELETE, etc.) to perform operations on resources.

[0074] Quota: refers to the threshold of the number of objects. When uploading an object, it is judged whether the threshold is exceeded. If it is exceeded, it is intercepted and an error is reported. If the threshold is not exceeded, it is passed.

[0075] Lifecycle expiration deletion: set object expiration rules for the storage bucket, and automatically detect and delete expired objects in the storage bucket in the background. The expiration time is in days.

[0076] Object time index: an index record key-val pair with object upload time as the key prefix. The time index of all objects in the storage bucket can be sorted by time according to the time index.

[0077] Bucket shard: each storage bucket has some rados shards that record the indexes of the objects in the bucket. The index records (also known as listing indexes) of the objects in the bucket are recorded in the form of kv on these rados shards, which are used for object listing.

[0078] Multi-version: To address scenarios where objects with the same name may be overwritten in distributed object storage, or objects cannot be restored after being accidentally deleted by users, the object storage multi-version management function generates a random version number when uploading an object. It then processes uploaded files with the same name based on the file's version number and file name. This allows for the storage of multiple versions, ensuring that the object presents the latest version. Historical versions of the object at any point in time can be downloaded from the bucket.

[0079] Object stripes: Object storage divides the entire object into multiple stripes of a fixed size (4MB by default) and writes them to the storage pool. The first stripe is called the object header, which records not only data but also object metadata. The remaining stripes are called tail stripes.

[0080] Lifecycle rules include: settings for the entire bucket or prefix, which applies to all objects in the entire bucket. Prefixes only apply to objects that match the prefix. Settings apply to the current version or historical versions. The current version is the latest version. When the multi-version feature is not enabled, the object only has the latest version. Historical versions are for bucket multi-versions. When the bucket multi-version feature is enabled, a new version will be generated when the same-name object is uploaded repeatedly. The latest version is uploaded last, and the others are historical versions. If the lifecycle rule is set for historical versions, it will only take effect on historical version objects. Set an expiration time. This refers to the time from the time the object is uploaded for the object to expire and be automatically deleted after the expiration.

[0081] The current lifecycle background thread defaults to scanning the bucket shards of the bucket with the lifecycle rule set at midnight. It determines whether the object has expired based on the object information (object name, version number, creation time, etc.) recorded in the listed object index and deletes the expired objects. However, because the object index is arranged in lexicographical order by object name, a full scan is required to find and delete all expired objects. Even if all objects in the entire bucket are not expired, a full scan is required for identification and processing, which is very inefficient. The current mechanism cannot meet the needs of some scenarios where rapid writing within a short period of time is needed and rapid deletion is also required. In addition, a large number of scanning operations will greatly affect the metadata processing capabilities, thereby affecting the normal read and write business performance of the system.

[0082] Based on this, an embodiment of the present invention proposes an accelerated solution for lifecycle expiration deletion in a distributed object storage system. This solution adds an object time index to each object uploaded to the storage system, sorted by object upload time. The lifecycle background processing thread can quickly locate and delete expired objects based on the object time index, eliminating the need for a full scan. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also significantly reduces metadata processing resource usage, improving the overall read and write performance of the storage system. This solution provides customers with a good user experience, enriches the functionality of object storage, and enhances the market competitiveness of distributed object products.

[0083] Specifically, the embodiment of the present invention provides a method for managing object data lifecycle, such as Figure 1 As shown, the method provided in the embodiment of the present invention can be executed by an electronic device, which is a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to these. The terminal device and the electronic device can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention. The method includes:

[0084] S101. Obtain a first object upload time of an expired object in a target bucket, where the object upload time of the expired object is determined based on the current time and a preset deletion time period.

[0085] Among them, the target bucket represents a storage container for storing and managing multiple objects. An expired object refers to an object that has reached the deletion time limit according to the preset lifecycle policy of the target bucket. The object upload time represents the specific time point when the object is uploaded to the target bucket. The preset deletion time period refers to the time set when the target bucket or object is created to determine when the object should be deleted. It can be a fixed time, such as 10 days. For example, the preset deletion time period set for the target bucket is 10, and the current time is 2024.12.21, then the upload time of the first object is determined to be 2024.12.11, and objects on and before this date should be deleted.

[0086] S102. Determine multiple objects of a time index according to the time index prefix from the index table of the target bucket, wherein the index table includes an enumeration index and a time index;

[0087] In the embodiment of the present application, a time index is added to each object uploaded to the storage bucket, and the time index is recorded in the form of an omap kv on the bucket shard. The objects in the entire storage bucket can be sorted according to the uploading time based on the time index. The earlier the uploading time, the higher the sorting position. Since the index table includes multiple indexes, in the embodiment of the present application, the multiple objects of the time index are determined from the index table through the time index prefix, and the multiple objects can be arranged according to the uploading time in chronological order.

[0088] The index information of each object of the time index is a kv pair, k includes: a time index prefix, an object uploading time, and can further include: an object name and a version number, and v includes but is not limited to: an object name, a version number, an object size, an object creation time, and a data fingerprint (such as an object md5 value), and can further include: a dynamically changing popularity level and a priority.

[0089] For example, the composition form of the time index is as follows:

[0090] Key: 0x80 + "2006_" + object uploading time + "_" + object name + "_" + version number;

[0091] Val: a structure body recording key information of the object;

[0092] struct obj_time_index

[0093] {

[0094] string obj_name; / / object name,

[0095] string instance; / / version number, empty when multi-version is not enabled,

[0096] uint64 size; / / object size,

[0097] utime ctime; / / object creation time,

[0098] string etag; / / object md5 value

[0099] }

[0100] 0x80 + "2006_" is used as the time index prefix, only indexes with the prefix are listed, and all time indexes on the shard can be grouped together with the prefix.

[0101] S103, determining index information of a target object with the same first object uploading time from the multiple objects of the time index based on the respective corresponding second object uploading time of the multiple objects.

[0102] In the embodiment of the present invention, based on the corresponding second object upload time of multiple objects, the target object with the same upload time as the first object is determined to obtain the index information of the target object, that is, the kv information of each target object.

[0103] Furthermore, each target bucket maintains multiple bucket shards that record the rados shards of the object indexes within them. Therefore, based on the upload time of the second object corresponding to each of the multiple objects, the index information of the target object with the same upload time as the first object can be determined from the multiple objects with the time index corresponding to each bucket shard.

[0104] Furthermore, the time index is arranged in chronological order, so the target object can be confirmed simply by directly matching the first several objects, which greatly improves the efficiency of determining the target object.

[0105] S104: Delete the object data corresponding to the target object according to the index information of the target object.

[0106] According to the index information of the target object, the object data corresponding to the target object is obtained, and the object data corresponding to the target object is deleted.

[0107] Furthermore, when deleting the object data corresponding to the target object, the time index and enumeration index of the target object can also be deleted. It should be noted that this operation can also be performed on all deletion operations, such as delete obj, lifecycle object deletion, batch object deletion, etc.

[0108] It can be seen that in the embodiment of the present invention, the first object upload time of the expired object of the target bucket is obtained, and the time index prefix in the index table is used to efficiently locate multiple objects related to the first object upload time. By comparing the second object upload time of these objects with the first object upload time, the index information of the target object is accurately determined, and the corresponding object data is deleted accordingly. According to the object time index, the expired object can be directly and quickly found for deletion without the need for a full scan. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces the metadata processing resource usage, thereby improving the overall read and write performance of the storage system.

[0109] A possible implementation of the embodiment of the present invention is to delete corresponding object data according to index information of the target object, including:

[0110] Determining a first data fingerprint according to the index information of the target object;

[0111] Obtain the object data corresponding to the target object according to the index information of the target object;

[0112] Determining whether the first data fingerprint and the second data fingerprint of the object data are consistent;

[0113] If so, delete the object data corresponding to the target object.

[0114] The index information includes a data fingerprint, which can be generated based on the important content of the object data, such as an MD5 value. Furthermore, based on the index information of the target object, a first data fingerprint can be determined. Then, after the object data corresponding to the actual stored target object is found through the index information, a second data fingerprint corresponding to the object data is obtained. If the two data fingerprints are identical, the object data is correct and can be deleted. If they are inconsistent, a prompt is given and the index information in the multiple index information that needs to be processed in this management is deleted.

[0115] A possible implementation of an embodiment of the present invention is to obtain the first object upload time of an expired object of a target bucket, including: determining the log information of the target bucket based on multiple lifecycle log shards, where the target bucket is any one of multiple storage buckets, and the log information of the target bucket represents the periodic management rules of the object storage of the target bucket; and obtaining the first object upload time of the expired object of the target bucket based on the log information of the target bucket.

[0116] In this embodiment of the present invention, multiple lifecycle log shards are configured. Each lifecycle log shard contains periodic management rules for multiple storage buckets, each of which is hashed in key-value format on the lifecycle log shard. After traversing multiple lifecycle log shards, the log information for the target bucket is determined. Based on the log information for the target bucket, the first object upload time of the expired object in the target bucket is obtained.

[0117] A possible implementation of the embodiment of the present invention further includes:

[0118] Regularly obtain the popularity level and priority of each object in the target bucket, where the popularity level is determined based on at least one of the object's access time, access method, and access frequency, and the priority indicates the importance of the object;

[0119] Determine the adjustment time based on the heat level and priority;

[0120] Determine the dynamic object upload time based on the object's corresponding adjustment time and object upload time;

[0121] Generate an adjustment request for the object that needs to be adjusted based on the dynamic object upload time, so that the user can confirm whether the adjustment is needed;

[0122] After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

[0123] The method for determining each object's popularity level includes determining the object's popularity level based on at least one of the object's access duration, access method, and access frequency. Access duration T represents the duration of time the object was accessed within a historical period (e.g., within the past N days), where the historical period can be customized by the user. Access method M indicates whether the object was directly read or executed when accessed within the historical period. Access frequency F refers to the number of times the object was accessed within the historical period. For access duration, a first score S1 corresponding to the access duration is determined based on a preset correspondence between durations and numerical values. The first score ranges from 0 to 1. Multiple access methods and corresponding fixed-value scores are pre-set. The average of the fixed scores corresponding to the access methods within the historical period is used as the second score S2 corresponding to the access method. The second score ranges from 0 to 1. For access frequency, a third score S3 = min(1, F / Fmax), where Fmax is a preset threshold, is used to determine the third score. If only one parameter is included, the score alone is used as the popularity level. If multiple parameters are included, a weighted calculation is performed to obtain a total score, which serves as the popularity level. For example, let's determine the popularity level of each object based on its access duration, access method, and access frequency. The popularity level is calculated as wT × S1 + wM × S2 + wF × S3, where wT is the weight of access time, wM is the weight of access method, and wF is the weight of access frequency. wT + wM + wF = 1.

[0124] The priority is a user-defined setting, ranging from 0 to 1.

[0125] After obtaining the heat level and priority, if there is a heat level greater than the preset heat level threshold or the priority is greater than the priority threshold, the adjustment time is determined to be the maximum adjustment time, and the maximum adjustment time is set by the user according to actual conditions; if there is no heat level greater than the preset heat level threshold, and there is no priority greater than the priority threshold, a weighted calculation is performed based on the heat level and priority to obtain a comprehensive value, and the adjustment time is determined according to the comprehensive value and the time adjustment relationship, where the time adjustment relationship is a correspondence between multiple value ranges and multiple adjustment times.

[0126] After obtaining the adjustment time, the dynamic object upload time is determined based on the object's corresponding adjustment time and object upload time, that is, the object upload time is postponed. Further, after user confirmation, the original object upload time can also be recorded in the index information. At the same time as the update, related information such as fingerprint information will be synchronously updated.

[0127] It can be seen that in the embodiment of the present invention, the heat level of each object in the target bucket is regularly evaluated, and the adjustment time is determined according to the heat level and priority of the object, and then the object upload time is dynamically adjusted. The storage period of important or hot objects can be extended according to their heat and priority, making periodic management more flexible and improving user experience.

[0128] A possible implementation of the embodiment of the present invention further includes: obtaining an object deletion request for the object to be deleted; and deleting the head stripe data, enumeration index and time index, and tail stripe data corresponding to the object to be deleted in sequence according to the object deletion request.

[0129] See also Figure 2 , receives an object deletion request (all deletion operations); deletes the object header; deletes the object's enumeration index and time index; deletes the object's tail stripe data; and returns a response.

[0130] Specifically, when a user or system needs to delete an object in a bucket, a deletion request is sent to the object storage system. The request usually contains information about the object to be deleted, such as the object name, version, and possible target bucket information, which can uniquely locate the object to be deleted. In the object storage system, each object includes header stripe data and tail stripe data. Deleting the object header associated with the object to be deleted ensures that the object's metadata no longer occupies storage space and cannot be accessed through the system interface. Then, the object's enumeration index and time index are deleted to ensure the reliability of the index. Deleting the tail stripe data associated with the object to be deleted ensures that the object's data no longer occupies storage space and cannot be accessed through the system interface. Furthermore, a response can be returned to the requester to indicate the success or failure of the object deletion operation.

[0131] A possible implementation method of an embodiment of the present invention also includes: obtaining an object upload request; parsing the object upload request to obtain the object's bucket information, user identity, object name, upload time, object version, and object data; assembling metadata based on the bucket information, user identity, object name, upload time, and object version; dividing the object data into multiple stripes of data; and writing multiple first stripes of data from the multiple stripes of data into the tail stripe data; the multiple stripes include multiple first stripes of data and second stripes of data; according to the metadata, recording the enumeration index and time index corresponding to the object upload request on the bucket shard; and writing the header stripe data based on the metadata and the second stripe data.

[0132] See also Figure 3 , receives the client's object upload request; parses the request parameters and assembles the object's metadata; receives the object data uploaded by the client and writes the tail stripe data first; records the enumeration index and time index on the bucket shard; writes the object header stripe; and responds to the client.

[0133] Specifically, when a user uploads an object to be stored to the object storage system through a client, the system receives an object upload request. The system parses the parameters in the request to obtain the object's bucket information, user identity, object name, upload time, object version, and object data. The object's metadata is assembled based on the bucket information, user identity, object name, upload time, and object version. To improve write efficiency and fault tolerance, the file is split into multiple stripes, and the object data is divided into multiple stripes. To quickly confirm a successful write, the tail stripe data of the file is written first. Object storage systems usually divide buckets into multiple shards to improve storage and access efficiency. Index information about the newly uploaded object is recorded on the corresponding bucket shard, including an enumeration index and a time index (used to find objects in chronological order). After completing the writing of the tail stripe data and the recording of the index information, the head stripe data of the object is written. After confirming that the object data has been successfully stored, a successful upload response is returned to the client.

[0134] Based on any of the above embodiments, the present invention provides another object data lifecycle management method. Figure 4 ,include:

[0135] Background thread started;

[0136] Loop through the buckets recorded on each lifecycle log shard;

[0137] According to the log information, obtain the end_time of the expired object being processed (which can be calculated based on the expiration time of the lifecycle rule and the current time);

[0138] Loop through the shards of the bucket;

[0139] Use 0x80 + "2006_" as the prefix and start marker, and 0x80 + "2006_" + end_time as the end_marker to list the time index on the bucket shards;

[0140] Process the listed object time indexes in sequence;

[0141] Obtain metadata information of deleted objects based on the listed object time index information;

[0142] Get the metadata information of the object. If the object does not exist, discard the log.

[0143] Determine whether the md5 values ​​of the objects are the same:

[0144] If the MD5 values ​​are different, the log is discarded. This means that the object existed during the scan, but a new object was uploaded later, overwriting the old object and creating an object with the same name, but with different content.

[0145] If the md5 values ​​are the same, delete the object data and metadata;

[0146] Determine whether all lifecycle log shards have been processed:

[0147] If the processing is completed, enter wait until the next processing cycle;

[0148] If not completed, continue processing.

[0149] In summary, the embodiments of the present invention propose an object time index recording method, an accelerated method for lifecycle expiration deletion, and an object time index pruning and deletion method. Specifically, the method for recording the time index of uploaded objects in the storage bucket is recorded together with the object listing index, which avoids additional IO interactions and performance impact, and can implement sorting of objects in the storage bucket by upload time based on the index; the accelerated method for lifecycle expiration deletion lists objects based on the recorded time index, which can implement listing objects in order of upload time, and directly and quickly find expired objects for deletion, which greatly improves the speed and efficiency of lifecycle processing, avoids full enumeration every day, and greatly reduces metadata processing resource usage, thereby improving the overall read and write performance of the storage system. The object time index pruning method prunes and deletes together with the object deletion listing index, which avoids additional IO interactions and performance impact.

[0150] A possible implementation of an embodiment of the present invention is to delete object data corresponding to a target object according to index information of the target object, including:

[0151] Determine the number of target objects to be deleted for the current user;

[0152] If the number is not greater than the preset number, the object data corresponding to the target object is deleted according to the index information of the target object;

[0153] If the number is greater than the preset number, determining whether the remaining available resources corresponding to the current user are less than the preset resource amount;

[0154] If it is not less than the preset resource amount, the object data corresponding to the target object is deleted according to the index information of the target object;

[0155] If the resource amount is less than the preset amount, multiple optimal resource providers are determined based on the user information and resource borrowing amount of multiple other users. The user information includes the remaining resource amount and the importance of the storage object. The resource borrowing amount is determined based on the quantity and the remaining available resource amount.

[0156] Sending borrowing requests to multiple optimal resource providing users;

[0157] After receiving the consent information of the target optimal resource provider, in the case of the target resource quantity, the object data corresponding to the target object is deleted according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource quantity is determined by the resource borrowing quantity and the remaining available resource quantity;

[0158] After completing the deletion operation of the object data corresponding to the target object, the borrowed target optimal resource is returned to the user.

[0159] The target bucket corresponds to the current user. The preset quantity and resource amount are user-defined values. When a large number of a user's objects expire and their storage pool is running low on space, the system considers the needs of other users to ensure the selection of the most suitable resource provider for borrowing resources. Once the objects are deleted or the resource shortage is alleviated, the borrowing of resources is immediately terminated and the resources are returned to other users.

[0160] The process of determining the amount of resource borrowing may include: obtaining data on historical deletion operations, including the number, type, and size of deleted objects, as well as resource usage when performing deletion operations; using machine learning methods to train a model using the data of historical deletion operations to obtain a prediction model, which can predict the required amount of resource borrowing based on the current number of objects to be deleted and the remaining available resources.

[0161] Based on the user information of each other user, users whose stored objects are of importance exceeding a threshold are removed from the multiple users. The remaining users are then identified as the optimal resource providers, with the highest number of users having at least a preset amount of remaining resources greater than the borrowed amount. A borrowing request is then sent to the optimal resource providers, with the user whose information is first received being the target optimal resource provider. Resources from this user are then borrowed to delete the object data corresponding to the target object within the target amount of resources. After the deletion operation is complete, the resources are returned.

[0162] It can be seen that in the embodiment of the present invention, when deleting the object data corresponding to the target object, the resource status and needs of the current user are fully considered; when there are many objects to be deleted and the amount of resources is insufficient, resources are borrowed from other users to complete the deletion operation. Specifically, if the number is large and the remaining available resources are insufficient, the optimal resource provider is determined by comprehensively considering the user information and resource borrowing amount of multiple other users, and a borrowing request is sent; the optimal resource provider who agrees to the information the earliest is selected and the target resource amount is determined; after completing the deletion operation, the borrowed resources are returned in a timely manner to ensure the recycling of resources and the efficiency of the deletion operation.

[0163] The following is an introduction to an object data lifecycle management device provided by an embodiment of the present invention. The device described below and the method described above can be referenced to each other. The device of this embodiment is set in an electronic device. Figure 5 , Figure 5 1 is a block diagram of a device according to one embodiment of the present invention, comprising:

[0164] An acquisition module 210 is configured to acquire a first object upload time of an expired object in a target bucket, where the object upload time of the expired object is determined based on a current time and a preset deletion time period;

[0165] A first determining module 220 is configured to determine, from an index table of a target bucket, a plurality of objects of a time index according to a time index prefix, wherein the index table includes an enumeration index and a time index;

[0166] A second determining module 230 is configured to determine, from the multiple objects indexed by time, index information of a target object having the same upload time as the first object based on the respective corresponding second object upload times of the multiple objects;

[0167] The deletion module 240 is configured to delete the object data corresponding to the target object according to the index information of the target object.

[0168] In one achievable manner, the deletion module 240 is used to: determine a first data fingerprint based on the index information of the target object; obtain object data corresponding to the target object based on the index information of the target object; determine whether the first data fingerprint and the second data fingerprint of the object data are consistent; and if so, delete the object data corresponding to the target object.

[0169] In one achievable method, the acquisition module 210 is used to determine the log information of the target bucket based on multiple lifecycle log shards, where the target bucket is any storage bucket among multiple storage buckets, and the log information of the target bucket represents the periodic management rules of the object storage of the target bucket; based on the log information of the target bucket, the first object upload time of the expired object of the target bucket is obtained.

[0170] In one achievable method, the present invention further includes:

[0171] An update module is configured to periodically obtain a popularity level and priority of each object in a target bucket, wherein the popularity level is determined based on at least one of the object's access duration, access mode, and access frequency, and the priority indicates the object's importance;

[0172] Determine the adjustment time based on the heat level and priority;

[0173] Determine the dynamic object upload time based on the object's corresponding adjustment time and object upload time;

[0174] Generate an adjustment request for the object that needs to be adjusted based on the dynamic object upload time, so that the user can confirm whether the adjustment is needed;

[0175] After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

[0176] In one implementable manner, the deletion module is further configured to: obtain an object deletion request for an object to be deleted;

[0177] According to the object deletion request, the head stripe data, enumeration index and time index, and tail stripe data corresponding to the object to be deleted are deleted in sequence.

[0178] In one achievable method, the present invention further includes:

[0179] The storage module is used to: obtain object upload requests;

[0180] Parse the object upload request to obtain the object's bucket information, user identity, object name, upload time, object version, and object data;

[0181] Assemble metadata based on bucket information, user identity, object name, upload time, and object version;

[0182] Dividing the object data into a plurality of strips of data; and writing a plurality of first strips of data from the plurality of strips of data into the tail strip of data; the plurality of strips of data including a plurality of first strips of data and a second strip of data;

[0183] Based on the metadata, the enumeration index and time index corresponding to the object upload request are recorded on the bucket shard;

[0184] The head stripe data is written based on the metadata and the second stripe data.

[0185] In one achievable manner, the deletion module 240 is further configured to:

[0186] Determine the number of target objects to be deleted for the current user;

[0187] If the number is not greater than the preset number, the object data corresponding to the target object is deleted according to the index information of the target object;

[0188] If the number is greater than the preset number, determining whether the remaining available resources corresponding to the current user are less than the preset resource amount;

[0189] If it is not less than the preset resource amount, the object data corresponding to the target object is deleted according to the index information of the target object;

[0190] If the resource amount is less than the preset amount, multiple optimal resource providers are determined based on the user information and resource borrowing amount of multiple other users. The user information includes the remaining resource amount and the importance of the storage object. The resource borrowing amount is determined based on the quantity and the remaining available resource amount.

[0191] Sending borrowing requests to multiple optimal resource providing users;

[0192] After receiving the consent information of the target optimal resource provider, in the case of the target resource quantity, the object data corresponding to the target object is deleted according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource quantity is determined by the resource borrowing quantity and the remaining available resource quantity;

[0193] After completing the deletion operation of the object data corresponding to the target object, the borrowed target optimal resource is returned to the user.

[0194] An embodiment of the present invention provides an electronic device, such as Figure 6 As shown, Figure 6 The electronic device 300 shown includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may further include a transceiver 304. It should be noted that in practical applications, the number of transceivers 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present invention.

[0195] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0196] Bus 302 may include a path for transmitting information between the above components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0197] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0198] The memory 303 is used to store application code for executing the solution of the present invention, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the above method embodiment.

[0199] Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0200] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0201] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the corresponding contents of the aforementioned method embodiment are implemented.

[0202] It should be understood that although the steps in the flowcharts of the drawings are shown in sequence according to the indication of the arrows, the steps are not necessarily executed in sequence according to the indication of the arrows. Unless explicitly stated herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, at least part of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0203] The above merely describes some embodiments of the present application, and it should be pointed out that for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the scope of protection of the present application.

Claims

1. A method for managing object data lifecycle, characterized in that: Executed by an electronic device, the object data lifecycle management method includes: Obtain a first object upload time of an expired object in a target bucket, where the object upload time of the expired object is determined based on the current time and a preset deletion time period; Determining, from an index table of the target bucket, a plurality of objects of a time index according to a time index prefix, wherein the index table includes an enumeration index and a time index; Based on the second object upload time corresponding to each of the multiple objects, determining index information of a target object having the same upload time as the first object from the multiple objects indexed by the time; Deleting object data corresponding to the target object according to the index information of the target object; The deleting the object data corresponding to the target object according to the index information of the target object includes: Determine the number of target objects to be deleted for the current user; If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object; If the amount is greater than the preset amount, determining whether the remaining available resources corresponding to the current user are less than the preset amount; If the amount is not less than the preset resource amount, deleting the object data corresponding to the target object according to the index information of the target object; If the amount of resources is less than the preset amount, multiple optimal resource providing users are determined based on user information of multiple other users and the amount of resources borrowed, wherein the user information includes the remaining amount of resources and the importance of the storage object, and the amount of resources borrowed is determined based on the amount and the remaining available resources; Sending a borrowing request to the plurality of optimal resource providing users; After receiving consent information from a target optimal resource provider, in the case of a target resource amount, deleting the object data corresponding to the target object according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource amount is determined by the resource borrowing amount and the remaining available resource amount; After completing the deletion operation of the object data corresponding to the target object, the borrowed resources of the target optimal resource provider are returned.

2. The object data lifecycle management method according to claim 1, characterized in that: The deleting corresponding object data according to the index information of the target object includes: Determining a first data fingerprint according to the index information of the target object; Obtaining object data corresponding to the target object according to the index information of the target object; Determining whether the first data fingerprint is consistent with the second data fingerprint of the object data; If so, the object data corresponding to the target object is deleted.

3. The object data lifecycle management method according to claim 1, wherein: Get the first upload time of the expired object in the target bucket, including: Determining log information of a target bucket according to the multiple lifecycle log shards, where the target bucket is any one of the multiple storage buckets, and the log information of the target bucket represents a periodic management rule of object storage of the target bucket; According to the log information of the target bucket, the first object upload time of the expired object in the target bucket is obtained.

4. The object data lifecycle management method according to claim 3, characterized in that: Also includes: Periodically obtaining the popularity level and priority of each object in the target bucket, wherein the popularity level is determined based on at least one of the access duration, access mode, and access frequency of the object, and the priority indicates the importance of the object; Determining an adjustment time based on the heat level and priority; Determine the dynamic object upload time based on the object's corresponding adjustment time and object upload time; generating an adjustment request for the object that needs to be adjusted according to the upload time of the dynamic object, so that the user can confirm whether adjustment is required; After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

5. The object data lifecycle management method according to claim 1, characterized in that: Also includes: Get the object deletion request of the object to be deleted; According to the object deletion request, the head stripe data, the enumeration index and the time index, and the tail stripe data corresponding to the object to be deleted are deleted in sequence.

6. The object data lifecycle management method according to claim 1, characterized in that: Also includes: Get object upload request; Parsing the object upload request to obtain the object's storage bucket information, user identity, object name, upload time, object version, and object data; Assemble metadata based on bucket information, user identity, object name, upload time, and object version; dividing the object data into a plurality of strips of data; and writing a plurality of first strips of data from the plurality of strips of data into the tail stripe of data; the plurality of strips of data including a plurality of first strips of data and a second stripe of data; According to the metadata, record the enumeration index and time index corresponding to the object upload request on the bucket shard; Head stripe data is written based on the metadata and the second stripe data.

7. An object data lifecycle management device, characterized in that: include: An acquisition module, configured to acquire a first object upload time of an expired object in a target bucket, where the object upload time of the expired object is determined based on a current time and a preset deletion time period; A first determining module is configured to determine, from an index table of the target bucket, a plurality of objects of a time index according to a time index prefix, wherein the index table includes an enumeration index and a time index; a second determining module configured to determine, from the plurality of objects indexed by the time, index information of a target object having the same upload time as the first object based on the respective corresponding second object upload times of the plurality of objects; A deletion module, configured to delete the object data corresponding to the target object according to the index information of the target object; The deletion module is used to: determine the number of target objects to be deleted by the current user; If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object; If the amount is greater than the preset amount, determining whether the remaining available resources corresponding to the current user are less than the preset amount; If the amount is not less than the preset resource amount, deleting the object data corresponding to the target object according to the index information of the target object; If the amount of resources is less than the preset amount, multiple optimal resource providing users are determined based on user information of multiple other users and the amount of resources borrowed, wherein the user information includes the remaining amount of resources and the importance of the storage object, and the amount of resources borrowed is determined based on the amount and the remaining available resources; Sending a borrowing request to the plurality of optimal resource providing users; After receiving consent information from a target optimal resource provider, in the case of a target resource amount, deleting the object data corresponding to the target object according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource amount is determined by the resource borrowing amount and the remaining available resource amount; After completing the deletion operation of the object data corresponding to the target object, the borrowed resources of the target optimal resource provider are returned.

8. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: execute the steps of the object data lifecycle management method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the steps of the object data lifecycle management method according to any one of claims 1 to 6.

Images