Object data life cycle management method and device, equipment and medium

Abstract

The invention relates to the field of data processing, in particular to an object data life cycle management method and device, equipment and a medium, the object data life cycle management method is executed by electronic equipment, and the object data life cycle management method comprises the steps of obtaining first object uploading time of an expired object of a target bucket, by means of the time index prefixes in the index table, a plurality of objects related to the uploading time of the first object can be efficiently positioned, the index information of the target object is accurately determined by comparing the uploading time of the second object and the uploading time of the first object of the objects, the corresponding object data is deleted accordingly, and the user experience is improved. According to the object time index, the overdue object can be directly and rapidly found to be deleted, full-amount scanning is not needed, the processing speed and efficiency of life cycle overdue deletion are greatly improved, metadata processing resource occupation is greatly reduced, and the overall read-write performance of a storage system is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an object data lifecycle management method, device, equipment and medium. Background Art

[0002] The current distributed object storage system can accommodate massive amounts of object data, but since the storage space is capped and each storage bucket also has specification limits, once the specifications are exceeded, some problems will arise, such as insufficient metadata and data space, the need for expansion, or excessive metadata resulting in poor reconstruction performance, etc., which seriously affect customer use.

[0003] Generally speaking, as time goes by, more and more useless data will be accumulated, and the useless old data needs to be deleted. However, there are thousands of objects in a bucket, and manual deletion is impossible. Objects under the specified prefix or directory need to be automatically deleted. The current distributed object storage system already supports the function of automatic deletion upon lifecycle expiration. Users only need to set lifecycle rules for the storage bucket, and the lifecycle background thread will automatically delete expired objects according to the configured rules.

[0004] The current lifecycle background thread scans the bucket shards of the bucket with the lifecycle rules set at midnight by default, determines whether the object has expired based on the object information recorded in the listed object index, and deletes the expired objects. However, since the object index is arranged in lexicographic order by object name, a full scan is required to find and delete all expired objects. Even if all objects in the entire bucket are not expired, a full scan is required to identify and process them, and the processing efficiency is very low. For some scenarios where fast writing in a short period of time and quick deletion are required, the current mechanism cannot meet the requirements, and a large number of scanning operations will greatly affect the metadata processing capabilities, thereby affecting the normal read and write business performance of the system. Summary of the invention

[0005] The purpose of the present invention is to provide an object data lifecycle management method, device, equipment and medium, which can directly and quickly find expired objects and delete them according to the object time index without full scanning. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces metadata processing resource usage and improves the overall read and write performance of the storage system.

[0006] In a first aspect, a method for managing an object data lifecycle is provided, which is executed by an electronic device. The method for managing an object data lifecycle includes:

[0007] Obtain the first object upload time of the expired object of the target bucket, where the object upload time of the expired object is determined based on the current time and a preset deletion time period;

[0008] Determine, from the index table of the target bucket, a plurality of objects of the time index according to the time index prefix, wherein the index table includes an enumeration index and a time index;

[0009] Based on the second object upload time corresponding to each of the multiple objects, determine the index information of the target object having the same upload time as the first object from the multiple objects of the time index;

[0010] According to the index information of the target object, the object data corresponding to the target object is deleted.

[0011] In a preferred example, the present invention can be further configured as follows: deleting corresponding object data according to the index information of the target object includes:

[0012] Determining a first data fingerprint according to the index information of the target object;

[0013] Acquire object data corresponding to the target object according to the index information of the target object;

[0014] Determining whether the first data fingerprint is consistent with the second data fingerprint of the object data;

[0015] If so, the object data corresponding to the target object is deleted.

[0016] In a preferred example, the present invention can be further configured as follows: obtaining the first object upload time of the expired object of the target bucket includes:

[0017] Determine log information of a target bucket according to the multiple lifecycle log shards, where the target bucket is any storage bucket among the multiple storage buckets, and the log information of the target bucket represents a periodic management rule of object storage of the target bucket;

[0018] According to the log information of the target bucket, the first object upload time of the expired object of the target bucket is obtained.

[0019] In a preferred example, the present invention can be further configured as follows:

[0020] Periodically obtain the heat level and priority of each object in the target bucket, wherein the heat level is determined based on at least one of the access duration, access mode, and access frequency of the object, and the priority indicates the importance of the object;

[0021] Determining an adjustment time according to the heat level and priority;

[0022] Determine the dynamic object upload time according to the adjustment time corresponding to the object and the object upload time;

[0023] Generate an adjustment request for the object that needs to be adjusted according to the dynamic object upload time, so that the user can confirm whether adjustment is needed;

[0024] After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

[0025] In a preferred example, the present invention can be further configured as follows:

[0026] Get the object deletion request of the object to be deleted;

[0027] According to the object deletion request, the head stripe data, the enumeration index and the time index, and the tail stripe data corresponding to the object to be deleted are deleted in sequence.

[0028] In a preferred example, the present invention can be further configured as follows:

[0029] Get object upload request;

[0030] Parsing the object upload request to obtain the object's storage bucket information, user identity, object name, upload time, object version, and object data;

[0031] Assemble metadata based on bucket information, user identity, object name, upload time, and object version;

[0032] Dividing the object data into a plurality of strips of data; and writing a plurality of first strips of data from the plurality of strips of data into the tail strip of data; the plurality of strips of data includes a plurality of first strips of data and a second strip of data;

[0033] According to the metadata, record the enumeration index and time index corresponding to the object upload request on the bucket shard;

[0034] The head stripe data is written according to the metadata and the second stripe data.

[0035] In a preferred example, the present invention can be further configured as follows: deleting the object data corresponding to the target object according to the index information of the target object includes:

[0036] Determine the number of target objects to be deleted for the current user;

[0037] If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object;

[0038] If the number is greater than the preset number, determining whether the remaining available resource amount corresponding to the current user is less than the preset resource amount;

[0039] If it is not less than the preset resource amount, deleting the object data corresponding to the target object according to the index information of the target object;

[0040] If it is less than the preset resource amount, multiple optimal resource providing users are determined according to user information of multiple other users and resource borrowing amount, wherein the user information includes the remaining resource amount and the importance of the storage object, and the resource borrowing amount is determined based on the amount and the remaining available resource amount;

[0041] Sending a borrowing request to the plurality of optimal resource providing users;

[0042] After receiving the consent information of the target optimal resource provider, in the case of the target resource amount, deleting the object data corresponding to the target object according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource amount is determined by the resource borrowing amount and the remaining available resource amount;

[0043] After completing the deletion operation of the object data corresponding to the target object, the borrowed resources of the target optimal resource provider are returned.

[0044] In a second aspect, an object data lifecycle management device is provided, comprising:

[0045] An acquisition module, configured to acquire a first object upload time of an expired object of a target bucket, wherein the object upload time of the expired object is determined based on a current time and a preset deletion time period;

[0046] A first determination module is used to determine multiple objects of time index according to the time index prefix from the index table of the target bucket, wherein the index table includes enumeration index and time index;

[0047] A second determining module, configured to determine, from the multiple objects of the time index, index information of a target object having the same upload time as the first object based on the respective corresponding second object upload times of the multiple objects;

[0048] The deleting module is used to delete the object data corresponding to the target object according to the index information of the target object.

[0049] In a third aspect, an electronic device is provided, including:

[0050] one or more processors;

[0051] Memory;

[0052] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the object data lifecycle management method shown in any possible implementation of the first aspect.

[0053] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded by a processor and executes the steps of the object data lifecycle management method shown in any possible implementation method of the first aspect.

[0054] In a fifth aspect, a computer program product is provided, including a computer program, which, when executed by a processor, implements operations corresponding to the object data lifecycle management method shown in any possible implementation manner in the first aspect.

[0055] In summary, the object data lifecycle management method provided by the present invention includes the following beneficial technical effects:

[0056] The electronic device obtains the first object upload time of the expired object in the target bucket, and uses the time index prefix in the index table to efficiently locate multiple objects related to the first object upload time. By comparing the second object upload time of these objects with the first object upload time, the index information of the target object is accurately determined, and the corresponding object data is deleted accordingly. According to the object time index, the expired object can be directly and quickly found for deletion without the need for a full scan. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces the metadata processing resource usage, and improves the overall read and write performance of the storage system.

[0057] In addition, the present invention also provides an object data lifecycle management device, equipment and medium, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0059] Figure 1 This is a flow chart of an object data lifecycle management method provided by an embodiment of the present invention;

[0060] Figure 2It is a schematic diagram of a process of object deletion provided by an embodiment of the present invention;

[0061] Figure 3 It is a schematic diagram of a process of object storage provided by an embodiment of the present invention;

[0062] Figure 4 This is a flow chart of an object data lifecycle management method provided by an embodiment of the present invention;

[0063] Figure 5 It is a structural diagram of an object data lifecycle management device provided by an embodiment of the present invention;

[0064] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the present invention.

[0066] It should be noted that in the optional embodiments of the present invention, the object information and other related data involved, when the embodiments of the present invention are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present invention involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant department, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0069] In order to better understand and illustrate the solutions of the embodiments of the present invention, some technical terms involved in the embodiments of the present invention are briefly explained below.

[0070] Distributed object storage system: refers to a distributed storage system for unstructured data objects.

[0071] Bucket: A bucket in object storage is a logical container for storing objects;

[0072] User: Before operating data, you must first create a user, then create a bucket under the user, and finally operate the data in the bucket. Multiple users can be created in the object storage system, and the data under different users are isolated from each other.

[0073] RESTful API: It is a design style and development method for network applications. It relies on stateless client-server communication and uses HTTP protocol methods (such as GET, PUT, POST, DELETE, etc.) to perform operations on resources.

[0074] Quota: refers to the threshold of the number of objects. When uploading an object, it is determined whether it exceeds the threshold. If it exceeds the threshold, it will be intercepted and an error will be reported. If it does not exceed the threshold, it will be released.

[0075] Lifecycle expiration deletion: Set object expiration rules for the bucket, and the background will automatically detect and delete expired objects in the bucket. The expiration time is in days.

[0076] Object time index: An index with the object upload time as the key prefix records key-val pairs. The time index of all objects in the bucket can be sorted by time based on this.

[0077] Bucket sharding: Each storage bucket has corresponding rados shards that record the index of the objects in it. The index records of the objects in the bucket (also known as enumeration index) are recorded on these rados shards in the form of kv for object enumeration.

[0078] Multi-version: For the scenario of how to restore objects with the same name in distributed object storage or objects that are accidentally deleted by users, the multi-version management function of object storage generates a random version number when uploading an object, and processes the upload of files with the same name based on the version number and file name of the file. Multiple versions can be saved, and the object presents the latest version. The historical version of the object at any time can be downloaded from the bucket.

[0079] Object stripe: Object storage divides the entire object into multiple stripes according to a fixed size (4M by default) and writes them into the storage pool. The first stripe is also called the object header. The object header records not only data but also object metadata. The remaining stripes are called tail stripes.

[0080] Lifecycle rules include: setting for the entire bucket or prefix, which is effective for all objects in the entire bucket. Prefixes are effective only for objects that match the prefix. Settings are effective for the current version or historical version. The current version is the latest version. When the multi-version function is not enabled, the object has only the latest version; the historical version is for the multi-version function of the bucket. When the multi-version function of the bucket is enabled, a new version will be generated when the same-name object is uploaded repeatedly. The latest version is uploaded last, and the others are historical versions. If the lifecycle rule is set for the historical version, it will only take effect on the historical version object. Set the expiration time. It means how long it will take for the object to expire from the time the object is uploaded, and it will be automatically deleted after the expiration.

[0081] The current lifecycle background thread scans the bucket shards of the bucket with the lifecycle rules set at midnight by default, determines whether the object has expired based on the object information (object name, version number, creation time, etc.) recorded in the listed object index, and deletes the expired objects. However, since the object index is arranged in lexicographic order by object name, a full scan is required to find and delete all expired objects. Even if all objects in the entire bucket are not expired, a full scan is required to identify and process them, and the processing efficiency is very low. For some scenarios where fast writing in a short period of time and quick deletion are required, the current mechanism cannot meet the requirements, and a large number of scanning operations will greatly affect the metadata processing capabilities, thereby affecting the normal read and write business performance of the system.

[0082] Based on this, the embodiment of the present invention proposes an acceleration solution for lifecycle expiration deletion of a distributed object storage system. This solution adds an object time index to each object uploaded to the storage system. The index is ordered by the object upload time. The lifecycle background processing thread can directly and quickly find the expired object and delete it according to the object time index without full scanning. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces the metadata processing resource usage and improves the overall read and write performance of the storage system. It brings a good user experience to customers, enriches the functions of object storage, and improves the market competitiveness of distributed object products.

[0083] Specifically, an embodiment of the present invention provides an object data lifecycle management method, such as Figure 1 As shown, the method provided in the embodiment of the present invention can be executed by an electronic device, and the electronic device is a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The terminal device and the electronic device can be directly or indirectly connected through wired or wireless communication, and the embodiment of the present invention is not limited here. The method includes:

[0084] S101, obtaining a first object upload time of an expired object in a target bucket, where the object upload time of the expired object is determined based on the current time and a preset deletion time period;

[0085] Among them, the target bucket represents a storage container for storing and managing multiple objects. An expired object refers to an object that has reached the deletion time limit according to the preset lifecycle policy of the target bucket. The object upload time indicates the specific time point when the object is uploaded to the target bucket. The preset deletion time period refers to the time set when the target bucket or object is created to determine when the object should be deleted. It can be a fixed time, such as 10 days. Exemplarily, the preset deletion time period set for the target bucket is 10, and the current time is 2024.12.21, then the determined upload time of the first object is 2024.12.11, and objects on and before this date should be deleted.

[0086] S102, determining multiple objects of the time index according to the time index prefix from the index table of the target bucket, wherein the index table includes a list index and a time index;

[0087] In the embodiment of the present invention, a time index is added to each object uploaded to the storage bucket, and the time index is recorded in the form of omap kv on the bucket shard. The objects in the entire storage bucket can be sorted by upload time according to the time index. The earlier the upload time, the higher the sorting position. Since the index table includes multiple indexes, in the embodiment of the present invention, multiple objects of the time index are determined from the index table through the time index prefix, and multiple objects can be arranged in the order of time upload.

[0088] The index information of each object in the time index is a kv pair, where k includes the time index prefix, object upload time, and may also include the object name and version number. v includes but is not limited to the object name, version number, object size, object creation time, data fingerprint (such as object MD5 value), and of course may also include dynamically changing heat level and priority.

[0089] For example, the time index is composed as follows:

[0090] Key: 0x80+“2006_”+object upload time+“_”+object name+“_”+version number;

[0091] Val: structure, recording the key information of the object;

[0092] struct obj_time_index

[0093] {

[0094] string obj_name; / / object name,

[0095] string instance; / / version number, empty when multiple versions are not enabled.

[0096] uint64 size; / / object size,

[0097] utime ctime; / / Object creation time,

[0098] string etag; / / object md5 value

[0099] }

[0100] Among them, 0x80+"2006_" is used as the prefix for listing time indexes, and only indexes with this prefix are listed. Moreover, with this prefix, all time indexes on the shard can be grouped together.

[0101] S103, based on the respective corresponding second object upload times of the multiple objects, determining index information of a target object having the same upload time as the first object from the multiple objects indexed by time;

[0102] In the embodiment of the present invention, based on the corresponding second object upload time of each of the multiple objects, the target object with the same upload time as the first object is determined to obtain the index information of the target object, that is, the kv information of each target object.

[0103] Furthermore, each target bucket maintains multiple bucket shards, recording the rados shards of the object index therein. Therefore, based on the respective corresponding second object upload times of the multiple objects, the index information of the target object with the same upload time as the first object can be determined from the multiple objects of the time index corresponding to each bucket shard.

[0104] Furthermore, the time index is arranged in chronological order, so the target object can be confirmed simply by directly matching the first several objects, which greatly improves the efficiency of determining the target object.

[0105] S104: Delete object data corresponding to the target object according to the index information of the target object.

[0106] According to the index information of the target object, the object data corresponding to the target object is obtained, and the object data corresponding to the target object is deleted.

[0107] Furthermore, when deleting the object data corresponding to the target object, the time index and enumeration index of the target object can also be deleted. It should be noted that this operation can also be performed on all deletion operations, such as delete obj, lifecycle object deletion, batch object deletion, etc.

[0108] It can be seen that in the embodiment of the present invention, the first object upload time of the expired object of the target bucket is obtained, and the time index prefix in the index table is used to efficiently locate multiple objects related to the first object upload time. By comparing the second object upload time of these objects with the first object upload time, the index information of the target object is accurately determined, and the corresponding object data is deleted accordingly. According to the object time index, the expired object can be directly and quickly found for deletion without the need for a full scan. This not only greatly improves the processing speed and efficiency of lifecycle expiration deletion, but also greatly reduces the metadata processing resource usage, thereby improving the overall read and write performance of the storage system.

[0109] A possible implementation of the embodiment of the present invention is to delete corresponding object data according to index information of the target object, including:

[0110] Determining a first data fingerprint according to the index information of the target object;

[0111] According to the index information of the target object, obtain the object data corresponding to the target object;

[0112] Determining whether the first data fingerprint and the second data fingerprint of the object data are consistent;

[0113] If so, delete the object data corresponding to the target object.

[0114] The index information includes a data fingerprint, which can be a data fingerprint generated based on the important content of the object data, and can be an md5 value, etc. Then, the first data fingerprint can be determined based on the index information of the target object; and then the object data corresponding to the actually stored target object is found through the index information, and the second data fingerprint corresponding to the object data is obtained. If the two data fingerprints are the same, it means that the object data is correct and is deleted. If the two are inconsistent, a prompt is given and the index information of the multiple index information that needs to be processed in this management is deleted.

[0115] A possible implementation method of an embodiment of the present invention is to obtain the first object upload time of an expired object of a target bucket, including: determining the log information of the target bucket according to multiple lifecycle log shards, the target bucket being any storage bucket among multiple storage buckets, the log information of the target bucket representing the periodic management rules of the object storage of the target bucket; and obtaining the first object upload time of the expired object of the target bucket according to the log information of the target bucket.

[0116] In an embodiment of the present invention, multiple lifecycle log shards are provided, and each lifecycle log shard has periodic management rules for multiple storage buckets, which are all hashed in the form of kv on the lifecycle log shard. After traversing multiple lifecycle log shards, the log information of the target bucket is determined. According to the log information of the target bucket, the first object upload time of the expired object of the target bucket is obtained.

[0117] A possible implementation of the embodiment of the present invention further includes:

[0118] Periodically obtain the heat level and priority of each object in the target bucket, wherein the heat level is determined based on at least one of the access time, access mode, and access frequency of the object, and the priority indicates the importance of the object;

[0119] Determine the adjustment time based on the heat level and priority;

[0120] Determine the dynamic object upload time according to the adjustment time corresponding to the object and the object upload time;

[0121] Generate an adjustment request for the object that needs to be adjusted according to the dynamic object upload time, so that the user can confirm whether adjustment is needed;

[0122] After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

[0123] The method of determining the heat level of each object includes: determining the heat level of each object based on at least one of the access time, access method, and access frequency of the object. The access time T represents the access time of the object in a historical period (such as within the past N days), wherein the historical period can be customized by the user. The access method M refers to whether the object is directly read or executed when accessing the object in the historical period. The access frequency F refers to the number of times the object is accessed in the historical period. For the access time, according to the correspondence between the preset time and the value, the first score S1 corresponding to the access time is determined, and the first score range is 0-1. Pre-set multiple access methods and corresponding fixed value scores, and the average of the fixed scores corresponding to the access methods in the historical period is used as the second score S2 corresponding to the access method, and the second score range is 0-1. For the access frequency, the third score S3=min(1,F / Fmax), wherein Fmax is the set threshold, and the third score is obtained. If only one parameter is included, only the score is used as the heat level. If multiple parameters are included, a weighted calculation is performed to obtain the total score as the heat level. For example, taking the determination of the heat level of each object based on the access time, access method, and access frequency of the object as an example, the heat level = wT×S1+wM×S2+wF×S3, wT: the weight of the access time; wM: the weight of the access method; wF: the weight of the access frequency. wT+wM+wF=1.

[0124] The priority is a user-defined setting, ranging from 0 to 1.

[0125] After obtaining the heat level and priority, if there is a heat level greater than the preset heat level threshold or the priority is greater than the priority threshold, the adjustment time is determined to be the maximum adjustment time, and the maximum adjustment time is set by the user according to actual conditions; if there is no heat level greater than the preset heat level threshold, and there is no priority greater than the priority threshold, a weighted calculation is performed based on the heat level and priority to obtain a comprehensive value, and the adjustment time is determined according to the comprehensive value and the time adjustment relationship, wherein the time adjustment relationship is a correspondence between multiple value ranges and multiple adjustment times.

[0126] After obtaining the adjustment time, the dynamic object upload time is determined according to the adjustment time and object upload time corresponding to the object, that is, the object upload time is postponed. Further, after user confirmation, the original object upload time can also be recorded in the index information. At the same time as the update, related information such as fingerprint information will be updated synchronously.

[0127] It can be seen that in the embodiment of the present invention, the heat level of each object in the target bucket is regularly evaluated, and the adjustment time is determined according to the heat level and priority of the object, and then the object upload time is dynamically adjusted. The storage period of important or high-heat objects can be extended according to their heat and priority, making periodic management more flexible and improving user experience.

[0128] A possible implementation of the embodiment of the present invention further includes: obtaining an object deletion request for the object to be deleted; and deleting the head stripe data, enumeration index and time index, and tail stripe data corresponding to the object to be deleted in sequence according to the object deletion request.

[0129] See also Figure 2 , receive an object deletion request (all deletion operations); delete the object header; delete the object's enumeration index and time index; delete the object's tail stripe data; return a response.

[0130] Specifically, when a user or system needs to delete an object in a bucket, a deletion request is sent to the object storage system. The request usually contains information about the object to be deleted, such as the object name, version, and possible target bucket information that can uniquely locate the object to be deleted. In the object storage system, each object includes header stripe data and tail stripe data. Deleting the object header associated with the object to be deleted ensures that the metadata of the object no longer occupies storage space and cannot be accessed through the system interface; then deleting the enumeration index and time index of the object to ensure the reliability of the index; deleting the tail stripe data associated with the object to be deleted makes the object data no longer occupy storage space and cannot be accessed through the system interface. Furthermore, a response can be returned to the requester to indicate the success or failure of the object deletion operation.

[0131] A possible implementation method of an embodiment of the present invention also includes: obtaining an object upload request; parsing the object upload request to obtain the object's bucket information, user identity, object name, upload time, object version, and object data; assembling metadata based on the bucket information, user identity, object name, upload time, and object version; dividing the object data into multiple stripes of data; and writing multiple first stripes of data from the multiple stripes of data into the tail stripe data; the multiple stripes of data include multiple first stripes of data and second stripes of data; according to the metadata, recording the enumeration index and time index corresponding to the object upload request on the bucket shard; according to the metadata and the second stripe data, writing the head stripe data.

[0132] See also Figure 3 , receive the client's object upload request; parse the request parameters and assemble the object's metadata; receive the object data uploaded by the client and write the tail stripe data first; record the enumeration index and time index on the bucket shard; write the object header stripe; and respond to the client.

[0133] Specifically, when a user uploads an object to be stored to the object storage system through a client, the system receives an object upload request. Parse the parameters in the request to obtain the object's bucket information, user identity, object name, upload time, object version, and object data. Assemble the object's metadata based on the bucket information, user identity, object name, upload time, and object version; in order to improve write efficiency and fault tolerance, the file will be divided into multiple stripes, and the object data will be divided into multiple stripes of data. In order to quickly confirm the success of the write, the tail stripe data of the file is written first; the object storage system usually divides the bucket into multiple shards to improve the efficiency of storage and access; record the index information about the newly uploaded object on the corresponding bucket shard, including the enumeration index and time index (used to find objects in chronological order); after completing the writing of the tail stripe data and the recording of the index information, write the head stripe data of the object; after confirming that the object data has been successfully stored, return a response of successful upload to the client.

[0134] Based on any of the above embodiments, the present invention provides another object data lifecycle management method. Figure 4 ,include:

[0135] Background thread started;

[0136] Loop through the buckets recorded on each lifecycle log shard;

[0137] According to the log information, get the end_time of the expired object processed this time (it can be calculated based on the expiration time of the lifecycle rule and the current time);

[0138] Loop through the shards of the bucket;

[0139] Use 0x80+"2006_" as prefix and start marker, and 0x80+"2006_"+end_time as end_marker to list the time index on the bucket shards;

[0140] Process the listed object time indexes in sequence;

[0141] Obtain the metadata information of the deleted object according to the listed object time index information;

[0142] Get the metadata information of the object. If the object does not exist, discard the log.

[0143] Determine whether the md5 values ​​of the objects are the same:

[0144] If the md5 values ​​are different, the log is discarded; this means that the object still existed during the scan, and then a new object was uploaded, overwriting the old object, creating an object with the same name, but with different content from the original object;

[0145] If the md5 values ​​are the same, delete the object data and metadata;

[0146] Determine whether all lifecycle log shards have been processed:

[0147] If the processing is completed, enter the wait state until the next processing cycle;

[0148] If not completed, continue processing.

[0149] In summary, the embodiments of the present invention propose an object time index recording method, an acceleration method for lifecycle expiration deletion, and an object time index pruning and deletion method. Specifically, the method for recording the time index of uploaded objects in the storage bucket is recorded together with the object listing index, which avoids additional IO interactions and performance impact, and can implement sorting of objects in the storage bucket by upload time based on the index; the acceleration method for lifecycle expiration deletion lists objects according to the recorded time index, which can implement listing of objects in order of upload time, and directly and quickly find expired objects for deletion, which greatly improves the speed and efficiency of lifecycle processing, avoids full enumeration every day, and also greatly reduces metadata processing resource usage, improving the overall read and write performance of the storage system. The object time index pruning method prunes and deletes together with the object deletion and listing index, which avoids additional IO interactions and performance impact.

[0150] A possible implementation of the embodiment of the present invention is to delete object data corresponding to the target object according to the index information of the target object, including:

[0151] Determine the number of target objects to be deleted for the current user;

[0152] If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object;

[0153] If the number is greater than the preset number, determining whether the remaining available resources corresponding to the current user are less than the preset resources;

[0154] If it is not less than the preset resource amount, the object data corresponding to the target object is deleted according to the index information of the target object;

[0155] If the resource amount is less than the preset amount, multiple optimal resource providing users are determined based on the user information of multiple other users and the resource borrowing amount. The user information includes the remaining amount of resources and the importance of storage objects. The resource borrowing amount is determined based on the quantity and the remaining available resources.

[0156] Sending borrowing requests to multiple optimal resource providing users;

[0157] After receiving the consent information of the target optimal resource provider, in the case of the target resource amount, the object data corresponding to the target object is deleted according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource amount is determined by the resource borrowing amount and the remaining available resource amount;

[0158] After completing the deletion operation of the object data corresponding to the target object, the borrowed target optimal resource is returned to provide the user with resources.

[0159] The user corresponding to the target bucket is the current user. The preset number and preset resource amount are the values ​​preset by the user. When a large number of a user's objects expire and the storage pool space is tight, the situation of other users is comprehensively considered to ensure that the most suitable resource provider is selected to borrow resources. After the deletion is completed or the resource shortage is alleviated, the borrowing of resources is immediately stopped and these resources are returned to other users.

[0160] The process of determining the amount of resource borrowing may include: obtaining data on historical deletion operations, including the number, type, and size of deleted objects, and resource usage when performing deletion operations; using a machine learning method to train a model using the data on historical deletion operations to obtain a prediction model, which can predict the required amount of resource borrowing based on the current number of objects to be deleted and the remaining available resources.

[0161] According to the user information of each other user, other users whose storage object importance exceeds the importance threshold are removed from multiple other users; according to the remaining resource amounts of the remaining other users, the users with the highest resource amount at least greater than the resource borrowing amount are determined as multiple optimal resource providing users. Then, a borrowing request is sent to multiple optimal resource providing users, and the user with the first unified information received is used as the target optimal resource providing user, and then the user's resources are borrowed, so as to delete the object data corresponding to the target object under the target resource amount. After the deletion operation is completed, the resources are returned.

[0162] It can be seen that in the embodiment of the present invention, when deleting the object data corresponding to the target object, the resource status and needs of the current user are fully considered; when there are many objects to be deleted and the amount of resources is insufficient, resources are borrowed from other users to complete the deletion operation. Specifically, if the number is large and the remaining available resources are insufficient, the optimal resource provider is determined by comprehensively considering the user information and resource borrowing amount of multiple other users, and a borrowing request is sent; the optimal resource provider who agrees to the information the earliest is selected and the target resource amount is determined; after completing the deletion operation, the borrowed resources are returned in a timely manner to ensure the recycling of resources and the efficiency of the deletion operation.

[0163] An object data lifecycle management device provided by an embodiment of the present invention is introduced below. The device described below and the method described above can be referred to each other. The device of this embodiment is set in an electronic device. Figure 5 , Figure 5 : is a structural block diagram of a device according to one embodiment of the present invention, comprising:

[0164] An acquisition module 210 is used to acquire a first object upload time of an expired object of a target bucket, where the object upload time of the expired object is determined based on a current time and a preset deletion time period;

[0165] A first determination module 220 is used to determine multiple objects of time index according to the time index prefix from the index table of the target bucket, wherein the index table includes an enumeration index and a time index;

[0166] A second determination module 230, configured to determine, from the multiple objects of the time index, index information of a target object having the same upload time as the first object based on the respective corresponding second object upload times of the multiple objects;

[0167] The deleting module 240 is used to delete the object data corresponding to the target object according to the index information of the target object.

[0168] In one implementable manner, the deletion module 240 is used to: determine a first data fingerprint based on index information of the target object; obtain object data corresponding to the target object based on the index information of the target object; determine whether the first data fingerprint and the second data fingerprint of the object data are consistent; if so, delete the object data corresponding to the target object.

[0169] In one implementable manner, the acquisition module 210 is used to determine the log information of the target bucket based on multiple lifecycle log shards, where the target bucket is any storage bucket among multiple storage buckets, and the log information of the target bucket represents the periodic management rules of the object storage of the target bucket; based on the log information of the target bucket, the first object upload time of the expired object of the target bucket is obtained.

[0170] In one achievable manner, the present invention further includes:

[0171] An update module is used to: regularly obtain the heat level and priority of each object in the target bucket, wherein the heat level is determined based on at least one of the access duration, access mode, and access frequency of the object, and the priority indicates the importance of the object;

[0172] Determine the adjustment time based on the heat level and priority;

[0173] Determine the dynamic object upload time according to the adjustment time corresponding to the object and the object upload time;

[0174] Generate an adjustment request for the object that needs to be adjusted according to the dynamic object upload time, so that the user can confirm whether adjustment is needed;

[0175] After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

[0176] In an implementable manner, the deletion module is further used to: obtain an object deletion request of the object to be deleted;

[0177] According to the object deletion request, the head stripe data, enumeration index and time index, and tail stripe data corresponding to the object to be deleted are deleted in sequence.

[0178] In one achievable manner, the present invention further includes:

[0179] The storage module is used to: obtain an object upload request;

[0180] Parse the object upload request to obtain the object's bucket information, user identity, object name, upload time, object version, and object data;

[0181] Assemble metadata based on bucket information, user identity, object name, upload time, and object version;

[0182] Divide the object data into a plurality of strips of data; and write a plurality of first strips of data from the plurality of strips of data into the tail strip of data; the plurality of strips of data includes a plurality of first strips of data and a second strip of data;

[0183] Based on the metadata, record the enumeration index and time index corresponding to the object upload request on the bucket shard;

[0184] The head stripe data is written according to the metadata and the second stripe data.

[0185] In an implementable manner, the deletion module 240 is further configured to:

[0186] Determine the number of target objects to be deleted for the current user;

[0187] If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object;

[0188] If the number is greater than the preset number, determining whether the remaining available resources corresponding to the current user are less than the preset resources;

[0189] If it is not less than the preset resource amount, the object data corresponding to the target object is deleted according to the index information of the target object;

[0190] If the resource amount is less than the preset amount, multiple optimal resource providing users are determined based on the user information of multiple other users and the resource borrowing amount. The user information includes the remaining amount of resources and the importance of storage objects. The resource borrowing amount is determined based on the quantity and the remaining available resources.

[0191] Sending borrowing requests to multiple optimal resource providing users;

[0192] After receiving the consent information of the target optimal resource provider, in the case of the target resource amount, the object data corresponding to the target object is deleted according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource amount is determined by the resource borrowing amount and the remaining available resource amount;

[0193] After completing the deletion operation of the object data corresponding to the target object, the borrowed target optimal resource is returned to provide the user with resources.

[0194] An embodiment of the present invention provides an electronic device, such as Figure 6 As shown, Figure 6 The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present invention.

[0195] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0196] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0197] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0198] The memory 303 is used to store application code for executing the solution of the present invention, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the above method embodiment.

[0199] Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0200] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0201] An embodiment of the present invention provides a computer program product, including a computer program, which implements the corresponding contents of the aforementioned method embodiment when the computer program is executed by a processor.

[0202] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0203] The above are only some embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for managing object data lifecycle, characterized in that: Executed by an electronic device, the object data lifecycle management method includes: Obtain a first object upload time of an expired object of a target bucket, where the object upload time of the expired object is determined based on a current time and a preset deletion time period; Determine, from the index table of the target bucket, a plurality of objects of the time index according to the time index prefix, wherein the index table includes an enumeration index and a time index; Based on the second object upload time corresponding to each of the multiple objects, determine the index information of the target object having the same upload time as the first object from the multiple objects of the time index; According to the index information of the target object, the object data corresponding to the target object is deleted.

2. The object data lifecycle management method according to claim 1, characterized in that: The deleting corresponding object data according to the index information of the target object includes: Determining a first data fingerprint according to the index information of the target object; Acquire object data corresponding to the target object according to the index information of the target object; Determining whether the first data fingerprint is consistent with the second data fingerprint of the object data; If so, the object data corresponding to the target object is deleted.

3. The object data lifecycle management method according to claim 1, characterized in that: Get the first object upload time of the expired object in the target bucket, including: Determine log information of a target bucket according to the multiple lifecycle log shards, where the target bucket is any storage bucket among the multiple storage buckets, and the log information of the target bucket represents a periodic management rule of object storage of the target bucket; According to the log information of the target bucket, the first object upload time of the expired object of the target bucket is obtained.

4. The object data lifecycle management method according to claim 3, characterized in that: Also includes: Periodically obtain the heat level and priority of each object in the target bucket, wherein the heat level is determined based on at least one of the access duration, access mode, and access frequency of the object, and the priority indicates the importance of the object; Determining an adjustment time according to the heat level and priority; Determine the dynamic object upload time according to the adjustment time corresponding to the object and the object upload time; Generate an adjustment request for the object that needs to be adjusted according to the dynamic object upload time, so that the user can confirm whether adjustment is needed; After receiving the user's confirmation information, the index table of the target bucket is updated according to the dynamic object upload time.

5. The object data lifecycle management method according to claim 1, characterized in that: Also includes: Get the object deletion request of the object to be deleted; According to the object deletion request, the head stripe data, the enumeration index and the time index, and the tail stripe data corresponding to the object to be deleted are deleted in sequence.

6. The object data lifecycle management method according to claim 1, characterized in that: Also includes: Get object upload request; Parsing the object upload request to obtain the object's storage bucket information, user identity, object name, upload time, object version, and object data; Assemble metadata based on bucket information, user identity, object name, upload time, and object version; Dividing the object data into a plurality of strips of data; and writing a plurality of first stripe data from the plurality of stripe data into the tail stripe data; the plurality of stripe data includes a plurality of first stripe data and a second stripe data; According to the metadata, record the enumeration index and time index corresponding to the object upload request on the bucket shard; The head stripe data is written according to the metadata and the second stripe data.

7. The object data lifecycle management method according to any one of claims 1 to 6, characterized in that: The deleting object data corresponding to the target object according to the index information of the target object includes: Determine the number of target objects to be deleted for the current user; If the number is not greater than the preset number, deleting the object data corresponding to the target object according to the index information of the target object; If the number is greater than the preset number, determining whether the remaining available resource amount corresponding to the current user is less than the preset resource amount; If it is not less than the preset resource amount, deleting the object data corresponding to the target object according to the index information of the target object; If it is less than the preset resource amount, multiple optimal resource providing users are determined according to user information of multiple other users and resource borrowing amount, wherein the user information includes the remaining resource amount and the importance of the storage object, and the resource borrowing amount is determined based on the amount and the remaining available resource amount; Sending a borrowing request to the plurality of optimal resource providing users; After receiving the consent information of the target optimal resource provider, in the case of the target resource amount, deleting the object data corresponding to the target object according to the index information of the target object; the target optimal resource provider is the optimal resource provider whose consent information is received earliest; the target resource amount is determined by the resource borrowing amount and the remaining available resource amount; After completing the deletion operation of the object data corresponding to the target object, the borrowed resources of the target optimal resource provider are returned.

8. An object data lifecycle management device, characterized in that: include: An acquisition module, configured to acquire a first object upload time of an expired object of a target bucket, wherein the object upload time of the expired object is determined based on a current time and a preset deletion time period; A first determination module is used to determine multiple objects of time index according to the time index prefix from the index table of the target bucket, wherein the index table includes enumeration index and time index; A second determining module, configured to determine, from the multiple objects of the time index, index information of a target object having the same upload time as the first object based on the respective corresponding second object upload times of the multiple objects; The deleting module is used to delete the object data corresponding to the target object according to the index information of the target object.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to: execute the steps of the object data lifecycle management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the steps of the object data lifecycle management method according to any one of claims 1 to 7.

Images