A data deletion method, apparatus, device, and computer-readable storage medium

Abstract

The present application discloses a data deletion method, apparatus, device, and computer-readable storage medium, which relate to the field of expiration deletion. The method includes: when an object in an object storage bucket is moved to the recycle bin, an index shard of the object storage bucket is established; the object information and deletion time of the object are saved to the index shard through an omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time; the object to be deleted is determined according to the deletion time, and the object to be deleted and the corresponding omap key-value pair are deleted. When an object is deleted, an index shard is established for the object storage bucket, and the object information and deletion time of the object are saved in the index shard in the form of an omap key-value pair. The key includes the deletion time and object name of the object, and the keys are sorted according to the deletion time. The object to be deleted is determined according to the deletion time of the object, without having to traverse all the objects in the recycle bin, reducing the traversal operation.

Description

Technical Field

[0001] This application relates to the field of expiration deletion, and in particular, to a data deletion method, apparatus, device, and computer-readable storage medium. Background Art

[0002] Distributed object storage is suitable for storing unstructured data such as pictures and videos, and has the characteristics of high-speed direct access to disks by SAN (Storage Area Network and SAN Protocols) and distributed sharing by NAS (Network Attached Storage), providing a storage architecture with high performance, high reliability, cross-platform, and secure data sharing. With the increasing size of Internet unstructured data, distributed object storage has been widely developed and applied.

[0003] The document materials temporarily deleted by users will be stored in the recycle bin, and the document materials in the recycle bin can be restored. Currently, when implementing the recycle bin function in object storage, it is mainly achieved through multi-versions or creating hidden directories within the bucket. For the multi-version method, the objects in the recycle bin are historical version objects, and the expiration deletion of the recycle bin objects is achieved by setting a lifecycle policy for the expiration time of the historical versions of the bucket; for the method of creating hidden directories within the bucket, an expiration deletion lifecycle policy with the recycle bin directory as the prefix is set. That is, currently, users can only delete objects through the object lifecycle. By scanning all objects in the bucket recycle bin through the lifecycle thread, checking the time when the object was added to the recycle bin and comparing it with the current time to detect whether the recycle bin object has expired, and if it has expired, deleting the object. Since all objects in the bucket recycle bin need to be traversed, it will cause a large number of query operations on the index pool, seriously affecting the performance of the cluster metadata pool.

[0004] Therefore, how to solve the above technical problems should be the focus of attention of those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data deletion method, apparatus, device, and computer-readable storage medium to reduce query operations and improve system stability.

[0006] To solve the above technical problems, this application provides a data deletion method, including:

[0007] When an object in an object storage bucket is moved to the recycle bin, an index shard of the object storage bucket is established;

[0008] Save the object information and deletion time of the object to the index shard through the omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time;

[0009] Determine the object to be deleted according to the deletion time, and delete the object to be deleted and the corresponding omap key-value pair.

[0010] Optionally, the determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes:

[0011] Obtain the retention time of the object in the recycle bin and the current time;

[0012] Determine the expiration time node of the object in the recycle bin according to the current time and the retention time;

[0013] Use a thread to traverse the keys on the index shard that are before the expiration time node;

[0014] Delete the object to be deleted and the omap key-value pair corresponding to the traversed key.

[0015] Optionally, the scanning period of the thread is at the minute level.

[0016] Optionally, before deleting the object to be deleted and the omap key-value pair corresponding to the traversed key, further include:

[0017] Judge whether the verification information in the index shard is consistent with the target verification information in the header object, where the verification information is located in the object metadata of the value in the omap key-value pair, the target verification information is located in the metadata of the header object, and the verification information corresponds to the traversed key;

[0018] If the verification information is consistent with the target verification information, execute the step of deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.

[0019] Optionally, the obtaining the retention time of the object in the recycle bin includes:

[0020] Obtain the bucket metadata;

[0021] Determine the retention time according to the bucket metadata.

[0022] Optionally, the determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes:

[0023] Obtain the object name and bucket name of the object to be deleted;

[0024] Determine the deletion time of the object to be deleted in the header object according to the object name and the bucket name;

[0025] Determine the corresponding target deletion time in the index shard according to the deletion time in the header object;

[0026] Delete the object to be deleted and the omap key-value pair corresponding to the target deletion time.

[0027] Optionally, the determining the deletion time of the object to be deleted in the header object according to the object name and the bucket name includes:

[0028] Determine the header object according to the object name and the bucket name;

[0029] Determine the metadata of the object according to the header object;

[0030] Determine the deletion time in the header object according to the metadata.

[0031] Optionally, when multiple recycle bins delete objects simultaneously, the establishing the index shard of the object storage bucket includes:

[0032] Establish the index shard corresponding to each object storage bucket.

[0033] This application also provides a data deletion device, including:

[0034] A establishing module, configured to establish an index shard of the object storage bucket when an object in the object storage bucket moves to the recycle bin;

[0035] A saving module, configured to save the object information and deletion time of the object to the index shard through an omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time;

[0036] A deleting module, configured to determine the object to be deleted according to the deletion time, and delete the object to be deleted and the corresponding omap key-value pair.

[0037] This application also provides a data deletion device, including:

[0038] A memory, configured to store a computer program;

[0039] A processor, configured to implement the steps of any one of the above data deletion methods when executing the computer program.

[0040] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned data deletion methods are implemented.

[0041] A data deletion method provided by the present application includes: when an object in an object storage bucket is moved to a recycle bin, establishing an index shard of the object storage bucket; saving the object information and deletion time of the object to the index shard through an omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time; determining the object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key-value pair.

[0042] It can be seen that when an object in the object storage bucket of the data deletion method in the present application is moved to the recycle bin, an index shard is established for the object storage bucket, and the object information and deletion time of the object are saved in the index shard in the form of an omap key-value pair. The key in the omap key-value pair includes the deletion time and the object name of the object, and the keys are sorted according to the deletion time. Then, the object to be deleted is determined according to the deletion time of the object, and the object to be deleted and the corresponding omap key-value pair are deleted together, without traversing all the objects in the recycle bin, reducing a large number of traversal operations on the metadata pool, reducing the pressure on the metadata pool, thereby reducing cluster I / O and improving system stability.

[0043] In addition, the present application also provides a device, a device and a computer-readable storage medium having the above advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a flowchart of a data deletion method provided by an embodiment of the present application;

[0046] Figure 2 It is a flowchart of an automatic deletion operation provided by an embodiment of the present application;

[0047] Figure 3 It is another flowchart of an automatic deletion operation provided by an embodiment of the present application;

[0048] Figure 4A flowchart for performing a manual deletion operation provided by an embodiment of the present application;

[0049] Figure 5 A structural block diagram of a data deletion device provided by an embodiment of the present application;

[0050] Figure 6 A structural block diagram of a data deletion device provided by an embodiment of the present application. Detailed implementation manners

[0051] To enable those skilled in the art to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0052] As described in the background art section, currently, users can only delete objects through the object life cycle. By scanning all objects in the bucket recycle bin through the life cycle thread, comparing the time when the object was added to the recycle bin with the current time, and detecting whether the recycle bin object has expired. If it has expired, the object will be deleted. Since all objects in the bucket recycle bin need to be traversed, a large number of query operations on the index pool will be caused, seriously affecting the performance of the cluster metadata pool.

[0053] In view of this, the present application provides a data deletion method. Please refer to Figure 1 which includes:

[0054] Step S101: When an object in the object storage bucket is moved to the recycle bin, an index shard of the object storage bucket is established.

[0055] Object-based Storage (OBS) is a computer data storage architecture that manages data as objects, as opposed to other storage architectures (such as file systems that manage data at the file level) and block storage that manages data as blocks within blocks and sectors. Each object typically includes the data itself, a variable number of metadata, and a globally unique identifier.

[0056] An object storage bucket is a container for storing objects in OBS. Object storage provides a flat storage method based on buckets and objects. All objects in the bucket are at the same logical level, removing the multi-level tree directory structure in the file system. Each bucket has its own storage class, access permission, region of belonging, etc. attributes. Users can create buckets with different storage classes and access permissions in different regions and configure more advanced attributes to meet the storage requirements of different scenarios.

[0057] When an object in an object bucket is moved to the recycle bin, that is, the object is deleted and placed in the recycle bin. An object bucket corresponds to an index shard, and the index shard is used to record the objects in the object bucket and the related information of the objects. The index shard is established in the metadata pool.

[0058] Step S102: Save the object information and deletion time of the object to the index shard through the omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time.

[0059] The object information includes the object name and object metadata, and the object name corresponds to the object one by one. The omap key-value pair includes a key (keyword) and a value (the value of the keyword). The key includes the deletion time and the object name, and the value includes the object metadata. The time is used as a prefix in the key, and the keys are sorted according to the deletion time, including sorting in ascending order of the deletion time or sorting in descending order of the deletion time, which can be set by oneself specifically and is not specifically limited in this application. Among them, the deletion time is the time when the object is moved to the recycle bin.

[0060] Step S103: Determine the object to be deleted according to the deletion time, and delete the object to be deleted and the corresponding omap key-value pair.

[0061] The object is saved in the data pool, and the index shard records the relevant information of the objects in the data pool. When deleting, the objects in the data pool and the corresponding information on the index shard are deleted together, so as to ensure that the data pool and the index shard are consistent. The specific deletion operations include manual deletion and automatic deletion, which are introduced separately below.

[0062] Automatic deletion operation of objects and omap key-value pairs:

[0063] Please refer to Figure 2 , the determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes:

[0064] Step S201: Obtain the retention time of the object in the recycle bin and the current time.

[0065] The retention time is the time that the object in the recycle bin can be saved in the recycle bin. When the retention time is exceeded, the object expires. If the retention time is not exceeded, the object can continue to be saved in the recycle bin. The current time is the current time node.

[0066] The obtaining the retention time of the object in the recycle bin includes:

[0067] Obtain the bucket metadata;

[0068] Determine the retention time according to the bucket metadata.

[0069] The retention time of the object is included in the bucket metadata. Therefore, after obtaining the bucket metadata, the retention time of the object can be obtained.

[0070] Step S202: Determine the expiration time node of the object in the recycling station according to the current time and the retention time.

[0071] Subtract the retention time from the current time to obtain the expiration time node of the object in the recycling station. Objects moved to the recycling station before the expiration time node have expired and need to be deleted; objects moved to the recycling station after the expiration time node are still stored in the recycling station and do not need to be deleted. For example, assume that the retention time of the object in the recycling station is seven days and the current time is 8:00 am on May 29th. Then the expiration time node is 8:00 am on May 22nd. That is, objects moved to the recycling station before 8:00 am on May 22nd have expired and need to be deleted, and objects moved to the recycling station after 8:00 am on May 22nd continue to be retained.

[0072] Step S203: Use a thread to traverse the keys on the index shard that are before the expiration time node.

[0073] The keys are arranged in chronological order. Therefore, when the thread traverses, it does not need to traverse all the keys. It only needs to traverse the keys whose deletion time is before the expiration time node. The end mark of the traversal is the expiration time node, thus reducing a large number of unnecessary traversal operations.

[0074] In the prior art, the life cycle of the object in the recycling station is executed in days, so the expiration time can only be in days. To achieve more refined cycle management of the object in the recycling station, in an embodiment of the present application, the scanning cycle of the thread is at the minute level, which can achieve life cycle management of the recycling station object at the hour level and meet the more fine-grained needs of users. For example, the scanning cycle of the thread can be 10 minutes, or 20 minutes, etc., which can be set by oneself. Correspondingly, the retention time of the object in the recycling station can be several hours. For example, assume that the retention time of the object in the recycling station is 5 hours and the current time is 8:00 am on May 29th. Then the expiration time node is 3:00 am on May 29th. That is, objects moved to the recycling station before 3:00 am on May 29th have expired and need to be deleted, and objects moved to the recycling station after 3:00 am on May 29th continue to be retained.

[0075] Step S204: Delete the object to be deleted and the omap key-value pair corresponding to the traversed key.

[0076] The key includes the deletion time and the object name. The object to be deleted can be determined through the object name in the traversed key, and then the object to be deleted is deleted from the data pool, and the omap key-value pair where the traversed key is located on the index shard is deleted. The information of the object recorded on the index shard is consistent with the object in the data pool.

[0077] When automatically deleting objects and omap key-value pairs, please refer to Figure 3 , in an embodiment of the present application, the determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes:

[0078] Step S301: Obtain the retention time of the object in the recycle bin and the current time.

[0079] Step S302: Determine the expiration time node of the object in the recycle bin according to the current time and the retention time.

[0080] Step S303: Use a thread to traverse the keys on the index shard that are before the expiration time node.

[0081] Step S304: Determine whether the verification information in the index shard is consistent with the target verification information in the header object. The verification information is in the object metadata of the value in the omap key-value pair, and the target verification information is in the metadata of the header object. The verification information corresponds to the traversed key.

[0082] The header object records the metadata of the object. The verification information of the object in the metadata is called the target verification information. The omap key-value pair includes a key and a value. The value includes the verification information of the object. That is, each key corresponds to a verification information, and each object name corresponds to a verification information.

[0083] It should be noted that the specific category of the verification information in the present application is not limited and can be selected by itself. For example, the verification information can be a tag, or an MD5 fingerprint information, etc.

[0084] Step S305: If the verification information is consistent with the target verification information, execute the step of deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.

[0085] Step S306: Delete the object to be deleted and the omap key-value pair corresponding to the traversed key.

[0086] Step S307: When the verification information is inconsistent with the target verification information, stop the deletion operation.

[0087] It should be noted that for steps S301, S302, S303, and S306, please refer to the above steps S201, S202, S203, and S204, and no detailed description will be given here.

[0088] In this embodiment, before deleting the object to be deleted and the omap key-value pair, it is verified whether the verification information in the index shard is consistent with the target verification information in the head object. Only when the two are consistent will the deletion operation be performed, which can prevent the objects in the recycle bin from being accidentally deleted.

[0089] Manual deletion operation of object and omap key-value pair:

[0090] Please refer to Figure 4 , where determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes:

[0091] Step S401: Obtain the object name and bucket name of the object to be deleted.

[0092] The bucket name is also the ID of the bucket, and the bucket name is unique. The object to be deleted stored in the data pool can be determined through the object name and the bucket name.

[0093] Step S402: Determine the deletion time of the object to be deleted in the head object according to the object name and the bucket name.

[0094] The head object includes the object name, bucket name, and object metadata, and the object metadata is recorded in the xattr of the head object.

[0095] As an implementable manner, determining the deletion time of the object to be deleted in the head object according to the object name and the bucket name includes:

[0096] Step S4021: Determine the head object according to the object name and the bucket name;

[0097] Step S4022: Determine the metadata of the object according to the head object;

[0098] Step S4023: Determine the deletion time in the head object according to the metadata.

[0099] The object metadata in the head object and the object metadata in the value are not exactly the same. The object metadata in the head object includes the deletion time of the object, some custom metadata, object attribute information, manifest (recording the tail object of the object), access control information, etc.; the object metadata in the value includes the size of the object, etc.

[0100] Step S403: Determine the corresponding target deletion time in the index shard according to the deletion time located in the head object.

[0101] The deletion time of the object is included in both the object metadata in the head object and the key of the omap key-value pair. Therefore, the corresponding target deletion time in the omap key-value pair can be found by obtaining the deletion time in the object metadata of the head object.

[0102] Step S404: Delete the object to be deleted and the omap key-value pair corresponding to the target deletion time.

[0103] Find the corresponding target deletion time in the omap key-value pair, that is, find the corresponding key, so as to obtain the object name corresponding to the object to be deleted. Delete the object to be deleted according to the object name, and delete the corresponding omap key-value pair according to the target deletion time, so that the information of the object recorded on the index shard is consistent with the object in the data pool.

[0104] In the data deletion method of the present application, when an object in the object storage bucket is moved to the recycle bin, an index shard is established for the object storage bucket, and the object information and deletion time of the object are saved in the index shard in the form of omap key-value pairs. The key in the omap key-value pair includes the deletion time and object name of the object, and the keys are sorted according to the deletion time. Then, the object to be deleted is determined according to the deletion time of the object, and the object to be deleted and the corresponding omap key-value pair are deleted together, without traversing all the objects in the recycle bin, reducing a large number of traversal operations on the metadata pool, reducing the pressure on the metadata pool, thereby reducing cluster IO and improving system stability.

[0105] Based on any of the above embodiments, in an embodiment of the present application, when multiple recycle bins delete objects simultaneously, the establishment of the index shard of the object storage bucket includes:

[0106] Establish the index shard corresponding to each object storage bucket.

[0107] The object storage bucket corresponds to the index shard one by one. The deletion method of the objects in each recycle bin is the same. Specifically, reference can be made to the above description. The deletion thread in this embodiment can support the operation of multiple buckets simultaneously, support higher concurrency and shorter execution cycles, and support hourly expiration deletion management.

[0108] The following describes the data deletion method in the present application with a specific case.

[0109] Step 1: When deleting an object in the object storage bucket, move the object in the object storage bucket to the recycle bin, and create a new index shard corresponding to the object storage bucket in the metadata pool.

[0110] Step 2: Map the object information (object name and object metadata) to the index shard. The object information is stored on the index shard using omap key-value pairs. The omap key is the deletion time + object name, and the value is the object metadata information, which is used to traverse the recycle bin objects and verify the object usage during deletion. The purpose of sorting the index records by time is achieved by using time as the prefix for the key.

[0111] Step 3: Manual deletion: According to the object name and bucket name, obtain the head object information, view the time when the object was moved to the recycle bin, obtain the key on the recycle bin index shard based on this time, and delete the object and the index information simultaneously;

[0112] Automatic deletion: Obtain the bucket metadata information, view the retention time of the objects in the recycle bin, and calculate when the objects added to the recycle bin have expired. The recycle bin object deletion thread uses the calculated time as the end marker during traversal, scans the objects on the recycle bin index shard of the object storage bucket, obtains the expired recycle bin objects, and compares whether the verification information in the index is consistent with the object metadata information. If they are consistent, delete the object and delete the index record simultaneously. If they are inconsistent, do not delete.

[0113] Next, the data deletion device provided in the embodiments of the present application will be introduced. The data deletion device described below can be correspondingly referred to the data deletion method described above.

[0114] Figure 5 is the structural block diagram of the data deletion device provided in the embodiments of the present application. Refer to Figure 5 The data deletion device may include:

[0115] A creation module 100, configured to create an index shard of the object storage bucket when an object in the object storage bucket is moved to the recycle bin;

[0116] A saving module 200, configured to save the object information and deletion time of the object to the index shard through omap key-value pairs, where the key in the omap key-value pairs includes the deletion time and the object name, and the key is sorted according to the deletion time;

[0117] A deletion module 300, configured to determine the object to be deleted according to the deletion time, and delete the object to be deleted and the corresponding omap key-value pair.

[0118] The data deletion device in this embodiment is used to implement the aforementioned data deletion method. Therefore, the specific implementation in the data deletion device can be seen in the embodiment part of the data deletion method in the previous text. For example, the establishment module 100, the storage module 200, and the deletion module 300 are respectively used to implement steps S101, S102, and S103 in the above data deletion method. Therefore, its specific implementation can refer to the descriptions of the corresponding individual embodiments and will not be elaborated here.

[0119] When an object in the object storage bucket is moved to the recycle bin in the data deletion device of this application, an index shard is established for the object storage bucket, and the object information and deletion time of the object are saved in the index shard in the form of omap key-value pairs. The key in the omap key-value pair includes the deletion time and object name of the object, and the keys are sorted according to the deletion time. Then, the object to be deleted is determined according to the deletion time of the object, and the object to be deleted and the corresponding omap key-value pair are deleted together, without traversing all the objects in the recycle bin, reducing a large number of traversal operations on the metadata pool, reducing the pressure on the metadata pool, thereby reducing cluster IO and improving system stability.

[0120] Optionally, when performing the automatic deletion operation, the deletion module 300 includes:

[0121] The first acquisition unit is used to acquire the retention time of the object in the recycle bin and the current time;

[0122] The first determination unit is used to determine the expiration time node of the object in the recycle bin according to the current time and the retention time;

[0123] The traversal unit is used to traverse the keys on the index shard that are before the expiration time node by using a thread;

[0124] The first deletion unit is used to delete the object to be deleted and the omap key-value pair corresponding to the traversed key.

[0125] Optionally, when the traversal unit traverses the keys on the index shard that are before the expiration time node by using a thread, the scanning period of the thread is at the minute level.

[0126] In the prior art, the life cycle of objects in the recycle bin is executed on a daily basis, so the expiration time can only be in days. In this embodiment, the scanning cycle of the process is at the minute level, and the life cycle management of recycle bin objects at the hour level can be achieved, meeting the user's more fine-grained requirements. For example, the scanning cycle of the thread can be 10 minutes, or 20 minutes, etc., which can be set by the user. Correspondingly, the retention time of objects in the recycle bin can be several hours. For example, assuming that the retention time of an object in the recycle bin is 5 hours and the current time is 8:00 am on May 29th, the expiration time node is 3:00 am on May 29th. That is, objects moved to the recycle bin before 3:00 am on May 29th have expired and need to be deleted, while objects moved to the recycle bin after 3:00 am on May 29th continue to be retained.

[0127] Optionally, in an embodiment of the present application, the deletion module 300 further includes:

[0128] A judgment unit, configured to judge whether the verification information in the index slice is consistent with the target verification information in the header object. The verification information is located in the object metadata of the value in the omap key-value pair, the target verification information is located in the metadata of the header object, and the verification information corresponds to the traversed key;

[0129] An execution unit, configured to, if the verification information is consistent with the target verification information, execute the step of deleting the object to be deleted corresponding to the traversed key and the omap key-value pair;

[0130] A stop unit, configured to stop the deletion operation when the verification information is inconsistent with the target verification information.

[0131] In this embodiment, before deleting the object to be deleted and the omap key-value pair, it is verified whether the verification information in the index slice is consistent with the target verification information in the header object. Only when the two are consistent, the deletion operation is performed, which can avoid the objects in the recycle bin from being accidentally deleted.

[0132] Optionally, the first acquisition unit includes:

[0133] An acquisition subunit, configured to acquire bucket metadata;

[0134] A first determination subunit, configured to determine the retention time according to the bucket metadata.

[0135] Optionally, when performing a manual deletion operation, the deletion module 300 includes:

[0136] A second acquisition unit, configured to acquire the object name and bucket name of the object to be deleted;

[0137] A second determination unit, configured to determine the deletion time of the object to be deleted in the header object according to the object name and the bucket name;

[0138] A third determination unit, configured to determine a target deletion time corresponding to the index shard according to the deletion time in the header object;

[0139] A second deletion unit, configured to delete the object to be deleted and the omap key-value pair corresponding to the target deletion time.

[0140] Optionally, the second determination unit includes:

[0141] A second determination subunit, configured to determine the header object according to the object name and the bucket name;

[0142] A third determination subunit, configured to determine the metadata of the object according to the header object;

[0143] A fourth determination subunit, configured to determine the deletion time in the header object according to the metadata.

[0144] Optionally, when multiple recycle bins delete objects simultaneously, the establishment module 100 is specifically configured to establish the index shard corresponding to each object storage bucket.

[0145] The object storage bucket and the index shard are in one-to-one correspondence. The deletion method of the objects in each recycle bin is the same. For specific reference, please refer to the above description. The deletion thread in this embodiment can support operating on multiple buckets simultaneously, support higher concurrency and shorter execution cycles, and support expiration deletion management at the hourly level.

[0146] Next, the data deletion device provided in the embodiment of the present application will be introduced. The data deletion device described below can be correspondingly referred to the data deletion method described above.

[0147] Figure 6 It is a structural block diagram of a data deletion device provided in an embodiment of the present application. A data deletion device includes:

[0148] A memory 11, configured to store a computer program;

[0149] A processor 12, configured to implement the steps of the data deletion method described in any of the above embodiments when executing the computer program.

[0150] When the data deletion device in this application moves an object in an object storage bucket to the recycle bin, it creates an index shard for the object storage bucket, and saves the object information and deletion time of the object in the index shard in the form of omap key-value pairs. The key in the omap key-value pair includes the deletion time and object name of the object, and the keys are sorted according to the deletion time. Then, it determines the objects to be deleted based on the deletion time of the objects, and deletes the objects to be deleted together with the corresponding omap key-value pairs, without having to traverse all the objects in the recycle bin, reducing a large number of traversal operations on the metadata pool, reducing the pressure on the metadata pool, thereby reducing cluster I / O and improving system stability.

[0151] The computer-readable storage medium provided by the embodiments of this application will be introduced below. The computer-readable storage medium described below can be mutually referred to in correspondence with the data deletion method described above.

[0152] A computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the data deletion method described in any one of the above embodiments.

[0153] When the object in the object storage bucket of the computer-readable storage medium in this application moves to the recycle bin, it creates an index shard for the object storage bucket, and saves the object information and deletion time of the object in the index shard in the form of omap key-value pairs. The key in the omap key-value pair includes the deletion time and object name of the object, and the keys are sorted according to the deletion time. Then, it determines the objects to be deleted based on the deletion time of the objects, and deletes the objects to be deleted together with the corresponding omap key-value pairs, without having to traverse all the objects in the recycle bin, reducing a large number of traversal operations on the metadata pool, reducing the pressure on the metadata pool, thereby reducing cluster I / O and improving system stability.

[0154] The various embodiments in this specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0155] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0156] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0157] The data deletion method, device, equipment, and computer-readable storage medium provided in this application have been introduced in detail above. Specific examples are used herein to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data deletion method, characterized in that, including: When an object in an object bucket is moved to the recycle bin, an index shard of the object bucket is established; Save the object information and deletion time of the object to the index shard through an omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time; Determine the object to be deleted according to the deletion time, and delete the object to be deleted and the corresponding omap key-value pair.

2. The data deletion method according to claim 1, wherein The determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes: Obtain the retention time of the object in the recycle bin and the current time; Determine the expiration time node of the object in the recycle bin according to the current time and the retention time; Use a thread to traverse the keys on the index shard that are before the expiration time node; Delete the object to be deleted and the omap key-value pair corresponding to the traversed key.

3. The data deletion method according to claim 2, wherein The scanning period of the thread is at the minute level.

4. The data deletion method according to claim 2, wherein Before deleting the object to be deleted and the omap key-value pair corresponding to the traversed key, it further includes: Judge whether the verification information in the index shard is consistent with the target verification information in the header object, where the verification information is located in the object metadata of the value in the omap key-value pair, the target verification information is located in the metadata of the header object, and the verification information corresponds to the traversed key; If the verification information is consistent with the target verification information, execute the step of deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.

5. The data deletion method according to claim 2, wherein The obtaining the retention time of the object in the recycle bin includes: Obtain bucket metadata; Determine the retention time according to the bucket metadata.

6. The data deletion method according to claim 1, characterized in that The determining the object to be deleted according to the deletion time and deleting the object to be deleted and the corresponding omap key-value pair includes: Obtain the object name and bucket name of the object to be deleted; Determine the deletion time of the object to be deleted in the header object according to the object name and the bucket name; Determine the corresponding target deletion time in the index shard according to the deletion time in the header object; Delete the object to be deleted and the omap key-value pair corresponding to the target deletion time.

7. The data deletion method according to claim 6, wherein The determining the deletion time of the object to be deleted in the header object according to the object name and the bucket name includes: Determine the header object according to the object name and the bucket name; Determine the metadata of the object according to the header object; Determine the deletion time in the header object according to the metadata.

8. The data deletion method according to any one of claims 1 to 7, characterized in that, When multiple recycle bins delete objects simultaneously, the establishing the index shard of the object bucket includes: Establish the index shard corresponding to each object bucket.

9. A data deletion device, characterized in that, including: An establishing module, configured to establish an index shard of the object bucket when an object in the object bucket is moved to the recycle bin; A saving module, configured to save the object information and deletion time of the object to the index shard through an omap key-value pair, where the key in the omap key-value pair includes the deletion time and the object name, and the keys are sorted according to the deletion time; A deletion module, configured to determine an object to be deleted according to the deletion time, and delete the object to be deleted and the corresponding omap key-value pair.

10. A data deletion device, characterized in that, Comprising: A memory, configured to store a computer program; A processor, configured to implement the steps of the data deletion method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data deletion method according to any one of claims 1 to 8 are implemented.

Images