Barrel deleting method and device based on object storage gateway

Abstract

The invention provides a bucket deletion method and device based on an object storage gateway, which can improve the bucket deletion efficiency. The method for deleting the bucket based on the object storage gateway comprises the following steps: in response to a request for deleting the bucket, counting the number of objects cached in the bucket to obtain the total number of the objects in the bucket; when the total number of the objects in the bucket is smaller than a preset number, deleting entry metadata of the bucket, and removing the bucket from a bucket list of the user; after the buckets are removed from the bucket list of the user, identifying to-be-cleared data of the buckets; wherein the to-be-cleared data comprises instance metadata of the buckets and index metadata of the buckets; and asynchronously deleting the identified instance metadata of the buckets and the index metadata of the buckets to complete deletion of the buckets of the object storage gateway.

Description

Technical Field

[0001] The present invention relates to the technical field of object storage, and in particular to a method and device for deleting a bucket based on an object storage gateway. Background Art

[0002] With the development of the Internet, the amount of data on the Internet is increasing, and distributed object storage has come into being. As a new network storage architecture, distributed object storage is suitable for the storage of unstructured data such as pictures and videos, and has the characteristics of high-speed direct access to disks for block storage and distributed sharing for file storage. It has been widely developed and applied on the Internet. At present, distributed object storage systems provide a wealth of bucket and object management functions, especially bucket index sharding, which allows the number of objects stored in a single bucket to reach billions or even tens of billions. However, deleting such a bucket with a huge number of index shards takes a lot of time, causing the client to time out and report an error. Then the object storage gateway continues to process the deletion work, which may cause the client's response to be inconsistent with the actual action of the object storage gateway, resulting in unnecessary operation and maintenance work. Summary of the invention

[0003] In order to solve the above technical problems, the present invention is proposed. The embodiments of the present invention provide a method and device for deleting a bucket based on an object storage gateway, which can improve the efficiency of bucket deletion.

[0004] According to one aspect of the present invention, a method for deleting a bucket based on an object storage gateway is provided, comprising: in response to a request to delete the bucket, counting the number of objects cached in the bucket to obtain the total number of objects in the bucket; when the total number of objects in the bucket is less than a preset number, deleting the entry metadata of the bucket and removing the bucket from the user's bucket list; after removing the bucket from the user's bucket list, identifying data to be cleared from the bucket; wherein the data to be cleared includes instance metadata of the bucket and index metadata of the bucket; and asynchronously deleting the identified instance metadata of the bucket and index metadata of the bucket to complete the deletion of the bucket of the object storage gateway.

[0005] In one embodiment, in response to a request to delete a bucket, the number of objects cached in the bucket is counted to obtain the total number of objects in the bucket, including: when a user uploads an object, the total number of objects cached in the bucket is increased by 1; when a user or a life cycle deletes an object, the total number of objects cached in the bucket is reduced by 1; wherein, when the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list, including: when the total number of objects in the bucket is less than 0, it is determined that the bucket has been emptied; when it is determined that the bucket has been emptied, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list.

[0006] In one embodiment, when the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list, including: when the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list through a bucket unlink operation; after the bucket is removed from the user's bucket list, the bucket deletion method based on the object storage gateway also includes: returning information that the bucket is successfully removed from the user's bucket list to the client to prompt the user that the bucket is successfully removed.

[0007] In one embodiment, the object storage gateway-based bucket deletion method also includes: caching the number of objects in the bucket using a distributed cache; based on the distributed cache, when the number of objects in the target object storage gateway increases or decreases, sending an update request to other object storage gateways among the multiple object storage gateways except the target object storage gateway; wherein the target object storage gateway is any one of the multiple object storage gateways; based on the update request, the other object storage gateways except the target object storage gateway increase or decrease their own number of objects to make the total number of objects in the buckets cached in the multiple object storage gateways consistent.

[0008] In one embodiment, after a bucket is removed from a user's bucket list, data to be cleared in the bucket is identified, including: after the bucket is removed from the user's bucket list, instance metadata of all buckets in the current cluster is obtained; based on at least one of the instance metadata, a name of the bucket and an ID of the bucket are obtained; wherein the name of the bucket and the ID of the bucket correspond to the instance metadata; based on the name of the bucket and the ID of the bucket, bucket information of the bucket is obtained; when the bucket has been removed from the user's bucket list, the bucket information of the bucket is deleted to issue an error prompt; wherein the error prompt is used to prompt that the bucket information of the bucket does not exist; when an error prompt is obtained, the example metadata and index metadata of the corresponding bucket are determined to be data to be cleared.

[0009] In one embodiment, after removing the bucket from the user's bucket list, identifying the data to be cleared in the bucket also includes: when the original bucket is removed from the user's bucket list and a bucket with the same name is created, obtaining bucket information of the bucket with the same name; the original bucket is any bucket among all the buckets in the current cluster; based on the bucket information of the bucket with the same name, obtaining the ID of the bucket with the same name; when the ID of the bucket with the same name is inconsistent with the ID indicated by the instance metadata of the original bucket, determining that the instance metadata and corresponding index metadata of the original bucket are data to be cleared.

[0010] In one embodiment, the object storage gateway-based bucket deletion method also includes: when the ID of the bucket with the same name is consistent with the ID indicated by the instance metadata of the original bucket, determining that the bucket is in use, and not cleaning up the instance metadata and corresponding index metadata.

[0011] In one embodiment, after a bucket is removed from a user's bucket list, data to be cleared in the bucket is identified, including: after the bucket is removed from the user's bucket list, in response to a scheduled task, periodically identifying data to be cleared in the bucket to delete the periodically identified instance metadata of the bucket and the index metadata of the bucket; or after the bucket is removed from the user's bucket list, in response to a delete command, identifying data to be cleared in the bucket in real time to delete the instance metadata of the bucket and the index metadata of the bucket identified in real time.

[0012] In one embodiment, after counting the number of objects cached in the bucket in response to a request to delete the bucket to obtain the total number of objects in the bucket, the object storage gateway-based bucket deletion method also includes: when the total number of objects in the bucket is greater than a preset number, determining that the bucket is not empty; when it is determined that the bucket is not empty, returning a verification result that the bucket is not empty to the client; and based on the verification result that the bucket is not empty, issuing an emptying prompt to prompt the user that the bucket is not empty.

[0013] According to another aspect of the present invention, a bucket deletion device based on an object storage gateway is provided, including: a statistical module, used to count the number of objects cached in the bucket in response to a request to delete the bucket to obtain the total number of objects in the bucket; a removal module, used to delete the entry metadata of the bucket and remove the bucket from the user's bucket list when the total number of objects in the bucket is less than a preset number; an identification module, used to identify the data to be cleared in the bucket after removing the bucket from the user's bucket list; wherein the data to be cleared includes the instance metadata of the bucket and the index metadata of the bucket; a deletion module, used to asynchronously delete the instance metadata and index metadata of the identified bucket to complete the deletion of the bucket of the object storage gateway.

[0014] The object storage gateway-based bucket deletion method and device provided by the present invention, after receiving a request to delete the bucket, checks whether the bucket meets the removal conditions according to whether the total number of objects in the bucket is less than a preset number, thereby improving the processing efficiency of checking whether the bucket can be deleted, and after removing the entry metadata of the bucket, identifies the remaining data to be cleared in the bucket, and then deletes the data to be cleared, avoiding separate deletion of bucket shards, improving deletion efficiency, reducing the time consumption of deleting the entire bucket, improving user experience, and reducing the operation and maintenance workload caused by bucket deletion timeout. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a flowchart of a method for deleting a bucket based on an object storage gateway provided by an exemplary embodiment of the present invention.

[0017] Figure 2 It is a flow chart of the working principle of a bucket deletion system based on an object storage gateway provided by an exemplary embodiment of the present invention.

[0018] Figure 3 The figure is a schematic diagram of an asynchronous deletion process of an asynchronous cleaning module provided by an exemplary embodiment of the present invention.

[0019] Figure 4 It is a structural diagram of a bucket deletion device based on an object storage gateway provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0020] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0021] The object storage gateway (RADOS Gateway, RGW) is a key component in the Ceph storage system, which is used to provide object storage services. Through the object storage gateway, users can use the HTTP protocol to access and operate the object storage in the Ceph cluster. When deleting a bucket in Ceph object storage, all objects in the bucket must be cleared before the bucket itself can be deleted. Therefore, the first step after the object storage gateway receives the request to delete the bucket is to verify whether the bucket is empty. Since there are many index shards in the bucket, when the bucket is indeed empty or the remaining objects happen to be located on the shards with relatively low numbers, it is necessary to verify that each shard is empty to determine whether the bucket is empty or not, which will take a lot of time. In addition, after verifying that the bucket is indeed empty, it is necessary to delete the bucket's entry, instance, and index metadata. Since there are many bucket shards, that is, a lot of index metadata, they need to be deleted one by one, which takes a lot of time, causing the client to wait for timeout errors, and then the object storage gateway continues to process the deletion work, which may cause the client's response to be inconsistent with the actual action of the object storage gateway, resulting in unnecessary operation and maintenance work. Therefore, the present invention counts the number of objects in the bucket and performs distributed caching of each gateway, so as to directly determine whether the number of objects in the bucket is 0 when checking whether the bucket is empty, thereby improving the processing efficiency of checking whether it is empty; after deleting the bucket entry metadata, asynchronously delete the instance and index metadata of the bucket, thereby improving the efficiency of deleting the bucket index metadata in the bucket deletion interface. Through the above two aspects of optimization, the bucket deletion interface effectively avoids direct operations on shards, thereby greatly improving the deletion speed, improving user experience, and reducing unnecessary bucket deletion operation and maintenance work.

[0022] Figure 1 is a flowchart of a method for deleting a bucket based on an object storage gateway provided by an exemplary embodiment of the present invention, such as Figure 1 As shown in the figure, the bucket deletion method based on the object storage gateway includes:

[0023] Step 100: In response to a request to delete a bucket, count the number of objects cached in the bucket to obtain the total number of objects in the bucket.

[0024] When a request to delete a bucket is received, the number of objects in the bucket (total_object_num) is counted in real time through the normal object upload and download statistics to determine whether the bucket is empty. This improves the verification efficiency and avoids the tedious verification according to bucket fragments.

[0025] Step 200: When the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list.

[0026] The preset number can be set to 0. When the total number of objects in the bucket is less than 0, the bucket can be quickly determined to be empty. After the bucket is determined to be empty, the bucket can be removed from the user's bucket list, and the client can be directly returned to notify the user that the bucket has been successfully removed. For example, a user deletes a bucket with 1000 shards and 100 million objects. When deleting, the total_object_num is not 0, and the bucket is reported as not empty. Then the user deletes all objects in the bucket by calling the delete interface. At this time, total_object_num is equal to 0. The delete bucket interface is called again, and the check is empty, so the bucket can proceed to the next removal operation.

[0027] Step 300: After removing the bucket from the user's bucket list, identify the data to be cleared in the bucket, wherein the data to be cleared includes the instance metadata of the bucket and the index metadata of the bucket.

[0028] After the above operations, the bucket entry metadata has been deleted, and the bucket has been removed from the user bucket list, but the bucket instance and index metadata have not been deleted. At this time, active identification is used to discover and clean up the remaining metadata information, which is the data to be cleared, including the bucket instance metadata and index metadata.

[0029] Optionally, the instance of a bucket usually refers to descriptive information related to the bucket instance itself, which may include the creation time, owner, access rights, and storage location of the bucket. This information is essential for managing and maintaining the bucket, and helps the system understand the basic properties and status of the bucket. The index metadata of a bucket is usually related to data indexing, and may include the organization of data, index structure, and how to quickly locate data objects based on indexes. The entry metadata of a bucket refers to the actual data objects stored in the bucket, such as files, pictures, videos, etc. These data are the specific content uploaded by users and wish to be stored in the cloud storage service.

[0030] Step 400: Asynchronously delete the instance metadata of the identified bucket and the index metadata of the bucket to complete the deletion of the bucket of the object storage gateway.

[0031] The asynchronous deletion method can be: when the bucket instance and index metadata that need to be cleaned are obtained, the index metadata is cleaned first. The number of ceph rados objects corresponding to the index metadata is the number of index shards of the bucket. Since the number of shards may reach more than 100,000, it is proposed to use multi-threading to clean the index metadata. For example, if a bucket has 100,000 shards, the naming number of its index metadata is 0-99999. A thread pool of 100 threads is opened to first process the deletion tasks of 0-99. Then whichever thread completes the deletion work will continue to execute the deletion operation of the next index until all 100,000 indexes are deleted. After deleting the corresponding index metadata, the instance metadata is deleted. Then continue to clean the next one to achieve asynchronous deletion. Compared with the record mode, the asynchronous deletion solution does not need to increase the interaction with the bottom layer in the bucket deletion operation interface, and does not need to make records at the bottom layer for subsequent cleaning work. Asynchronous (Asynchronous) is a work or communication method opposite to synchronous (Synchronous). It is mainly reflected in not performing operations or processing data in a strict order or timeline. The core characteristics of asynchronous methods are non-blocking and concurrency, that is, the completion of one operation does not immediately lead to the start of the next operation, but allows other tasks to be performed while waiting for the operation to complete.

[0032] In one embodiment, step 100 (in response to a request to delete a bucket, counting the number of objects cached in the bucket to obtain the total number of objects in the bucket) may include: when a user uploads an object, the total number of objects cached in the bucket increases by 1; when a user or a lifecycle deletes an object, the total number of objects cached in the bucket decreases by 1; wherein, step 200 (when the total number of objects in the bucket is less than a preset number, deleting the entry metadata of the bucket and removing the bucket from the user's bucket list) may include: when the total number of objects in the bucket is less than 0, determining that the bucket has been emptied; when it is determined that the bucket has been emptied, deleting the entry metadata of the bucket and removing the bucket from the user's bucket list.

[0033] When a user uploads an object, the total_object_num of the objects in the bucket is increased by 1. When a user or lifecycle deletes an object, the total_object_num of the objects in the bucket is decreased by 1. Lifecycle means that the deletion command is automatically executed periodically through a scheduled task, and user deletion means that the user actively deletes the object. According to the operations of uploading and deleting objects in the bucket (uploading an object means increasing 1, deleting an object means decreasing 1), real-time updates are performed to ensure the real-time and accurate number of objects. When responding to a request to delete a bucket, the total number of objects in the bucket can be quickly counted to determine whether the bucket is empty. When the total number of objects in the bucket is less than 0, it can be determined that the bucket has been emptied, which improves the efficiency of the empty verification process. In the object number cache, the uploaded object is increased by 1, and the deleted object is decreased by 1. The object number cache updates the underlying storage to ensure the real-time and accurate number of objects. The object number cache obtains the underlying storage from the ceph underlying storage, so the number of objects cached in the bucket can be obtained in real time, providing a data basis for subsequent verification of whether the bucket is empty.

[0034] In one embodiment, step 200 (when the total number of objects in the bucket is less than a preset number, deleting the bucket's entry metadata and removing the bucket from the user's bucket list) may include: when the total number of objects in the bucket is less than a preset number, deleting the bucket's entry metadata, and removing the bucket from the user's bucket list through a bucket unlink operation; after removing the bucket from the user's bucket list, the object storage gateway-based bucket deletion method may also include: returning information that the bucket has been successfully removed from the user's bucket list to the client to prompt the user that the bucket has been successfully removed.

[0035] The bucket unlink operation is used to remove a bucket from the user's bucket list. Bucket usually refers to a storage bucket, and unlink is often used to indicate the removal of links between files or directories. Therefore, the bucket unlink operation can delete one or more objects (files) from the storage bucket, which can be regarded as "disconnecting" the links between these objects and the storage bucket. After the bucket is removed, it is directly returned to the client to notify the user that the bucket has been successfully deleted, so as to reduce the time occupied by the bucket deletion interface.

[0036] In one embodiment, the method for deleting a bucket based on an object storage gateway may further include: caching the number of objects in the bucket using a distributed cache; based on the distributed cache, when the number of objects in the target object storage gateway increases or decreases, sending an update request to other object storage gateways among the multiple object storage gateways except the target object storage gateway; wherein the target object storage gateway is any one of the multiple object storage gateways; based on the update request, the other object storage gateways except the target object storage gateway increase or decrease the number of their own objects to make the total number of objects in the buckets cached in the multiple object storage gateways consistent.

[0037] The object number cache uses a distributed cache method to ensure that the cache of the value in multiple object storage gateways is consistent. The object number cache between each object storage gateway adopts an active push method to achieve cache consistency of each gateway. For example, there are three object storage gateways. One of the object storage gateways performs the operation of deleting objects. At this time, the number of objects is reduced by 1, and an update request is sent to the other two object storage gateways, so that the object number cache values ​​of the other two object storage gateways are also reduced by 1. When one of the object storage gateways uploads an object, the object number is increased by 1, and an update request is sent to the other two object storage gateways, so that the object number cache values ​​of the other two object storage gateways are also increased by 1. The same operation is performed when other storage gateways change. The object number cache writes the value to the underlying ceph storage system in real time to ensure that the object number cache value can be obtained in time when the object storage gateway is restarted or horizontally expanded.

[0038] In one embodiment, step 300 (identifying the data to be cleared in the bucket after removing the bucket from the user's bucket list) includes: after removing the bucket from the user's bucket list, obtaining instance metadata of all buckets in the current cluster; obtaining the name of the bucket and the ID of the bucket based on at least one instance metadata; wherein the name of the bucket and the ID of the bucket correspond to the instance metadata; obtaining bucket information of the bucket based on the name of the bucket and the ID of the bucket; when the bucket has been removed from the user's bucket list, deleting the bucket information of the bucket to issue an error prompt; wherein the error prompt is used to prompt that the bucket information of the bucket does not exist; when an error prompt is obtained, determining that the example metadata and index metadata of the corresponding bucket are data to be cleared.

[0039] When the task is executed, first list all bucket instance metadata in the current cluster, then get the bucket name and bucket ID based on the obtained instance metadata, and then get the bucketinfo (bucket information) of the bucket. When the bucket has been deleted, deleting bucketinfo will report an error "does not exist". At this time, it can be determined that the instance and its corresponding index are data to be cleared.

[0040] In one embodiment, step 300 (identifying the data to be cleared in the bucket after removing the bucket from the user's bucket list) may also include: when the original bucket is removed from the user's bucket list and a bucket with the same name is created, obtaining bucket information of the bucket with the same name; the original bucket is any bucket among all the buckets in the current cluster; based on the bucket information of the bucket with the same name, obtaining the ID of the bucket with the same name; when the ID of the bucket with the same name is inconsistent with the ID indicated by the instance metadata of the original bucket, determining that the instance metadata and corresponding index metadata of the original bucket are data to be cleared.

[0041] When a bucket is deleted and a bucket with the same name is created again, bucketinfo can be obtained at this time, but the bucket ID in the instance metadata is inconsistent with the bucket ID in the newly obtained bucketinfo. Therefore, it can be determined that the instance and its corresponding index are data to be cleared; only when the bucket ID in the obtained bucketinfo is consistent with the bucket ID in the instance metadata, the bucket is in use and no cleanup operation is performed. For example, the bucket name is test, the bucket ID obtained through instance is 34b10dbf-b1de-4cfe-b4c0-90b3427565c8.186530062.20, and the bucket ID obtained through bucketinfo is 34b10dbf-b1de-4cfe-b4c0-90b3427565c8.186530062.30. The two are different, so the instance is the metadata of the bucket to be deleted, and then the bucket index metadata is obtained through the ID in the instance, and the two are deleted separately. Asynchronously process the instance and index metadata of the bucket, effectively utilize the index sharding feature of the bucket, start multiple threads to delete the index metadata in parallel, and improve efficiency.

[0042] In one embodiment, the bucket deletion method based on the object storage gateway may also include: when the ID of the bucket with the same name is consistent with the ID indicated by the instance metadata of the original bucket, determining that the bucket is in use, and not cleaning up the instance metadata and the corresponding index metadata.

[0043] When the ID of the bucket with the same name is consistent with the ID indicated by the instance metadata of the original bucket, the instance metadata and corresponding index metadata of the bucket are still in use. Therefore, the instance metadata and corresponding index metadata in use are not cleaned up temporarily to ensure normal use by users.

[0044] In one embodiment, step 300 (identifying the data to be cleared in the bucket after the bucket is removed from the user's bucket list) may include: after the bucket is removed from the user's bucket list, in response to a scheduled task, periodically identifying the data to be cleared in the bucket to delete the instance metadata of the bucket and the index metadata of the bucket that are periodically identified; or after the bucket is removed from the user's bucket list, in response to a delete command, identifying the data to be cleared in the bucket in real time to delete the instance metadata of the bucket and the index metadata of the bucket that are identified in real time.

[0045] The deletion command can be executed by the admin command line. The query command of the admin command line is radosgw-adminbucket stale-instances list. Internally, it obtains all instance metadata, obtains bucketinfo according to the bucket name, and then compares the bucket ID in the instance metadata with the bucket ID in bucketinfo. If they are inconsistent, they are metadata to be cleaned up and then displayed. The cleanup command of the admin command line is radosgw-admin bucket stale-instances delete. Internally, it filters out all instance and index metadata to be deleted in the same way as the query command, and then starts multi-threading for cleanup.

[0046] In one embodiment, after step 100 (in response to a request to delete a bucket, counting the number of objects cached in the bucket to obtain the total number of objects in the bucket), the bucket deletion method based on the object storage gateway may also include: when the total number of objects in the bucket is greater than a preset number, determining that the bucket is not empty; when it is determined that the bucket is not empty, returning a verification result that the bucket is not empty to the client; based on the verification result that the bucket is not empty, issuing an emptying prompt to prompt the user that the bucket is not empty.

[0047] When the bucket is not empty, the check result is returned to the client to notify the user to empty the bucket. For example, only when the total number of objects in the bucket is zero, the bucket is determined to be empty, and the user is notified to delete the object to make the number of objects in the bucket zero.

[0048] In one embodiment, the present invention provides a bucket deletion system based on an object storage gateway, including a bucket empty verification module, an object number cache module, and an asynchronous cleanup module. Figure 2 FIG. 1 is a flowchart of the working principle of a bucket deletion system based on an object storage gateway provided by an exemplary embodiment of the present invention. Figure 2As shown in the figure, the object number cache module mainly caches the number of objects in the bucket, and updates in real time according to operations such as uploading and deleting objects in the bucket (uploading an object adds 1, deleting an object subtracts 1), and maintains the cache of the number of objects in the bucket to ensure the real-time and accurate number of objects. The bucket check module is responsible for judging whether the bucket is empty based on the number of objects cached in the object number cache module. It directly judges based on the existing statistical data, and the judgment efficiency is high, which can improve the overall deletion efficiency. The bucket check module verifies whether the bucket is empty and deletes the entry metadata based on the request to delete the bucket, and then removes the bucket from the list through the bucket unlink operation and returns it to the client. The asynchronous cleanup module is mainly used to identify the legacy data generated in the deletion operation. The legacy data includes the instance and index metadata of the bucket, and asynchronously cleans the bucket instance and index metadata.

[0049] Figure 3 FIG. 1 is a schematic diagram of an asynchronous deletion process of an asynchronous cleaning module provided by an exemplary embodiment of the present invention. Figure 3 As shown, the asynchronous deletion process of the asynchronous cleanup module is: first obtain all instances in the cluster (step 31), screen instances without entries (step 32), then obtain the corresponding index metadata (step 33), and finally delete the instance and index metadata (step 34), realizing asynchronous multi-threaded concurrent deletion and improving deletion efficiency.

[0050] Figure 4 is a structural diagram of a bucket deletion device based on an object storage gateway provided by an exemplary embodiment of the present invention, such as Figure 4 As shown, the bucket deletion device 4 based on the object storage gateway includes: a statistical module 41, which is used to count the number of objects cached in the bucket in response to a request to delete the bucket to obtain the total number of objects in the bucket; a removal module 42, which is used to delete the entry metadata of the bucket and remove the bucket from the user's bucket list when the total number of objects in the bucket is less than a preset number; an identification module 43, which is used to identify the data to be cleared in the bucket after removing the bucket from the user's bucket list; wherein the data to be cleared includes the instance metadata of the bucket and the index metadata of the bucket; a deletion module 44, which is used to asynchronously delete the instance metadata and index metadata of the identified bucket to complete the deletion of the bucket of the object storage gateway.

[0051] The bucket deletion device based on the object storage gateway provided by the present invention, after receiving a request to delete the bucket, verifies whether the bucket meets the removal condition according to whether the total number of objects in the bucket is less than a preset number, thereby improving the processing efficiency of verifying whether the bucket can be deleted, and after removing the entry metadata of the bucket, identifies the remaining data to be cleared in the bucket, and then deletes the data to be cleared, avoiding separate deletion of bucket shards, improving deletion efficiency, reducing the time consumption of deleting the entire bucket, improving user experience, and reducing the operation and maintenance workload caused by bucket deletion timeout.

[0052] In one embodiment, the statistics module 41 can be configured as follows: when a user uploads an object, the total number of objects cached in the bucket increases by 1; when the user or the life cycle deletes the object, the total number of objects cached in the bucket decreases by 1; wherein, the removal module 42 can be correspondingly configured as follows: when the total number of objects in the bucket is less than 0, it is determined that the bucket has been emptied; when it is determined that the bucket has been emptied, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list.

[0053] In one embodiment, the removal module 42 can be configured to: when the total number of objects in the bucket is less than a preset number, delete the entry metadata of the bucket, and remove the bucket from the user's bucket list through a bucket unlink operation; after removing the bucket from the user's bucket list, the bucket deletion method based on the object storage gateway also includes: returning information that the bucket has been successfully removed from the user's bucket list to the client to prompt the user that the bucket has been successfully removed.

[0054] In one embodiment, the bucket deletion device 4 based on the object storage gateway can be configured as follows: the number of objects in the bucket is cached using a distributed cache; based on the distributed cache, when the number of objects in the target object storage gateway increases or decreases, an update request is sent to other object storage gateways among the multiple object storage gateways except the target object storage gateway; wherein the target object storage gateway is any one of the multiple object storage gateways; based on the update request, the other object storage gateways except the target object storage gateway increase or decrease the number of their own objects to make the total number of objects in the buckets cached in the multiple object storage gateways consistent.

[0055] In one embodiment, the identification module 43 can be configured as follows: after removing the bucket from the user's bucket list, obtain the instance metadata of all buckets in the current cluster; based on at least one instance metadata, obtain the name of the bucket and the ID of the bucket; wherein the name of the bucket and the ID of the bucket correspond to the instance metadata; based on the name of the bucket and the ID of the bucket, obtain the bucket information of the bucket; when the bucket has been removed from the user's bucket list, delete the bucket information of the bucket to issue an error prompt; wherein the error prompt is used to prompt that the bucket information of the bucket does not exist; when an error prompt is obtained, determine that the example metadata and index metadata of the corresponding bucket are data to be cleared.

[0056] In one embodiment, the identification module 43 can be configured as follows: when the original bucket is removed from the user's bucket list and a bucket with the same name is created, the bucket information of the bucket with the same name is obtained; the original bucket is any bucket among all the buckets in the current cluster; based on the bucket information of the bucket with the same name, the ID of the bucket with the same name is obtained; when the ID of the bucket with the same name is inconsistent with the ID indicated by the instance metadata of the original bucket, the instance metadata and corresponding index metadata of the original bucket are determined to be data to be cleared.

[0057] In one embodiment, the bucket deletion device 4 based on the object storage gateway can be configured as follows: when the ID of the bucket with the same name is consistent with the ID indicated by the instance metadata of the original bucket, the bucket is determined to be in use, and the instance metadata and corresponding index metadata are not cleaned up.

[0058] In one embodiment, the identification module 43 can be configured to: after removing the bucket from the user's bucket list, in response to a scheduled task, periodically identify the data to be cleared in the bucket to delete the instance metadata of the bucket and the index metadata of the bucket that are periodically identified; or after removing the bucket from the user's bucket list, in response to a delete command, identify the data to be cleared in the bucket in real time to delete the instance metadata of the bucket and the index metadata of the bucket that are identified in real time.

[0059] In one embodiment, the bucket deletion device 4 based on the object storage gateway can be configured as follows: when the total number of objects in the bucket is greater than a preset number, it is determined that the bucket is not empty; when it is determined that the bucket is not empty, the verification result of the bucket not being empty is returned to the client; based on the verification result of the bucket not being empty, an emptying prompt is issued to prompt the user that the bucket is not empty.

[0060] An embodiment of the present invention provides a bucket deletion device based on an object storage gateway. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From a hardware perspective, in addition to the CPU, memory, network interface, and non-volatile memory, the device in the embodiment where the device is located can generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, as a device in a logical sense, it is formed by the CPU of the device where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution.

[0061] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the bucket deletion method based on the object storage gateway of any of the above embodiments.

[0062] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the object storage gateway-based bucket deletion method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0063] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing processor executable instructions; and a processor for executing the bucket deletion method based on an object storage gateway of any of the above embodiments.

[0064] In addition, an embodiment of the present invention may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the object storage gateway-based bucket deletion method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for deleting a bucket based on an object storage gateway, characterized in that: include: In response to a request to delete a bucket, count the number of objects cached in the bucket to obtain a total number of objects in the bucket; When the total number of objects in the bucket is less than a preset number, deleting the entry metadata of the bucket and removing the bucket from the user's bucket list; After removing the bucket from the user's bucket list, identifying the data to be cleared in the bucket; wherein the data to be cleared includes instance metadata of the bucket and index metadata of the bucket; Asynchronously delete the instance metadata and index metadata of the identified bucket to complete the bucket deletion of the object storage gateway.

2. The method for deleting a bucket based on an object storage gateway according to claim 1, characterized in that: In response to a request to delete a bucket, count the number of objects cached in the bucket to obtain the total number of objects in the bucket, including: When a user uploads an object, the total number of objects cached in the bucket increases by 1; When a user or lifecycle deletes an object, the total number of cached objects in the bucket is reduced by 1; When the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list, including: When the total number of objects in the bucket is less than 0, it is determined that the bucket is empty; When it is determined that the bucket is empty, the bucket entry metadata is deleted and the bucket is removed from the user's bucket list.

3. The method for deleting a bucket based on an object storage gateway according to claim 1, characterized in that: When the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list, including: When the total number of objects in the bucket is less than a preset number, the entry metadata of the bucket is deleted, and the bucket is removed from the user's bucket list through a bucket unlink operation; After removing the bucket from the user's bucket list, the object storage gateway-based bucket deletion method also includes: The information that the bucket is successfully removed from the user's bucket list is returned to the client to prompt the user that the bucket is successfully removed.

4. The method for deleting a bucket based on an object storage gateway according to claim 1, characterized in that: The method for deleting a bucket based on the object storage gateway also includes: The number of objects in the bucket is cached using distributed cache; Based on the distributed cache, when the number of objects in the target object storage gateway increases or decreases, an update request is sent to other object storage gateways among the multiple object storage gateways except the target object storage gateway; wherein the target object storage gateway is any one of the multiple object storage gateways; Based on the update request, other object storage gateways except the target object storage gateway increase or decrease the number of their own objects to make the total number of objects in the buckets cached in the multiple object storage gateways consistent.

5. The method for deleting a bucket based on an object storage gateway according to claim 1, characterized in that: After removing a bucket from the user's bucket list, identify the bucket's data to be cleared, including: After removing the bucket from the user's bucket list, obtain the instance metadata of all buckets in the current cluster; According to at least one of the instance metadata, obtaining a bucket name and a bucket ID; wherein the bucket name and the bucket ID correspond to the instance metadata; Get the bucket information of the bucket according to the bucket name and bucket ID; When the bucket has been removed from the user's bucket list, the bucket information of the bucket is deleted to issue an error prompt; wherein the error prompt is used to prompt that the bucket information of the bucket does not exist; When an error message is received, it is determined that the sample metadata and index metadata of the corresponding bucket are data to be cleared.

6. The method for deleting a bucket based on an object storage gateway according to claim 5, characterized in that: After a bucket is removed from the user's bucket list, the bucket's data to be cleared also includes: When the original bucket is removed from the user's bucket list and a bucket with the same name is created, the bucket information of the bucket with the same name is obtained; the original bucket is any bucket among all the buckets in the current cluster; Based on the bucket information of the bucket with the same name, obtain the ID of the bucket with the same name; When the ID of the bucket with the same name is inconsistent with the ID indicated by the instance metadata of the original bucket, it is determined that the instance metadata and the corresponding index metadata of the original bucket are data to be cleared.

7. The method for deleting a bucket based on an object storage gateway according to claim 6, characterized in that: The method for deleting a bucket based on the object storage gateway also includes: When the ID of the bucket with the same name is consistent with the ID indicated by the instance metadata of the original bucket, it is determined that the bucket is in use, and the instance metadata and the corresponding index metadata are not cleared.

8. The method for deleting a bucket based on an object storage gateway according to claim 1, characterized in that: After removing a bucket from the user's bucket list, identify the bucket's data to be cleared, including: After removing the bucket from the user's bucket list, in response to the scheduled task, periodically identify the data to be cleared in the bucket to delete the instance metadata and index metadata of the bucket periodically identified; or After the bucket is removed from the user's bucket list, in response to a delete command, the data to be cleared in the bucket is identified in real time to delete the instance metadata of the bucket and the index metadata of the bucket identified in real time.

9. The method for deleting a bucket based on an object storage gateway according to claim 1, characterized in that: After counting the number of objects cached in the bucket in response to the request to delete the bucket to obtain the total number of objects in the bucket, the method for deleting the bucket based on the object storage gateway further includes: When the total number of objects in the bucket is greater than a preset number, determining that the bucket is not empty; When it is determined that the bucket is not empty, the verification result that the bucket is not empty is returned to the client; According to the verification result that the bucket is not emptied, an emptying prompt is issued to remind the user that the bucket is not emptied.

10. A bucket deletion device based on an object storage gateway, characterized in that: include: A statistics module, configured to count the number of objects cached in the bucket in response to a request to delete the bucket, so as to obtain a total number of objects in the bucket; A removal module, configured to delete the entry metadata of the bucket and remove the bucket from the user's bucket list when the total number of objects in the bucket is less than a preset number; An identification module, used to identify the data to be cleared in the bucket after the bucket is removed from the bucket list of the user; wherein the data to be cleared includes the instance metadata of the bucket and the index metadata of the bucket; The deletion module is used to asynchronously delete the instance metadata and index metadata of the identified bucket to complete the bucket deletion of the object storage gateway.

Images