Bucket data processing method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a storage bucket data processing method and device, equipment, a storage medium and a product. Comprising the following steps: in response to operation of inputting deletion starting time and deletion speed by a tenant and executing a command of deleting a bucket, generating a deletion request at the deletion starting time; according to the storage bucket name and the tenant identifier, judging whether the storage bucket instance information and the tenant information are in a cache or not; if the bucket instance information and the tenant information are both in the cache, obtaining the number of fragments of the bucket according to the bucket instance information; according to the fragment number of the storage bucket, obtaining all storage bucket fragment index information; obtaining each storage bucket fragment according to the index information of each storage bucket fragment; determining all deletion threads according to the deletion speed; all the storage bucket fragments are put into a pre-created thread pool, so that all deletion threads complete deletion operation on all objects under all the storage bucket fragments; and updating the tenant information and deleting the storage bucket name. The effect of improving the deleting speed is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a storage bucket data processing method, device, equipment, storage medium and product. Background Art

[0002] When tenants use the distributed object storage system, there are specific business scenarios where tenants need to upload a large number of small files to the object storage bucket every month. For example, in some data-intensive enterprises or Internet businesses, a large number of log files, image cache files, etc. are stored in the form of small files. When files are uploaded using different buckets every month and the files in the previous month's bucket are no longer needed in the next month, a large number of objects in the bucket will be deleted.

[0003] In the prior art, the data deletion scheme for a distributed object storage super-large bucket is performed through a third-party tool or a tool provided by the object storage. The objects in the bucket are first read out and then deleted. After the objects in the bucket are deleted, the bucket is deleted.

[0004] However, both deletion methods are single-process deletion, and the deletion speed is slow. It is often the case that the speed of deleting objects is slower than the speed of uploading objects, which results in the inability to write data and affects the normal operation of the business. Summary of the invention

[0005] The embodiments of the present application provide a storage bucket data processing method, apparatus, device, storage medium and product to achieve the effect of improving the deletion speed.

[0006] In a first aspect, an embodiment of the present application provides a bucket data processing method, which is applied to a server, comprising: in response to a tenant inputting a deletion start time and a deletion speed and executing a command to delete a bucket, generating a deletion request at the deletion start time, wherein the deletion request includes a bucket name and a tenant identifier; judging whether the bucket instance information is in the cache according to the bucket name; judging whether the tenant information is in the cache according to the tenant identifier; if both the bucket instance information and the tenant information are in the cache, obtaining the number of shards of the bucket according to the bucket instance information; obtaining all bucket shard index information according to the number of shards of the bucket; obtaining each bucket shard according to the index information of each bucket shard; determining all deletion threads according to the deletion speed; placing all bucket shards into a pre-created thread pool so that all deletion threads complete the deletion operation on all objects under each bucket shard; updating the tenant information and deleting the bucket instance information; deleting the bucket name under the tenant.

[0007] In one possible implementation, all bucket shards are placed in a pre-created thread pool so that all deletion threads complete the deletion operation on all objects under each bucket shard, including: traversing all bucket shards; for any bucket shard, placing the deletion task into a pre-created thread pool so that all deletion threads in the pre-created thread pool process it; for any deletion thread, enumerating all objects under each bucket shard; traversing all objects under each bucket shard, and calling the deletion interface of each object to perform the deletion operation.

[0008] In one possible implementation, all deletion threads are determined according to the deletion speed, including: obtaining the current amount of available memory and the amount of memory occupied by a pre-created thread pool; determining the effective amount of memory of the pre-created thread pool according to the current amount of available memory and the amount of memory occupied; determining the maximum number of deletion threads according to the effective amount of memory and the average amount of memory occupied by each deletion thread; determining the theoretical number of deletion threads according to the deletion speed and the average deletion speed of each deletion thread; determining the smaller value between the maximum number of deletion threads and the theoretical number of deletion threads as the number of deletion threads; and allocating all deletion threads in the pre-created thread pool according to the number of deletion threads.

[0009] In one possible implementation, the formula for determining the maximum number of deletion threads is:

[0010]

[0011] Where N M Indicates the maximum number of deletion threads; M e Indicates the amount of available memory; M t Indicates the average memory usage of each preset deletion thread; Indicates a floor operation.

[0012] In a possible implementation, the formula for determining the theoretical number of deletion threads is:

[0013]

[0014] Where N V Indicates the theoretical number of deletion threads; V d Indicates the deletion speed; V t Indicates the average deletion speed of each preset deletion thread; Indicates a round-up operation.

[0015] In a possible implementation, after all deletion threads complete the deletion operation on all objects under each bucket shard, it also includes: traversing each bucket shard to determine whether there is an object in the undeleted operation state; if there is an object in the undeleted operation state, the object is placed in a pre-created thread pool to perform the deletion operation.

[0016] In a possible implementation, all objects under each bucket shard are traversed, and the deletion interface of each object is called to perform the deletion operation, including: traversing all objects under each bucket shard; for any object, determining whether the creation time of the object is later than the deletion start time; if the creation time of the object is later than the deletion start time, not deleting the object; if the creation time of the object is not later than the deletion start time, calling the deletion interface of each object to perform the deletion operation.

[0017] In a second aspect, an embodiment of the present application provides a storage bucket data processing device, which is applied to a server, including:

[0018] The deletion request generation module is used to generate a deletion request at the deletion start time in response to the tenant inputting the deletion start time and the deletion speed and executing the command to delete the bucket, wherein the deletion request includes the bucket name and the tenant identifier.

[0019] The bucket location determination module is used to determine whether the bucket instance information is in the cache according to the bucket name.

[0020] The tenant information location determination module is used to determine whether the tenant information is in the cache according to the tenant identifier.

[0021] The bucket shard number acquisition module is used to obtain the bucket shard number according to the bucket instance information if the bucket instance information and the tenant information are both in the cache.

[0022] The bucket shard index information acquisition module is used to obtain all bucket shard index information according to the number of shards of the bucket.

[0023] Each bucket shard acquisition module is used to acquire each bucket shard according to each bucket shard index information.

[0024] The deletion thread determination module is used to determine all deletion threads according to the deletion speed.

[0025] The object deletion module is used to put all bucket shards into a pre-created thread pool so that all deletion threads can complete the deletion operation on all objects under each bucket shard.

[0026] The tenant information update module is used to update tenant information and delete bucket instance information.

[0027] Bucket name deletion module, used to delete bucket names under a tenant.

[0028] In a third aspect, an embodiment of the present application provides a server, comprising: a memory, a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect above and / or various possible implementations of the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0031] The bucket data processing method, apparatus, device, storage medium and product provided in the embodiments of the present application first obtain the number of shards of the bucket according to the bucket instance information, then read multiple bucket shards of the bucket at the same time, and then start multiple deletion threads for the multiple bucket shards to delete the objects in each bucket shard, thereby changing the original single-process deletion to multi-process deletion. By using a thread pool, a certain number of deletion threads are created in advance and put into the pool for management. When a deletion operation needs to be performed, the deletion thread is directly obtained from the thread pool for use, thereby avoiding the resource overhead caused by frequent creation and destruction of deletion threads, and improving resource utilization efficiency and deletion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 A schematic diagram of a scenario of a storage bucket data processing method provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a method for processing bucket data provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a storage bucket data processing device provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0037] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0039] First, the terms involved in this application are explained:

[0040] Bucket: A data container for storing objects, usually the basic storage unit provided by cloud storage services.

[0041] In order to clearly understand the technical solution of this application, the solution of the prior art is first introduced in detail. In the current distributed object storage system, there is the following business scenario: tenants upload a large number of small files to bucket A of the object storage every month, and a new bucket B will be created the next month, and then continue to upload a large number of small files to bucket B. Since the storage space of the object storage is limited, and the tenants no longer need the files in bucket A the next month, at this time, while deleting objects in bucket A, they upload objects to bucket B. Since the current distributed storage object system deletes large numbers of small files slowly, there will be a problem that the speed of deleting objects is slower than the speed of uploading objects, which will cause the object storage space to be completely filled until data cannot be written.

[0042] In response to the above technical problems, the inventors thought of first obtaining the number of shards of the bucket based on the bucket instance information, then reading multiple bucket shards of the bucket at the same time, and then starting multiple deletion threads for the multiple bucket shards to delete the objects in each bucket shard, thereby changing the original single-process deletion to multi-process deletion, which can improve the deletion performance and ensure the normal operation of the object storage system.

[0043] Based on the above creative findings, the inventor proposed the technical solution of the present application.

[0044] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0045] Figure 1 A schematic diagram of a scenario of a method for processing bucket data provided in an embodiment of the present application. Figure 1 As shown, the specific application scenario of the present application includes: a receiving device 101, a processor 102 and a display device 103.

[0046] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the storage bucket data processing method. In other feasible implementations of the present application, the above architecture may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0047] In a specific implementation process, the receiving device 101 may be an input / output interface or a communication interface, which is used to receive the deletion start time and deletion speed input by the tenant and execute a command to delete the storage bucket.

[0048] The processor 102 may generate a deletion request at the deletion start time in response to the tenant inputting the deletion start time and the deletion speed and executing the command to delete the bucket. According to the bucket name, determine whether the bucket instance information is in the cache; according to the tenant identifier, determine whether the tenant information is in the cache; if both the bucket instance information and the tenant information are in the cache, obtain the number of shards of the bucket according to the bucket instance information; according to the number of shards of the bucket, obtain all bucket shard index information; according to the index information of each bucket shard, obtain each bucket shard; according to the deletion speed, determine all deletion threads; put all bucket shards into a pre-created thread pool so that all deletion threads complete the deletion operation on all objects under each bucket shard; update the tenant information and delete the bucket instance information; delete the bucket name under the tenant.

[0049] The display device 103 can be used to display the result after the bucket instance information and the bucket name are deleted.

[0050] It should be understood that the above-mentioned processor can be implemented by the processor reading instructions in the memory and executing the instructions, or it can be implemented by a chip circuit.

[0051] Figure 2 Schematic diagram of the process of the bucket data processing method provided in the embodiment of the present application. Figure 2 As shown, the method includes steps S201 to S210:

[0052] S201: In response to the tenant inputting a deletion start time and a deletion speed and executing a command to delete a bucket, a deletion request is generated at the deletion start time, wherein the deletion request includes a bucket name and a tenant identifier.

[0053] Specifically, the tenant executes the command to delete the bucket through the console or supporting tools. The tenant enters the deletion start time and sets the deletion speed before executing the operation to delete the bucket.

[0054] For example, the tenant enters the deletion start time, such as starting the deletion at 24:00 a.m. when the business pressure is relatively low. Since the bucket deletion time is relatively long, the tenant can choose to perform 32x deletion from 22:00 to 5:00 the next day, and 1x deletion from 5:00 to 22:00.

[0055] S202: Determine whether the bucket instance information is in the cache according to the bucket name.

[0056] Specifically, after receiving the bucket name, the server searches the cache for matching bucket instance information according to a preset search mechanism. If not in the cache, the server obtains the bucket instance information from the storage pool and puts the bucket instance information into the cache.

[0057] S203: Determine whether the tenant information is in the cache according to the tenant identifier.

[0058] Specifically, after receiving the tenant identifier, the server searches the cache for matching tenant information according to a preset search mechanism. If not in the cache, the server obtains the tenant information from the storage pool and puts the tenant information into the cache.

[0059] S204: If the bucket instance information and the tenant information are both in the cache, the number of shards of the bucket is obtained according to the bucket instance information.

[0060] The bucket instance information includes: bucket name, tenant name, bucket ID, bucket creation time, bucket data storage location, bucket shard number, etc.

[0061] Among them, tenant information includes tenant identification, tenant name, secret key, tenant permissions, sub-users and other information.

[0062] Specifically, after confirming that the bucket instance information and the tenant information are both in the cache, the number of shards of the bucket is accurately extracted based on the bucket instance information.

[0063] S205: Obtain all bucket shard index information according to the number of bucket shards.

[0064] Among them, a unique index number is assigned to each shard of the bucket in sequence, starting from 0 and continuing to the number of shards minus 1, or its index information is determined based on factors such as the physical location of the shard in the storage pool and the creation time.

[0065] S206: Obtain each bucket shard according to the shard index information of each bucket.

[0066] Specifically, in a distributed storage environment, the shard content is obtained by interacting with the storage node through network communication, according to the storage node address, storage path, etc. pointed to by the index information; or in a local storage system, the corresponding shard data is read according to the disk storage area corresponding to the index information.

[0067] S207: Determine all deletion threads according to the deletion speed.

[0068] The deletion thread may be understood as an execution unit that can independently execute the deletion task.

[0069] Specifically, since a deletion may last for a week or even a month, the tenant can set different deletion speeds in different time periods. According to the different deletion speeds set by the tenant, different numbers of deletion threads are allocated in the thread pool, including steps Sa1 to Sa6:

[0070] Sa1: Get the current amount of available memory and the amount of memory occupied by the pre-created thread pool.

[0071] Specifically, if the total memory of the thread pool is M t (Unit: Bytes) The current amount of memory used can be obtained through the system monitoring tool as M u (Unit: Byte), then the current available memory M a The formula M a =M t -M u The thread pool itself will also occupy a certain amount of memory space during operation for thread management, such as storing thread status information, task queues, etc. Suppose the estimated amount of memory occupied by thread pool management related operations is M p (Unit: Byte).

[0072] Sa2: Determine the effective memory size of the pre-created thread pool based on the current available memory and occupied memory.

[0073] Specifically, the calculation formula for the effective memory amount Me that can be used by the thread to perform the deletion operation is M e =M a -M p .

[0074] Sa3: Determine the maximum number of deletion threads based on the effective memory amount and the average memory occupied by each deletion thread.

[0075] Specifically, determine the estimated average memory usage of each deletion thread during the deletion operation, set as m t (Unit: Bytes). This value can be determined based on experience data from similar deletion operations or analysis of the characteristics of this deletion task. For example, based on previous tests of bucket data deletion operations, it was found that each thread needs an average of 100MB of memory to complete the task. t It can be set to 100MB.

[0076] Specifically, based on the effective amount of memory that can be used by threads to perform deletion operations and the estimated average amount of memory occupied by each deletion thread, the formula for determining the maximum number of deletion threads is:

[0077]

[0078] Where N M Indicates the maximum number of deletion threads; M e Indicates the amount of available memory; M t Indicates the average memory usage of each preset deletion thread; Indicates a floor operation.

[0079] Sa4: Determine the theoretical number of deletion threads based on the deletion speed and the preset average deletion speed of each deletion thread.

[0080] Specifically, the deletion speed of tenant input is set to V d (Unit: bytes / second), which indicates the amount of data that you want to delete in a unit of time. Assume that the average deletion speed that each deletion thread can achieve under ideal conditions is V t (Unit: Bytes / second). This speed may be affected by many factors, such as the read and write speed of the storage device, network bandwidth, etc.

[0081] Specifically, in order to achieve the deletion speed V required by the tenant d , the formula to determine the theoretical number of deletion threads is:

[0082]

[0083] Where N V Indicates the theoretical number of deletion threads; V d Indicates the deletion speed; V t Indicates the average deletion speed of each preset deletion thread; Indicates a round-up operation.

[0084] Sa5: The smaller value between the maximum number of deletion threads and the theoretical number of deletion threads is determined as the number of deletion threads.

[0085] Specifically, the number of parallel deletion threads N is determined by comprehensively considering the number of parallel deletion threads that can be supported by effective memory and the number of parallel deletion threads based on the deletion speed requirement. f The maximum number of deletion threads N should be taken M and the theoretical number of deletion threads N V The smaller value of N f =min(N M ,N V ).

[0086] Sa6: Allocate all deletion threads in the pre-created thread pool based on the number of deletion threads.

[0087] Specifically, according to the determined number of deletion threads, the server will select a corresponding number of available threads from the pre-created thread pool and assign them to perform the bucket deletion task.

[0088] For example, if the determined number of deletion threads is 10, the server will search and mark 10 threads in an idle state in the thread pool, and then associate these threads with each shard deletion task of the bucket.

[0089] S208: Put all bucket shards into a pre-created thread pool, so that all deletion threads complete the deletion operation on all objects under each bucket shard.

[0090] Specifically, the pre-created thread pool is a mechanism that centrally manages thread resources and can coordinate task execution. After receiving the bucket shards, it allocates tasks according to the scheduling rules. Each deletion thread obtains the corresponding bucket shard from the thread pool, and then deletes the objects in the shard in turn.

[0091] Exemplarily, for example, assuming there are 5 deletion threads and 10 bucket shards, the thread pool may reasonably allocate these 10 shards to 5 threads, for example, thread 1 is responsible for processing shards 1 and 2, thread 2 is responsible for processing shards 3 and 4, and so on. Then each deletion thread will go deep into the shard it is responsible for, and perform deletion operations on various objects such as files and data blocks contained in the shard one by one according to the established order or data structure characteristics.

[0092] Specifically, it includes steps Sb1 to Sb4:

[0093] Sb1: Traverse all bucket shards.

[0094] Sb2: For any bucket shard, put the deletion task into the pre-created thread pool so that all deletion threads in the pre-created thread pool can process it.

[0095] Specifically, after traversing each specific bucket shard, the deletion task for the shard will be added to the pre-created thread pool, and the thread pool will assign the task to the idle deletion thread for execution based on its internal scheduling strategy.

[0096] Sb3: For any deletion thread, list all objects under each bucket shard.

[0097] Specifically, when the deletion thread receives a deletion task for a bucket shard, it needs to first clarify all objects contained in the shard and list the objects under the shard.

[0098] Sb4: Traverse all objects under each bucket shard and call the deletion interface of each object to perform the deletion operation.

[0099] Specifically, after all objects under each bucket shard are identified, the deletion thread will perform a traversal operation again to process each object.

[0100] Specifically, for each object, the deletion action is implemented by calling its corresponding deletion interface, wherein the deletion interface of each object is usually pre-defined by the storage pool and encapsulates the specific code logic for deleting the object.

[0101] S209: Update tenant information and delete bucket instance information.

[0102] Specifically, after deleting all bucket shards, the statistics of the number of objects and the total size of objects under the tenant are updated, and the bucket instance information is deleted.

[0103] S210: Delete the bucket name under the tenant.

[0104] In summary, we first obtain the number of bucket shards according to the bucket instance information, then read multiple bucket shards of the bucket at the same time, and then start multiple deletion threads for multiple bucket shards to delete objects in each bucket shard, changing the original single-process deletion to multi-process deletion. By using the thread pool, a certain number of deletion threads are created in advance and put into the pool for management. When the deletion operation needs to be performed, the deletion thread is directly obtained from the thread pool for use, avoiding the resource overhead caused by frequent creation and destruction of deletion threads, and improving resource utilization efficiency and deletion performance.

[0105] In another embodiment of the present application, although the deletion thread performs the deletion operation on all objects under each bucket shard, it is still possible that some objects are not successfully deleted due to various reasons, such as temporary system failure, permission problems, or special attributes of the object itself that cause deletion to be blocked, so an additional monitoring and verification mechanism is required for secondary confirmation and processing. Specifically, after all deletion threads complete the deletion operation on all objects under each bucket shard, steps S301 to S302 are also included:

[0106] S301: traverse each bucket shard to determine whether there is an object in an undeleted operation state.

[0107] Specifically, by accessing each bucket shard in turn, the system can go deep into each shard to check the deletion status of the objects therein. A status check is performed on all objects in the shard to determine whether any object is in a non-deleted state.

[0108] S302: If there is an object in a non-deletion operation state, the object is placed in a pre-created thread pool to perform a deletion operation.

[0109] Specifically, when it is determined that there are objects in an undeleted operation state, after the undeleted objects are placed in the thread pool, the thread pool will allocate idle deletion threads to process these objects according to its own internal scheduling rules.

[0110] In summary, by conducting secondary inspections and targeted processing of objects that may not have been successfully deleted, data residue problems caused by various unexpected situations are minimized. Only when all objects under all bucket shards are indeed completely deleted, subsequent operations such as updating tenant information and deleting bucket instance information can be performed based on accurate storage status, avoiding a series of problems such as chaotic tenant permission management, storage resource statistics errors, and redundant information occupying storage resources caused by data inconsistency, thereby ensuring the accuracy and integrity of the data in the entire storage system during the bucket deletion process and the stable operation of the entire system.

[0111] In another embodiment of the present application, while deleting the storage bucket, an object upload operation may be performed, and step Sb4 specifically includes steps S401 to S404:

[0112] S401: Traverse all objects under each bucket shard.

[0113] S402: For any object, determine whether the creation time of the object is later than the deletion start time.

[0114] Specifically, after traversing each object, the system obtains the creation time information of the object and compares it with the preset bucket deletion start time.

[0115] The creation time information clearly identifies the moment when the object is first generated in the storage system.

[0116] S403: If the creation time of the object is later than the deletion start time, the object is not deleted.

[0117] Specifically, when it is determined that the creation time of an object is later than the deletion start time, the system skips the deletion operation on the object.

[0118] S404: If the creation time of the object is not later than the deletion start time, the deletion interface of each object is called to perform the deletion operation.

[0119] Specifically, for objects whose creation time is earlier than or equal to the deletion start time, they are data that already existed in the bucket before the bucket deletion operation was started, and should be deleted according to the established bucket deletion plan. At this time, the system will call the corresponding deletion interface of each object to perform the deletion operation.

[0120] In summary, the function of accurately distinguishing and processing objects created at different times in the same bucket in a targeted manner not only ensures that the bucket deletion operation can proceed normally and effectively cleans up old objects that should be deleted, but also avoids accidental deletion of new objects uploaded during the deletion process, thereby maintaining the integrity of the bucket data and business continuity.

[0121] Figure 3 This is a schematic diagram of the structure of the storage bucket data processing device provided in the embodiment of the present application. Figure 3 As shown, the bucket data processing device includes: a deletion request generation module 301, a bucket location judgment module 302, a tenant information location judgment module 303, a bucket shard number acquisition module 304, a bucket shard index information acquisition module 305, each bucket shard acquisition module 306, a deletion thread determination module 307, an object deletion module 308, a tenant information update module 309 and a bucket name deletion module 310.

[0122] The deletion request generation module 301 is used to generate a deletion request at the deletion start time in response to the tenant inputting the deletion start time and the deletion speed and executing the command to delete the bucket, wherein the deletion request includes the bucket name and the tenant identifier.

[0123] The bucket location determination module 302 is used to determine whether the bucket instance information is in the cache according to the bucket name.

[0124] The tenant information location determination module 303 is used to determine whether the tenant information is in the cache according to the tenant identifier.

[0125] The bucket shard number acquisition module 304 is used to acquire the bucket shard number according to the bucket instance information if both the bucket instance information and the tenant information are in the cache.

[0126] The bucket shard index information acquisition module 305 is used to obtain all bucket shard index information according to the number of shards of the bucket.

[0127] The bucket shard acquisition module 306 is used to acquire each bucket shard according to the bucket shard index information.

[0128] The deletion thread determination module 307 is used to determine all deletion threads according to the deletion speed.

[0129] The object deletion module 308 is used to put all bucket shards into a pre-created thread pool so that all deletion threads complete the deletion operation on all objects under each bucket shard.

[0130] The tenant information updating module 309 is used to update the tenant information and delete the bucket instance information.

[0131] The bucket name deletion module 310 is used to delete a bucket name under a tenant.

[0132] In one possible implementation, the object deletion module 308 is specifically used to traverse all bucket shards; for any bucket shard, put the deletion task into a pre-created thread pool so that all deletion threads in the pre-created thread pool can process it; for any deletion thread, list all objects under each bucket shard; traverse all objects under each bucket shard, and call the deletion interface of each object to perform the deletion operation.

[0133] In one possible implementation, the deletion thread determination module 307 is specifically used to obtain the current amount of available memory and the amount of memory occupied by the pre-created thread pool; determine the effective amount of memory of the pre-created thread pool based on the current amount of available memory and the amount of memory occupied; determine the maximum number of deletion threads based on the effective amount of memory and the average amount of memory occupied by each preset deletion thread; determine the theoretical number of deletion threads based on the deletion speed and the preset average deletion speed of each deletion thread; determine the smaller value between the maximum number of deletion threads and the theoretical number of deletion threads as the number of deletion threads; and allocate all deletion threads in the pre-created thread pool based on the number of deletion threads.

[0134] In a possible implementation, the formula for the deletion thread determination module 307 to determine the maximum number of deletion threads is:

[0135]

[0136] Where N MIndicates the maximum number of deletion threads; M e Indicates the amount of available memory; M t Indicates the average memory usage of each preset deletion thread; Indicates a floor operation.

[0137] In a possible implementation, the formula for determining the theoretical number of threads to be deleted by the deletion thread determination module 307 is:

[0138]

[0139] Where N V Indicates the theoretical number of deletion threads; V d Indicates the deletion speed; V t Indicates the average deletion speed of each preset deletion thread; Indicates a round-up operation.

[0140] In a possible implementation, the bucket data processing device also includes a verification module for traversing each bucket shard to determine whether there is an object in an undeleted operation state; if there is an object in an undeleted operation state, the object is placed in a pre-created thread pool to perform a deletion operation.

[0141] In a possible implementation, the object deletion module 308 is also specifically used to traverse all objects under each bucket shard; for any object, determine whether the creation time of the object is later than the deletion start time; if the creation time of the object is later than the deletion start time, do not delete it; if the creation time of the object is not later than the deletion start time, call the deletion interface of each object to perform the deletion operation.

[0142] The storage bucket data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0143] Figure 4 This is a schematic diagram of the structure of the server provided in the embodiment of the present application. Figure 4 As shown, the server provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the server also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus 404.

[0144] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above method.

[0145] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0146] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0147] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0148] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0149] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0150] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0151] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0152] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0153] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0157] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0158] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for processing bucket data, characterized in that: Applicable to servers, including: In response to the tenant inputting a deletion start time and a deletion speed and executing a command to delete the bucket, generating a deletion request at the deletion start time, wherein the deletion request includes a bucket name and a tenant identifier; According to the bucket name, determine whether the bucket instance information is in the cache; According to the tenant identifier, determining whether the tenant information is in the cache; If the bucket instance information and the tenant information are both in the cache, obtaining the number of shards of the bucket according to the bucket instance information; According to the number of shards of the bucket, obtain all bucket shard index information; Obtain each bucket shard according to the shard index information of each bucket; According to the deletion speed, determining all deletion threads; Put all bucket shards into a pre-created thread pool, so that all deletion threads complete the deletion operation on all objects under each bucket shard; Update the tenant information and delete the bucket instance information; Delete the bucket name under the tenant.

2. The method according to claim 1, characterized in that The step of placing all bucket shards into a pre-created thread pool so that all deletion threads complete the deletion operation on all objects under each bucket shard includes: Traverse all the bucket shards; For any bucket shard, placing the deletion task into the pre-created thread pool so that all deletion threads in the pre-created thread pool process it; For any deletion thread, list all objects under each bucket shard; All objects under the bucket shards are traversed, and the deletion interface of each object is called to perform the deletion operation.

3. The method according to claim 1, characterized in that Determining all deletion threads according to the deletion speed includes: Obtain the current amount of available memory and the amount of memory occupied by the pre-created thread pool; Determine the effective memory amount of the pre-created thread pool according to the current available memory amount and the occupied memory amount; Determine the maximum number of deletion threads according to the effective memory amount and the preset average memory occupied by each deletion thread; Determine the theoretical number of deletion threads according to the deletion speed and the preset average deletion speed of each deletion thread; The smaller value between the maximum number of deletion threads and the theoretical number of deletion threads is determined as the number of deletion threads; All the deletion threads are allocated in the pre-created thread pool according to the number of deletion threads.

4. The method according to claim 3, characterized in that The formula for determining the maximum number of deletion threads is: Where N M Indicates the maximum number of deletion threads; M e Indicates the effective memory amount; M t Indicates the average memory usage of each preset deletion thread; Indicates a floor operation.

5. The method according to claim 3, characterized in that: The formula for determining the theoretical number of deleted threads is: Where N V V represents the theoretical number of deletion threads; d Represents the deletion speed; V t Indicates the average deletion speed of each preset deletion thread; Indicates a round-up operation.

6. The method according to claim 1, characterized in that After all the deletion threads complete the deletion operation on all the objects under the bucket shards, the method further includes: Traversing each of the storage bucket shards to determine whether there is an object in an undeleted operation state; If there is an object in a non-deletion operation state, the object is placed in the pre-created thread pool to perform the deletion operation.

7. The method according to claim 2, characterized in that The traversing all objects under each bucket shard and calling the deletion interface of each object to perform the deletion operation includes: Traverse all objects under each bucket shard; For any object, determine whether the creation time of the object is later than the deletion start time; If the creation time of the object is later than the deletion start time, the object will not be deleted; If the creation time of the object is not later than the deletion start time, the deletion interface of each object is called to perform the deletion operation.

8. A storage bucket data processing device, characterized in that: Applicable to servers, including: a deletion request generating module, configured to generate a deletion request at the deletion start time in response to the tenant inputting a deletion start time and a deletion speed and executing an operation of a command to delete a storage bucket, wherein the deletion request includes a storage bucket name and a tenant identifier; A bucket location determination module, used to determine whether the bucket instance information is in the cache according to the bucket name; A tenant information location determination module, used to determine whether the tenant information is in the cache according to the tenant identifier; A bucket shard number acquisition module, configured to acquire the number of shards of the bucket according to the bucket instance information if both the bucket instance information and the tenant information are in the cache; A bucket shard index information acquisition module, used to obtain all bucket shard index information according to the number of shards of the bucket; Each bucket shard acquisition module is used to acquire each bucket shard according to each bucket shard index information; A deletion thread determination module, used to determine all deletion threads according to the deletion speed; An object deletion module is used to put all bucket shards into a pre-created thread pool so that all deletion threads complete the deletion operation on all objects under each bucket shard; A tenant information update module, used to update the tenant information and delete the bucket instance information; The bucket name deletion module is used to delete the bucket name under the tenant.

9. A server, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images