Bucket sharding processing method, apparatus, device, and medium

Abstract

The application discloses a bucket fragmentation processing method and device, equipment and medium, and relates to the technical field of computers. The method comprises the following steps: determining whether there is a to-be-processed bucket fragment in a current task queue; if there is, extracting a current to-be-processed bucket fragment from the current task queue and obtaining a next task queue; performing first preset locking processing on the next task queue and performing re-fragmentation on the current to-be-processed bucket fragment to obtain a plurality of target bucket fragments; performing partitioning on all first objects based on the number of the target bucket fragments to obtain a plurality of groups of second objects, and transferring each group of second objects to a corresponding target bucket fragment; performing first preset unlocking processing on the next task queue, updating the next task queue to the current task queue, and jumping back to the step of determining whether there is a to-be-processed bucket fragment in the current task queue. The re-fragmentation can reduce the negative influence on front-end business.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a bucket fragmentation processing method, apparatus, equipment, and medium. Background Technology

[0002] Currently, distributed object storage systems support dynamic resharding of bucket indexes. When the number of objects in a bucket reaches a threshold, the bucket index is automatically resharded. The indexes of all objects on the original shards are recalculated and mapped to new bucket index shards. The original bucket index shards are then deleted, and the bucket information is updated. However, this approach has significant drawbacks. The default number of bucket shards is 1024, and each bucket shard can hold 100,000 object indexes. When the number of objects in a bucket exceeds 1024 * 100,000, resharding begins. During resharding, the bucket is locked. Then, following the sharding order, all object indexes on each shard are listed, calculated, and a new destination index shard is obtained. The object index is then written to the destination index shard. The bucket is only unlocked after all objects have been migrated. During the locking process, all read and write requests to the bucket are blocked and fail. Through actual testing, resharding a bucket containing 1024 objects into 2048 shards took 56 minutes, during which all front-end services were blocked. Moreover, the longer the number of objects and the longer the migration time, the longer the resharding process lasts, resulting in a greater impact on front-end services and a very poor user experience.

[0003] In summary, how to reduce the negative impact of resharding on front-end business is a problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a bucket sharding processing method, apparatus, device, and medium that can reduce the negative impact of re-sharding on front-end services. The specific solution is as follows:

[0005] In a first aspect, this application discloses a bucket fragmentation processing method, including:

[0006] Determine if there is a bucket shard to be processed in the current task queue. If so, extract the bucket shard to be processed from the current task queue and obtain the next task queue.

[0007] The next task queue is subjected to a first preset locking process, and the current bucket fragment to be processed is re-sharded based on the number of target objects corresponding to the current bucket fragment to be processed, so as to obtain several target bucket fragments.

[0008] Determine the number of shards in the target bucket shard, and partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard;

[0009] Perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there are any unprocessed bucket shards in the current task queue.

[0010] Optionally, before determining whether there are any pending bucket shards in the current task queue, the method further includes:

[0011] The preset task detection module filters out bucket shards to be processed based on the current number of objects and the target number of objects required for each bucket shard in the current bucket.

[0012] Save the fragments of the bucket to be processed to the current task queue.

[0013] Optionally, the bucket sharding method further includes:

[0014] Monitor whether there is a third object that needs to be written to the currently pending bucket shard;

[0015] If it exists, determine whether the current bucket shard to be processed meets the preset object writing requirements. If it does, write the third object to the current bucket shard to be processed.

[0016] Optionally, determining whether the current bucket shard to be processed meets the preset object writing requirements, and if so, writing the third object to the current bucket shard to be processed, includes:

[0017] Determine whether a preset identifier exists on the current bucket shard to be processed; otherwise, write the third object into the current bucket shard to be processed.

[0018] Optionally, after determining whether a preset identifier exists on the current fragment of the bucket to be processed, the method further includes:

[0019] If it exists, perform a second preset locking process on the currently pending bucket fragment, and obtain the locking result of the second preset locking process;

[0020] Determine whether the locking result meets the preset requirements. If it does, write the third object into the current bucket shard to be processed.

[0021] Optionally, the step of determining whether the locking result meets preset requirements, and if so, writing the third object into the current bucket sharding process, further includes:

[0022] Clear the preset identifier on the currently pending bucket fragment.

[0023] Optionally, the process of resharding the current bucket shard based on the target object quantity requirement corresponding to the current bucket shard to obtain several target bucket shards further includes:

[0024] The preset identifier used to characterize the current bucket fragment being fragmented is recorded on the current bucket fragment.

[0025] Secondly, this application discloses a bucket-slicing processing apparatus, comprising:

[0026] The judgment module is used to determine whether there are any pending bucket fragments in the current task queue;

[0027] The bucket shard extraction module is used to extract the current bucket shard to be processed from the current task queue if it exists, and to obtain the next task queue.

[0028] The resharding module is used to perform a first preset locking process on the next task queue, and to reshard the current bucket to be processed based on the number of target objects corresponding to the current bucket to be processed to obtain several target buckets.

[0029] The object transfer module is used to determine the number of shards in the target bucket shard, partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard;

[0030] The update module is used to perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there is a bucket shard to be processed in the current task queue.

[0031] Thirdly, this application discloses an electronic device, including:

[0032] Memory, used to store computer programs;

[0033] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed bucket sharding method.

[0034] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed bucket sharding processing method.

[0035] As can be seen, this application determines whether there is a pending bucket shard in the current task queue. If so, it extracts the current pending bucket shard from the current task queue and obtains the next task queue. It then performs a first preset locking process on the next task queue and re-shards the current pending bucket shard based on the target object quantity requirement corresponding to the current pending bucket shard, to obtain several target bucket shards. It determines the number of shards in each target bucket shard and partitions all first objects in the current pending bucket shard to obtain several groups of second objects corresponding to the number of shards. Each group of second objects is then transferred to the corresponding target bucket shard. Finally, it performs a first preset unlocking process on the next task queue and updates the next task queue to the current task queue, then jumps back to the step of determining whether there is a pending bucket shard in the current task queue. Therefore, this application determines whether there is a pending bucket shard in the current task queue. If so, it extracts the current pending bucket shard from the current task queue, obtains the next task queue, and then performs a first preset locking process on the next task queue. This ensures that only the current pending bucket shard is re-sharded at any given time. The pending bucket shards in the next task queue can continue to perform related tasks for the front-end business. This greatly reduces the blocking time for the front-end business and reduces the re-sharding time. It improves upon the shortcomings of existing technologies that severely block front-end read and write businesses during bucket re-sharding, optimizes the scheme for dynamically increasing bucket shards, provides users with a better user experience, and greatly enhances the market competitiveness of the distributed object storage system. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a bucket fragmentation processing method disclosed in this application;

[0038] Figure 2 This is a flowchart of a specific bucket fragmentation processing method disclosed in this application;

[0039] Figure 3 This is a flowchart of a specific bucket fragmentation processing method disclosed in this application;

[0040] Figure 4 This is a schematic diagram of the structure of a bucket segmentation processing device disclosed in this application;

[0041] Figure 5This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] Currently, distributed object storage systems support dynamic resharding of bucket indexes. When the number of objects in a bucket reaches a threshold, the bucket index is automatically resharded. The indexes of all objects on the original shards are recalculated and mapped to new bucket index shards. The original bucket index shards are then deleted, and the bucket information is updated. However, this approach has significant drawbacks. The default number of bucket shards is 1024, and each bucket shard can hold 100,000 object indexes. When the number of objects in a bucket exceeds 1024 * 100,000, resharding begins. During resharding, the bucket is locked. Then, following the sharding order, all object indexes on each shard are listed, calculated, and a new destination index shard is obtained. The object index is then written to the destination index shard. The bucket is only unlocked after all objects have been migrated. During the locking process, all read and write requests to the bucket are blocked and fail. Through actual testing, resharding a bucket containing 1024 objects into 2048 shards took 56 minutes, during which all front-end services were blocked. Moreover, the longer the number of objects and the longer the migration time, the longer the resharding process lasts, resulting in a greater impact on front-end services and a very poor user experience.

[0044] Therefore, this application provides a bucket sharding processing scheme that can reduce the negative impact of re-sharding on front-end business.

[0045] See Figure 1 As shown in the embodiment of this application, a bucket fragmentation processing method is disclosed, including:

[0046] Step S11: Determine whether there is a bucket shard to be processed in the current task queue. If so, extract the bucket shard to be processed from the current task queue and obtain the next task queue.

[0047] In this embodiment, the background thread can periodically scan the current task queue to determine if there is a pending bucket shard. If it does not exist, it means there is no need to re-shard. If it does exist, the current pending bucket shard is extracted from the current task queue, and the next task queue is retrieved. It is understood that the current task queue may contain only one pending bucket shard or multiple pending bucket shards. For example, it can be set to scan the current task queue every three minutes to check for pending bucket shards.

[0048] Step S12: Perform a first preset locking process on the next task queue, and resegment the current bucket fragment based on the target object quantity requirement corresponding to the current bucket fragment to be processed, so as to obtain several target bucket fragments.

[0049] In this embodiment, a first preset locking process is applied to the next task queue, preventing the extraction of other bucket shards from the queue. This ensures that only one bucket shard (the currently pending bucket shard) can be re-sharded at a time, while other bucket shards can still function normally. For example, in front-end read / write tasks, re-sharding only one bucket shard at a time significantly reduces the impact on front-end services. It can be understood that when the currently pending bucket shard is being re-sharded, a preset identifier is recorded on it. This identifier indicates that the currently pending bucket shard is undergoing re-sharding, preventing it from processing front-end services and blocking writes to front-end applications or related objects.

[0050] In this embodiment, the current bucket shard is resharded based on the target object quantity requirement corresponding to the current bucket shard, to obtain several target bucket shards. For example, based on the corresponding target object quantity requirement, the current bucket shard needs to be resharded into 8 shards, i.e., 8 target bucket shards. During resharding, for example, the system default of 1024 bucket shards is used to evenly divide the object name range.

[0051] Step S13: Determine the number of shards in the target bucket shard, and partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard.

[0052] In this embodiment, the number of shards in the target bucket is determined. For example, if the number of shards is 8, all first objects in the current bucket shard to be processed can be divided into 8 groups. All first objects can be divided into 8 groups, resulting in 8 groups of second objects. Each group of second objects is then transferred to the corresponding target bucket shard. During the transfer process, the container storing shards on the bucket shard is read, and the smallest element greater than or equal to the current object name is found. If the starting object name of the current element is greater than or equal to the object name to be written, and the ending object name is greater than the object name of the second object to be written, then the second object is written to the bucket shard containing this pair of starting objects. Other metadata information of the bucket is updated. The preset identifier on the current bucket shard to be processed, indicating that the current bucket shard to be processed is undergoing resharding, is deleted, and the current bucket shard to be processed is unlocked. The 7 newly generated shards and the start and end element names of the objects on the shards are updated in the container of the bucket metadata information. Simultaneously, the start and end elements of the original shards are also updated. The start and end dates of objects on each bucket shard are unknown, along with their corresponding shard numbers. These are stored as containers in the bucket metadata. The target bucket shard number is determined by comparing the size of the object names. The original method of obtaining the shard number from the object name through hashing has been modified; instead, the start and end object names of objects on each shard are stored as containers in the bucket metadata.

[0053] Step S14: Perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there are unprocessed bucket shards in the current task queue.

[0054] In this embodiment, after each group of second objects is transferred to the corresponding target bucket shard, the next task queue can be subjected to the first preset unlocking process, and the next task queue can be updated to the current task queue. Then, the process jumps back to the step of determining whether there is a bucket shard to be processed in the current task queue. That is, the current bucket shard to be processed can be determined from the bucket shards to be processed, and then several target bucket shards of the current bucket shard to be processed can be obtained.

[0055] As can be seen, this application determines whether there is a pending bucket shard in the current task queue. If so, it extracts the current pending bucket shard from the current task queue and obtains the next task queue. It then performs a first preset locking process on the next task queue and re-shards the current pending bucket shard based on the target object quantity requirement corresponding to the current pending bucket shard, to obtain several target bucket shards. It determines the number of shards in each target bucket shard and partitions all first objects in the current pending bucket shard to obtain several groups of second objects corresponding to the number of shards. Each group of second objects is then transferred to the corresponding target bucket shard. Finally, it performs a first preset unlocking process on the next task queue and updates the next task queue to the current task queue, then jumps back to the step of determining whether there is a pending bucket shard in the current task queue. Therefore, this application determines whether there is a pending bucket shard in the current task queue. If so, it extracts the current pending bucket shard from the current task queue, obtains the next task queue, and then performs a first preset locking process on the next task queue. This ensures that only the current pending bucket shard is re-sharded at any given time. The pending bucket shards in the next task queue can continue to perform related tasks for the front-end business. This greatly reduces the blocking time for the front-end business and reduces the re-sharding time. It improves upon the shortcomings of existing technologies that severely block front-end read and write businesses during bucket re-sharding, optimizes the scheme for dynamically increasing bucket shards, provides users with a better user experience, and greatly enhances the market competitiveness of the distributed object storage system.

[0056] See Figure 2 As shown in the figure, this application discloses a specific bucket fragmentation processing method, including:

[0057] Step S21: Based on the current number of objects and the target number requirement of each bucket shard in the current bucket, the preset task detection module filters out the bucket shards to be processed and saves the bucket shards to be processed to the current task queue.

[0058] In this embodiment, the preset task detection module can periodically update the target object quantity requirement, for example, every 3 minutes. It can also update the current object quantity of a bucket shard to the user quota (i.e., the target object quantity requirement). During this process, it obtains the object quantity of each bucket, which is equivalent to obtaining the total object quantity of all bucket shards within the bucket. Based on the quota of each bucket shard, it can determine whether the object quantity exceeds the maximum threshold for the number of objects on a single shard. If it does, the bucket shard is added to the task queue for processing. Here, the distributed object storage system refers to a distributed storage system for unstructured data objects. A bucket in object storage is a logical container for storing objects available to the user. Each bucket is further divided into several bucket shards, and each bucket shard stores the index of all objects within the bucket. An object can only be mapped to one bucket shard.

[0059] Step S22: Determine whether the pending bucket shard exists in the current task queue. If it exists, extract the current pending bucket shard from the current task queue and obtain the next task queue.

[0060] Step S23: Perform a first preset locking process on the next task queue, and resegment the current bucket fragment based on the target object quantity requirement corresponding to the current bucket fragment to be processed, so as to obtain several target bucket fragments.

[0061] Step S24: Determine the number of shards in the target bucket shard, and partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard.

[0062] Step S25: Perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there are unprocessed bucket shards in the current task queue.

[0063] For a more detailed explanation of the processing steps S22, S23, S24 and S25, please refer to the aforementioned disclosed embodiments, which will not be repeated here.

[0064] Therefore, this application identifies bucket shards that need resharding, saves these bucket shards as pending bucket shards in the current task queue, and if it is subsequently determined that there are pending bucket shards in the current task queue, it selects one pending bucket shard, reshards it, and performs the first locking process on the next task queue. This ensures that only one bucket shard is resharding at any given time, improving upon the shortcomings of existing bucket index resharding processes that severely block front-end business read and write operations, optimizing the dynamic addition of bucket indexes, providing customers with a better user experience, and greatly enhancing the market competitiveness of the storage system.

[0065] See Figure 3 As shown in the figure, this application discloses a specific bucket fragmentation processing method, including:

[0066] Step S31: Determine whether there is a bucket shard to be processed in the current task queue. If so, extract the bucket shard to be processed from the current task queue and obtain the next task queue.

[0067] Step S32: Perform a first preset locking process on the next task queue, and resegment the current bucket fragment based on the target object quantity requirement corresponding to the current bucket fragment to be processed, so as to obtain several target bucket fragments.

[0068] In this embodiment, the target object quantity requirement corresponding to the current bucket shard is obtained. If the target object quantity requirement requires the current bucket shard to be redivided into 10 shards, then the current bucket shard to be processed is redivided into 10 target bucket shards.

[0069] Step S33: Record the preset identifier used to characterize that the current bucket fragment to be processed is being fragmented on the current bucket fragment to be processed.

[0070] In this embodiment, it is monitored whether there is a third object that needs to be written to the current bucket shard; if there is, it is determined whether the current bucket shard meets the preset object writing requirements; if it does, the third object is written to the current bucket shard.

[0071] In this embodiment, determining whether the current bucket shard meets the preset object writing requirements, and if so, writing the third object to the current bucket shard, specifically includes: determining whether there is a preset identifier on the current bucket shard; otherwise, writing the third object to the current bucket shard.

[0072] In this embodiment, after determining whether a preset identifier exists on the current bucket shard to be processed, the method further includes: if it exists, performing a second preset locking process on the current bucket shard to be processed and obtaining the locking result of the second preset locking process; determining whether the locking result meets preset requirements, and if it does, writing the third object to the current bucket shard to be processed. In the third object writing process, firstly, the bucket shard number to which the third object is to be written is determined, and then whether there is a preset identifier for sharding on the shard is read; if there is no preset identifier, the third object is written normally; if there is a preset identifier, it means that the shard is performing a bucket sharding task, and then the second locking process is attempted on the bucket shard. If the locking is successful, it means that the shard resharding has been completed, then the preset identifier for sharding is cleared, then the bucket information is updated, a new bucket shard is obtained again, and the third object is written.

[0073] In this embodiment, the process of determining whether the locking result meets the preset requirements and writing the third object into the current bucket shard if it does not meet the requirements also includes: clearing the preset identifier on the current bucket shard.

[0074] Step S34: Determine the number of shards in the target bucket shard, and partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard.

[0075] In this embodiment, for example, if the target bucket shard has 16 shards, then all the first objects in the current bucket shard to be processed are partitioned to obtain 16 groups of second objects, and then each group of second objects is transferred to the corresponding target bucket shard.

[0076] Step S35: Perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there are unprocessed bucket shards in the current task queue.

[0077] For a more detailed explanation of the processing steps S31, S32 and S35, please refer to the aforementioned disclosed embodiments, which will not be repeated here.

[0078] Therefore, this application modifies the one-time dynamic addition of shards to all buckets under a bucket to adding individual shards to the task queue and performing dynamic shard addition, which greatly reduces the shard fission time, significantly reduces the impact on front-end business, and improves the efficiency of bucket re-sharding.

[0079] See Figure 4 As shown in the figure, this application discloses a bucket sharding processing device, including:

[0080] Module 11 is used to determine whether there are any pending bucket fragments in the current task queue;

[0081] Bucket shard extraction module 12 is used to extract the current bucket shard to be processed from the current task queue if it exists, and to obtain the next task queue;

[0082] The resharding module 13 is used to perform a first preset locking process on the next task queue, and to reshard the current bucket to be processed based on the number of target objects corresponding to the current bucket to be processed to obtain several target buckets.

[0083] The object transfer module 14 is used to determine the number of shards in the target bucket shard, partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard.

[0084] Update module 15 is used to perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there is a bucket shard to be processed in the current task queue.

[0085] In this embodiment, in the bucket shard extraction module 12, if there is a bucket shard to be processed in the current task queue, the bucket shard to be processed is extracted from the current task queue. It can be understood that the next task queue is different from the current task queue. The next task queue does not contain the bucket shard to be processed, and the current task queue may contain one bucket shard to be processed or may contain multiple bucket shards to be processed.

[0086] In this embodiment, in the resharding module 13, a first preset locking process is performed on the next task queue to ensure that no more bucket shards can be extracted from the next task queue at the current time. This ensures that only one bucket shard is resharded at any given time, while other bucket shards can process their corresponding front-end services, reducing blocking and negative impact on front-end services. Before resharding the bucket shard based on the target object quantity requirement corresponding to the bucket shard, a second preset locking process can be performed on the bucket shard, and a preset identifier can be recorded on the bucket shard to indicate that the bucket shard is currently undergoing resharding and cannot process front-end services at this time.

[0087] In this embodiment, if the current bucket fragment to be processed is re-divided into 8 target bucket fragments in the re-sharding module 13, then in the object transfer module 14, all the first objects in the current bucket fragment to be processed are evenly divided into 8 groups to obtain 8 groups of second objects, and then each group of second objects is transferred to the corresponding target bucket fragment. If the current bucket fragment to be processed is re-divided into 10 target bucket fragments in the re-sharding module 13, then in the object transfer module 14, all the first objects in the current bucket fragment to be processed are evenly divided into 10 groups to obtain 10 groups of second objects, and then each group of second objects is transferred to the corresponding target bucket fragment.

[0088] In this embodiment, in the update module 15, after each group of second objects is transferred to the corresponding target bucket shard, the next task queue is subjected to the first preset unlocking process, and the next task queue is updated to the current task queue. It can also be determined again whether there is a bucket shard to be processed in the current task queue.

[0089] As can be seen, this application determines whether there is a pending bucket shard in the current task queue. If so, it extracts the current pending bucket shard from the current task queue and obtains the next task queue. It then performs a first preset locking process on the next task queue and re-shards the current pending bucket shard based on the target object quantity requirement corresponding to the current pending bucket shard, to obtain several target bucket shards. It determines the number of shards in each target bucket shard and partitions all first objects in the current pending bucket shard to obtain several groups of second objects corresponding to the number of shards. Each group of second objects is then transferred to the corresponding target bucket shard. Finally, it performs a first preset unlocking process on the next task queue and updates the next task queue to the current task queue, then jumps back to the step of determining whether there is a pending bucket shard in the current task queue. Therefore, this application determines whether there is a pending bucket shard in the current task queue. If so, it extracts the current pending bucket shard from the current task queue, obtains the next task queue, and then performs a first preset locking process on the next task queue. This ensures that only the current pending bucket shard is re-sharded at any given time. The pending bucket shards in the next task queue can continue to perform related tasks for the front-end business. This greatly reduces the blocking time for the front-end business and reduces the re-sharding time. It improves upon the shortcomings of existing technologies that severely block front-end read and write businesses during bucket re-sharding, optimizes the scheme for dynamically increasing bucket shards, provides users with a better user experience, and greatly enhances the market competitiveness of the distributed object storage system.

[0090] In some specific embodiments, the bucket slicing processing device further includes:

[0091] The bucket sharding filtering unit is used to filter bucket shards to be processed based on the current number of objects and the target number of objects in each bucket shard in the current bucket, through a preset task detection module.

[0092] The storage unit is used to save the fragments of the bucket to be processed to the current task queue.

[0093] In some specific embodiments, the bucket slicing processing device further includes:

[0094] The object writing unit is used to monitor whether there is a third object that needs to be written to the current bucket shard; if there is, it determines whether the current bucket shard meets the preset object writing requirements; if it does, the third object is written to the current bucket shard.

[0095] In some specific embodiments, the object writing unit includes:

[0096] The identifier determination unit is used to determine whether a preset identifier exists on the current bucket fragment to be processed; otherwise, the third object is written to the current bucket fragment to be processed.

[0097] In some specific embodiments, the object writing unit includes:

[0098] The third object writing unit is used to perform a second preset locking process on the current bucket shard to be processed if it exists, and obtain the locking result of the second preset locking process; determine whether the locking result meets the preset requirements, and if it does, write the third object to the current bucket shard to be processed.

[0099] In some specific embodiments, the third object writing unit includes:

[0100] The identifier clearing unit is used to clear the preset identifier on the current bucket fragment to be processed.

[0101] In some specific embodiments, the bucket slicing processing device further includes:

[0102] The identification recording unit is used to record the preset identifier, which indicates that the current bucket fragment to be processed is being fragmented, on the current bucket fragment to be processed.

[0103] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the bucket sharding processing method performed by the electronic device disclosed in any of the foregoing embodiments.

[0104] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0105] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0106] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0107] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the bucket sharding processing method disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0108] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the method steps performed during the bucket sharding process disclosed in any of the foregoing embodiments.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The present invention has provided a detailed description of a bucket segmentation processing method, apparatus, device, and medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for processing bucket segments, characterized in that, include: Determine if there is a bucket shard to be processed in the current task queue. If so, extract the bucket shard to be processed from the current task queue and obtain the next task queue. The next task queue is subjected to a first preset locking process, and the current bucket fragment to be processed is re-sharded based on the number of target objects corresponding to the current bucket fragment to be processed, so as to obtain several target bucket fragments. Determine the number of shards in the target bucket shard, and partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards. Then, transfer each group of second objects to the corresponding target bucket shard. Perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there are any unprocessed bucket shards in the current task queue; The step of transferring each group of the second objects to the corresponding target bucket shard includes: Read the container storing the shards on the target bucket shard, and find the smallest element in the container that is not less than the object name of the second object to be written. If the starting object name of the smallest element is not less than the object name of the second object to be written, and the ending object name of the smallest element is greater than the object name of the second object to be written, then write the second object to be written into the target bucket shard corresponding to the smallest element.

2. The bucket segmentation method according to claim 1, characterized in that, Before determining whether there are any pending bucket shards in the current task queue, the method further includes: The preset task detection module filters out bucket shards to be processed based on the current number of objects and the target number of objects required for each bucket shard in the current bucket. Save the fragments of the bucket to be processed to the current task queue.

3. The bucket segmentation method according to claim 1, characterized in that, Also includes: Monitor whether there is a third object that needs to be written to the currently pending bucket shard; If it exists, determine whether the current bucket shard to be processed meets the preset object writing requirements. If it does, write the third object to the current bucket shard to be processed.

4. The bucket segmentation method according to claim 3, characterized in that, The step of determining whether the current bucket shard to be processed meets the preset object writing requirements, and if so, writing the third object to the current bucket shard to be processed, includes: Determine whether a preset identifier exists on the current bucket shard to be processed; otherwise, write the third object into the current bucket shard to be processed.

5. The barrel segmentation method according to claim 4, characterized in that, After determining whether a preset identifier exists on the current bucket fragment to be processed, the method further includes: If it exists, perform a second preset locking process on the currently pending bucket fragment, and obtain the locking result of the second preset locking process; Determine whether the locking result meets the preset requirements. If it does, write the third object into the current bucket shard to be processed.

6. The bucket segmentation method according to claim 5, characterized in that, The step of determining whether the locking result meets the preset requirements, and if so, writing the third object into the current bucket sharding process, further includes: Clear the preset identifier on the currently pending bucket fragment.

7. The bucket segmentation method according to any one of claims 4 to 6, characterized in that, The process of resharding the current bucket partition based on the target object quantity requirement corresponding to the current bucket partition to obtain several target bucket partitions also includes: The preset identifier used to characterize the current bucket fragment being fragmented is recorded on the current bucket fragment.

8. A barrel-slicing processing device, characterized in that, include: The judgment module is used to determine whether there are any pending bucket fragments in the current task queue; The bucket shard extraction module is used to extract the current bucket shard to be processed from the current task queue if it exists, and to obtain the next task queue. The resharding module is used to perform a first preset locking process on the next task queue, and to reshard the current bucket to be processed based on the target object quantity requirement corresponding to the current bucket to be processed to obtain several target buckets. The object transfer module is used to determine the number of shards in the target bucket shard, partition all the first objects in the current bucket shard to be processed to obtain several groups of second objects corresponding to the number of shards, and then transfer each group of second objects to the corresponding target bucket shard. The update module is used to perform a first preset unlocking process on the next task queue, update the next task queue to the current task queue, and then jump back to the step of determining whether there is a bucket shard to be processed in the current task queue. The bucket segmentation processing device is specifically used for: Read the container storing the shards on the target bucket shard, and find the smallest element in the container that is not less than the object name of the second object to be written. If the starting object name of the smallest element is not less than the object name of the second object to be written, and the ending object name of the smallest element is greater than the object name of the second object to be written, then write the second object to be written into the target bucket shard corresponding to the smallest element.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the bucket sharding method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the bucket sharding method as described in any one of claims 1 to 7.

Images