Automatic barrel fragmentation method and device, medium and equipment

By monitoring the number of object indexes in the bucket in a distributed object storage system and using multiple sub-threads to perform parallel reslicing, the problem of reslicing in the prior art is solved, and business performance and user experience are improved.

CN119002818APending Publication Date: 2024-11-22SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411156194.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When the number of object indexes in the bucket reaches the threshold, the reshaping process will cause long-term locking, blocking write requests, and affecting the performance of front-end business.

Method used

By monitoring the number of object indexes in the bucket, if the preset threshold is reached, multiple subthreads are allocated to copy the original shard object index to the target shard, parallel reshaping is achieved, reducing the time-consuming reshaping.

Benefits of technology

It significantly improves the replication speed of object index in the bucket, reduces the time to reslicing, reduces the negative impact on front-end services, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002818A_ABST
    Figure CN119002818A_ABST
Patent Text Reader

Abstract

The invention provides an automatic barrel fragmentation method and device, a medium and equipment. The method comprises the following steps: monitoring the number of object indexes in each bucket in the process of uploading an object to a distributed object storage system; according to the monitored number of the object indexes in each bucket, determining whether the bucket accords with a preset re-fragmentation rule; if yes, recording the fragmented tasks corresponding to the buckets into a task list; determining an object storage gateway client which is currently in a to-be-distributed task state; allocating each fragmented task in the task list to an object storage gateway client which is currently in a to-be-allocated task state, so that the object storage gateway client which obtains the fragmented task through allocation copies each object index in an original fragment in a bucket corresponding to the fragmented task to a target fragment through a plurality of sub-threads, and re-fragmentation of the barrel is realized. According to the invention, the time consumption of re-fragmentation can be reduced, and the negative influence on the front-end service is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of distributed object storage systems, and in particular to a method and apparatus, medium, and equipment for automatic bucket sharding. Background Art

[0002] With the development of the Internet, the amount of data on the Internet is increasing, and distributed object storage systems have emerged. As a new network storage architecture, distributed object storage systems are suitable for the storage of unstructured data such as pictures and videos, and have the characteristics of fast storage, high-speed direct access to disks, and distributed sharing of file storage. They have been widely developed and applied on the Internet. At present, the distributed object storage system already supports the dynamic re-sharding function of bucket object indexes, that is, when the number of object indexes in a bucket reaches the threshold, it will automatically trigger the re-sharding of the bucket object index, that is, all object indexes on the original shards in the bucket will be recalculated and mapped to the new shards, and then all object indexes on the original shards will be deleted, and finally the bucket information will be updated.

[0003] The existing sharding scheme has great drawbacks. For example, the number of shards in a bucket is 1024, and each shard can carry 100,000 object indexes. When the number of object indexes in the bucket is detected to be greater than 124 million, the bucket will be re-sharded. During the re-sharding period, the bucket must be locked, and then all object indexes on each shard will be listed from 0 to 1023 in the order of the original shards. After a certain allocation calculation, a new destination shard is obtained, and then the object index is written to the destination shard. The bucket will not be unlocked until all object indexes are copied. During the locking period, all write requests to the bucket will be blocked and fail. Moreover, as the number of object indexes in the bucket increases, the longer the transfer time, the longer the duration of re-sharding, the greater the impact on the front-end business, and the worse the user experience.

[0004] From the above, it can be seen that how to reduce the time consumption of re-sharding and thus reduce the negative impact on front-end business is a problem to be solved. Summary of the invention

[0005] In order to solve at least one of the above technical problems, an embodiment of the present invention provides a method and device, medium, and equipment for automatic bucket segmentation.

[0006] According to the first aspect, the automatic bucket segmentation method provided by the embodiment of the present invention includes:

[0007] When objects are uploaded to the distributed object storage system, the number of object indexes in each bucket is monitored;

[0008] According to the number of object indexes in each monitored bucket, determine whether the bucket meets the preset re-sharding rules; if so, record the sharding task corresponding to the bucket into the task list;

[0009] Determine the object storage gateway client that is currently in the state of waiting for task assignment;

[0010] Each sharding task in the task list is assigned to an object storage gateway client that is currently in a task-to-be-assigned state, so that the object storage gateway client that is assigned the sharding task copies each object index in the original shard in the bucket corresponding to the sharding task to the target shard through multiple sub-threads, thereby realizing re-sharding of the bucket corresponding to the sharding task.

[0011] In one embodiment, the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task-to-be-assigned state includes:

[0012] If the number of shard tasks in the task list is less than or equal to the number of object storage gateway clients currently in the state of waiting for task assignment, each shard task is assigned to an object storage gateway client currently in the state of waiting for task assignment, and each assigned task is assigned to a different object storage gateway client; after each shard task assignment, the corresponding relationship between the shard task and the object storage gateway client is recorded in the log pool, and the previous corresponding relationship of the object storage gateway client is deleted;

[0013] If the number of shard tasks in the task list is greater than the number of object storage gateway clients currently in the state of pending task allocation, a shard task is allocated to each object storage gateway client currently in the state of pending task allocation, and the shard tasks allocated to each object storage gateway client are different; when an object storage gateway client completes a shard task and returns to the state of pending task allocation, the remaining unallocated shard tasks are allocated one by one to the object storage gateway clients that have returned to the state of pending task allocation; after each shard task allocation, the corresponding relationship between the shard task and the object storage gateway client is recorded in the log pool, and the previous corresponding relationship of the object storage gateway client is deleted.

[0014] In one embodiment, determining whether the bucket complies with a preset resharding rule according to the number of object indexes in each bucket monitored includes:

[0015] If the preset resharding rule is that the number of object indexes in the bucket reaches a first threshold, then determine whether the number of object indexes in each monitored bucket reaches the first threshold; if it reaches the first threshold, the bucket meets the preset resharding rule; if it does not reach the first threshold, the bucket does not meet the preset resharding rule;

[0016] And / or, if the preset re-sharding rule is that the number of object indexes on a single shard in a bucket reaches a second threshold, then the number of object indexes on a single shard in the bucket is calculated based on the number of object indexes in each bucket and the number of original shards in the bucket, and it is determined whether the number of object indexes on a single shard in the bucket reaches the second threshold; if the second threshold is reached, the bucket complies with the preset re-sharding rule; if the second threshold is not reached, the bucket does not comply with the preset re-sharding rule.

[0017] In one embodiment, before assigning each shard task in the task list to an object storage gateway client that is currently in a task-to-be-assigned state, the method further includes:

[0018] Determine whether the current time is within the scheduled time;

[0019] If it is within the scheduling time, the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state is performed;

[0020] If it is not within the scheduled time, wait until the scheduled time is reached and then execute the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state.

[0021] In one embodiment, the status of each shard in each bucket includes a state of not yet repartitioned, a state of being repartitioned, and a state of repartition completed; wherein:

[0022] The shards that have not yet been repartitioned are newly created shards that have not yet been repartitioned. Read and write operations are not blocked for the shards that have not yet been repartitioned. The shards that are being repartitioned are the original shards that are being repartitioned. Any operations are blocked for the shards that are being repartitioned. The shards that have been repartitioned are the original shards that have been repartitioned. Read and write operations are not blocked for the shards that have been repartitioned. Before the shards that have been repartitioned are cleared, if there are write requests, the write data is written to both the original shard that has been repartitioned and the corresponding target shard.

[0023] In one embodiment, the process of the object storage gateway client copying each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads includes:

[0024] N sub-threads and one main thread are started; wherein each thread is responsible for copying the object indexes of M / N original shards to the target shard; each sub-thread notifies the main thread after completing the work of copying the object index of an original shard to the target shard, so that the main thread allocates the next original shard that needs to be re-divided to the sub-thread, until the sub-thread completes the work of copying the object indexes of M / N original shards to the target shard; M is the number of original shards in the bucket, and N is the number of threads; the process of each sub-thread copying the object index of an original shard to the target shard includes: marking the state of the original shard as being re-divided to block any operation on the original shard; enumerating all the object indexes on the original shard in turn, calculating the ID of the target shard corresponding to each object index, copying each object index to the target shard of the corresponding ID, until all the object indexes on the original shard are copied to the corresponding target shard, marking the state of the original shard as being re-divided to release the blocking operation of the original shard;

[0025] When N sub-threads have successfully completed the repartitioning of all the original shards corresponding to them, they return a message indicating that the bucket has been repartitioned successfully and clean up the original shards.

[0026] In one embodiment, the process in which the object storage gateway client copies each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads also includes: if any sub-thread fails to successfully complete the resharding of any of its own original shards, a message indicating that the resharding of the bucket has failed is returned, and each target shard is cleaned up so that the bucket uses its own original shard to store object indexes.

[0027] According to the second aspect, the automatic barrel slicing device provided by the embodiment of the present invention includes:

[0028] The quantity monitoring module is used to monitor the number of object indexes in each bucket during the process of uploading objects to the distributed object storage system;

[0029] The task recording module is used to determine whether the bucket meets the preset re-sharding rule according to the number of object indexes in each bucket monitored; if so, the sharding task corresponding to the bucket is recorded in the task list;

[0030] A client determination module, used to determine an object storage gateway client that is currently in a state of waiting for task assignment;

[0031] The task assignment module is used to assign each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state, so that the object storage gateway client that is assigned the shard task can copy each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads, thereby realizing re-sharding of the bucket corresponding to the shard task.

[0032] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.

[0033] According to a fourth aspect, an embodiment of the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.

[0034] The automatic bucket sharding method, device, medium, and equipment provided by the embodiment of the present invention monitor the number of object indexes in each bucket during the process of uploading an object to a distributed object storage system. When the bucket is determined to meet the preset re-sharding rule based on the number of object indexes in each bucket monitored, the sharding task corresponding to the bucket is recorded in the task list, and each sharding task in the task list is assigned one by one to the object storage gateway client currently in the state of to-be-assigned tasks. In this way, the object storage gateway client assigned with the sharding task copies each object index in the original shard in the bucket corresponding to the sharding task to the target shard through multiple sub-threads, thereby realizing the re-sharding of the bucket corresponding to the sharding task. It can be seen that using multiple sub-threads to process the re-sharding of multiple shards of a bucket can significantly improve the copying speed of the object index in the bucket and reduce the time consumption of re-sharding, thereby solving the problem of the single-thread execution of bucket re-sharding in the prior art, and the slow processing speed affecting the customer's write operation, thereby reducing the negative impact on the front-end business. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the automatic bucket segmentation method in one embodiment of the present invention;

[0036] Figure 2 The figure is a structural block diagram of an automatic barrel slicing device in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] In a first aspect, an embodiment of the present invention provides a method for automatically sharding a bucket, see Figure 1 The method comprises the following steps S110 to S140:

[0038] S110, in the process of uploading the object to the distributed object storage system, monitoring the number of object indexes in each bucket;

[0039] That is, after an object is uploaded to the distributed object storage system for storage, an object index corresponding to the object will be generated, and the object index will be placed on a shard in a bucket. Therefore, in the process of uploading objects to the distributed object storage system, the number of object indexes in some buckets will change, so the number of object indexes in each bucket is monitored.

[0040] Among them, there will be multiple buckets in the distributed object storage system, each bucket has multiple shards, and each shard stores multiple object indexes.

[0041] S120, determining whether the bucket meets the preset re-sharding rule according to the number of object indexes in each bucket monitored; if so, recording the sharding task corresponding to the bucket into the task list;

[0042] Among them, the preset re-sharding rules can be set as needed.

[0043] In one embodiment, determining whether a bucket complies with a preset resharding rule based on the number of object indexes in each monitored bucket may include: if the preset resharding rule is that the number of object indexes in the bucket reaches a first threshold, determining whether the number of object indexes in each monitored bucket reaches the first threshold; if the first threshold is reached, the bucket complies with the preset resharding rule; if the first threshold is not reached, the bucket does not comply with the preset resharding rule.

[0044] That is to say, if the preset re-sharding rule is that the number of object indexes in the bucket reaches a first threshold, then it is necessary to determine whether the number of object indexes in each bucket reaches the first threshold; if it reaches the first threshold, it is considered to comply with the preset re-sharding rule, otherwise it does not comply with the preset re-sharding rule.

[0045] In one embodiment, determining whether a bucket complies with a preset resharding rule based on the number of object indexes in each bucket monitored may include: if the preset resharding rule is that the number of object indexes on a single shard in the bucket reaches a second threshold, then calculating the number of object indexes on a single shard in the bucket based on the number of object indexes in each bucket and the number of original shards in the bucket, and determining whether the number of object indexes on a single shard in the bucket reaches the second threshold; if the second threshold is reached, the bucket complies with the preset resharding rule; if the second threshold is not reached, the bucket does not comply with the preset resharding rule.

[0046] That is to say, if the preset resharding rule is that the number of object indexes on a single shard in a bucket reaches the second threshold, it can be considered that the average number of object indexes on each shard reaches the second threshold. Therefore, it is necessary to divide the number of object indexes in each bucket by the number of original shards in the bucket to obtain the number of object indexes on a single shard in the bucket, and then determine whether the number of object indexes on a single shard in the bucket reaches the second threshold. If so, it meets the preset resharding rule, otherwise it does not meet the preset resharding rule.

[0047] For example, the preset resharding rule is to reshard when the number of object indexes on a single shard reaches 10w. At this time, a bucket has 512 shards and the number of objects reaches 51.2 million. It can be seen that the number of object indexes per shard reaches 10w. According to the preset resharding rule, the bucket needs to be resharded. At this time, the sharding task of the bucket is recorded, and the bucket information of the bucket and the number of original shards are recorded in the sharding task.

[0048] S130, determining an object storage gateway client that is currently in a state of waiting for task assignment;

[0049] Among them, the object storage gateway client is the radosgw client.

[0050] S140. Assign each shard task in the task list to an object storage gateway client that is currently in a task state to be assigned, so that the object storage gateway client that is assigned the shard task copies each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads, thereby realizing re-sharding of the bucket corresponding to the shard task.

[0051] In one embodiment, the step of assigning each shard task in the task list to an object storage gateway client currently in a task-to-be-assigned state in S140 may include:

[0052] If the number of shard tasks in the task list is less than or equal to the number of object storage gateway clients currently in the state of waiting for task assignment, each shard task is assigned to an object storage gateway client currently in the state of waiting for task assignment, and each assigned task is assigned to a different object storage gateway client; after each shard task assignment, the corresponding relationship between the shard task and the object storage gateway client is recorded in the log pool, and the previous corresponding relationship of the object storage gateway client is deleted.

[0053] For example, if there are three shard tasks in the task list and four object storage gateway clients are currently in the task-to-be-assigned state, the three shard tasks are assigned one by one to the three object storage gateway clients currently in the task-to-be-assigned state. The corresponding relationship between the three shard tasks and the three object storage gateway clients currently in the task-to-be-assigned state is recorded. If one of the three object storage gateway clients currently in the task-to-be-assigned state has just completed other shard tasks and has returned to the task-to-be-assigned state, the previous corresponding relationship of the object storage gateway client is deleted, thereby ensuring the real-time accuracy of the recorded corresponding relationship.

[0054] In one embodiment, the step of assigning each shard task in the task list to an object storage gateway client currently in a task-to-be-assigned state in S140 may include:

[0055] If the number of shard tasks in the task list is greater than the number of object storage gateway clients currently in the state of tasks to be assigned, a shard task is assigned to each object storage gateway client currently in the state of tasks to be assigned, and the shard tasks assigned to each object storage gateway client are different; when an object storage gateway client completes a shard task and returns to the state of tasks to be assigned, the remaining unassigned shard tasks are assigned one by one to the object storage gateway clients that have returned to the state of tasks to be assigned; after each shard task is assigned, the corresponding relationship between the shard task and the object storage gateway client is recorded in the log pool, and the previous corresponding relationship of the object storage gateway client is deleted.

[0056] For example, there are 3 radosgw clients in the state of waiting for task assignment in the cluster, with IDs 1, 2, and 3, and there are 4 buckets that need to be resharded. The sharding task of the first bucket is assigned to the radosgw client with ID 1, and the corresponding relationship is recorded in the log pool, i.e., the log pool; the sharding task of the second bucket is assigned to the radosgw client with ID 2, and the corresponding relationship is recorded in the log pool; finally, the sharding task of the third bucket is assigned to the radosgw client with ID 3, and the corresponding relationship is recorded in the log pool. When a radosgw client completes the sharding task to restore to the state of waiting for task assignment, the sharding task of the fourth bucket is assigned. This ensures that all radosgw clients only reshard one bucket at a time point, and that the load is balanced as much as possible between radosgw clients. When reading and adding and deleting records of radosgw clients executing sharding tasks in the log pool, the corresponding rados objects need to be locked to avoid dirty data.

[0057] After assigning a sharding task to a radosgw client, you need to mark the radosgw client as being in a sharding task and no longer accepting new sharding tasks, and then start formally resharding the corresponding bucket.

[0058] In one embodiment, before allocating each shard task in the task list to an object storage gateway client that is currently in a task to be allocated state, the method may further include:

[0059] Determine whether the current time is within the scheduled time;

[0060] If it is within the scheduling time, the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state is performed;

[0061] If it is not within the scheduled time, wait until the scheduled time is reached and then execute the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state.

[0062] For example, considering the impact of sharding on performance, we choose to re-divide the buckets of all sharding tasks between 0:00 and 4:00 in the morning. At this time, we will check the current time and determine whether it meets the requirements of 0:00 to 4:00 in the morning. If it does, the sharding task will be scheduled to the object storage gateway client; if it does not, it will not be scheduled, but will wait until the execution time to schedule it.

[0063] In one embodiment, the status of each shard in each bucket includes a state of not yet resharded, a state of being resharded, and a state of resharding completed; wherein: a shard in a state of not yet resharded is a newly created shard that has not been resharded, and read and write operations are not blocked for the shard in a state of not yet resharded; a shard in a state of being resharded is an original shard that is being resharded, and any operation is blocked for the shard in a state of being resharded; a shard in a state of resharding completed is an original shard that has been resharded, and read and write operations are not blocked for the shard in a state of resharding completed, and before the shard in a state of being resharded is cleared, when there is a write request, the write data is simultaneously written to the original shard that has been resharded and the corresponding target shard.

[0064] That is to say, the shard status is recorded for each shard in the bucket. There are three types of shard status: not yet repartitioned, being repartitioned, and repartition completed. This status can be recorded on the rados object of the shard. The not yet repartitioned state is the initial state of the shard. At this time, the shard has been created but has not been repartitioned yet, and normal read and write operations can be performed; the being repartitioned state means that the shard is being repartitioned, and the index data stored in the shard is being copied to the new shard. At this time, the write operation is blocked; the repartition completed state means that the index data on the shard has been completely copied to the new shard, and it still retains the original index data in case a shard fails to be repartitioned later and the original shard can be rolled back and used again.

[0065] The relationship between the three sharding states and data writing:

[0066] (1) When there is index information that needs to be written to a shard in the "not yet repartitioned state", the business is not blocked, and the object index is directly written to the shard in this state. The initial state of a shard after creation is the "not yet repartitioned state". A shard in this state will not block any read or write business. Since the data distributed on it has not yet been copied to the new shard, the data on the shard will not be lost regardless of whether the entire bucket sharding is successful or not;

[0067] (2) When there is index information that needs to be written to a shard in the "re-sharding state", the service is temporarily blocked. For a shard in the "re-sharding state", since the index data distributed on it is being copied to the new shard, the write operation cannot determine the progress of its copying. At this time, writing may cause the loss of the data in the new shard. Therefore, it is necessary to wait for the shard in this state to complete the re-sharding before processing. The waiting time is about 4 seconds;

[0068] (3) When there is index information that needs to be written to the shard in the "re-sharding completed state", the information needs to be copied to the corresponding new shard at the same time. Since the data distributed on the shard in the "re-sharding completed state" has been copied to the new shard, the information needs to be written to the shard and the corresponding new shard, so that the new shard is used when the entire bucket is successfully sharded, and the old shard is used when the sharding fails, without losing data.

[0069] In one embodiment, the process of the object storage gateway client copying each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads may include:

[0070] N sub-threads and one main thread are started; wherein each thread is responsible for copying the object indexes of M / N original shards to the target shard; each sub-thread notifies the main thread after completing the work of copying the object index of an original shard to the target shard, so that the main thread allocates the next original shard that needs to be re-divided to the sub-thread, until the sub-thread completes the work of copying the object indexes of M / N original shards to the target shard; M is the number of original shards in the bucket, and N is the number of threads; the process of each sub-thread copying the object index of an original shard to the target shard includes: marking the state of the original shard as being re-divided to block any operation on the original shard; enumerating all the object indexes on the original shard in turn, calculating the ID of the target shard corresponding to each object index, copying each object index to the target shard of the corresponding ID, until all the object indexes on the original shard are copied to the corresponding target shard, marking the state of the original shard as being re-divided to release the blocking operation of the original shard;

[0071] When N sub-threads have successfully completed the repartitioning of all the original shards corresponding to them, they return a message indicating that the bucket has been repartitioned successfully and clean up the original shards.

[0072] That is to say, the resharding process first creates a new shard and stores the index data when resharding. Then the current number of shards of the bucket, that is, the number of original shards, is obtained, and then multiple sub-threads are started according to the number of original shards. Each sub-thread is responsible for the resharding of multiple original shards in turn. The number of sub-threads can be determined according to the execution time, the number of current shards of the bucket, etc. For example, the current bucket has 512 shards, 16 sub-threads are started, and each sub-thread is responsible for the resharding of 32 original shards. Among them, each sub-thread migrates the object index in an original shard to the new shard. When the object index on the responsible original shard is completely copied to the new shard, that is, the target shard, the next shard is resharded. The main thread is responsible for scheduling the shards to be executed to the sub-threads in turn. The sub-threads are responsible for copying the object index on the scheduled shards to the new shards in turn, and after completion, notify the main thread to wait for the next scheduling.

[0073] Among them, the repartitioning work of the child thread is roughly as follows: first, the state of an original shard that it is responsible for is set to the repartitioning state. At this time, the original shard will be in a temporarily locked state and cannot be written or deleted. Then the child thread lists all the object indexes on the original shard in turn, and determines the ID of the new shard corresponding to each object index through certain calculations, and writes the object index to the new shard with the corresponding ID. The above process is executed repeatedly until all the object indexes on the original shard are listed and copied to the corresponding new shard. Finally, the state of the original shard is modified to the "repartitioning completed state". According to the above introduction, the shard in the "repartitioning completed state" can perform read and write operations. When performing write operations, the object index to be written needs to be written to both the original shard and the new shard. After the child thread completes the repartition of a certain original shard, it notifies the main thread, and then waits for the main thread to schedule the IDs of other original shards to be repartitioned. For example, when the number of object indexes on an original shard is 100,000, 1,000 object indexes are listed each time, and then hashed according to the number of new shards and the object name to obtain the ID of the new shard corresponding to each object index, and then the object index is written to the new shard. After 10 cycles, the replication of all object indexes of the original shard can be completed. The repartitioning time of the entire shard is about 4 seconds. When the Ceph underlying layer uses the nvme type interface, the time will be reduced to about 1 second, and users will not perceive the business impact of sharding.

[0074] It is understandable that only after all child threads have successfully completed the repartitioning of all the original shards corresponding to them, the repartitioning of the bucket is considered successful, and only then can the original shards be cleared.

[0075] In one embodiment, the method provided by the embodiment of the present invention may further include:

[0076] If any child thread fails to successfully complete the repartitioning of any of its original shards, a message indicating that the repartitioning of the bucket has failed is returned, and each target shard is cleaned up so that the bucket uses its own original shards to store object indexes.

[0077] It can be seen that as long as the repartitioning of any original shard of any child thread fails, the repartitioning of the bucket is considered to have failed, so it is necessary to roll back to the original shard and use its own original shard to store the object index.

[0078] It is understandable that the object storage bucket sharding function is one of the specific manifestations of the high scalability of the distributed object storage system. By sharding the data and distributing it to different physical nodes, the total storage capacity of the system can be infinitely increased to cope with the large-scale data growth needs. Each shard can independently process read and write requests on different servers or cluster nodes, thereby avoiding single-point overload and improving the throughput and response speed of the overall system. However, user services will be blocked during the sharding process, especially when the number of objects is tens of millions or even hundreds of millions, which may cause long-term bucket reading and writing to be unavailable, and the experience is extremely poor. The embodiment of the present invention opens multiple sub-threads for each sharding task, and concurrently processes the re-sharding of multiple original shards, ensuring that the data of the original shards is quickly migrated to the new shards to reduce the blocking of user services; setting the status on each shard, not blocking the services of the shards that have not yet been re-sharded, and only the shards that are being re-sharded will remain locked for a very short time, which is about 5 seconds.

[0079] It can be seen that the embodiment of the present invention proposes a method for efficient automatic resharding of buckets of object storage. By refining the resharding process, the sharding efficiency can be improved and the blocking of user services can be reduced. Among them, multiple sub-threads are used to perform the resharding of the original shards in parallel to improve the overall processing speed. Moreover, each sub-thread is responsible for maintaining the sharding status of a certain original shard. Only the shard in the "being resharded state" will temporarily affect the service, avoiding the situation where the entire bucket read and write service is blocked when the bucket is resharded, thereby improving the user experience. In other words, by processing the sharding tasks in parallel, the sharding duration is greatly reduced. In addition, the sharding status is refined, and the user service is blocked to a certain extent at the granularity of the shard. Compared with the original sharding solution with bucket as the granularity and single thread, the sharding efficiency is greatly improved, the availability and scalability of the bucket are improved, and the user experience is improved.

[0080] In summary, the general process of the embodiment of the present invention includes: when a bucket needs to be re-sharded, multiple sub-threads are started to copy the object index from the original shard to the target shard, and each sub-thread locks the shard it is responsible for during migration, and only supports query operations at this time; because the sharding granularity of the bucket is fine, each bucket usually contains multiple shards, so each sub-thread only operates the object index on one shard at a time point, so that multiple sub-threads are used to process the re-sharding of multiple shards of a bucket at the same time, that is, with the help of multiple sub-threads, efficient parallel re-sharding of the bucket is achieved, and the speed of copying the object index in the bucket is significantly improved, thereby solving the problem of the single-thread execution of bucket re-sharding in the prior art, and the slow processing speed affects the customer's write operation. Moreover, the blocking granularity is reduced to the shard, that is, only the shard being re-sharded is briefly blocked, instead of blocking all the services of the entire bucket, so that the re-sharding of a single shard can be basically completed in about 5 seconds, and the user service can be reduced from the original blocking of dozens of minutes to less than 5 seconds.

[0081] In a second aspect, an embodiment of the present invention provides an automatic barrel slicing device, see Figure 2 , the device 100 comprises:

[0082] The quantity monitoring module 110 is used to monitor the quantity of object indexes in each bucket during the process of uploading the object to the distributed object storage system;

[0083] The task recording module 120 is used to determine whether the bucket meets the preset re-sharding rule according to the number of object indexes in each bucket monitored; if so, record the sharding task corresponding to the bucket into the task list;

[0084] The client determination module 130 is used to determine the object storage gateway client that is currently in a state of waiting for task assignment;

[0085] The task assignment module 140 is used to assign each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state, so that the object storage gateway client that is assigned the shard task can copy each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads, thereby realizing re-sharding of the bucket corresponding to the shard task.

[0086] In one embodiment, the task allocation module is specifically used to: if the number of shard tasks in the task list is less than or equal to the number of object storage gateway clients currently in the state of waiting for task allocation, then allocate each shard task to an object storage gateway client currently in the state of waiting for task allocation, and each allocated task is allocated to a different object storage gateway client; after each shard task allocation, record the corresponding relationship between the shard task and the object storage gateway client in the log pool, and delete the previous corresponding relationship of the object storage gateway client; if the number of shard tasks in the task list is greater than the number of object storage gateway clients currently in the state of waiting for task allocation, then allocate a shard task to each object storage gateway client currently in the state of waiting for task allocation, and the shard tasks allocated to each object storage gateway client are different; when an object storage gateway client completes a shard task and returns to the state of waiting for task allocation, allocate the remaining unallocated shard tasks one by one to the object storage gateway client restored to the state of waiting for task allocation; after each shard task allocation, record the corresponding relationship between the shard task and the object storage gateway client in the log pool, and delete the previous corresponding relationship of the object storage gateway client.

[0087] In one embodiment, the task recording module is specifically used for: if the preset resharding rule is that the number of object indexes in the bucket reaches a first threshold, then determining whether the number of object indexes in each monitored bucket reaches the first threshold; if the first threshold is reached, the bucket complies with the preset resharding rule; if the first threshold is not reached, the bucket does not comply with the preset resharding rule; and / or, if the preset resharding rule is that the number of object indexes on a single shard in the bucket reaches a second threshold, then calculating the number of object indexes on a single shard in the bucket based on the number of object indexes in each bucket and the number of original shards in the bucket, and determining whether the number of object indexes on a single shard in the bucket reaches the second threshold; if the second threshold is reached, the bucket complies with the preset resharding rule; if the second threshold is not reached, the bucket does not comply with the preset resharding rule.

[0088] In one embodiment, before the task assignment module assigns each shard task in the task list to an object storage gateway client that is currently in a state of waiting for task assignment, it is also used to: determine whether the current time is within the scheduling time; if it is within the scheduling time, execute the step of assigning each shard task in the task list to an object storage gateway client that is currently in a state of waiting for task assignment; if it is not within the scheduling time, wait until the scheduling time is entered and then execute the step of assigning each shard task in the task list to an object storage gateway client that is currently in a state of waiting for task assignment.

[0089] In one embodiment, the status of each shard in each bucket includes a state of not yet resharded, a state of being resharded, and a state of resharding completed; wherein: a shard in a state of not yet resharded is a newly created shard that has not been resharded, and read and write operations are not blocked for the shard in a state of not yet resharded; a shard in a state of being resharded is an original shard that is being resharded, and any operation is blocked for the shard in a state of being resharded; a shard in a state of resharding completed is an original shard that has been resharded, and read and write operations are not blocked for the shard in a state of resharding completed, and before the shard in a state of being resharded is cleared, when there is a write request, the write data is simultaneously written to the original shard that has been resharded and the corresponding target shard.

[0090] In one embodiment, the object storage gateway client specifically includes:

[0091] A thread start module is used to start N sub-threads and a main thread; wherein each thread is responsible for copying the object indexes of M / N original shards to the target shard; each sub-thread notifies the main thread after completing the work of copying the object index of an original shard to the target shard, so that the main thread allocates the next original shard to be re-divided to the sub-thread until the sub-thread completes the work of copying the object indexes of M / N original shards to the target shard; M is the number of original shards in the bucket, and N is the number of threads; the process of each sub-thread copying the object index of an original shard to the target shard includes: marking the state of the original shard as being re-divided to block any operation on the original shard; enumerating all the object indexes on the original shard in turn, calculating the ID of the target shard corresponding to each object index, copying each object index to the target shard of the corresponding ID, until all the object indexes on the original shard are copied to the corresponding target shard, marking the state of the original shard as being re-divided to release the blocking operation of the original shard;

[0092] The first feedback module is used to return a message indicating successful repartition of the bucket and clean up each original shard after all N sub-threads have successfully completed the repartition of all the original shards corresponding to them.

[0093] In one embodiment, the object storage gateway client further includes:

[0094] The second return module is used to return a message of repartition failure of the bucket if any child thread fails to successfully complete the repartition of any of its original slices, and clean up each target slice so that the bucket uses its own original slice to store object indexes.

[0095] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0096] In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0097] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0098] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0099] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0100] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0101] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0102] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0103] In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any embodiment of the specification.

[0104] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0105] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0107] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatic bucket sharding, It is characterized in that include: When objects are uploaded to the distributed object storage system, the number of object indexes in each bucket is monitored; According to the number of object indexes in each monitored bucket, determine whether the bucket meets the preset re-sharding rules; if so, record the sharding task corresponding to the bucket into the task list; Determine the object storage gateway client that is currently in the state of waiting for task assignment; Each sharding task in the task list is assigned to an object storage gateway client that is currently in a task-to-be-assigned state, so that the object storage gateway client that is assigned the sharding task copies each object index in the original shard in the bucket corresponding to the sharding task to the target shard through multiple sub-threads, thereby realizing re-sharding of the bucket corresponding to the sharding task.

2. The method according to claim 1, It is characterized in that The step of allocating each shard task in the task list to an object storage gateway client that is currently in a task-to-be-allocated state includes: If the number of shard tasks in the task list is less than or equal to the number of object storage gateway clients currently in the state of waiting for task assignment, each shard task is assigned to an object storage gateway client currently in the state of waiting for task assignment, and each assigned task is assigned to a different object storage gateway client; after each shard task assignment, the corresponding relationship between the shard task and the object storage gateway client is recorded in the log pool, and the previous corresponding relationship of the object storage gateway client is deleted; If the number of shard tasks in the task list is greater than the number of object storage gateway clients currently in the state of pending task allocation, a shard task is allocated to each object storage gateway client currently in the state of pending task allocation, and the shard tasks allocated to each object storage gateway client are different; when an object storage gateway client completes a shard task and returns to the state of pending task allocation, the remaining unallocated shard tasks are allocated one by one to the object storage gateway clients that have returned to the state of pending task allocation; after each shard task allocation, the corresponding relationship between the shard task and the object storage gateway client is recorded in the log pool, and the previous corresponding relationship of the object storage gateway client is deleted.

3. The method according to claim 1, It is characterized in that The step of determining whether the bucket complies with a preset resharding rule according to the number of object indexes in each bucket monitored includes: If the preset resharding rule is that the number of object indexes in the bucket reaches a first threshold, then determine whether the number of object indexes in each monitored bucket reaches the first threshold; if it reaches the first threshold, the bucket meets the preset resharding rule; if it does not reach the first threshold, the bucket does not meet the preset resharding rule; And / or, if the preset re-sharding rule is that the number of object indexes on a single shard in a bucket reaches a second threshold, then the number of object indexes on a single shard in the bucket is calculated based on the number of object indexes in each bucket and the number of original shards in the bucket, and it is determined whether the number of object indexes on a single shard in the bucket reaches the second threshold; if the second threshold is reached, the bucket complies with the preset re-sharding rule; if the second threshold is not reached, the bucket does not comply with the preset re-sharding rule.

4. The method according to claim 1, It is characterized in that Before allocating each shard task in the task list to an object storage gateway client currently in a task-to-be-allocated state, the method further includes: Determine whether the current time is within the scheduled time; If it is within the scheduling time, the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state is performed; If it is not within the scheduled time, wait until the scheduled time is reached and then execute the step of assigning each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state.

5. The method according to claim 1, It is characterized in that The status of each shard in each bucket includes the status of not yet repartitioned, the status of being repartitioned, and the status of repartition completed. Among them: The shards that have not yet been repartitioned are newly created shards that have not yet been repartitioned. Read and write operations are not blocked for the shards that have not yet been repartitioned. The shards that are being repartitioned are the original shards that are being repartitioned. Any operations are blocked for the shards that are being repartitioned. The shards that have been repartitioned are the original shards that have been repartitioned. Read and write operations are not blocked for the shards that have been repartitioned. Before the shards that have been repartitioned are cleared, if there are write requests, the write data is written to both the original shard that has been repartitioned and the corresponding target shard.

6. The method according to claim 5, It is characterized in that The process of the object storage gateway client copying each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads includes: N sub-threads and one main thread are started; wherein each thread is responsible for copying the object indexes of M / N original shards to the target shard; each sub-thread notifies the main thread after completing the work of copying the object index of an original shard to the target shard, so that the main thread allocates the next original shard that needs to be re-divided to the sub-thread, until the sub-thread completes the work of copying the object indexes of M / N original shards to the target shard; M is the number of original shards in the bucket, and N is the number of threads; the process of each sub-thread copying the object index of an original shard to the target shard includes: marking the state of the original shard as being re-divided to block any operation on the original shard; enumerating all the object indexes on the original shard in turn, calculating the ID of the target shard corresponding to each object index, copying each object index to the target shard of the corresponding ID, until all the object indexes on the original shard are copied to the corresponding target shard, marking the state of the original shard as being re-divided to release the blocking operation of the original shard; When N sub-threads have successfully completed the repartitioning of all the original shards corresponding to them, they return a message indicating that the bucket has been repartitioned successfully and clean up the original shards.

7. The method according to claim 6, It is characterized in that The process in which the object storage gateway client copies each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads also includes: if any sub-thread fails to successfully complete the re-sharding work of any of its own original shards, a message of re-sharding failure of the bucket is returned, and each target shard is cleaned up so that the bucket uses its own original shard to store object indexes.

8. An automatic barrel slicing device, It is characterized in that include: The quantity monitoring module is used to monitor the number of object indexes in each bucket during the process of uploading objects to the distributed object storage system; The task recording module is used to determine whether the bucket meets the preset re-sharding rule according to the number of object indexes in each bucket monitored; if so, the sharding task corresponding to the bucket is recorded in the task list; A client determination module, used to determine an object storage gateway client that is currently in a state of waiting for task assignment; The task assignment module is used to assign each shard task in the task list to an object storage gateway client that is currently in a task to be assigned state, so that the object storage gateway client that is assigned the shard task can copy each object index in the original shard in the bucket corresponding to the shard task to the target shard through multiple sub-threads, thereby realizing re-sharding of the bucket corresponding to the shard task.

9. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.