Object storage life cycle method and system for multi-thread load balancing

Abstract

The invention relates to the technical field of data storage, and particularly provides a multi-thread load balancing object storage life cycle method and system.The method comprises the steps that firstly, object listing screening and object processing of a life cycle are carried out separately, object listing is an object range set according to a life cycle rule, and preliminary listing is carried out on objects conforming to a naming rule; the screening is that the listed objects are screened according to object expiration time set according to a life cycle rule, and the screened objects are to-be-processed objects; the to-be-processed object is an object needing to be deleted or subjected to type conversion; the object enumeration screening and the object processing both adopt a multi-thread mode, and objects are evenly distributed to object processing threads after being screened out. Compared with the prior art, the method has the advantages that idle threads caused by uneven tasks can be avoided, the resource utilization rate is increased, and the processing rate of the life cycle is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and specifically provides a multi-threaded load-balanced object storage lifecycle method and system. Background Art

[0002] With the development of the Internet, the amount of data on the Internet is increasing, and distributed object storage has emerged. As a new network storage architecture, distributed object storage is suitable for the storage of unstructured data such as pictures and videos, and has the characteristics of high-speed direct access to disks for block storage and distributed sharing of file storage. It has been widely developed and applied in the Internet.

[0003] Currently, the distributed object storage system provides a bucket lifecycle function. Through configuration, each object storage gateway can perform bucket lifecycle checks, and for a single bucket, a single-threaded method is used to check and verify all objects in sequence from the index shard 0. This is very time-consuming for a single bucket with hundreds of millions of objects, and it takes weeks to process once. Even if multiple object storage gateways are used simultaneously, the lifecycle checks of multiple buckets are processed at the same time. For a single bucket with hundreds of millions of lifecycle checks, it is still single-threaded and the processing speed is still very slow. Some current multi-threaded processing methods and their shortcomings are introduced as follows:

[0004] (1) A multi-threaded processing scheme based on bucket sharding, where each thread processes objects on certain bucket shards. When the number of shards divided by the number of threads is equal to 1, the optimization degenerates into single-threaded processing and fails to achieve performance optimization. When the index is unevenly distributed, some threads process significantly more objects than other threads, resulting in a waste of thread resources. This scheme fails when the bucket index is not sharded.

[0005] (2) Start multiple threads to process the listed objects: After the objects are listed, they are assigned to each processing thread, which determines whether they need to be deleted or converted. As a result, some queues need to process more objects, while some queues need to process fewer objects, and the function of multiple threads cannot be fully utilized.

[0006] (3) Multiple processing threads share a task queue: When multiple threads obtain tasks, it is easy to cause resource contention. Sequential acquisition or lock acquisition will cause threads to wait and reduce performance.

[0007] Through the above solution, how to ensure that the tasks assigned to each thread in the multi-thread life cycle are as balanced as possible, so as to give full play to the advantages of multi-threading and avoid resource waste, is the key to improving the current object storage life cycle performance. How to improve performance, objects can be deleted or converted in a timely manner according to life cycle rules, thereby reducing the negative impact on user storage space and storage costs is a problem to be solved in this field. Summary of the invention

[0008] The present invention aims at the deficiencies of the above-mentioned prior art and provides a highly practical multi-threaded load-balanced object storage lifecycle method.

[0009] A further technical task of the present invention is to provide a reasonably designed, safe and applicable multi-threaded load-balanced object storage lifecycle system.

[0010] The technical solution adopted by the present invention to solve its technical problem is:

[0011] A multi-threaded load-balanced object storage lifecycle method, firstly, the object listing and screening of the lifecycle objects and the object processing are performed separately, the object listing is an object range set according to the lifecycle rule, and the objects that meet the naming rules are preliminarily listed;

[0012] The screening is to screen the listed objects according to the object expiration time set in the life cycle rule, and the screened objects are the objects to be processed; the objects to be processed are the objects that need to be deleted or converted to different types;

[0013] The object enumeration and screening and object processing are both performed in a multi-threaded manner, and after the objects are screened out, they are evenly distributed to the object processing threads.

[0014] Furthermore, in the object processing thread, first, each object processing thread has its own task queue, which stores the data statistics of the objects to be processed by the corresponding thread, and the task queues have their own unique digital numbers;

[0015] The data statistics include the number of objects to be deleted and the total size of objects to be converted.

[0016] Furthermore, the number of deleted objects is numdel , the formula is:

[0017]

[0018] When an object to be deleted is selected and added to the queue, 1 is added; when the corresponding thread obtains the task and deletes the object, 1 is subtracted;

[0019] The total size of the object to be converted statstrans , the formula is:

[0020]

[0021] When the object to be converted is screened out and added to the queue, the size of the object is added. When the corresponding thread obtains the task and converts the object, the size of the object is subtracted.

[0022] Furthermore, when the enumeration and screening thread screens out the objects to be processed, the number of objects to be deleted or the size of objects to be converted in all task queues is obtained, and then the objects are added to the corresponding task queue according to the principle of minimum task amount.

[0023] Furthermore, according to the principle of minimum task amount, when adding the object to the corresponding task queue;

[0024] Specifically: when objects are filtered out and need to be deleted, the number of objects to be deleted in all queues is obtained, and then the object is added to the task queue with the smallest number of objects to be deleted; when objects are filtered out and need to be converted, the size of objects to be converted in all queues is obtained, and then the object is added to the task queue with the smallest size of objects to be converted.

[0025] Furthermore, initially, the data statistics of each task queue are all initial values ​​0, and at this time, tasks are assigned in sequence according to the task queue numbers. When the data statistics information of multiple task queues are the same, the tasks are assigned to the queue with the smallest number according to the task queue numbers.

[0026] A multi-threaded load-balanced object storage lifecycle system, comprising an object enumeration and screening module, a task allocation module and an object processing module;

[0027] The object enumeration and screening module includes an enumeration and screening submodule and a status update submodule;

[0028] The task allocation module includes a task queue data statistics acquisition submodule, an allocation submodule and a data update submodule;

[0029] The object processing module includes an object deletion submodule, an object conversion submodule and a data update submodule.

[0030] Furthermore, the enumeration and filtering submodule is responsible for obtaining the lifecycle rules of the bucket, and obtaining the object list according to the limiting conditions specified by the rules, and then filtering out the objects that need to be deleted or converted by comparing the last modification time of the objects with the time set in the lifecycle rules;

[0031] The status update submodule is responsible for updating the bucket lifecycle execution status.

[0032] Further, the task queue data statistics acquisition submodule acquires data statistics information of all task queues, and then the allocation submodule adds the task to the task queue with the least tasks to be processed according to the data information of the task queue and the minimum task principle, and then updates the data statistics of the queue to which the task is added through the data update submodule. If the statistics information of multiple task queues is the same, the task is allocated to the queue with a smaller number;

[0033] The data statistical information includes the number of objects to be deleted and the total size of objects to be converted. The number of objects to be deleted is the number of objects to be deleted in the current task queue, and the total size of objects to be converted is the total size of objects to be converted in the current queue. The data statistical information of each task queue is added and deleted as tasks are inserted and consumed in the queue, and is executed by the task allocation module and the object processing module respectively.

[0034] Furthermore, the object processing module includes multiple threads for deleting and converting objects. Each thread obtains the object to be executed in its corresponding task queue, and then assigns it to the object deletion submodule and the object conversion submodule for processing according to the operation type, wherein the object deletion submodule is responsible for deleting expired objects, and the object conversion submodule is responsible for converting expired objects into a specified storage type;

[0035] After the object processing is completed, the data update submodule updates the data statistics information of the task queue corresponding to the thread, updates the task queue information, subtracts 1 from the number of objects to be deleted in the task queue after deleting the object, or subtracts the size of the converted object from the total size of objects to be converted in the task queue after converting the object to another type.

[0036] Compared with the prior art, the object storage lifecycle method and system of the present invention for multi-threaded load balancing have the following outstanding beneficial effects:

[0037] The present invention separates the object enumeration and screening process from the processing process. First, objects to be deleted or converted are screened out according to the life cycle rules, and then the objects are allocated to the task queues through the balanced allocation method provided in the present invention, ensuring that the number of objects to be deleted and the total size of objects to be converted in each queue are kept as balanced as possible.

[0038] In this way, each thread obtains the object in the task queue and performs deletion or type conversion operations. Through the balanced task allocation method, the tasks processed by each thread are kept balanced, avoiding the idleness of some threads caused by uneven tasks, improving resource utilization, and increasing the processing rate of the life cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Attached Figure 1 It is a flowchart of a multi-threaded load-balanced object storage lifecycle method;

[0041] Attached Figure 2 It is a framework diagram of a multi-threaded load-balanced object storage lifecycle system;

[0042] Attached Figure 3 The present invention is a schematic diagram of task queue data update in a multi-threaded load-balanced object storage lifecycle system. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] A best embodiment is given below:

[0045] like Figure 1 As shown, in this embodiment, a multi-threaded load-balanced object storage lifecycle method is firstly performed separately for object listing and object processing in the lifecycle. The object listing is to preliminarily list objects that meet the naming rules according to the object range set by the lifecycle rule, and the screening is to screen the listed objects according to the object expiration time set by the lifecycle rule, and the screened objects are the objects to be processed; the objects to be processed are the objects that need to be deleted or converted to different types.

[0046] Object enumeration and filtering and object processing can both be performed in a multi-threaded manner. After the objects are filtered out, they are evenly distributed to the object processing threads. First, each object processing thread has its own task queue, which stores data statistics of the objects to be processed by the corresponding thread, and the task queues have their own unique digital numbers; data statistics include the number of objects to be deleted and the total size of objects to be converted.

[0047] The number of objects to be deleted is numdel , the formula is:

[0048]

[0049] When an object to be deleted is selected and added to the queue, 1 is added; when the corresponding thread obtains the task and deletes the object, 1 is subtracted;

[0050] The total size of the objects to be converted statstrans , the formula is:

[0051]

[0052] When the object to be converted is screened out and added to the queue, the size of the object is added. When the corresponding thread obtains the task and converts the object, the size of the object is subtracted.

[0053] When the enumeration and screening thread screens out the objects to be processed, the number of objects to be deleted or the size of objects to be converted in all task queues is obtained, and then the objects are added to the corresponding task queue according to the principle of minimum task amount.

[0054] Specifically: when objects are filtered out and need to be deleted, the number of objects to be deleted in all queues is obtained, and then the object is added to the task queue with the smallest number of objects to be deleted; when objects are filtered out and need to be converted, the size of objects to be converted in all queues is obtained, and then the object is added to the task queue with the smallest size of objects to be converted. Initially, the data statistics of each task queue are all initial values ​​0. At this time, tasks are assigned in sequence according to the number of the task queue. Similarly, when the data statistics information of multiple task queues is the same, tasks are assigned to the queue with the smallest number according to the task queue number. By using the principle of minimum task quantity, it is ensured that the number of deletions and the size of conversions in each task queue are as balanced as possible.

[0055] If the object enumeration filters out an object that needs to be converted, and its size is 10M, then all the data statistics of the six task queues are obtained, where the sizes of the objects to be converted are 121M, 110M, 120M, 118M, 115M, and 117 respectively. According to the principle of minimum task volume, the 10M object to be allocated should be allocated to the task queue with a size of 110M. If the object enumeration filters out an object that is expired and needs to be deleted, then all the data statistics of the six task queues are obtained, where the numbers of objects to be deleted are 121, 121, 120, 121, 121, and 121 respectively. According to the principle of minimum task volume, the object to be allocated for deletion should be allocated to the task queue with 120 objects to be deleted.

[0056] Based on the above method, Figure 2-3 As shown, a multi-threaded load-balanced object storage lifecycle system in this embodiment includes an object enumeration and screening module, a task allocation module and an object processing module;

[0057] The object enumeration and filtering module includes the enumeration and filtering submodule and the status update submodule. The enumeration and filtering submodule is mainly responsible for obtaining the lifecycle rules of the bucket, and obtaining the object list according to the limiting conditions specified by the rules, and then filtering out the objects that need to be deleted or converted by comparing the last modification time of the object with the time set in the lifecycle rules; the status update submodule is responsible for updating the execution status of the bucket lifecycle to avoid resource waste and task contention caused by the simultaneous execution of multiple object storage gateways.

[0058] The task allocation module includes a task queue data statistics acquisition submodule, an allocation submodule, and a data update submodule. The task queue data statistics acquisition submodule is used to obtain data statistical information of all task queues, and then the allocation submodule adds tasks to the task queue with the least pending tasks according to the data information of the task queue and the minimum task principle. Then the data statistics of the queue to which the task is added are updated through the data update submodule. If the statistical information of multiple task queues is the same, the task is allocated to the queue with a smaller number.

[0059] The data statistics include the number of objects to be deleted and the total size of objects to be converted. The number of objects to be deleted is the number of objects to be deleted in the current task queue, and the total size of objects to be converted is the total size of objects to be converted in the current queue. The data statistics of each task queue are added and deleted as tasks are inserted and consumed in the queue, which are executed by the task allocation module and the object processing module respectively.

[0060] The object processing module includes an object deletion submodule, an object conversion submodule, and a data update submodule. This module contains multiple threads for deleting and converting objects. Each thread obtains the object to be executed in its corresponding task queue, and then assigns it to the object deletion submodule and the object conversion submodule for processing according to the operation type. The object deletion submodule is responsible for deleting expired objects, and the object conversion submodule is responsible for converting expired objects to the specified storage type.

[0061] After the object processing is completed, the data update submodule updates the data statistics of the task queue corresponding to the thread. To update the task queue information, after deleting the object, the number of objects to be deleted in the task queue is reduced by 1, or after the object is converted to another type, the total size of the objects to be converted in the task queue is reduced by the size of the converted object.

[0062] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0063] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-threaded load-balanced object storage lifecycle method, characterized in that: First, the object listing and screening of the life cycle and the object processing are performed separately. The object listing is an object range set according to the life cycle rule, and the objects that meet the naming rules are preliminarily listed; The screening is to screen the listed objects according to the object expiration time set in the life cycle rule, and the screened objects are the objects to be processed; the objects to be processed are the objects that need to be deleted or converted to different types; The object enumeration and screening and object processing are both performed in a multi-threaded manner, and after the objects are screened out, they are evenly distributed to the object processing threads.

2. According to the multi-threaded load-balanced object storage lifecycle method of claim 1, it is characterized in that: In the object processing thread, first, each object processing thread has its own task queue, which stores the data statistics of the objects to be processed by the corresponding thread, and the task queues have their own unique digital numbers; The data statistics include the number of objects to be deleted and the total size of objects to be converted.

3. The object storage lifecycle method for multi-threaded load balancing according to claim 2, characterized in that: The number of deleted objects is numdel , the formula is: When an object to be deleted is selected and added to the queue, 1 is added; when the corresponding thread obtains the task and deletes the object, 1 is subtracted; The total size of the object to be converted statstrans , the formula is: When the object to be converted is screened out and added to the queue, the size of the object is added. When the corresponding thread obtains the task and converts the object, the size of the object is subtracted.

4. According to the multi-threaded load-balanced object storage lifecycle method of claim 3, it is characterized in that: When the enumeration and filtering thread filters out the objects to be processed, the number of objects to be deleted or the size of objects to be converted in all task queues is obtained, and then the objects are added to the corresponding task queue according to the principle of minimum task amount.

5. The object storage lifecycle method for multi-threaded load balancing according to claim 4, characterized in that: When adding an object to the corresponding task queue according to the principle of minimum task amount; Specifically, when an object is screened out and needs to be deleted, the number of objects to be deleted in all queues is obtained, and then the object is added to the task queue with the smallest number of objects to be deleted; When an object that needs to be converted is screened out, the sizes of the objects to be converted in all queues are obtained, and then the object is added to the task queue with the smallest size of the objects to be converted.

6. The object storage lifecycle method for multi-threaded load balancing according to claim 5, characterized in that: Initially, the data statistics of each task queue are all initial values ​​0. At this time, tasks are assigned in sequence according to the task queue numbers. When the data statistics information of multiple task queues are the same, tasks are assigned to the queue with the smallest number according to the task queue numbers.

7. A multi-threaded load-balanced object storage lifecycle system, characterized in that: It includes object enumeration and screening module, task allocation module and object processing module; The object enumeration and screening module includes an enumeration and screening submodule and a status update submodule; The task allocation module includes a task queue data statistics acquisition submodule, an allocation submodule and a data update submodule; The object processing module includes an object deletion submodule, an object conversion submodule and a data update submodule.

8. The multi-threaded load-balanced object storage lifecycle system according to claim 7, characterized in that: The enumeration and filtering submodule is responsible for obtaining the lifecycle rules of the bucket, obtaining the object list according to the limiting conditions specified by the rules, and then filtering out the objects that need to be deleted or converted by comparing the last modification time of the objects with the time set in the lifecycle rules; The status update submodule is responsible for updating the bucket lifecycle execution status.

9. The multi-threaded load-balanced object storage lifecycle system according to claim 8, characterized in that: The task queue data statistics acquisition submodule acquires data statistics information of all task queues, and then the allocation submodule adds the task to the task queue with the least tasks to be processed according to the data information of the task queue and the minimum task principle, and then updates the data statistics of the queue to which the task is added through the data update submodule. If the statistics information of multiple task queues is the same, the task is allocated to the queue with a smaller number; The data statistical information includes the number of objects to be deleted and the total size of objects to be converted. The number of objects to be deleted is the number of objects to be deleted in the current task queue, and the total size of objects to be converted is the total size of objects to be converted in the current queue. The data statistical information of each task queue is added and deleted as tasks are inserted and consumed in the queue, and is executed by the task allocation module and the object processing module respectively.

10. The multi-threaded load-balanced object storage lifecycle system according to claim 9, characterized in that: The object processing module includes multiple threads for deleting and converting objects. Each thread obtains the object to be executed in its corresponding task queue, and then distributes it to the object deletion submodule and the object conversion submodule for processing according to the operation type. The object deletion submodule is responsible for deleting expired objects, and the object conversion submodule is responsible for converting expired objects to a specified storage type. After the object processing is completed, the data update submodule updates the data statistics information of the task queue corresponding to the thread, updates the task queue information, subtracts 1 from the number of objects to be deleted in the task queue after deleting the object, or subtracts the size of the converted object from the total size of objects to be converted in the task queue after converting the object to another type.

Images