Method, system and equipment for improving life cycle performance of object storage bucket and medium

Abstract

The invention discloses a method, a system, equipment and a medium for improving the life cycle performance of an object storage bucket, belongs to the technical field of life cycle management, and aims to solve the technical problems of how to improve the execution efficiency of the life cycle of the bucket and reduce the negative influence on the storage space and the storage cost of a user. According to the technical scheme, the method structurally comprises the steps that the life cycle of a barrel is started, and execution of the life cycle of the barrel is started, specifically, an object list is obtained according to the range of the action of a life cycle rule; the objects are distributed to task queues of all the threads in a balanced mode; and each thread obtains the object in the respective task queue and performs corresponding life cycle check.

Description

Technical Field

[0001] The present invention relates to the technical field of lifecycle management, and in particular to a method, system, device and medium for improving the lifecycle performance of an object storage bucket. Background Art

[0002] With the development of the Internet, the amount of data on the Internet is increasing, and distributed object storage has emerged. As a new network storage architecture, distributed object storage is suitable for the storage of unstructured data such as pictures and videos, and has the characteristics of high-speed direct access to disks for block storage and distributed sharing of file storage. It has been widely developed and applied in the Internet.

[0003] Currently, the distributed object storage system provides a bucket lifecycle function. Each object storage gateway performs a bucket lifecycle check, and for a single bucket, a single-threaded method is used to check and verify all objects in sequence from the index shard 0. This is very time-consuming for a single bucket with hundreds of millions of objects, and it takes weeks to process once. Even if multiple object storage gateways are used at the same time, the lifecycle checks of multiple buckets are processed at the same time. For a single bucket with hundreds of millions of lifecycle checks, it is still a single thread and the processing speed is still very slow.

[0004] Currently, there is a solution to perform multi-threaded processing based on bucket shards, where each thread processes objects on certain bucket shards. However, this solution has the following disadvantages:

[0005] ① When the number of shards divided by the number of threads is equal to 1, the optimization degenerates into single-thread processing and cannot achieve performance optimization and improvement;

[0006] ② In the case of uneven index distribution, the number of objects allocated by a thread may be significantly higher, and eventually only this thread is executing, which also degenerates into single-thread processing, resulting in a waste of thread resources;

[0007] ③ If the index data is not in the Ceph system and there is no bucket sharding, optimization cannot be achieved.

[0008] The bucket lifecycle function of object storage is one of the manifestations of its high flexibility and low cost. By configuring lifecycle rules for buckets, it can realize automatic processing of large amounts of data in the bucket, including deleting expired objects that are no longer used and converting files that need to be archived to devices with lower storage costs for retention, avoiding users from performing tedious inspections and conversions. However, since the bucket lifecycle function of some object storage currently uses a single thread, the processing speed is very slow for buckets with hundreds of millions of objects in the bucket. In some optimization schemes, multiple threads are enabled to process the lifecycle of a single bucket, and each thread processes several shards. However, when objects are unevenly distributed on bucket shards and the remainder of the number of bucket shards divided by the number of threads is small, the performance of this scheme degrades severely, and even degenerates into single-thread processing. In addition, it does not support the migration of index objects to other systems outside of Ceph, which results in a large difference between the object processing results seen by users and the lifecycle configuration, and the inability to respond in a timely and fast manner according to the rules, resulting in an increase in the user's storage capacity and cost, making the user experience extremely poor.

[0009] In summary, the prior art has the problem of slow processing speed when single-threaded bucket lifecycle management is performed at the bucket granularity, and the problem of performance degradation when multi-threading is enabled for lifecycle management at the bucket index shard granularity.

[0010] Therefore, how to improve the efficiency of bucket lifecycle execution and reduce the negative impact on user storage space and storage costs is a technical problem that needs to be solved urgently. Summary of the invention

[0011] The technical task of the present invention is to provide a method, system, device and medium for improving the life cycle performance of an object storage bucket, so as to solve the problem of how to improve the execution efficiency of the bucket life cycle and reduce the negative impact on user storage space and storage costs.

[0012] The technical task of the present invention is achieved in the following manner: a method for improving the performance of the life cycle of an object storage bucket, the method is to start the life cycle of the bucket and start executing the life cycle of the bucket, specifically as follows:

[0013] Get a list of objects based on the scope of the lifecycle rule;

[0014] Evenly distribute objects to the task queues of each thread;

[0015] Each thread obtains the objects in its own task queue and performs corresponding life cycle checks.

[0016] Preferably, when there are m objects in the bucket, n threads are set to process the objects, as follows:

[0017] List the objects in the bucket that meet the scope of the lifecycle rule;

[0018] Use hash algorithm to evenly distribute objects to the task queue of each thread;

[0019] Each thread detects that the task queue is not empty, and then obtains the task in the queue;

[0020] Determine whether the object is expired based on its last modification time and current time:

[0021] If it is expired, delete it or convert its type;

[0022] If it has not expired, no processing will be done.

[0023] More optimally, objects are evenly distributed to the task queues of each thread as follows:

[0024] The number of threads in the processing life cycle is num_thread. A public variable var1 is set for each bucket. The initial value is 0. When the first object is allocated, it is increased by 1. The task queue number is obtained through ref_thread = var1% num_thread. The thread number range is 0 <= ref_thread <num_thread;

[0025] The object is added to the task queue of the thread numbered ref_thread. Each subsequent time the object is assigned to a thread task, the public variable var1 is incremented by 1 until the public variable var1 is reset to 0 when the next bucket is processed. The initial value of var1 is 0.

[0026] Preferably, balanced allocation is achieved by counting and dividing by the number of threads to obtain the remainder, or by using a balanced allocation method using a hash of the object name.

[0027] A system for improving lifecycle performance of an object storage bucket, the system comprising:

[0028] Get list module, used to list all or part of the object list in the bucket; the object list is the object range specified in the lifecycle rule;

[0029] The task distribution module is used to distribute all acquired object lists to all thread task queues evenly according to the set rules. Each thread corresponds to its own numbered task list.

[0030] The lifecycle execution module is used for each thread to perform lifecycle checks and deletion or type conversion by obtaining objects in their respective task queues and corresponding rules.

[0031] Preferably, the acquisition list module acquires a list of objects within the scope of application of the lifecycle rule, specifically: if the lifecycle rule applies to objects prefixed with excel / , the acquisition list module acquires all object names prefixed with excel / in the corresponding bucket;

[0032] Among them, the object list is an ordered list or an unordered list.

[0033] Preferably, the task distribution module further uses balanced distribution of tasks, as follows:

[0034] (1) Set a public variable var1 with an initial value of 0;

[0035] (2) When the first object is allocated, add 1, and obtain the thread task queue to which it should be allocated by calculating ref_thread = var1% thread number. Each time an object is allocated, the public variable var1 increases by 1; where ref_thread represents the thread task queue number to which the object is allocated, and the value range is 0 <= ref_thread < thread number; var1 is a public variable, indicating the number of objects allocated; when the object of a new bucket is allocated, it is set to 0 and the count is restarted; the initial value of var1 is 0;

[0036] (3) All objects are evenly distributed to each task queue.

[0037] Preferably, the working process of the life cycle execution module is as follows:

[0038] (1) Respond to an object that has been placed in a task queue and obtain the object in the task queue and the corresponding life cycle rules;

[0039] (2) Calculate whether the corresponding object has expired based on the last modification time of the object and the expiration date configured in the rule:

[0040] If it is expired, it will be deleted or converted to a different type according to the configuration in the rule;

[0041] If it has not expired, no processing will be done;

[0042] (3) According to the logic of steps (1) and (2), each thread processes the objects in its own task queue in turn, as follows:

[0043] ①When the task queue is empty, the thread enters the waiting state;

[0044] ②When the task queue is not empty, execute step (2).

[0045] An electronic device comprising: a memory and at least one processor;

[0046] Wherein, the memory stores computer-executable instructions;

[0047] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for improving the lifecycle performance of the object storage bucket as described above.

[0048] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for improving the lifecycle performance of an object storage bucket as described above is implemented.

[0049] The method, system, device and medium for improving the lifecycle performance of an object storage bucket of the present invention have the following advantages:

[0050] (I) The present invention performs a list operation on the bucket through the main thread to obtain a list of objects to be processed by life cycle verification, and then distributes the objects in the object list to the task queues of each sub-thread in turn and in a balanced manner. Each sub-thread performs a life cycle check and makes corresponding processing, which solves the problem of slow processing speed in the prior art of single-thread execution of bucket life cycle management at the bucket granularity and performance degradation when multi-threading is enabled for life cycle processing at the bucket index shard granularity, thereby improving the execution performance of the bucket life cycle function of the object storage (RADOS Gateway);

[0051] (ii) The present invention improves the efficiency of bucket lifecycle execution, ensuring that no matter how many bucket shards there are, whether the indexes are evenly distributed on the shards, or even if the index data is not in the Ceph system, objects can be deleted or converted in a timely manner according to the lifecycle rules, thereby reducing the negative impact on user storage space and storage costs;

[0052] (III) The present invention opens multiple threads through object granularity, each thread processes the objects in its own task queue, improves processing efficiency through parallelism, and then distributes task queues to each thread in a balanced manner through the main thread, ensuring that multiple threads can be allocated the same number of objects under any circumstances (the difference in the number of objects processed between threads is no more than 1), avoiding degeneration into a single thread, thereby improving life cycle execution efficiency and avoiding performance degradation caused by uneven thread task distribution; at the same time, reducing operation and maintenance costs caused by untimely life cycle response;

[0053] (IV) The main thread of the present invention is used to obtain a list of objects within the scope of the life cycle, and then distributes them evenly to each sub-thread. The sub-thread verifies whether the object is expired according to the task distributed by the main thread. If it is expired, it will be deleted or converted. If it is not expired, it will not be processed. In order to evenly distribute objects to each sub-thread and ensure that the number of objects processed by the sub-thread is as balanced as possible, the present invention provides a balanced distribution method, that is, a method of dividing the number of distributed objects by the number of threads and taking the remainder, to ensure that the number of objects processed by each thread is as balanced as possible, and the difference in the number of objects processed by each thread is not greater than 1. Because multi-threading is at the object level, and through the object balanced configuration method, each thread can maintain simultaneous parallel processing as much as possible, thereby improving the processing capacity of the life cycle, avoiding the decline in processing performance caused by the uneven distribution of each thread, and at the same time avoiding thread waste, improving the utilization rate of system resources, and after the performance is improved, it can respond well to the user's life cycle rules, saving the user's storage cost and reducing the operation and maintenance workload caused by delayed triggering;

[0054] (V) The present invention enables a multi-threaded processing solution at the object granularity and cooperates with the object balanced allocation method to ensure that the difference in the number of objects processed between threads is no more than 1. Compared with single-thread processing, the processing performance is improved. Compared with multi-thread processing at the bucket sharding granularity, the performance degradation caused by the imbalance of the number of objects processed between threads can be avoided. It can even handle the scenario of index migration to a system that does not require sharding.

[0055] (VI) The present invention enables multithreading at the object granularity level, allocating a balanced number of objects to each thread, thus avoiding performance degradation caused by uneven allocation, and improving performance at the same time, thus avoiding performance degradation caused by uneven allocation, and quickly processing according to life cycle rules, thus saving users’ storage costs and avoiding unnecessary operation and maintenance work;

[0056] (VII) The present invention greatly improves the processing efficiency of the lifecycle function, avoids the waste of thread resources during the processing process, and is also conducive to reducing user storage costs and reducing operation and maintenance work caused by untimely lifecycle response. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention is further described below in conjunction with the accompanying drawings.

[0058] Attached Figure 1 A flowchart of a method for improving object storage bucket lifecycle performance;

[0059] Attached Figure 2 A schematic diagram of the structure of a system for improving the performance of object storage bucket lifecycle;

[0060] Attached Figure 3 This is a flowchart of the task distribution module. DETAILED DESCRIPTION

[0061] The method, system, device and medium for improving the lifecycle performance of an object storage bucket of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0062] Embodiment 1:

[0063] As attached Figure 1 As shown, this embodiment provides a method for improving the performance of the life cycle of an object storage bucket. The method is to start the life cycle of the bucket and start executing the life cycle of the bucket, as follows:

[0064] S1. Get a list of objects based on the scope of the lifecycle rule.

[0065] S2, distribute the objects evenly to the task queues of each thread;

[0066] S3. Each thread obtains the objects in its own task queue and performs corresponding life cycle checks.

[0067] In this embodiment, when there are m objects in the bucket, n threads are set to process the objects, as follows:

[0068] (1) List the objects in the bucket that meet the scope of the lifecycle rule;

[0069] (2) Evenly distribute objects to the task queues of each thread through a hash algorithm;

[0070] (3) When each thread detects that the task queue is not empty, it obtains the task in the queue;

[0071] (IV) Determine whether the object is expired based on the last modification time and current time:

[0072] If it is expired, delete it or convert its type;

[0073] If it has not expired, no processing will be done.

[0074] The balanced distribution of objects to the task queues of each thread in step S2 of this embodiment is specifically as follows:

[0075] S201, the number of threads in the processing life cycle is num_thread, a public variable var1 is set for each bucket, the initial value is 0, and 1 is added when the first object is allocated, and the task queue number is obtained through ref_thread=var1%num_thread, and the thread number range is 0<=ref_thread <num_thread;

[0076] S202, adding the object to the task queue of the thread numbered ref_thread, and adding 1 to the public variable var1 each time the object is subsequently assigned to the thread task, until the public variable var1 is reset to 0 when the next bucket is processed; wherein, the initial value of var1 is 0.

[0077] Through this method, objects can be evenly distributed to various threads for execution to the greatest extent, and the difference in the number of objects processed between threads is no more than 1. For example, there are 155 objects, 10 threads for processing life cycles, and the threads and corresponding thread task queues are numbered 0-9. The thread task queue of object 1 is 1%10=1, the thread task queue of object 2 is 1%10=2... The thread task queue of object 9 is 9%10=9, the thread task queue of object 10 is 10%10=0... The thread task queue of object 154 is 154%10=4, and the thread task queue of object 155 is 155%10=5. Therefore, the thread task queues are allocated 15, 16, 16, 16, 16, 15, 15, 15, 15 objects in sequence.

[0078] The balanced allocation in this embodiment is achieved by counting and dividing by the number of threads to obtain the remainder, or by using a balanced allocation method of object name hash.

[0079] Embodiment 2:

[0080] As attached Figure 2 As shown, this embodiment provides a system for improving the lifecycle performance of an object storage bucket, the system comprising:

[0081] Get list module, used to list all or part of the object list in the bucket; the object list is the object range specified in the lifecycle rule;

[0082] The task distribution module is used to distribute all acquired object lists to all thread task queues evenly according to the set rules. Each thread corresponds to its own numbered task list.

[0083] The lifecycle execution module is used for each thread to perform lifecycle checks and deletion or type conversion by obtaining objects in their respective task queues and corresponding rules.

[0084] The acquisition list module in this embodiment acquires a list of objects within the scope of application of the lifecycle rule. Specifically, if the lifecycle rule applies to objects with a prefix of excel / , the acquisition list module acquires all object names with a prefix of excel / in the corresponding bucket.

[0085] Among them, the object list is an ordered list or an unordered list.

[0086] As attached Figure 3As shown, the task distribution module in this embodiment also uses balanced distribution of tasks, which is as follows:

[0087] (1) Set a public variable var1 with an initial value of 0;

[0088] (2) When the first object is allocated, add 1, and obtain the thread task queue to which it should be allocated by calculating ref_thread = var1% thread number. Each time an object is allocated, the public variable var1 increases by 1; where ref_thread represents the thread task queue number to which the object is allocated, and the value range is 0 <= ref_thread < thread number; var1 is a public variable, indicating the number of objects allocated; when the object of a new bucket is allocated, it is set to 0 and the count is restarted; the initial value of var1 is 0;

[0089] (3) All objects are evenly distributed to each task queue.

[0090] The working process of the life cycle execution module in this embodiment is as follows:

[0091] (1) Respond to an object that has been placed in a task queue and obtain the object in the task queue and the corresponding life cycle rules;

[0092] (2) Calculate whether the corresponding object has expired based on the last modification time of the object and the expiration date configured in the rule:

[0093] If it is expired, it will be deleted or converted to a different type according to the configuration in the rule;

[0094] If it has not expired, no processing will be done;

[0095] (3) According to the logic of steps (1) and (2), each thread processes the objects in its own task queue in turn, as follows:

[0096] ①When the task queue is empty, the thread enters the waiting state;

[0097] ②When the task queue is not empty, execute step (2).

[0098] Embodiment 3:

[0099] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0100] Wherein, the memory stores computer-executable instructions;

[0101] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for improving the lifecycle performance of an object storage bucket as described in any one of the present inventions.

[0102] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0103] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0104] Embodiment 4:

[0105] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor, so that the processor executes the method for improving the performance of the life cycle of an object storage bucket in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0106] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0107] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0108] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0109] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving the lifecycle performance of an object storage bucket, characterized in that: This method starts the bucket lifecycle and begins executing the bucket lifecycle, as follows: Get a list of objects based on the scope of the lifecycle rule; Evenly distribute objects to the task queues of each thread; Each thread obtains the objects in its own task queue and performs corresponding life cycle checks.

2. The method for improving the lifecycle performance of an object storage bucket according to claim 1, characterized in that: When there are m objects in the bucket, set n threads to process the objects, as follows: List the objects in the bucket that meet the scope of the lifecycle rule; Use hash algorithm to evenly distribute objects to the task queue of each thread; Each thread detects that the task queue is not empty, and then obtains the task in the queue; Determine whether the object is expired based on its last modification time and current time: If it is expired, delete it or convert its type; If it has not expired, no processing will be done.

3. The method for improving the lifecycle performance of an object storage bucket according to claim 1 or 2, characterized in that: The objects are evenly distributed to the task queues of each thread as follows: The number of threads in the processing life cycle is num_thread. A public variable var1 is set for each bucket. The initial value is 0. When the first object is allocated, it is increased by 1. The task queue number is obtained through ref_thread = var1% num_thread. The thread number range is 0 <= ref_thread <num_thread; The object is added to the task queue of the thread numbered ref_thread. Each subsequent time the object is assigned to a thread task, the public variable var1 is incremented by 1 until the public variable var1 is reset to 0 when the next bucket is processed. The initial value of var1 is 0.

4. The method for improving the lifecycle performance of an object storage bucket according to claim 3, characterized in that: Balanced distribution is achieved by counting and dividing by the number of threads to obtain the remainder, or by using the object name hash method.

5. A system for improving the lifecycle performance of an object storage bucket, characterized in that: The system includes: Get list module, used to list all or part of the object list in the bucket; the object list is the object range specified in the lifecycle rule; The task distribution module is used to distribute all acquired object lists to all thread task queues evenly according to the set rules. Each thread corresponds to its own numbered task list. The lifecycle execution module is used for each thread to perform lifecycle checks and deletion or type conversion by obtaining objects in their respective task queues and corresponding rules.

6. The system for improving the lifecycle performance of an object storage bucket according to claim 5, characterized in that: The acquisition list module acquires a list of objects within the scope of application of the lifecycle rule. Specifically, if the lifecycle rule applies to objects with a prefix of excel / , the acquisition list module acquires all object names with a prefix of excel / in the corresponding bucket. Among them, the object list is an ordered list or an unordered list.

7. The system for improving the lifecycle performance of an object storage bucket according to claim 5 or 6, characterized in that: The task distribution module also uses balanced distribution of tasks, as follows: (1) Set a public variable var1 with an initial value of 0; (2) When the first object is allocated, add 1, and obtain the thread task queue to which it should be allocated by calculating ref_thread = var1% thread number. Each time an object is allocated, the public variable var1 increases by 1; where ref_thread represents the thread task queue number to which the object is allocated, and the value range is 0 <= ref_thread < thread number; var1 is a public variable, indicating the number of objects allocated; when the object of a new bucket is allocated, it is set to 0 and the count is restarted; the initial value of var1 is 0; (3) All objects are evenly distributed to each task queue.

8. The system for improving the lifecycle performance of an object storage bucket according to claim 7, characterized in that: The working process of the life cycle execution module is as follows: (1) Respond to an object that has been placed in a task queue and obtain the object in the task queue and the corresponding life cycle rules; (2) Calculate whether the corresponding object has expired based on the last modification time of the object and the expiration date configured in the rule: If it is expired, it will be deleted or converted to a different type according to the configuration in the rule; If it has not expired, no processing will be done; (3) According to the logic of steps (1) and (2), each thread processes the objects in its own task queue in turn, as follows: ①When the task queue is empty, the thread enters the waiting state; ②When the task queue is not empty, execute step (2).

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for improving the lifecycle performance of an object storage bucket as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method for improving the lifecycle performance of an object storage bucket as described in any one of claims 1 to 4 is implemented.

Images