Object Defragmentation Method, Device, Equipment and Storage Medium

Abstract

The present invention provides an object fragmentation sorting method, device, equipment and storage medium, which are applied to the technical field of object storage, and include: obtaining the number of updates of data in a preset address space in a cache pool in the previous unit time, and determining the space replacement frequency of the cache pool in the previous unit time according to the number of updates in the previous unit time; obtaining the first data volume written through to a data pool by a front-end server in the previous unit time and the total second data volume written, and determining the write-through ratio of the data pool according to the first data volume and the second data volume; determining the target value corresponding to each parameter of an object fragmentation sorting task according to the space replacement frequency and the write-through ratio; the target value of each parameter matches the current write service pressure situation of the front-end server; and performing an object fragmentation sorting task according to the target value corresponding to each parameter currently. By adopting the technical solution of the present invention, the space requirement and / or performance requirement required when the front-end server has a burst task can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of object storage, and particularly to an object fragmentation reorganization method, apparatus, device and storage medium. Background Art

[0002] In various storage application environments, the scenario of using cloud storage devices for storage (which can also be referred to as object-based cloud storage) is increasing. When storing, generally, the front-end server sends files of various sizes to the cloud storage device for storage. Such application scenarios require both a large amount of storage space and high performance requirements. Therefore, the cloud storage device needs to balance multiple requirements such as performance, redundancy security, and low cost. Based on this, in the small file application scenario, the cloud storage system usually adds a cache pool composed of multiple SSD (Solid State Disk) disks, and uses the better write performance of the SSD disks to temporarily store a large number of small file objects quickly sent by the upper-layer front-end server. Then, through an appropriate object merging strategy, small objects of various sizes are merged into larger objects and rewritten into a data pool with a larger capacity and lower cost composed of HDD (Hard Disk Drive) disks to meet the subsequent read and write requirements of the front-end server.

[0003] However, during the process of merging small objects in the cloud storage device, in case of an exception, or when the cloud storage device itself deletes expired data, or when the front-end server may upload files with the same name or delete some of the already written files, etc., object fragments of large objects will be generated in the data pool of the cloud storage device. The space occupied by this part of the object fragments is still considered to be in use by the front-end server, but in fact, it is already invalid occupied space that can be released. To address this problem, it is usually necessary to reorganize the object fragments in the data pool to release the invalid occupied space in the data pool. In the related art, when reorganizing the object fragments in the data pool, generally, a time is set in advance and then the object fragments are reorganized regularly.

[0004] However, the above-mentioned technology may have problems in some cases where it cannot meet the space requirements and / or performance requirements required by the front-end server's sudden tasks. Summary of the Invention

[0005] The present invention provides an object fragmentation reorganization method, apparatus, device and storage medium, which are used to solve the defect that in the prior art, setting a fixed time in advance to regularly reorganize object fragments may not meet the space requirements and / or performance requirements needed for the sudden tasks of the front-end server in some cases, and to achieve determining the parameter values of the object fragmentation reorganization task that match the write service pressure conditions of the data pool and the cache pool through the replacement frequency of the cache pool in the previous unit time and the write-through ratio of the data pool in the previous unit time, and accordingly executing the object fragmentation reorganization task can match the write service pressure conditions of the current data pool and the cache pool, so as to achieve the purpose that when there are sudden tasks on the front-end server, the data pool and the cache pool can also meet the required space requirements and / or performance requirements.

[0006] The present invention provides an object fragmentation reorganization method, including:

[0007] Obtain the number of updates of the data in the preset address space in the cache pool in the previous unit time, and determine the space replacement frequency of the cache pool in the previous unit time according to the number of updates in the previous unit time;

[0008] Obtain the first data volume written through to the data pool by the front-end server in the previous unit time and the second data volume written by the front-end server in total in the previous unit time, and determine the write-through ratio of the data written through to the data pool in the previous unit time according to the first data volume and the second data volume;

[0009] Determine the current target value corresponding to each parameter of the object fragmentation reorganization task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above object fragmentation reorganization task is a task of reorganizing the fragmented objects in the data pool, and the target value of each parameter matches the current write service pressure condition of the front-end server;

[0010] Execute the object fragmentation reorganization task according to the current target value corresponding to each parameter.

[0011] According to the object fragmentation reorganization method provided by the present invention, the above determining the current target value corresponding to each parameter of the object fragmentation reorganization task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool includes:

[0012] Judge whether the write-through ratio of the data pool exceeds the preset ratio threshold range to obtain a first judgment result; and judge whether the space replacement frequency of the cache pool is less than the preset frequency threshold to obtain a second judgment result;

[0013] Determine the current target value corresponding to each parameter of the object fragmentation reorganization task according to the first judgment result and the second judgment result.

[0014] An object fragmentation reorganization method provided by the present invention, the above preset ratio threshold range includes a lower limit value, and determining the target value corresponding to each parameter of the object fragmentation reorganization task according to the first judgment result and the second judgment result includes:

[0015] If the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, then obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the above parameters include at least one of the expiration time of the object fragmentation reorganization task, the threshold for fragmented objects to enter the fragmentation reorganization task queue, the number of threads of the object fragmentation reorganization task, the number of fragmented objects processed in parallel by a single thread, and the running time of the thread.

[0016] Perform at least one of the following adjustment operations on the historical value of the parameter of the object fragmentation reorganization task: reduce the historical value corresponding to the expiration time of the object fragmentation reorganization task to the target value, increase the historical value corresponding to the threshold for fragmented objects to enter the fragmentation reorganization task queue to the target value, increase the historical value corresponding to the number of threads of the object fragmentation reorganization task to the target value, increase the historical value corresponding to the number of fragmented objects processed in parallel by a single thread to the target value, increase the historical value corresponding to the running time of the thread to the target value.

[0017] An object fragmentation reorganization method provided by the present invention, the above preset ratio threshold range includes a lower limit value, and determining the target value corresponding to each parameter of the object fragmentation reorganization task according to the first judgment result and the second judgment result includes:

[0018] If the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value, and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, then obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the above parameters include at least one of the first cache pool usage threshold for triggering data write-through to the data pool and the second cache pool usage threshold for stopping data write-through to the data pool.

[0019] Perform at least one of the following adjustment operations on the historical value of the parameter of the object fragmentation reorganization task: reduce the historical value of the first cache pool usage threshold to the target value, increase the historical value of the second cache pool usage threshold to the target value.

[0020] An object fragmentation reorganization method provided by the present invention, the above preset ratio threshold range includes a lower limit value and an upper limit value, the upper limit value is greater than the lower limit value, and determining the target value corresponding to each parameter of the object fragmentation reorganization task according to the first judgment result and the second judgment result includes:

[0021] If the first judgment result is that the write-through ratio of the data pool is greater than the lower limit value and less than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, then obtain the historical values corresponding to each parameter of the object fragmentation task; the above parameters include at least one of the expiration time of the object fragmentation task and the threshold for a fragmented object to enter the fragmentation task queue;

[0022] Perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation task: increase the historical value corresponding to the expiration time of the object fragmentation task to the target value, and decrease the historical value corresponding to the threshold for a fragmented object to enter the fragmentation task queue to the target value.

[0023] According to an object fragmentation method provided by the present invention, the above preset ratio threshold range includes an upper limit value. According to the first judgment result and the second judgment result, determining the current target value corresponding to each parameter of the object fragmentation task includes:

[0024] If the first judgment result is that the write-through ratio of the data pool is greater than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, then obtain the historical values corresponding to each parameter of the object fragmentation task; the above parameters include at least one of the expiration time of the object fragmentation task, the threshold for a fragmented object to enter the fragmentation task queue, the number of threads of the object fragmentation task, the number of fragmented objects processed in parallel by a single thread, and the running time of the thread.

[0025] Perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation task: increase the historical value corresponding to the expiration time of the object fragmentation task to the target value, decrease the historical value corresponding to the threshold for a fragmented object to enter the fragmentation task queue to the target value, decrease the historical value corresponding to the number of threads of the object fragmentation task to the target value, decrease the historical value corresponding to the number of fragmented objects processed in parallel by a single thread to the target value, and decrease the historical value corresponding to the running time of the thread to the target value.

[0026] According to an object fragmentation method provided by the present invention, the above obtaining the number of updates of the data in the preset address space in the cache pool in the previous unit time includes:

[0027] Obtain the size of the target address space extracted from the cache pool corresponding to the previous unit time;

[0028] In the first address space, the second address space, and the third address space of the cache pool, sequentially select sub-address spaces of the size of the target address space; the above first address space, second address space, and third address space are respectively a section of address space corresponding to the front section, the middle section, and the rear section of the address coding of the cache pool;

[0029] Determine three segments of sub-address space as the preset address space, and count the number of updates of the data in the preset address space in the cache pool in the previous unit time.

[0030] The present invention also provides an object defragmentation device, including the following modules:

[0031] A space replacement frequency determination module, configured to obtain the number of updates of the data in the preset address space in the cache pool in the previous unit time, and determine the space replacement frequency of the cache pool in the previous unit time according to the number of updates in the previous unit time;

[0032] A write-through ratio determination module, configured to obtain the first data volume written through to the data pool by the front-end server in the previous unit time and the second data volume written by the front-end server in total in the previous unit time, and determine the write-through ratio of the data written through to the data pool in the previous unit time according to the first data volume and the second data volume;

[0033] A defragmentation task parameter determination module, configured to determine the target value corresponding to each parameter of the object defragmentation task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above object defragmentation task is a task of defragmenting the fragmented objects in the data pool, and the target value of each parameter matches the current write service pressure situation of the front-end server;

[0034] An object defragmentation module, configured to execute the object defragmentation task according to the target value corresponding to each parameter currently.

[0035] The present invention also provides a cloud storage device, which includes a cache pool, a data pool, a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the object defragmentation method as described in any one of the above is implemented.

[0036] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the object defragmentation method as described in any one of the above is implemented.

[0037] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the object defragmentation method as described in any one of the above is implemented.

[0038] The object fragmentation reorganization method, device, equipment and storage medium provided by the present invention obtain the number of updates of the data within a preset address space in the cache pool in the previous unit time, and determine the space replacement frequency of the cache pool in the previous unit time according to the number of updates in the previous unit time. At the same time, obtain the first data volume written through to the data pool by the front-end server in the previous unit time and the second data volume written in total by the front-end server in the previous unit time, and determine the write-through ratio of the data written through to the data pool in the previous unit time according to the first data volume and the second data volume. Then, determine the target value corresponding to each parameter of the object fragmentation reorganization task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool, and then execute the object fragmentation reorganization task according to the target value of each parameter currently; wherein, the object fragmentation reorganization task is a task of reorganizing the fragmented objects in the data pool, and the target value of each parameter thereof matches the current write service pressure situation of the front-end server. In this method, by calculating the replacement frequency of the cache pool in the previous unit time and the write-through ratio of the data pool in the previous unit time, the current write service pressure situation of the data pool and the cache pool in this cloud storage device and the current write service pressure situation of the front-end server can be intuitively reflected, so as to calculate the parameter values of the object fragmentation reorganization task that match the current write service pressure situation of the cloud storage device / front-end server accordingly. Then, when the object fragmentation reorganization task is executed according to this parameter value, if the front-end server has a sudden task, the cloud storage device can adaptively meet the space requirements and / or performance requirements required by the front-end server. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 It is a schematic diagram of the data flow direction of the cloud storage device.

[0041] Figure 2 It is a schematic diagram of the object fragmentation generation process.

[0042] Figure 3 It is a schematic diagram of the object writing form and the object fragmentation distribution.

[0043] Figure 4 It is a schematic diagram of the object fragmentation reorganization process in the related art.

[0044] Figure 5 It is one of the schematic diagrams of the process of the object fragmentation reorganization method provided by the present invention.

[0045] Figure 6 It is the second flowchart of the object fragmentation reorganization method provided by the present invention.

[0046] Figure 7 It is the flowchart of dynamically adjusting the parameters of the object fragmentation reorganization task provided by the present invention.

[0047] Figure 8 It is the third flowchart of the object fragmentation reorganization method provided by the present invention.

[0048] Figure 9 It is the schematic diagram of cache pool address encoding provided by the present invention.

[0049] Figure 10 It is the structural schematic diagram of the object fragmentation reorganization device provided by the present invention.

[0050] Figure 11 It is the structural schematic diagram of the cloud storage device provided by the present invention. Detailed implementation manners

[0051] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0052] To facilitate understanding of the technical solutions of the present invention, the following will first explain the proprietary terms that may be involved in the embodiments of the present invention.

[0053] Storage pool: A logically storage container where the front-end data is distributed in the storage system. It is usually formed by combining multiple OSDs (Object-Based Storage Systems) according to a certain strategy, including a replica pool and an erasure code pool, which are invisible to front-end users. Among them, the read and write performance of the replica pool is better, the data has multiple backups and the security is also better, but it occupies a large space, has a low space utilization rate and a higher usage cost. The erasure code pool is similar to the RAID (Redundant Arrays of Independent Disks) type of array storage. Through the erasure code algorithm, it ensures that the data has a certain redundancy, taking into account the performance requirements and the redundancy requirements of data security. It can improve the space utilization rate and reduce the usage cost.

[0054] Object cloud storage: Also known as object-based cloud storage, it usually treats each small file or shard of a large file as an object. After each object is written into the storage system, the address space of all objects in the storage pool is flatly distributed, and there is no hierarchical relationship between objects.

[0055] Object fragments: In the small file application scenario of object cloud storage, a cache pool composed of multiple SSD disks is usually added. This cache pool usually uses a replica pool (default three replicas). Taking advantage of the better write performance of SSD disks than HDD disks, it temporarily meets the write requirements of rapid and large-scale distribution from the front-end server. Then, an object merging strategy is formulated. When the merging conditions are met, small objects of various sizes are merged into larger objects and written into a larger-capacity data pool (default erasure code pool) composed of HDD disks. These large objects generated by merging will have object fragments when encountering abnormal merging processes, upper-layer overwriting (duplicate objects of different or the same sizes), expiration deletion, user deletion, etc. These fragments occupy real space and are not released, and cannot be reused when the front-end writes again, resulting in a large amount of space waste. Therefore, a task for organizing object fragments is initiated regularly.

[0056] Bucket: Also known as a storage bucket, it is the most basic logical unit visible to users for storing objects. Each user can have one or more Buckets to store objects of various sizes.

[0057] The following explains the relevant technical background of the present invention.

[0058] In various storage application environments, the scenarios of using cloud storage devices are increasing. For example, in large hospitals, there are many departments conducting examinations on different patients simultaneously. When each patient undergoes examinations such as B-ultrasound, X-ray, CT (Computed Tomography), and nuclear magnetic resonance, hundreds or thousands of high-definition images will be generated. And each outpatient department will generate a large number of various medical record reports, examination reports, etc., all of which will be written into the database, thus generating a large number of database files. Another example is applications in public security or traffic police. During the peak traffic period when the traffic flow is large, a large number of cameras on each road will generate a vast amount of passing vehicle images. All passing vehicle images need to be stored, and then a large number of images need to be extracted to determine whether there are real violations or illegalities. These analyses may be completed manually or by AI (Artificial Intelligence) software. In the above various application scenarios, whether it is picture files or database files, ultimately, the front-end server distributes various files of different sizes to the cloud storage system. Such application scenarios require both a large amount of storage space and high performance requirements. Therefore, cloud storage devices need to balance multiple requirements such as performance, redundancy security, and low cost. Based on this, in the small file application scenario, the cloud storage system usually adds a cache pool composed of multiple SSD disks. Utilizing the better write performance of SSD disks, it temporarily stores a large number of small file objects quickly distributed by the upper-layer front-end server. Then, through a suitable object merging strategy, small objects of various sizes are merged into larger objects and rewritten into a data pool with a larger capacity and lower cost composed of HDD disks to meet the subsequent read and write requirements of the front-end server.

[0059] When initializing the object cloud storage system / cloud storage device, referring to Figure 1 the data flow schematic diagram of the cloud storage device shown, generally, a data pool (HDD) and a cache pool (SSD) will be created respectively, and a small object threshold N (such as 256k, 1M, 4M, etc.) will be defined. When the size of a single data object is greater than N (i.e., a large object), it is directly written into the data pool composed of HDD disks; when the data object is less than N, it is first written into the cache pool composed of SSD disks, and then it can return to the front-end that the write has been successful, thus meeting the high-performance write requirements of the front-end. At the same time, when initializing the cloud storage system, a trigger strategy for merging small objects into large objects and an object fragmentation reorganization strategy for large objects will be preset. Referring to Figure 2Schematic diagram of the object fragmentation generation process. When the cloud storage system receives an object written from the front end, it first checks whether it is a small object. If it is a large object, it is directly written to the data pool; otherwise, it checks whether the cache pool is full. If the cache pool is not full, it is written to the cache pool; otherwise, it is written to the data pool. When the usage rate of the cache pool reaches the set threshold or the number of objects written to a single Bucket reaches the threshold, small objects will be merged into large objects. If the merge is successful, the generated large object is written to the data pool, and the small objects in the cache pool are deleted; if an exception occurs during the merge, the large object is added to the task queue for object fragmentation sorting and waits for the arrival of the fragmentation sorting cycle. During the operation of the system, in addition to the front-end server continuously writing new data, the storage system itself will have an expiration deletion program to delete expired data. The front end may also upload files with the same name or delete some of the written files, etc., which will all generate object fragments of large objects. The space occupied by this part of the object fragments is still considered to be in use from the perspective of the front end, but in fact, it is invalid occupied space that can be released.

[0060] Continue to refer to Figure 2 and Figure 3 Schematic diagram of the object writing form and object fragmentation distribution shown. Assume that the large object threshold is greater than 4M. Then all objects less than or equal to 4M will be judged as small objects and will try to be written to the cache pool first, while objects greater than 4M will be judged as non-small objects and written to the data pool. When the merge condition is met to generate a large object, each merged large object can be limited to a maximum of 16M, for example. At this time, object merging can be performed to generate a large object, and after the generation is successful, the small objects in the cache pool are deleted and the corresponding space is released. When the cache pool is not full, only 16M large objects are stored in the data pool. If the cache pool is full, in addition to 16M large objects, the data pool will also have objects of various sizes (such as small objects like pictures and databases) written through. When a large number of object fragments are caused by merge failures, various deletions, front-end overwriting, etc., it will result in a large amount of object fragment space of various sizes in the data pool. These spaces still appear to the front-end users as one by one 16M spaces that are actually occupied, but in fact, a large number of object fragments are in a state where they can be rewritten after being sorted. The sizes of these fragments may vary due to different deletion amounts, merge failures, front-end overwriting, and other reasons, such as 6M, 9M, 16M, etc. That is, the minimum size of these fragment spaces is the size of a small object, and the maximum size is the space size of a 16M large object.

[0061] For the above-generated object fragments, the cloud storage device can execute the object fragmentation sorting task to sort these object fragments and release the corresponding space. During the system initialization process before the cloud storage device executes the object fragmentation sorting task, refer to Figure 4In the schematic diagram of the object fragmentation reorganization process in the related art shown, a fragmentation expiration time T (i.e., the fragmentation reorganization period) is set, an object space utilization rate threshold D (default 50%, only objects with a utilization rate lower than D will enter the task queue to be reorganized) that triggers object fragments to enter the task queue to be reorganized, and the running time t of the object fragmentation reorganization task thread. If the time T has been reached, check whether the space utilization rate of each object fragment is lower than D. If it is lower than D, add it to the task queue. If the task queue is not empty, the management process of each node will start a thread for the object fragmentation reorganization task and start reorganizing each object fragment in the same Bucket in the task queue one by one. After all the object fragments in this Bucket have been reorganized, continue to process the object fragments in the next Bucket. Since for a storage system (such as a cloud storage device), quickly responding to the read and write requests of front-end users is the highest-priority task, the priority of the object fragmentation reorganization task is placed at a relatively low level in the system, which by default limits the speed of fragmentation reorganization. If the number of fragments is large, and the cache pool is continuously merging to generate large objects and writing them into the data pool, or the cache pool is full and front-end user data starts to be directly written through to the data pool, or front-end users are deleting old data, or the storage system itself is performing expiration deletion and other various read and write operations, it will all affect the speed of the object fragmentation reorganization task and the scheduling priority of each task, resulting in a very long fragmentation reorganization time. It may be that the running time t of the fragmentation reorganization thread has been reached (i.e., the fragmentation reorganization has timed out), and the object fragmentation reorganization task still has not been completed. Then, the current reorganization task will be forcibly stopped and left to restart a new round of reorganization tasks in the next cycle. If the fragmentation reorganization has not timed out, continue the fragmentation reorganization until it is completed.

[0062] However, in the above related technologies, all the parameters for defragmentation are set to a relatively small fixed value at one time during system initialization. Regardless of how the front-end service pressure changes and how many object fragments in the data pool enter the task queue to be defragmented, only one defragmentation thread is generated by one management process, and only one fragmented object in one Bucket is processed per unit time. Although this is beneficial for not affecting the response speed of business writes when the service pressure is high, it cannot quickly complete the object fragmentation task during idle times, nor can it adjust the speed of object fragmentation processing in real time to release the available space occupied by object fragments more quickly. If the system fails to complete the object fragmentation task as soon as possible during the idle period, and then encounters an increase in traffic pressure from the front-end server later, it will further affect the speed of the object fragmentation task. Even when a large number of small files are written through due to a sudden front-end pressure causing the cache pool performance to be unable to meet the requirements, the data pool cannot provide enough space and timely response speed to handle the temporary sudden write requirements. For example, assume that the total available capacity of the data pool is 10T. Theoretically, it can store 10 * 1024 * 1024M / 16M = 655360 large objects of 16M. According to the solution in the related technology, with the fixed single-threaded and single-object-fragment processing method, part of the object fragments are not processed in time. The space occupied by the fragmented objects is 1000 * 6M, 500 * 16M, and 200 * 9M. The total number of fragmented objects is 1700, accounting for 0.25% of the total number of objects 655360, and a total of 15.43G (1000 * 6 + 500 * 16 + 200 * 9 = 15800M) of space is invalidly occupied, accounting for 0.15% of the total available space. At a certain moment, the front-end server suddenly has a large number of burst write requirements for small objects, and the size of each small object is 16k. Then, because the object fragmentation task fails to be completed in time, 15800 * 1024k / 16k = 1011200 small objects cannot be written in time.

[0063] It can be seen that the related technologies may have problems in not being able to meet the space requirements and / or performance requirements needed for the sudden tasks of the front-end server in some cases. Based on this, the embodiments of the present invention provide an object fragmentation method, device, equipment, and storage medium, which can solve the above technical problems.

[0064] The following combines Figures 5 - 9 to describe the object fragmentation method of the embodiments of the present invention.

[0065] It should be noted that the execution subject of the embodiments of the present invention can be an object fragmentation device, or it can also be a cloud storage device or a cloud storage system. The following embodiments will take the cloud storage device as the execution subject for illustration.

[0066] Figure 5It is one of the schematic flowcharts of the object fragmentation sorting method provided by the present invention. As Figure 5 shown, the method includes the following steps:

[0067] S102, obtain the update times of the data in the preset address space in the cache pool within the previous unit time, and determine the space replacement frequency of the cache pool within the previous unit time according to the update times within the previous unit time.

[0068] Among them, a cache pool and a data pool can be preset in the cloud storage device to store data or data objects. The write performance of the cache pool is better than that of the data pool, but the memory space of the cache pool is smaller than that of the data pool. It can be used to temporarily store the data sent by the front-end server, while the data pool has a lower cost and a larger memory space, so more data can be stored. In this embodiment, the cache pool is mainly used to store the data of small objects, that is, the data of objects that occupy less memory space, and the data pool is mainly used to store the data of large objects, that is, the data of objects that occupy more memory space. When the memory space of the cache pool is insufficient, the data of small objects can also be stored appropriately.

[0069] Since there may be object fragmentation in the data pool during the process of storing object data for various reasons, it is necessary to start an object fragmentation sorting task to sort the fragments and release some memory space. In the related art, the method of starting the object fragmentation sorting task at a set time may not meet the performance requirements and space requirements when the front-end server has sudden tasks in some cases. Therefore, an improved way to execute the object fragmentation sorting task is proposed in this embodiment.

[0070] Specifically, in this step, the cloud storage device can first determine the previous unit time, and at the same time determine the preset address space of the cache pool, and then can count the update times of the data in the preset address space in the cache pool within the previous unit time. Here, the update times of the data refer to the sum of the number of times from idle to having data and the number of times from having data to idle in the preset address space within the previous unit time. The previous unit time can be a unit time before the current moment, and the length of the previous unit time can be specifically set according to the actual situation. For example, it can be 30 minutes, 1 hour, 24 hours, 2 days, etc. The preset address space of the cache pool refers to the preset memory space of the cache pool, which can be the entire address space corresponding to the cache pool address code, or a segment of the address space selected from the entire address space, or a combination of multiple segments of the address space selected from the entire address space, or other situations. Here, no specific limitation is made. In short, the update times of the data in the preset address space within the previous unit time can be counted.

[0071] After obtaining the number of data updates in the preset address space of the cache pool in the previous unit time, the number of updates can be directly used as the space replacement frequency of the cache pool in the previous unit time; alternatively, the number of data updates in the preset address space of the cache pool in multiple previous unit times (for example, counted separately one day or two days before the current moment) can be averaged or summed to determine the space replacement frequency of the cache pool in the previous unit time; or other methods can also be used. In short, as long as the space replacement frequency of the cache pool in the previous unit time can be counted, and this space replacement frequency can be denoted as P.

[0072] S104. Obtain the first data volume written through to the data pool by the front-end server in the previous unit time and the second data volume written in total by the front-end server in the previous unit time, and determine the write-through ratio of the data written through to the data pool in the previous unit time according to the first data volume and the second data volume.

[0073] In this step, the cloud storage device can count the data volume written through to the data pool by the front-end server in the same previous unit time as the cache pool, denoted as the first data volume. Situations such as insufficient memory space in the cache pool or large object data generated by the front-end server may cause the front-end server to directly write the data to the data pool instead of the cache pool when writing data, that is, write through to the data pool.

[0074] At the same time, the cloud storage device can count the total data volume written by the front-end server to the data pool and the cache pool in the same previous unit time, denoted as the second data volume.

[0075] After that, the ratio of the first data volume to the second data volume can be directly calculated, and the obtained ratio is directly used as the write-through ratio of the data pool in the previous unit time. Or, the first data volume and the second data volume in multiple previous unit times can also be counted, and after averaging, the final first data volume and second data volume are obtained, and then the ratio operation is performed to obtain the write-through ratio of the data pool in the previous unit time. Or other methods can also be used for calculation, which is not specifically limited here, as long as the write-through ratio of the data pool in the previous unit time can be calculated, and this write-through ratio can be denoted as Q.

[0076] S106. Determine the target value corresponding to each parameter of the object fragmentation and reorganization task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above object fragmentation and reorganization task is a task for reorganizing fragmented objects in the data pool, and the target value of each parameter matches the current write service pressure situation of the front-end server.

[0077] Among them, during the operation of the cloud storage device, system resources (including CPU resources, memory resources, cache pool space, data pool space, etc.) need to respond in real time to various sudden read and write tasks from many front-end servers at any time. The occupation of system resources by these tasks includes multiple subtasks such as calculation, scheduling, and waiting. Calculating the capacity required when the front-end server issues data reading or writing to capacity spaces such as memory, cache pool, and data pool, calculating the space replacement frequency P of the cache pool, calculating the write-through ratio Q, etc. all belong to calculation subtasks; there are usually dozens or hundreds of different task processes in the cloud storage device (such as various writing and reading, various timing tasks, various exception handling including disk error handling, etc.) running simultaneously. The priority of a certain process to obtain resources and run at different times is constantly changing. The CPU needs to timely perform scheduling tasks such as applying for and allocating system resources according to system requirements; after a certain task process obtains system resources, it completes data writing or reading. After waiting for the data writing or reading to be completed, it returns the completion result to the front-end server. The time during this period is all waiting time. Based on this, in this embodiment, it is proposed to realize adjusting the start conditions, sorting speeds, sorting times, and other parameter values of the object fragmentation sorting task in a timely manner according to different service pressures through the space replacement frequency of the cache pool in the previous unit time and the write-through ratio of the data pool in the previous unit time, so as to complete the object fragmentation sorting task in a timely manner and respond in a timely manner to the space and performance requirements of the sudden write tasks of the front-end server.

[0078] Specifically, in this step, the object fragmentation sorting task can correspond to a task queue. This task queue can include the object fragments to be sorted in the data pool. Generally, the data pool stores object data in a space of a fixed size. For example, an object's data is stored in a 16M space. Then, the memory space occupied by each object fragment in the task queue is generally less than 16M. In the related art, when performing the object fragmentation sorting task, the expiration time, execution time, etc. of the object fragmentation sorting task are generally set to fixed values during initialization, which causes the data pool to be unable to flexibly respond to the performance and space requirements of the sudden write tasks of the front-end server.

[0079] Based on this, after obtaining the space replacement frequency of the cache pool in the previous unit time and the write-through ratio of the data pool in the previous unit time, this embodiment can dynamically adjust the values of the parameters related to the object fragmentation task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool, and obtain the target value corresponding to each parameter, that is, adjust the value of each parameter from its historical value to the target value. Among them, the space replacement frequency of the cache pool in the previous unit time and the write-through ratio of the data pool in the previous unit time can comprehensively reflect the current write operation situation of the front-end server. For example, if the space replacement frequency of the cache pool is low and the write-through ratio of the data pool is also low, it indicates that the current write operation of the front-end server is less, the pressure is not high, and the write pressures of the data pool and the cache pool are both low, that is, they are relatively idle. At this time, the data pool can be made to complete the object fragmentation task as soon as possible, such as adjusting the value of the parameter of the due time of the object fragmentation task to the target value; for another example, if both the space replacement frequency of the cache pool and the write-through ratio of the data pool gradually increase, it indicates that the current write operation of the front-end server is gradually increasing and the pressure is also gradually increasing. There may be a situation of sudden write operations, and the write pressures of the data pool and the cache pool are both high. Therefore, the speed or time for the data pool to complete the object fragmentation task can be appropriately reduced, such as adjusting the value of the parameter of the due time of the object fragmentation task to the target value, so that the memory spaces and performances of the data pool and the cache pool can meet the write operation of the front-end server.

[0080] It can be understood that adjusting the values of the parameters related to the object fragmentation task to reach the target value, and this target value can be understood as the target value that matches or adapts to the current write operation pressure of the front-end server. The write operation pressure of the front-end server corresponds to the write pressures of both the data pool and the cache pool. Generally, the greater the write operation pressure of the front-end server, the greater the write pressures of both the data pool and the cache pool. Therefore, this target value can also be understood as the target value that matches / adapts to the current data write pressures of both the data pool and the cache pool, that is, the memory spaces and performances of the data pool and the cache pool can meet the current write operation pressure of the front-end server. In addition, for the adjustment of the values of the parameters of the object fragmentation task, it can be to adjust the value of one parameter, or adjust the values of multiple parameters or all parameters.

[0081] S108, perform the object fragmentation task according to the target value currently corresponding to each parameter.

[0082] In this step, after determining the target values of each parameter of the object fragmentation task, the object fragmentation can be performed according to the target values in the next fragmentation cycle, so that the memory spaces and performances of the data pool and the cache pool can meet the current write operation requirements of the front-end server.

[0083] In this embodiment, by obtaining the number of updates of the data in the preset address space in the cache pool within the previous unit time, and determining the space replacement frequency of the cache pool within the previous unit time according to the number of updates within the previous unit time, at the same time obtaining the first data volume written through to the data pool by the front-end server within the previous unit time and the second data volume written in total by the front-end server within the previous unit time, and determining the write-through ratio of the data written through to the data pool within the previous unit time according to the first data volume and the second data volume, then determining the target value corresponding to each parameter of the object fragmentation reorganization task currently according to the space replacement frequency of the cache pool and the write-through ratio of the data pool, and then executing the object fragmentation reorganization task according to the target value of each parameter currently; wherein, the object fragmentation reorganization task is a task of reorganizing the fragmented objects in the data pool, and the target value of each parameter thereof matches the current write service pressure situation of the front-end server. In this method, since the replacement frequency of the cache pool within the previous unit time and the write-through ratio of the data pool within the previous unit time are calculated, the current write service pressure situation of the data pool and the cache pool of this cloud storage device and the current write service pressure situation of the front-end server can be intuitively reflected, so as to calculate the parameter values of the object fragmentation reorganization task that match the current write service pressure situation of the cloud storage device / front-end server accordingly. Then, when the object fragmentation reorganization task is executed according to this parameter value and the front-end server has a sudden task, the cloud storage device can adaptively meet the space requirements and / or performance requirements required by the front-end server.

[0084] The following embodiments will illustrate the specific process of dynamically adjusting the values of the relevant parameters of the object fragmentation reorganization task through the space replacement frequency of the cache pool and the write-through ratio of the data pool.

[0085] Figure 6 is the second flowchart of the object fragmentation reorganization method provided by the present invention. As Figure 6 shown, the above step 106 may include the following steps:

[0086] Step 202, determining whether the space replacement frequency of the cache pool is less than the preset frequency threshold to obtain a first determination result; and determining whether the write-through ratio of the data pool exceeds the preset ratio threshold range to obtain a second determination result.

[0087] Among them, the preset frequency threshold corresponds to the threshold of the spatial replacement frequency, and its magnitude can be set according to the actual situation. For example, it can be set to 2, 3, 4, etc. The preset ratio threshold range corresponds to the threshold range of the write-through ratio of the data pool. The preset frequency threshold can include only one threshold or a threshold range composed of multiple thresholds. For example, it includes an upper limit value and a lower limit value. The preset ratio threshold range can be a threshold range composed of a lower limit value and an upper limit value. The magnitude of the preset ratio threshold range can also be set according to the actual situation. For example, it can be a threshold range [0~10%] composed of a lower limit value of 0 and an upper limit value of 10%. The specific upper limit value and lower limit value can be set according to the actual business.

[0088] After obtaining the write-through ratio Q of the data pool in the previous unit time, it can be determined whether Q exceeds the preset ratio threshold range to obtain a first judgment result. At the same time, after obtaining the spatial replacement frequency P of the cache pool in the previous unit time, it can be determined whether P is less than the preset frequency threshold to obtain a second judgment result.

[0089] It should be noted that the preset frequency threshold and the preset ratio threshold range here are both related to the data writing pressure conditions of both the cache pool and the data pool.

[0090] Step 204, according to the first judgment result and the second judgment result, determine the target value corresponding to each parameter of the object fragmentation reorganization task currently.

[0091] In this step, relevant parameters can be preset for the object fragmentation reorganization task. These relevant parameters can include, for example: the expiration time of the object fragmentation reorganization task, the threshold for fragmented objects to enter the fragmentation reorganization task queue, the number of threads of the object fragmentation reorganization task, the number of fragmented objects processed in parallel by a single thread, the running time of each thread, the first cache pool utilization rate threshold, the second cache pool utilization rate threshold, etc.

[0092] Among them, the expiration time of the object fragmentation reorganization task refers to the period for executing the object fragmentation reorganization task; the threshold for fragmented objects to enter the fragmentation reorganization task queue refers to the space utilization rate threshold of the object fragments. For example, if the space utilization rate of a certain object fragment is lower than the space utilization rate threshold, then the object fragment can enter the object fragmentation reorganization task queue to wait for fragmentation reorganization; the number of fragmented objects processed in parallel by a single thread refers to the number of fragmented object fragments merged by a single thread each time. For example, 5 object fragments are merged into one large object each time; the running time of each thread refers to the duration for a single thread to execute the object fragmentation reorganization task.

[0093] The first cache pool usage threshold refers to the cache pool usage threshold that triggers the data passthrough writing of the front-end server to the data pool; the second cache pool usage threshold refers to the cache pool usage threshold that stops the data passthrough writing of the front-end server to the data pool; the usage rate of the cache pool refers to the ratio of the memory occupied by the data currently stored in the cache pool to its total memory. When the writing pressure of the front-end server is relatively high and the speed at which the cache pool releases space after successfully merging large objects cannot keep up with the writing speed of the front-end server, the usage rate of the cache pool will continuously increase. When the usage rate of the cache pool is greater than the first cache pool usage threshold K, the usage rate of the cache pool may continue to increase until it is completely full, resulting in some or all of the objects of the front-end server being passthrough written to the data pool. Therefore, when passthrough writing occurs, the passthrough ratio Q value starts to increase from 0 until it reaches 100%. The larger the Q value, the greater the pressure on the front-end business. When the business pressure of the front-end server decreases and the usage rate of the cache pool drops below the second cache pool usage threshold k, the data passthrough writing to the data pool will stop, reducing the data writing pressure on the data pool to enhance the priority of the object fragmentation processing task and complete the object fragmentation as soon as possible.

[0094] In addition, for the expiration time of the above object fragmentation sorting task and the threshold for fragmented objects to enter the fragmentation sorting task queue, these two parameters are parameters related to the start condition of the object fragmentation sorting task. The number of threads of the object fragmentation sorting task, the number of fragmented objects processed in parallel by a single thread, and the running time of each thread are parameters related to the sorting speed and sorting time of the object fragmentation sorting task. The first cache pool usage threshold and the second cache pool usage threshold are parameters related to the space usage of the cache pool.

[0095] After obtaining the determination result of the space replacement frequency of the cache pool and the determination result of the passthrough ratio of the data pool, one or more parameters among the relevant parameters of the above object fragmentation sorting task can be adjusted. The adjustment method can be to increase or decrease the historical value of the parameter (i.e., the parameter value used in the previous execution cycle of the object fragmentation sorting task). After increasing or decreasing the historical value of the parameter, a target value matching the current write business pressure situation can be obtained.

[0096] In this embodiment, by comparing the space replacement frequency of the cache pool with the corresponding threshold and comparing the passthrough ratio of the data pool with the corresponding threshold range, and comprehensively determining the parameter value of the object fragmentation sorting task that matches the current front-end write business pressure through these two determination results, such a quantitative method can refine the process of determining the parameter value of the object fragmentation sorting task, enabling the object fragmentation sorting task to be executed in an optimal situation and also meeting the write business requirements of the front-end.

[0097] In the following embodiments, after determining the size relationship between the space replacement frequency of the above cache pool and the corresponding threshold and the size relationship between the write-through ratio of the data pool and the corresponding threshold range, several possible determination results will be described.

[0098] In some embodiments, referring to Figure 7 the flowchart of dynamically adjusting the parameters of the object fragmentation reorganization task shown in the figure, step 204 above, determining the current target value corresponding to each parameter of the object fragmentation reorganization task according to the first determination result and the second determination result, may include the following implementation manners:

[0099] To better understand the following implementation manners, first, the situation where the write-through ratio of the data pool and the space replacement frequency of the cache pool change as the written data increases will be described. When starting to write data, generally, the data will be first written into the cache pool and will not be written through to the data pool. Therefore, the write-through ratio of the data pool is relatively low at the beginning of writing data, such as equal to the lower limit value, and the space replacement frequency of the cache pool will gradually increase due to the continuous writing of data, but it will be less than the preset frequency threshold at the beginning. This scenario corresponds to Embodiment 1 below.

[0100] As the written data continues to increase, and at this time the capacity of the cache pool is still sufficient, then the data will continue to be written into the cache pool. At this time, the write-through ratio of the data pool is still relatively low, such as equal to the lower limit value, and due to the continuous increase of the written data in the cache pool, its space replacement frequency will also continue to increase, such as increasing to be greater than the preset frequency threshold. This scenario corresponds to Embodiment 2 below.

[0101] As the written data continues to increase, the capacity of the cache pool may be insufficient. Then, the written data will gradually be written through to the data pool. Therefore, the write-through ratio of the data pool will gradually increase, such as being greater than the lower limit value but less than the upper limit value. And at this time, due to the continuous increase of the written data, the space replacement frequency of the cache pool will continue to increase or remain unchanged, such as still being greater than the preset frequency threshold. This scenario corresponds to Embodiment 3 below.

[0102] As the written data continues to increase further, more data will be written through to the data pool until the data pool is full. At this time, the write-through ratio of the data pool will continue to increase, such as increasing to be greater than the upper limit value. And at this time, the data in the cache pool may already be full and the data in it will no longer be updated. Then, its space replacement frequency will decrease, that is, the space replacement frequency of the cache pool may be less than the preset frequency threshold. This scenario corresponds to Embodiment 4 below.

[0103] The above are several scenarios that may exist during the process of continuously writing data into the data pool and the cache pool. Next, the implementation manners of specifically adjusting the parameter values of the object fragmentation reorganization task in these scenarios will be described.

[0104] In the first implementation manner, the above preset ratio threshold range includes a lower limit value. If the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, then the historical values corresponding to each parameter of the object fragmentation reorganization task are obtained; the above parameters include at least one of the expiration time of the object fragmentation reorganization task, the threshold for fragmented objects to enter the fragmentation reorganization task queue, the number of threads of the object fragmentation reorganization task, the number of fragmented objects processed in parallel by a single thread, and the running time of the thread.

[0105] Perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation reorganization task: reduce the historical value of the expiration time of the object fragmentation reorganization task to the target value, increase the historical value of the threshold for fragmented objects to enter the fragmentation reorganization task queue to the target value, increase the historical value of the number of threads of the object fragmentation reorganization task to the target value, increase the historical value of the number of fragmented objects processed in parallel by a single thread to the target value, and increase the historical value of the running time of the thread to the target value.

[0106] Among them, the lower limit value of the preset ratio threshold range is assumed to be 0, the preset frequency threshold is assumed to be 2, the historical value of the expiration time of the object fragmentation reorganization task is assumed to be T, the historical value of the threshold for fragmented objects to enter the fragmentation reorganization task queue is assumed to be D (the default value of D is 50%), the historical value of the number of threads of the object fragmentation reorganization task is assumed to be m, the historical value of the number of fragmented objects processed in parallel by a single thread is assumed to be n, the historical value of the running time of each thread is assumed to be t, the first cache pool utilization rate threshold is denoted as K, and the second cache pool utilization rate threshold is denoted as k.

[0107] Specifically, if the write-through ratio Q = 0 and the space replacement frequency P is less than 2, it indicates that the data pool and the cache pool are relatively idle at this time, and the write service pressure on the front-end server is relatively small. Then, the following operations can be performed on the historical values of at least one parameter in the object fragmentation reorganization task to release the invalid space occupied by the object fragments as soon as possible during the idle time:

[0108] Reduce the expiration time of the object fragmentation reorganization task. Specifically, it can be reduced from the historical value T to the target value T1 (T1 < T), which can increase the number of times of object fragmentation processing within a certain period of time;

[0109] Increase the threshold for triggering fragmented objects to enter the object fragmentation reorganization task queue. Specifically, it can be increased from the historical value D to the target value D1 (D1 > D, such as D1 = 80%), which can lower the threshold for fragmented objects to enter the fragmentation reorganization task queue and enable more object fragments to enter the task queue for fragmentation reorganization;

[0110] Increase the number of threads for defragmentation. Specifically, it can be increased from the historical value m to the target value m1 (m1 > m).

[0111] Increase the number of object fragments processed in parallel by a single defragmentation thread. Specifically, it can be increased from the historical value n to the target value n1 (n1 > n), which can improve the defragmentation speed.

[0112] Increase the running time of a single defragmentation thread. Specifically, it can be increased from the historical value t to the target value t1 (t1 > t), which can extend the time for each fragment processing and release the space occupied by fragments more quickly.

[0113] In addition, for the first cache pool usage rate threshold and the second cache pool usage rate threshold in the parameters related to the object fragmentation task, the target values of these two parameters can remain their default values unchanged.

[0114] Through the operations of the above Embodiment 1, the speed of object fragmentation processing can be adjusted in real time when the data pool and the cache pool are relatively idle, and all the object fragments in the data pool can be processed as soon as possible, so as to release the available space invalidly occupied by the object fragments more quickly.

[0115] Embodiment 2: The above preset ratio threshold range includes a lower limit value. If the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value, and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, then obtain the historical value corresponding to each parameter of the object fragmentation task; the above parameters include at least one of the first cache pool usage rate threshold for triggering data write-through to the data pool and the second cache pool usage rate threshold for stopping data write-through to the data pool.

[0116] Perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation task: reduce the historical value of the first cache pool usage rate threshold to the target value, increase the historical value of the second cache pool usage rate threshold to the target value.

[0117] Among them, if the write-through ratio Q = 0 and the space replacement frequency P is greater than or equal to 2, it indicates that the write operation pressure of the front-end server is gradually increasing. Correspondingly, the business pressure of the cache pool is relatively large, and the pressure of the data pool temporarily meets the requirements or is relatively idle. Then, at least one of the following operations can be performed on the historical values of the parameters in the object fragmentation task to appropriately reduce the data write pressure on the cache pool:

[0118] Reduce the first cache pool usage rate threshold (the cache pool usage rate threshold for triggering data write-through to the data pool) K. Specifically, it can be reduced from the historical value to the target value. For example, reduce K from 90% to 80%, which can write more data of the front-end server to the data pool to appropriately reduce the data write pressure on the cache pool.

[0119] Increase the usage rate threshold k of the second cache pool (the cache pool usage rate threshold for stopping data write-through to the data pool). Specifically, it can be increased from the historical value to the target value. For example, increase k from 70% to 75%. This can relax the usage rate of the cache pool, enable more data to be written into the data pool, and appropriately reduce the data write pressure on the cache pool.

[0120] In addition, for the expiration time of the object fragmentation reorganization task, the threshold for fragmented objects to enter the fragmentation reorganization task queue, the number of threads of the object fragmentation reorganization task, the number of fragments processed in parallel by a single thread, and the running time of each thread, the target values of these parameters can all be set to the increased or decreased target values in Embodiment 1, that is, set to T1, D1, m1, n1, t1 and remain unchanged. This can ensure that the existing object fragments are sorted out as soon as possible when the write service pressure on the front-end server gradually increases.

[0121] Through the operations of Embodiment 1 above, it is possible to ensure that the existing object fragments are sorted out as soon as possible when the write service pressure on the front-end server gradually increases, and at the same time appropriately reduce the data write pressure on the cache pool.

[0122] Embodiment 3: The above preset ratio threshold range includes a lower limit value and an upper limit value, and the upper limit value is greater than the lower limit value. If the first judgment result is that the write-through ratio of the data pool is greater than the lower limit value and less than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, then obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the above parameters include at least one of the expiration time of the object fragmentation reorganization task and the threshold for fragmented objects to enter the fragmentation reorganization task queue;

[0123] Perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation reorganization task: increase the historical value corresponding to the expiration time of the object fragmentation reorganization task to the target value, and decrease the historical value corresponding to the threshold for fragmented objects to enter the fragmentation reorganization task queue to the target value.

[0124] Among them, assume that the upper limit value of the preset ratio threshold range is 10%. If 0 < write-through ratio Q ≤ 10% and the space replacement frequency P is greater than or equal to 2, it indicates that the business pressure on the front-end server continues to increase, the pressure on the cache pool is very high, and the data pool begins to directly bear a small part of the business write pressure of the front-end server. Then, the following operations can be performed on the historical values of at least one parameter in the object fragmentation reorganization task so that the data pool can meet the write service requirements of the front-end server in a timely manner:

[0125] Increase the expiration time of the object fragmentation reorganization task. Specifically, it can be increased from the historical value T1 to the target value T2, where T1 < T2 < T. This can appropriately reduce the number of object fragmentation processing times within a certain period of time;

[0126] Reduce the threshold for fragmented objects to enter the defragmentation task queue. Specifically, it can be reduced from the historical value D1 to the target value D2, where D < D2 < D1. For example, D2 = 60%, which can appropriately increase the threshold for object fragments to enter the defragmentation task queue, avoid excessive object fragments from entering the task queue for defragmentation, that is, reduce defragmentation;

[0127] In addition, for the number of threads for the object fragmentation task, the number of fragments processed in parallel by a single thread, and the running time of each thread, the target values of these parameters can all be set to the increased or decreased target values in Embodiment 1, that is, set to m1, n1, t1 and remain unchanged. At the same time, for the target values of the first cache pool utilization rate threshold and the second cache pool utilization rate threshold, these two parameters can be set to the decreased or increased target values in Embodiment 2, that is, K remains unchanged at 80%, and k remains unchanged at 75%.

[0128] Through the operations in the above Embodiment 3, it is convenient for the data pool to meet the business writes of the front-end server in a timely manner and extend the retention time of some object fragments.

[0129] Embodiment 4: The above preset ratio threshold range includes an upper limit value. If the first judgment result is that the write-through ratio of the data pool is greater than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, then obtain the historical value corresponding to each parameter of the object fragmentation task; the above parameters include at least one of the expiration time of the object fragmentation task, the threshold for fragmented objects to enter the defragmentation task queue, the number of threads for the object fragmentation task, the number of fragments processed in parallel by a single thread, and the running time of the thread;

[0130] Perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation task: increase the historical value corresponding to the expiration time of the object fragmentation task to the target value, decrease the historical value corresponding to the threshold for fragmented objects to enter the defragmentation task queue to the target value, decrease the historical value corresponding to the number of threads for the object fragmentation task to the target value, decrease the historical value corresponding to the number of fragments processed in parallel by a single thread to the target value, decrease the historical value corresponding to the running time of the thread to the target value.

[0131] Among them, if the tracing ratio Q is greater than 10% and the space replacement frequency P is less than 2, it indicates that the write operation pressure on the front-end server is very high at this time. Correspondingly, the write pressures on both the cache pool and the data pool are very high. The small objects in the cache pool are not merged into large objects in time, which leads to the decrease in the space replacement frequency P. The data pool may not be able to respond to the merge write requirements of the cache pool and the write operation pressure of the front-end server in time. In order to facilitate the data pool to fully respond to the business write operations of the front-end server or other read / write requirements, the priority of the object fragmentation task can be adjusted to the lowest. Specifically, the following operations can be performed on the historical values of at least one parameter in the object fragmentation task.

[0132] Increase the expiration time of the object fragmentation task. Specifically, it can be increased from the historical value T2 to the target value T, which can reduce the number of object fragment processing times within a certain period of time.

[0133] Decrease the threshold for triggering fragmented objects to enter the object fragmentation task queue. Specifically, it can be decreased from the historical value D2 to the target value D, which can increase the threshold for fragmented objects to enter the fragmentation task queue, enabling more object fragments to enter the task queue for fragmentation.

[0134] Decrease the number of threads for fragmentation. Specifically, it can be decreased from the historical value m1 to the target value m.

[0135] Decrease the number of object fragments processed in parallel by a single fragmentation thread. Specifically, it can be decreased from the historical value n1 to the target value n, which can reduce the fragmentation speed.

[0136] Decrease the running time of a single fragmentation thread. Specifically, it can be decreased from the historical value t1 to the target value t, which can shorten the time for each fragment processing.

[0137] In addition, for the first cache pool usage threshold and the second cache pool usage threshold among the parameters related to the object fragmentation task, the target values of these two parameters can be adjusted to their respective default values.

[0138] Through the operations of the above-mentioned Embodiment 4, when the write operation pressure on the front-end server is relatively high, the priority of the object fragmentation task can be adjusted to the lowest, ensuring that the data pool fully responds to the write operation pressure of the front-end server or other read / write requirements (such as the merge write requirements of the cache pool), and improving the space utilization rate of the cloud storage device.

[0139] In this embodiment, when the space replacement frequency of the cache pool and the write-through ratio of the data pool are at different values, the corresponding write service pressure conditions of the front end, the data pool, and the cache pool are determined. In this way, the parameter values of the object fragmentation reorganization task can be adjusted dynamically in a timely manner, and the space invalidly occupied by the object fragments can be released in a timely manner to meet the writing requirements of new service data and improve the space utilization rate of the cloud storage device.

[0140] In the actual process of counting the space replacement frequency P of the cache pool, it is found that since the space of the cache pool is written with small object data by the front-end server, after its total capacity reaches a certain write ratio R, it will trigger the merging of small objects into large objects, and when the number of small objects written in each Bucket storage bucket of the cache pool reaches a certain number S, it will also trigger the merging of large objects. After the large object merging is successful, the small objects will be deleted, and the released space will be used for front-end writing again. However, each address segment of the cache pool is not used by the same Bucket every time. The timing of small object merging in each Bucket is different, and the business pressure of the front end is also different in different periods. Therefore, the replacement frequency P of each segment of the address used is different in different time periods. If the replacement frequencies of all address spaces in the cache pool are counted, it will cause the system to calculate an excessive amount of data, which will not only lead to too long calculation time and affect the calculation of other tasks, but also affect the application and scheduling of other tasks for system resources due to occupying too much CPU resources. Based on this, this embodiment proposes a solution to obtain a part of the address space in the cache pool to count its space replacement frequency. The following embodiments will illustrate this process.

[0141] Figure 8 is the third flow chart of the object fragmentation reorganization method provided by the present invention. As Figure 9 shown, the step of "obtaining the update times of the data in the preset address space in the cache pool in the previous unit time" in the above step 102 may include the following steps:

[0142] Step 302, obtaining the size of the target address space correspondingly extracted in the cache pool in the previous unit time.

[0143] Among them, the size of the target address space extracted in the cache pool is smaller than the size of all address spaces in the cache pool. For the method of determining the size of the target address space, as an optional embodiment, it may be to determine the size of the target address space correspondingly extracted in the previous unit time according to the first space replacement frequency corresponding to the cache pool in the historical time; where the historical time is the historical time before the previous unit time, and the first space replacement frequency is proportional to the size of the target address space.

[0144] In other words, the greater the space replacement frequency of the historical time cache pool, the greater the write operation pressure on the cache pool, and the larger the size of the target address space to be extracted currently. This can ensure an accurate count of the current space replacement frequency of the cache pool. It can be understood that even if the size of the extracted target address space is increased, the memory occupied by the process of calculating the space replacement frequency is very small and will not have a significant impact on the performance of the cloud storage device.

[0145] Step 304: In the first-segment address space, the second-segment address space, and the third-segment address space of the cache pool, successively select sub-address spaces of the target address space size; the above-mentioned first-segment address space, second-segment address space, and third-segment address space are respectively one segment of address space corresponding to the front segment, middle segment, and rear segment of the address encoding of the cache pool.

[0146] In this step, the size of the extracted target address space obtained above can be the size of the entire extracted address space. Specifically, one or more segments of address space can be extracted from the cache pool to form the above-mentioned target address space size. Taking the example of evenly extracting three segments of address space, one sub-address space corresponding to the front segment, middle segment, and rear segment of the address encoding of the cache pool can be extracted respectively, and these three segments of address space form the target address space.

[0147] Exemplarily, referring to Figure 9 the schematic diagram of the cache pool address encoding shown, assuming the cache pool capacity is 1T, and assuming that the size of each extracted sub-address space can be 3G, 10G, 50G, etc. Taking 3G as an example, then select the address spaces corresponding to 3G of the front segment, 3G of the middle segment, and 3G of the rear segment of the cache pool address encoding (the size of x can be determined according to the address segmentation situation). Only the space replacement frequency of a total of 9G of space needs to be counted in this way, which is much faster than counting the space replacement frequency of 1T of capacity and occupies much less resources. Therefore, it can ensure an improvement in the efficiency of counting the replacement frequency on the basis of the accuracy of the counted space replacement frequency.

[0148] Step 306: Determine the three sub-address spaces as the preset address space, and count the number of updates of the data in the preset address space in the cache pool in the previous unit time.

[0149] In this step, after obtaining the three sub-address spaces extracted from the cache pool, these three sub-address spaces can be used as the preset address space extracted from the cache pool. Then, the number of updates of these three sub-address spaces in the previous unit time can be counted respectively, and the sum value after summing these three update numbers is used as the number of updates of the preset address space in the previous unit time. After that, the space replacement frequency of the preset address space in the cache pool in the previous unit time can be determined based on the number of updates of the preset address space in the previous unit time.

[0150] In this embodiment, by separately selecting a section of address space in the front section, middle section, and rear section of the cache pool address space to count the update times, and then counting the space replacement frequency, the selected address space for statistics is less. Therefore, on the basis of ensuring the accuracy of the counted space replacement frequency, the efficiency of counting the replacement frequency can be improved.

[0151] The object fragmentation reorganization device provided by the present invention will be described below. The object fragmentation reorganization device described below can be correspondingly referred to the object fragmentation reorganization method described above.

[0152] Figure 10 is a schematic structural diagram of the object fragmentation reorganization device provided by the present invention. Refer to Figure 10 As shown, the device may include:

[0153] A space replacement frequency determination module 410, configured to obtain the update times of the data in the preset address space in the cache pool in the previous unit time, and determine the space replacement frequency of the cache pool in the previous unit time according to the update times in the previous unit time;

[0154] A write-through ratio determination module 420, configured to obtain the first data volume written through to the data pool by the front-end server in the previous unit time and the second data volume written by the front-end server in the previous unit time in total, and determine the write-through ratio of the data written through to the data pool in the previous unit time according to the first data volume and the second data volume;

[0155] A fragmentation reorganization task parameter determination module 430, configured to determine the target value corresponding to each parameter of the object fragmentation reorganization task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above object fragmentation reorganization task is a task of reorganizing fragmented objects in the data pool, and the target value of each parameter matches the current write service pressure situation of the front-end server;

[0156] An object fragmentation reorganization module 440, configured to execute the object fragmentation reorganization task according to the target value corresponding to each parameter currently.

[0157] In some embodiments, the above fragmentation reorganization task parameter determination module 430 includes:

[0158] A judgment unit, configured to judge whether the write-through ratio of the data pool exceeds a preset ratio threshold range to obtain a first judgment result; and judge whether the space replacement frequency of the cache pool is less than a preset frequency threshold to obtain a second judgment result;

[0159] An object fragmentation reorganization task parameter determination unit, configured to determine the target value corresponding to each parameter of the object fragmentation reorganization task according to the first judgment result and the second judgment result.

[0160] In some embodiments, the above preset ratio threshold range includes a lower limit value. The object fragmentation reorganization task parameter determination unit is specifically configured to, if the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the parameters include at least one of the expiration time of the object fragmentation reorganization task, the threshold for a fragmented object to enter the fragmentation reorganization task queue, the number of threads of the object fragmentation reorganization task, the number of fragments processed in parallel by a single thread, and the running time of the thread; perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation reorganization task: reducing the historical value corresponding to the expiration time of the object fragmentation reorganization task to a target value, increasing the historical value corresponding to the threshold for a fragmented object to enter the fragmentation reorganization task queue to a target value, increasing the historical value corresponding to the number of threads of the object fragmentation reorganization task to a target value, increasing the historical value corresponding to the number of fragments processed in parallel by a single thread to a target value, and increasing the historical value corresponding to the running time of the thread to a target value.

[0161] In some embodiments, the above preset ratio threshold range includes a lower limit value. The object fragmentation reorganization task parameter determination unit is specifically configured to, if the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the parameters include at least one of the first cache pool usage rate threshold for triggering data write-through to the data pool and the second cache pool usage rate threshold for stopping data write-through to the data pool; perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation reorganization task: reducing the historical value of the first cache pool usage rate threshold to a target value, increasing the historical value of the second cache pool usage rate threshold to a target value.

[0162] In some embodiments, the above preset ratio threshold range includes a lower limit value and an upper limit value, and the upper limit value is greater than the lower limit value. The object fragmentation reorganization task parameter determination unit is specifically configured to, if the first judgment result is that the write-through ratio of the data pool is greater than the lower limit value and less than the upper limit value and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the parameters include at least one of the expiration time of the object fragmentation reorganization task and the threshold for a fragmented object to enter the fragmentation reorganization task queue; perform at least one of the following adjustment operations on the historical values of the parameters of the object fragmentation reorganization task: increasing the historical value corresponding to the expiration time of the object fragmentation reorganization task to a target value, reducing the historical value corresponding to the threshold for a fragmented object to enter the fragmentation reorganization task queue to a target value.

[0163] In some embodiments, the above preset ratio threshold range includes an upper limit value. The above object fragmentation reorganization task parameter determination unit is specifically configured to, if the first judgment result is that the write-through ratio of the data pool is greater than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, obtain the historical value corresponding to each parameter of the object fragmentation reorganization task; the above parameters include at least one of the expiration time of the object fragmentation reorganization task, the threshold for fragmented objects to enter the fragmentation reorganization task queue, the number of threads of the object fragmentation reorganization task, the number of fragments processed in parallel by a single thread, and the running time of the thread; perform at least one of the following adjustment operations on the historical value of the parameter of the object fragmentation reorganization task: increase the historical value corresponding to the expiration time of the object fragmentation reorganization task to the target value, decrease the historical value corresponding to the threshold for fragmented objects to enter the fragmentation reorganization task queue to the target value, decrease the historical value corresponding to the number of threads of the object fragmentation reorganization task to the target value, decrease the historical value corresponding to the number of fragments processed in parallel by a single thread to the target value, and decrease the historical value corresponding to the running time of the thread to the target value.

[0164] In some embodiments, the above space replacement frequency determination module 410 is specifically configured to obtain the size of the target address space extracted correspondingly in the cache pool in the previous unit time; in the first segment address space, the second segment address space, and the third segment address space of the cache pool, sequentially select sub-address spaces of the size of the target address space; the above first segment address space, second segment address space, and third segment address space are respectively a segment of address space corresponding to the front segment, the middle segment, and the rear segment of the address coding of the cache pool; determine the three sub-address spaces as the preset address space, and count the number of updates of the data in the preset address space in the cache pool in the previous unit time.

[0165] It should be noted here that the above device provided by the embodiments of the present invention can implement all the method steps implemented by the above method embodiments, and can achieve the same technical effects. The same parts and beneficial effects as those in the method embodiments will not be specifically described in this embodiment.

[0166] Figure 11 An example of the physical structure diagram of a cloud storage device is shown in Figure 11As shown, the cloud storage device may include: a processor 510 , a communications interface 520 , a memory 530 and a communication bus 540 , wherein the processor 510 , the communications interface 520 , and the memory 530 communicate with each other via the communication bus 540 . The processor 510 can call the logic instructions in the memory 530 to execute the object defragmentation method, which includes: obtaining the number of updates of data in a preset address space in the cache pool in the previous unit time, and determining the space replacement frequency of the cache pool in the previous unit time based on the number of updates in the previous unit time; obtaining the first data volume written to the data pool by the front-end server in the previous unit time and the second data volume written in total by the front-end server in the previous unit time, and determining the write-through ratio of data to the data pool in the previous unit time based on the first data volume and the second data volume; determining the current target value corresponding to each parameter of the object defragmentation task based on the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above-mentioned object defragmentation task is a task for sorting fragmented objects in the data pool, and the target value of each parameter matches the current write business pressure of the front-end server; and executing the object defragmentation task according to the current target value corresponding to each parameter.

[0167] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the object defragmentation method provided by the above methods, which includes: obtaining the number of updates of data in a preset address space in a cache pool in the previous unit time, and determining the space replacement frequency of the cache pool in the previous unit time based on the number of updates in the previous unit time; obtaining a first data amount written to a data pool by a front-end server in the previous unit time and a second data amount written in total by the front-end server in the previous unit time, and determining the write-through ratio of data to the data pool in the previous unit time based on the first data amount and the second data amount; determining the current target value corresponding to each parameter of the object defragmentation task based on the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above object defragmentation task is a task for sorting fragmented objects in the data pool, and the target value of each parameter matches the current write business pressure of the front-end server; and executing the object defragmentation task based on the current target value corresponding to each parameter.

[0169] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the object defragmentation method provided by the above-mentioned methods, the method comprising: obtaining the number of updates of data in a preset address space in a cache pool in the previous unit time, and determining the space replacement frequency of the cache pool in the previous unit time based on the number of updates in the previous unit time; obtaining a first amount of data written to a data pool by a front-end server in the previous unit time and a second amount of data written in total by the front-end server in the previous unit time, and determining the write-through ratio of data to the data pool in the previous unit time based on the first amount of data and the second amount of data; determining the current target value corresponding to each parameter of the object defragmentation task based on the space replacement frequency of the cache pool and the write-through ratio of the data pool; the above-mentioned object defragmentation task is a task for sorting fragmented objects in the data pool, and the target value of each parameter matches the current write business pressure of the front-end server; and executing the object defragmentation task based on the current target value corresponding to each parameter.

[0170] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for object defragmentation, characterized in that: include: Obtaining the number of updates of data in a preset address space in the cache pool in the previous unit time, and determining the space replacement frequency of the cache pool in the previous unit time according to the number of updates in the previous unit time; the number of updates refers to the sum of the number of times the preset address space changes from idle to having data and the number of times the preset address space changes from having data to idle in the previous unit time; Obtaining a first amount of data that is written through to the data pool by the front-end server in the previous unit time and a second amount of data that is written in total by the front-end server in the previous unit time, and determining a write-through ratio of data to the data pool in the previous unit time according to the first amount of data and the second amount of data; the write-through to the data pool means that the front-end server directly writes the data to the data pool when writing data; Determining a target value currently corresponding to each parameter of the object defragmentation task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool; The object defragmentation task is a task for defragmenting the fragmented objects in the data pool, and the target value of each parameter matches the current write business pressure of the front-end server; the parameters of the object defragmentation task include at least one of the following: the expiration time of the object defragmentation task, the threshold for the fragmented objects to enter the defragmentation task queue, the number of threads of the object defragmentation task, the number of fragments processed in parallel by a single thread, the running time of each thread, the first buffer pool usage threshold for triggering data write-through to the data pool, and the second buffer pool usage threshold for stopping data write-through to the data pool; The object defragmentation task is performed according to the target value currently corresponding to each parameter.

2. The object defragmentation method according to claim 1, characterized in that: Determining the target value currently corresponding to each parameter of the object defragmentation task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool includes: Determine whether the write-through ratio of the data pool exceeds a preset ratio threshold range to obtain a first determination result; and determine whether the space replacement frequency of the buffer pool is less than a preset frequency threshold to obtain a second determination result; According to the first judgment result and the second judgment result, a target value currently corresponding to each parameter of the object defragmentation task is determined.

3. The object defragmentation method according to claim 2, characterized in that: The preset ratio threshold range includes a lower limit value, and determining the target value currently corresponding to each parameter of the object defragmentation task according to the first judgment result and the second judgment result includes: If the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, then obtaining the historical value corresponding to each parameter of the object defragmentation task; the parameters include at least one of the expiration time of the object defragmentation task, the threshold for the fragmented object to enter the defragmentation task queue, the number of threads of the object defragmentation task, the number of fragments processed in parallel by a single thread, and the running time of the thread; Perform at least one of the following adjustment operations on the historical values ​​of the parameters of the object defragmentation task: reduce the historical value corresponding to the expiration time of the object defragmentation task to the target value, increase the historical value corresponding to the threshold of the fragmented object entering the defragmentation task queue to the target value, increase the historical value corresponding to the number of threads of the object defragmentation task to the target value, increase the historical value corresponding to the number of fragments processed in parallel by a single thread to the target value, and increase the historical value corresponding to the running time of the thread to the target value.

4. The object defragmentation method according to claim 2, characterized in that: The preset ratio threshold range includes a lower limit value, and determining the target value currently corresponding to each parameter of the object defragmentation task according to the first judgment result and the second judgment result includes: If the first judgment result is that the write-through ratio of the data pool is equal to the lower limit value, and the second judgment result is that the space replacement frequency of the buffer pool is not less than the preset frequency threshold, then obtaining the historical value corresponding to each parameter of the object defragmentation task; the parameter includes at least one of a first buffer pool usage rate threshold for triggering data write-through to the data pool and a second buffer pool usage rate threshold for stopping data write-through to the data pool; At least one of the following adjustment operations is performed on the historical value of the parameter of the object defragmentation task: reducing the historical value of the first buffer pool usage threshold to a target value, and increasing the historical value of the second buffer pool usage threshold to a target value.

5. The object defragmentation method according to claim 2, characterized in that: The preset ratio threshold range includes a lower limit value and an upper limit value, the upper limit value is greater than the lower limit value, and determining the target value currently corresponding to each parameter of the object defragmentation task according to the first judgment result and the second judgment result includes: If the first judgment result is that the write-through ratio of the data pool is greater than the lower limit value and less than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is not less than the preset frequency threshold, then obtaining the historical value corresponding to each parameter of the object defragmentation task; the parameter includes at least one of the expiration time of the object defragmentation task and the threshold value for the fragmented object to enter the defragmentation task queue; Perform at least one of the following adjustment operations on the historical value of the parameter of the object defragmentation task: increase the historical value corresponding to the expiration time of the object defragmentation task to the target value, and reduce the historical value corresponding to the threshold of the fragmented object entering the defragmentation task queue to the target value.

6. The object defragmentation method according to claim 2, characterized in that: The preset ratio threshold range includes an upper limit value, and determining the target value currently corresponding to each parameter of the object defragmentation task according to the first judgment result and the second judgment result includes: If the first judgment result is that the write-through ratio of the data pool is greater than the upper limit value, and the second judgment result is that the space replacement frequency of the cache pool is less than the preset frequency threshold, then obtaining the historical value corresponding to each parameter of the object defragmentation task; the parameters include at least one of the expiration time of the object defragmentation task, the threshold for the fragmented object to enter the defragmentation task queue, the number of threads of the object defragmentation task, the number of fragments processed in parallel by a single thread, and the running time of the thread; Perform at least one of the following adjustment operations on the historical values ​​of the parameters of the object defragmentation task: increase the historical value corresponding to the expiration time of the object defragmentation task to the target value, reduce the historical value corresponding to the threshold of the fragmented object entering the defragmentation task queue to the target value, reduce the historical value corresponding to the number of threads of the object defragmentation task to the target value, reduce the historical value corresponding to the number of fragments processed in parallel by a single thread to the target value, and reduce the historical value corresponding to the running time of the thread to the target value.

7. The object defragmentation method according to any one of claims 1 to 6, characterized in that: The obtaining of the number of updates of data in a preset address space in the buffer pool in the previous unit time includes: Obtaining the size of the target address space corresponding to the extraction in the buffer pool in the previous unit time; In the first address space, the second address space and the third address space of the buffer pool, sub-address spaces of the size of the target address space are selected in sequence; the first address space, the second address space and the third address space are address spaces corresponding to the front section, the middle section and the back section of the address code of the buffer pool respectively; The three sub-address spaces are determined as the preset address spaces, and the number of updates of the data in the preset address spaces in the cache pool in the previous unit time is counted.

8. An object defragmentation device, characterized in that: include: A space replacement frequency determination module is used to obtain the number of updates of data in a preset address space in a cache pool in the previous unit time, and determine the space replacement frequency of the cache pool in the previous unit time according to the number of updates in the previous unit time; the number of updates refers to the sum of the number of times the preset address space changes from idle to having data and the number of times the preset address space changes from having data to idle in the previous unit time; A write-through ratio determination module, used to obtain a first amount of data written through to the data pool by the front-end server in the previous unit time and a second amount of data written in total by the front-end server in the previous unit time, and determine a write-through ratio of data written through to the data pool in the previous unit time according to the first amount of data and the second amount of data; the write-through to the data pool means that the front-end server directly writes the data to the data pool when writing data; A defragmentation task parameter determination module, used to determine the target value currently corresponding to each parameter of the object defragmentation task according to the space replacement frequency of the cache pool and the write-through ratio of the data pool; the object defragmentation task is a task for arranging fragmented objects in the data pool, and the target value of each parameter matches the current write business pressure of the front-end server; the parameters of the object defragmentation task include at least one of the following: the expiration time of the object defragmentation task, the threshold for fragmented objects to enter the defragmentation task queue, the number of threads of the object defragmentation task, the number of fragments processed in parallel by a single thread, the running time of each thread, the first cache pool usage threshold for triggering data write-through to the data pool, and the second cache pool usage threshold for stopping data write-through to the data pool; The object defragmentation module is used to execute the object defragmentation task according to the target value currently corresponding to each parameter.

9. A cloud storage device, comprising a cache pool, a data pool, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the object defragmentation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the object defragmentation method according to any one of claims 1 to 7 is implemented.

Images