A garbage collection method of a storage system and an electronic device

Abstract

The application discloses a garbage collection method of a storage system and electronic equipment, and relates to the technical field of storage systems.The method controls the concurrency of garbage collection tasks based on the relationship between the concurrency of garbage collection tasks and the number of IOs processed by the storage system, controls garbage collection of the storage system according to the determined concurrency, reduces the concurrency of garbage collection tasks when the concurrency of garbage collection tasks increases and the number of IOs processed by the storage system decreases, thereby reducing the impact on the business peak period and reducing the time delay of user business, controls garbage collection according to the increased concurrency of garbage collection tasks when the concurrency of garbage collection tasks increases and the number of IOs processed by the storage system remains unchanged, thereby realizing reasonable utilization of the user business valley period, and additionally, the garbage collection function of the storage system is started after the storage pool is established, the impact of the garbage collection function on the performance of the storage system after the garbage collection function is started is relieved, and the efficiency of garbage collection is improved.

Description

Technical Field

[0001] This invention relates to the field of storage system technology, and more particularly to a method for garbage collection in a storage system and an electronic device. Background Technology

[0002] To address the fragmentation problem of storage pools and enable the recycling of physical space, garbage collection functionality is typically incorporated into data storage systems.

[0003] The garbage collection function controls its activation and deactivation based on a threshold determined by the ratio of free blocks to total blocks in the storage system. Garbage collection will not be performed until the ratio reaches the threshold. The activation of garbage collection indicates the existence of a large number of non-free blocks. Therefore, processing these non-free blocks consumes significant resources, reducing the available resources for front-end input / output (IO). In other words, the performance of the storage system changes drastically after garbage collection is activated. During peak user periods, this garbage collection function can cause a significant increase in latency, severely impacting user experience.

[0004] Therefore, how to mitigate the impact of garbage collection on storage system performance and reduce the impact on user business is a technical problem that urgently needs to be solved by those in this field. Summary of the Invention

[0005] This invention provides a garbage collection method and electronic device for a storage system, which at least solves the problem in related technologies that after the garbage collection function is activated, the performance of the storage system will change drastically; during peak user business periods, the garbage collection function will cause a significant increase in user business latency, which will seriously affect user experience.

[0006] This invention provides a method for garbage collection in a storage system, comprising:

[0007] In response to the completion of storage pool creation, the garbage collection function of the storage system is started;

[0008] Compare the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system;

[0009] If the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs in the next cycle is less than the number of inputs and outputs in the previous cycle, then the storage system is controlled to perform garbage collection tasks in cycles following the next cycle with a task concurrency less than that of the next cycle.

[0010] If the second relationship is that the number of inputs and outputs in two adjacent cycles is the same, then the control storage system will perform garbage collection tasks in the cycle following the next cycle with a higher task concurrency than the next cycle.

[0011] The beneficial effects of this invention are as follows: First, in this method, the concurrency of garbage collection tasks is controlled based on the relationship between the concurrency of garbage collection tasks and the number of inputs and outputs processed by the storage system. Garbage collection in the storage system is controlled according to the determined concurrency of garbage collection tasks, thus achieving garbage collection of the storage system. Second, in the process of controlling the concurrency of garbage collection tasks based on the relationship between the concurrency of garbage collection tasks and the number of inputs and outputs processed by the storage system, when the concurrency of garbage collection tasks increases, the number of inputs and outputs processed by the storage system decreases, thereby reducing the concurrency of garbage collection tasks and reducing the impact of garbage collection tasks on peak business periods, thus reducing the latency of user services. Third, when the concurrency of garbage collection tasks increases, the storage system... If the number of inputs and outputs processed by the system is the same, the concurrency of garbage collection tasks is increased. Garbage collection is controlled based on the increased concurrency of garbage collection tasks, thus achieving reasonable utilization of user business off-peak periods. In addition, compared with the method of controlling the start and stop of garbage collection function based on the threshold determined by the ratio of free blocks to total blocks in the storage system, the start of garbage collection function indicates the existence of a large number of non-free blocks, that is, the performance of the storage system changes drastically after the garbage collection function is started. However, in the method provided by this invention, the garbage collection function of the storage system is started after the storage pool is established, avoiding a large number of non-free blocks waiting to be collected at the same time. Therefore, it alleviates the impact on the performance of the storage system after the garbage collection function is started and improves the efficiency of garbage collection.

[0012] The present invention also provides an electronic device, comprising:

[0013] Memory, used to store computer programs;

[0014] A processor, used to implement the garbage collection method of the above-described storage system when executing a computer program. Attached Figure Description

[0015] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a garbage collection method for a storage system provided in an embodiment of the present invention;

[0017] Figure 2This is an overall flowchart of a garbage collection method for a storage system provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0019] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] Garbage collection functionality is typically implemented as a background task, controlled by a threshold. This threshold is usually a specific ratio of free blocks to total blocks. The background task periodically monitors this threshold, and actual garbage collection is not performed until it is reached. Only after the threshold is met will it begin querying and modifying metadata on valid data, migrating data, or reclaiming invalid space. Under garbage collection, the number of free blocks generally increases. When the threshold is no longer met, garbage collection stops and resumes only when the conditions are met again. This garbage collection functionality also brings a series of problems, including the need to reserve space, leading to low actual physical space utilization for users. Performance issues are the most significant. During peak business periods, garbage collection can cause a substantial and unmitigated increase in latency, severely impacting user experience.

[0021] As described earlier, the garbage collection function only starts working after the ratio of free blocks to total blocks reaches a certain threshold. This leads to a sharp performance change before and after reaching the threshold because the garbage collection function consumes a large amount of Central Processing Unit (CPU) and memory resources, resulting in fewer available resources for handling front-end I / O. Furthermore, the garbage collection function not only uses a single threshold as a reference for starting and stopping tasks, but also uses this threshold as the sole reference for controlling task concurrency. A large number of free blocks results in low concurrency and slow collection speed, while a small number of free blocks increases task concurrency and improves collection speed. This also means that to ensure a certain number of available blocks, the fewer free blocks there are, the more resources garbage collection preempts, leading to worse front-end performance. Therefore, in real-world production environments, the garbage collection function exhibits a sharp performance drop as the amount of written data increases, gradually deteriorating until it reaches a steady state.

[0022] To mitigate the performance impact of garbage collection, especially to reduce latency during peak user periods, this invention provides a novel garbage collection method for storage systems.

[0023] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 A flowchart of a garbage collection method for a storage system provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0024] S11: In response to the completion of storage pool creation, start the garbage collection function of the storage system;

[0025] S12: Compare the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system;

[0026] S13: Use different strategies to perform garbage collection tasks based on the second relation.

[0027] Specifically, step S13 includes:

[0028] S130: If the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs in the next cycle is less than the number of inputs and outputs in the previous cycle, then control the storage system to perform garbage collection tasks in the cycle after the next cycle according to the task concurrency of less than the next cycle.

[0029] S131: If the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs of the two adjacent cycles are the same, then control the storage system to perform garbage collection tasks in the cycle after the next cycle with a task concurrency greater than that of the next cycle.

[0030] Before activating the garbage collection function of the storage system, check whether the storage system has a mechanism that relies on a threshold to start and stop the garbage collection function. If so, first cancel the mechanism that relies on a threshold to start and stop the garbage collection function, and then use the garbage collection method provided by this invention to perform the garbage collection task.

[0031] The method for controlling the start and stop of garbage collection is based on a threshold determined by the ratio of free blocks to total blocks in the storage system. The start of garbage collection indicates the existence of a large number of non-free blocks, which means that the performance of the storage system changes drastically after the garbage collection function is started. However, in the method provided by this invention, the garbage collection function of the storage system is started as soon as the storage pool is established, which avoids a large number of non-free blocks waiting to be reclaimed at the same time. Therefore, it alleviates the impact on the performance of the storage system after the garbage collection function is started and improves the efficiency of garbage collection.

[0032] From the moment the storage pool is created, the all-flash pool module of the storage system controls the initiation of the garbage collection function. The statistical period for the number of inputs and outputs of the storage system and the initial concurrency of garbage collection tasks are set. The duration of the period and the initial concurrency of garbage collection tasks are not limited and are determined based on actual conditions. Generally, the initial concurrency of garbage collection tasks is set to the minimum concurrency. The basic unit of garbage collection is grain, and multiple grains can be collected concurrently. The minimum concurrency can be an empirical value, such as the concurrency of one grain, or a measured value, such as the grain concurrency without affecting performance. The storage system can only start from the minimum concurrency when it starts. After the garbage collection function is initiated, the storage system is controlled to execute garbage collection tasks according to the initial garbage collection concurrency. Simultaneously, the number of inputs and outputs processed by the storage system within that period is obtained. In implementation, obtaining the number of inputs and outputs processed by the storage system includes:

[0033] Obtain the input / output stream processing path of the storage system;

[0034] Set up a statistics module on the input / output stream processing path to count the number of inputs and outputs;

[0035] The number of inputs and outputs processed by the storage system is obtained through the statistics module.

[0036] By embedding statistical functions into the front-end input / output stream processing path, user business processes are decoupled, thus avoiding any impact on business operations.

[0037] To compare the first relationship between task concurrency in two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, after initiating garbage collection of the storage system and before comparing the first relationship between task concurrency in two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, the following steps are also included:

[0038] Since the garbage collection function is detected to be started, garbage collection tasks are executed according to the current task concurrency within the current period, and the number of inputs and outputs processed by the storage system within the current period is obtained under the current task concurrency.

[0039] Control the current task concurrency to increase by a preset step size to obtain a new current task concurrency;

[0040] In the next cycle of the current cycle, perform garbage collection tasks according to the new current task concurrency, and under the new current task concurrency, obtain the number of inputs and outputs processed by the storage system in the next cycle of the current cycle.

[0041] The current cycle is the first cycle after the garbage collection function is started. Therefore, the current task volume corresponding to this cycle is set to the minimum concurrency level described above. During the first cycle, garbage collection tasks are executed at the minimum concurrency level, and the number of inputs and outputs processed by the storage system during the first cycle is obtained at this minimum concurrency level. In the second cycle, the concurrency level of the garbage collection task is increased by a preset step size. The preset step size is not limited; for example, if the preset step size is 1, the minimum concurrency level is increased by 1 to become the garbage collection task concurrency level for the second cycle, and the number of inputs and outputs processed by the storage system during the second cycle is obtained at this concurrency level.

[0042] After obtaining the task concurrency and the number of inputs and outputs processed by the storage system for each of two adjacent cycles, the task concurrency of these two cycles can be compared to obtain the first relationship; and the number of inputs and outputs processed by the storage system in these two cycles can be compared to obtain the second relationship.

[0043] If the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs in the next cycle is less than the number of inputs and outputs in the previous cycle, then the storage system is controlled to perform garbage collection tasks in cycles following the next cycle with a task concurrency less than that of the next cycle.

[0044] If two adjacent cycles are the first and second cycles after the garbage collection function is started, and the task concurrency in the first cycle is less than that in the second cycle, and the number of inputs and outputs processed by the storage system in the second cycle is less than that in the first cycle, then it is determined that increasing the garbage collection task affects the inputs and outputs processed by the storage system, which in turn affects the business. Therefore, in order to reduce the impact of the garbage collection task on the user's business, the storage system is controlled to perform garbage collection tasks with a lower task concurrency than that in the second cycle in cycles after the second cycle.

[0045] To improve the accuracy and precision of the determined concurrency levels, in implementation, controlling the storage system to perform garbage collection tasks in subsequent cycles at a lower task concurrency level than the previous cycle includes:

[0046] The concurrency of the task in the next cycle is used as the concurrency of the current task. The concurrency of the current task is controlled to decrease by a preset step size and a new concurrency of the current task is obtained so as to execute the garbage collection task with a concurrency of less than that of the next cycle.

[0047] Control the storage system to perform garbage collection tasks according to the new current task concurrency in the first cycle after the previous cycle.

[0048] If the task concurrency in the second cycle is reduced by a preset step size to obtain a new current task concurrency, this new current task concurrency is used as the task concurrency in the third cycle, and the storage system is controlled to perform garbage collection tasks according to the new current task concurrency in the third cycle.

[0049] After controlling the storage system to perform garbage collection tasks according to the new current task concurrency in the first cycle following the next cycle, it also includes:

[0050] Take the first period after the next period as the current period, and return the steps to get the number of inputs and outputs processed by the storage system in the next period under the new current task concurrency.

[0051] After comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, it also includes:

[0052] If the first relationship is that the task concurrency of the previous cycle is greater than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs in the next cycle is greater than the number of inputs and outputs in the previous cycle, then return to the step of taking the task concurrency of the next cycle as the current task concurrency, controlling the current task concurrency to decrease according to the preset step size and obtaining the new current task concurrency.

[0053] If the second relationship is that the number of inputs and outputs in two adjacent cycles is the same, then the control storage system will perform garbage collection tasks according to the task concurrency of the next cycle within the first preset duration.

[0054] In this method, the task concurrency of the next cycle is reduced by a preset step size. The reduced garbage collection concurrency is used as the garbage collection concurrency of the first cycle following the next cycle. The method continues to acquire the first and second relationships between two adjacent cycles. If, in the case that the task concurrency of the previous cycle is still greater than that of the next cycle, and the second relationship indicates that the input / output count of the next cycle is greater than that of the previous cycle, then the garbage collection task is still affecting user services. Therefore, the task concurrency is further reduced, and the analysis of the first and second relationships between two adjacent cycles continues until the input / output counts of two adjacent cycles are detected to be the same. Then, the storage system is controlled to execute the garbage collection task according to the task concurrency of the next cycle within a first preset time period. The first preset time period is not limited and is determined based on the actual situation.

[0055] Since user services vary across different periods, resulting in different input / output volumes processed by the storage system, this embodiment does not directly use the previous concurrency value. Instead, it redetermines a new concurrency value based on the storage system's concurrency and the amount of input / output processed, thus improving the accuracy of concurrency determination. Furthermore, by controlling the reduced concurrency according to a preset step size, the precision of concurrency determination is further enhanced.

[0056] After detecting that the number of inputs and outputs in two adjacent cycles of the second relationship is the same, and controlling the storage system to execute the garbage collection task according to the task concurrency of the next cycle within a first preset time period, in order to adapt the garbage collection task concurrency to the actual business scenario, after controlling the storage system to execute the garbage collection task according to the task concurrency of the next cycle within the first preset time period, the following is also included:

[0057] Get the current task concurrency within the first preset time period;

[0058] The current task concurrency is controlled to decrease by a preset step size and a new current task concurrency is obtained; the steps of comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system are returned.

[0059] In this method, after executing according to the determined concurrency level for a period of time, the concurrency level of the garbage collection task is further controlled according to the first and second relationships, so that the concurrency level of the garbage collection task can be adjusted according to the actual business scenario.

[0060] The above describes reducing task concurrency by a preset step size, thereby ensuring that the number of inputs and outputs processed by adjacent memory cycles is the same. In practice, to improve the efficiency of ensuring that the number of inputs and outputs processed by adjacent memory cycles is the same, the method further includes:

[0061] After controlling the storage system to perform garbage collection tasks with a lower task concurrency level than the next cycle in the cycle following the next cycle, and continuing to compare the first relationship between the task concurrency levels of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, if it is detected that the number of inputs and outputs of two adjacent cycles is the same, then the first task concurrency level corresponding to when the control storage system performs garbage collection tasks with a lower task concurrency level than the next cycle in the cycle following the next cycle is obtained; and the second task concurrency level corresponding to when the number of inputs and outputs of two adjacent cycles is the same is obtained.

[0062] Obtain the concurrency difference between multiple sets of concurrency levels for the first task and the second task;

[0063] Deep learning models are trained using multiple sets of first-task concurrency and concurrency differences;

[0064] Get the current first task concurrency when the control storage system performs garbage collection tasks in a period following the next period with a task concurrency lower than that of the next period;

[0065] Input the current first task concurrency into the completed deep learning model;

[0066] The current concurrency difference is output by the completed deep learning model;

[0067] The target concurrency is obtained by subtracting the current concurrency from the current concurrency of the first task.

[0068] Adjust the task concurrency to the target concurrency and execute the garbage collection task according to the target concurrency.

[0069] In this method, a deep learning model is trained by obtaining multiple sets of concurrency differences between the concurrency of the first task and the concurrency of the second task. The deep learning model then predicts the change in task concurrency when the number of inputs and outputs in two adjacent cycles are the same. Based on the predicted concurrency, the concurrency of garbage collection is controlled, which improves the efficiency of making the number of inputs and outputs processed by the storage system the same in two adjacent cycles and reduces the impact of garbage collection concurrency on user services.

[0070] The above describes how to determine the task concurrency in the period following two adjacent cycles when the first relationship is that the task concurrency in the preceding cycle is less than that in the following cycle, and the second relationship is that the number of inputs and outputs in the following cycle is less than that in the preceding cycle. Below, we continue to discuss how to determine the task concurrency in the period following two adjacent cycles when the first relationship is that the task concurrency in the preceding cycle is less than that in the following cycle, and the second relationship is that the number of inputs and outputs in the two adjacent cycles is the same. Specifically:

[0071] If the system detects that the task concurrency in the preceding cycle is less than that in the following cycle, and the second relationship is that the number of inputs and outputs in the two adjacent cycles is the same, then the storage system will execute garbage collection tasks with a higher task concurrency in cycles following the following cycle. If the task concurrency increases while the number of inputs and outputs processed by the storage system remains unchanged, then the garbage collection task concurrency will be increased, thus achieving reasonable utilization of off-peak periods.

[0072] In implementation, controlling the storage system to perform garbage collection tasks with a higher task concurrency level than the previous cycle in subsequent cycles includes:

[0073] The concurrency of the task in the next cycle is used as the concurrency of the current task. The process returns to the step of increasing the concurrency of the current task by a preset step size to obtain a new concurrency of the current task, so as to execute the garbage collection task with a concurrency greater than that of the next cycle.

[0074] Control the storage system to perform garbage collection tasks according to the new current task concurrency in the first cycle after the previous cycle.

[0075] For example, the storage system is controlled to perform garbage collection tasks in the third cycle, which is the cycle following the second cycle, with a lower task concurrency than the second cycle. To improve the accuracy of the determined concurrency, the task concurrency of the next cycle is increased by a preset step size to obtain a new current task concurrency. This new current task concurrency is then used as the task concurrency for the third cycle, and the storage system is controlled to perform garbage collection tasks according to the new current task concurrency in the third cycle.

[0076] After controlling the storage system to perform garbage collection tasks according to the new current task concurrency in the first cycle following the next cycle, it also includes:

[0077] Take the first cycle after the next cycle as the current cycle, and return the steps to get the number of inputs and outputs processed by the storage system in the next cycle of the current cycle under the new current task concurrency.

[0078] In this method, the concurrency of the task in the next cycle is increased by a preset step size, and the increased garbage collection concurrency is used as the garbage collection concurrency of the first cycle after the next cycle. The method continues to obtain the first and second relationships of the two adjacent cycles, and controls the concurrency of the garbage collection task according to the first and second relationships, so that the concurrency of the garbage collection task can be adjusted according to the actual business scenario.

[0079] In practice, business operations do sometimes stop. To further utilize these low-traffic periods, the garbage collection methods used in the storage system also include:

[0080] If the number of inputs and outputs of the storage system is equal to 0 for multiple consecutive cycles, the service is stopped, and the maximum concurrency among the preset task concurrency is obtained.

[0081] The control storage system performs garbage collection tasks at maximum concurrency within a second preset time period.

[0082] The second preset duration and maximum concurrency are not limited and can be determined based on actual conditions. In this method, if the number of inputs and outputs in the storage system is equal to 0 for multiple consecutive periods, the front-end service is considered to have stopped. After determining that the front-end service has stopped, this method executes a garbage collection task at the highest concurrency level, thus utilizing the off-peak period and providing more free blocks for subsequent user service processing.

[0083] In practice, a situation might arise where high concurrency in front-end services causes the storage system to process zero input / output operations, leading to the mistaken conclusion that front-end user services have stopped. To avoid this, in implementation, determining service stoppage includes:

[0084] If the number of inputs and outputs of the storage system is detected to be 0 for the first time, the garbage collection function of the storage system will be turned off.

[0085] If, after the garbage collection function is turned off, the number of inputs and outputs of the storage system is detected to be equal to 0 for multiple consecutive cycles, the service is determined to be stopped.

[0086] After confirming the business has stopped, and before retrieving the maximum concurrency from the preset task concurrency, the following steps are also included:

[0087] Enables the garbage collection function of the control storage system.

[0088] This method completely stops garbage collection when the first count reaches 0. It then checks if the count remains 0 for several cycles after the initial stop; if so, the service is considered to have stopped. This improves the accuracy of determining service termination.

[0089] User business operations may change after a garbage collection task is executed at maximum concurrency. To minimize the impact of garbage collection on user business operations and ensure its efficiency, the implementation includes the following steps after the storage system executes the garbage collection task at maximum concurrency for a second preset period:

[0090] Use the maximum concurrency as the current task concurrency;

[0091] Starting from the execution of the garbage collection task at maximum concurrency, the steps include obtaining the number of inputs and outputs processed by the storage system within a third preset time period, and returning to the steps of comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system.

[0092] The third preset duration is not limited and is determined based on the actual situation. In this method, after running at the maximum concurrency, the concurrency of the garbage collection task is re-determined after a period of time to reduce the impact of garbage collection tasks on user services and ensure the efficiency of garbage collection.

[0093] To enable those skilled in the art to better understand the garbage collection method of the storage system provided by the present invention, the above process will continue to be described below with reference to the accompanying drawings and specific embodiments. Figure 2 A flowchart illustrating an overall process for a garbage collection method in a storage system, as provided in an embodiment of the present invention, is shown below. Figure 2 As shown, the method includes:

[0094] S14: Full flash pool created successfully, garbage collection function started;

[0095] S15: Waste collection runs with minimal concurrency;

[0096] S16: Count the number of front-end inputs and outputs processed by the storage system after running one cycle at the lowest concurrency level;

[0097] S17: Increase the concurrency of garbage collection tasks;

[0098] S18: Count the number of front-end inputs and outputs after a new concurrent operation cycle;

[0099] S19: Compare with the previous statistics to determine if they are basically consistent with the previous ones; if yes, return to step S17; if no, proceed to step S20.

[0100] S20: If a value less than the previous value is detected, reduce the concurrency of garbage collection tasks;

[0101] S21: Reduce the number of front-end inputs and outputs after a concurrent operation cycle;

[0102] S22: Continue to compare with the previous statistics to determine if it is basically consistent with the previous one; if not, return to step S20; if yes, proceed to step S23.

[0103] S23: After running at the current concurrency level for a period of time, re-initiate the change of concurrency level; return to step S17;

[0104] S24: If the number of inputs and outputs processed by the storage system is 0 within multiple cycles, which meets the interruption criteria, then garbage collection will run at the highest concurrency.

[0105] In implementation, (1) firstly, the working mechanism of starting and stopping garbage collection based on threshold is cancelled, and the garbage collection function is started after the full flash pool is created;

[0106] (2) Implement an IO statistics function on the front-end IO stream processing path to count the number of front-end IOs processed within a unit period, for example, the statistics period is 5 seconds;

[0107] (3) The garbage collection function starts working with the lowest concurrency after startup and counts the number of front-end IOs processed per unit cycle under this concurrency intensity;

[0108] (4) After a statistical period, the task concurrency of the garbage collection function is increased by 1, and the number of front-end IOs processed in the new period is counted. If the number of IOs counted after the end of the period is basically consistent with (3), it means that the current garbage collection concurrency will not affect the business pressure of the front end, and the task concurrency can be increased.

[0109] (5) Repeat step (4) and compare the number of I / Os with the number of I / Os in the previous statistical cycle at the end of each statistical cycle;

[0110] (6) Until the number of IOs in the latest statistical period is less than the number of IOs in the previous period, it indicates that the increased concurrency of garbage collection tasks has affected the front-end IO processing capacity. It is necessary to reduce the concurrency and continue to count the number of IOs in the period after the reduction.

[0111] (7) Repeat step (6) until the number of IOs in the latest statistical period is basically the same as the previous one. Then, garbage collection can be performed continuously for a period of time with this concurrency, such as for a duration of 1 minute.

[0112] (8) After running for 1 minute, the user load may increase on the basis of the existing task concurrency. Therefore, continue to reduce the concurrency and compare the number. If the number of IOs after reducing the concurrency is the same as before the reduction, then steps (4), (5) and (6) can be executed repeatedly. If the number of IOs after reducing the concurrency is higher than before the reduction, then step (6) needs to be executed repeatedly.

[0113] (9) If the number of IOs counted within the period is 0, it is considered that the current front-end business has stopped. Garbage collection can be performed at the highest concurrency and re-detected after a period of time.

[0114] The garbage collection method for storage systems provided by this invention solves the performance impact caused by using a single threshold as a monitoring condition in related technologies, especially the inability to recognize the actual scenario and the impact on peak user business periods. By dynamically acquiring the front-end IO pressure and actively adjusting the control mechanism of garbage collection task concurrency, it achieves reasonable utilization of the low user business pressure period and the staggering of the peak user business pressure period. While ensuring the efficiency of garbage collection, it greatly improves system performance, alleviates the performance fluctuation problem during operation, and mitigates the performance impact of the all-flash stack storage pool module after garbage collection is started by optimizing the start timing of garbage collection.

[0115] This invention provides a method for dynamically adjusting the concurrency of garbage collection tasks after initiation, judging the current business pressure based on the front-end IO processing volume under different concurrency levels, and determining the appropriate garbage collection task concurrency. By comprehensively considering multiple thresholds to determine the concurrency intensity of garbage collection tasks, and optimizing the start and stop timing of garbage collection tasks, the impact of garbage collection on performance throughout the entire business operation is optimized. In particular, it achieves adaptation and identification to different business scenarios, ensuring that the storage system can provide reliable performance regardless of whether the business is in a peak or off-peak period.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0117] Embodiments of the present invention also provide a waste recycling device for a storage system, comprising:

[0118] The startup module is used to initiate the garbage collection function of the storage system in response to the completion of storage pool creation;

[0119] The comparison module is used to compare the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system;

[0120] The first control module is used to control the storage system to perform garbage collection tasks in the period following the next period if the task concurrency of the previous period is less than that of the next period in the first relationship of two adjacent periods, and if the number of inputs and outputs in the next period is less than that of the previous period in the second relationship, then the storage system is controlled to perform garbage collection tasks with a task concurrency of less than that of the next period in the period following the next period.

[0121] The second control module is used to control the storage system to perform garbage collection tasks in the period following the next period with a task concurrency greater than that of the next period if the number of inputs and outputs in two adjacent cycles are the same.

[0122] In some embodiments, the waste collection device of the storage system further includes:

[0123] The first acquisition module is used to execute garbage collection tasks according to the current task concurrency within the current period from the time the garbage collection function is detected to start, and to acquire the number of inputs and outputs processed by the storage system within the current period under the current task concurrency.

[0124] The third control module is used to control the current task concurrency to increase according to a preset step size to obtain a new current task concurrency.

[0125] The second acquisition module is used to execute garbage collection tasks according to the new current task concurrency in the next cycle of the current cycle, and to acquire the number of inputs and outputs processed by the storage system in the next cycle of the current cycle under the new current task concurrency.

[0126] In some embodiments, the second control module includes:

[0127] The first module is used to take the task concurrency of the next cycle as the current task concurrency, and return to trigger the third control module to obtain a task concurrency greater than that of the next cycle to execute the garbage collection task.

[0128] The fourth control module is used to control the storage system to perform garbage collection tasks according to the new current task concurrency in the first cycle after the previous cycle;

[0129] The waste recycling unit of the storage system also includes:

[0130] The second module is used to take the first cycle after the next cycle as the current cycle and return to trigger the second acquisition module.

[0131] In some embodiments, the first control module includes:

[0132] The third acquisition module is used to take the task concurrency of the next cycle as the current task concurrency, control the current task concurrency to decrease according to a preset step size and acquire a new current task concurrency, so as to obtain a task concurrency smaller than that of the next cycle to execute the garbage collection task.

[0133] The fifth control module is used to control the storage system to perform garbage collection tasks according to the new current task concurrency in the first cycle after the previous cycle;

[0134] The third module is used to take the first cycle after the next cycle as the current cycle and return to trigger the second acquisition module;

[0135] The waste recycling unit of the storage system also includes:

[0136] The detection module is used to trigger the third acquisition module if, when the first relationship is that the task concurrency of the previous cycle is greater than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs in the next cycle is greater than the number of inputs and outputs in the previous cycle, it returns.

[0137] The sixth control module is used to control the storage system to perform garbage collection tasks according to the task concurrency of the next cycle within a first preset time period if the second relationship is that the number of inputs and outputs in two adjacent cycles is the same.

[0138] In some embodiments, the waste collection device of the storage system further includes:

[0139] The fourth acquisition module is used to acquire the current task concurrency within the first preset time period;

[0140] The seventh control module is used to control the current task concurrency to decrease according to a preset step size and obtain the new current task concurrency; it returns to the trigger comparison module.

[0141] In some embodiments, the waste collection device of the storage system further includes:

[0142] The determination and acquisition module is used to determine the service to stop when the number of inputs and outputs of the storage system is equal to 0 for multiple consecutive periods, and to acquire the maximum concurrency among the preset task concurrency.

[0143] The eighth control module is used to control the storage system to perform garbage collection tasks at the maximum concurrency level within a second preset time period.

[0144] In some embodiments, the waste collection device of the storage system further includes:

[0145] The fourth module is used to set the maximum concurrency as the current task concurrency.

[0146] The fifth acquisition module is used to acquire the number of inputs and outputs processed by the storage system within a third preset time period after the garbage collection task is executed at the maximum concurrency level, and return the result to the trigger comparison module.

[0147] For a description of the features of the garbage collection device in the embodiment of the storage system, please refer to the relevant description of the garbage collection method in the embodiment of the storage system, which will not be repeated here.

[0148] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in the garbage collection method embodiments of any of the above-described storage systems.

[0149] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the garbage collection method embodiments of any of the above storage systems when running.

[0150] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0151] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the garbage collection method for a storage system.

[0152] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the garbage collection method embodiments of any of the above storage systems.

[0153] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] The above provides a detailed description of a garbage collection method and electronic device for a storage system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for garbage collection in a storage system, characterized in that, include: In response to the completion of storage pool creation, the garbage collection function of the storage system is started; Compare the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system; If the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs in the next cycle is less than the number of inputs and outputs in the previous cycle, then the storage system is controlled to perform garbage collection tasks in cycles following the next cycle with a task concurrency less than that of the next cycle. If the second relationship is that the number of inputs and outputs in two adjacent cycles is the same, then the control storage system performs garbage collection tasks in the cycle following the next cycle at a task concurrency greater than that of the next cycle. Also includes: If the number of inputs and outputs of the storage system is equal to 0 for multiple consecutive cycles, the service is stopped, and the maximum concurrency among the preset task concurrency is obtained. The control storage system performs garbage collection tasks according to the maximum concurrency within a second preset time period; If the number of inputs and outputs of the storage system is detected to be 0 for the first time, the garbage collection function of the storage system will be turned off. If, after the garbage collection function is turned off, the number of inputs and outputs of the storage system is detected to be equal to 0 for multiple consecutive cycles, the service is determined to be stopped. After determining that the service has stopped, and before obtaining the maximum concurrency among the preset task concurrency levels, the process further includes: Enables the garbage collection function of the control storage system.

2. The waste recycling method for a storage system according to claim 1, characterized in that, After initiating the garbage collection function of the storage system and before comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, the method further includes: Since the garbage collection function is detected to be started, the garbage collection task is executed according to the current task concurrency in the current cycle, and the number of inputs and outputs processed by the storage system in the current cycle is obtained under the current task concurrency. The current task concurrency is controlled to increase by a preset step size to obtain a new current task concurrency; In the next cycle of the current cycle, the garbage collection task is executed according to the new current task concurrency, and under the new current task concurrency, the number of inputs and outputs processed by the storage system in the next cycle of the current cycle is obtained.

3. The waste recycling method for the storage system according to claim 2, characterized in that, Controlling the storage system to perform garbage collection tasks in cycles following the next cycle at a higher task concurrency level than the next cycle includes: The task concurrency of the next cycle is used as the current task concurrency. The process returns to the step of controlling the current task concurrency to increase by a preset step size to obtain a new current task concurrency, so as to obtain a task concurrency greater than that of the next cycle and perform garbage collection tasks. The control storage system performs garbage collection tasks according to the new current task concurrency in the first cycle following the last cycle; After the control storage system performs a garbage collection task according to the new current task concurrency in the first cycle following the latter cycle, the method further includes: Take the first cycle after the last cycle as the current cycle, and return to the step of obtaining the number of inputs and outputs processed by the storage system in the next cycle of the current cycle under the new current task concurrency.

4. The waste recycling method for the storage system according to claim 2, characterized in that, The control storage system performs garbage collection tasks in cycles following the next cycle at a task concurrency level less than that of the next cycle, including: The task concurrency of the next cycle is used as the current task concurrency. The current task concurrency is controlled to decrease according to a preset step size and a new current task concurrency is obtained so as to obtain a task concurrency smaller than that of the next cycle to perform garbage collection tasks. The control storage system performs garbage collection tasks according to the new current task concurrency in the first cycle following the last cycle; After the control storage system performs a garbage collection task according to the new current task concurrency in the first cycle following the latter cycle, the method further includes: Take the first cycle after the last cycle as the current cycle, and return to the step of obtaining the number of inputs and outputs processed by the storage system in the next cycle of the current cycle under the new current task concurrency. After comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, the following is also included: If the first relationship is that the task concurrency of the previous cycle is greater than the task concurrency of the next cycle in two adjacent cycles, and if the second relationship is that the number of inputs and outputs in the next cycle is greater than the number of inputs and outputs in the previous cycle, then return to the step of taking the task concurrency of the next cycle as the current task concurrency, controlling the current task concurrency to decrease according to a preset step size and obtaining a new current task concurrency. If the second relationship is that the number of inputs and outputs in two adjacent cycles is the same, then the storage system is controlled to perform garbage collection tasks according to the task concurrency of the next cycle within a first preset time period.

5. The waste recycling method for the storage system according to claim 4, characterized in that, After controlling the storage system to perform garbage collection tasks according to the task concurrency of the next cycle within a first preset time period, the method further includes: Get the current task concurrency within the first preset time period; Control the current task concurrency to decrease according to the preset step size and obtain a new current task concurrency; return to the step of comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system.

6. The waste recycling method for a storage system according to claim 1, characterized in that, The number of inputs and outputs processed by the storage system includes: Obtain the input / output stream processing path of the storage system; Set up a statistics module on the input / output stream processing path to count the number of inputs and outputs; The statistics module obtains the number of inputs and outputs processed by the storage system.

7. The waste recycling method for a storage system according to claim 6, characterized in that, After the control storage system performs the garbage collection task according to the maximum concurrency within a second preset time period, the method further includes: Use the maximum concurrency as the current task concurrency; Starting from the start of the garbage collection task at the maximum concurrency level, the number of inputs and outputs processed by the storage system is obtained within a third preset time period, and the step of comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system is returned.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the garbage collection method for the storage system as described in any one of claims 1 to 7 when executing the computer program.

Images