Garbage collection method of storage system and electronic equipment

Abstract

The invention discloses a garbage collection method of a storage system and electronic equipment, and relates to the technical field of storage systems. According to the method, the concurrency amount of the garbage collection tasks is controlled based on the relation between the concurrency amount of the garbage collection tasks and the number of IO processed by a storage system, and garbage collection of the storage system is controlled according to the determined concurrency amount; when the concurrency of the garbage collection tasks is increased and the IO quantity processed by the storage system is reduced, the concurrency of the garbage collection tasks is reduced, so that the influence on the service peak period is reduced, and the time delay of the user service is reduced; when the concurrent quantity of the garbage collection tasks is increased and the IO quantity processed by the storage system is the same, garbage collection is controlled according to the increased concurrent quantity of the garbage collection tasks, so that reasonable utilization of user services in a valley period is realized; in addition, after the storage pool is established, the garbage recycling function of the storage system is started, the impact on the performance of the storage system after the garbage recycling function is started is relieved, and the garbage recycling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of storage systems, and in particular to a garbage collection method and electronic equipment for a storage system. Background Art

[0002] In order to solve the problem of storage pool fragmentation and realize the recycling of physical space, a garbage collection function is usually installed in the data storage system.

[0003] The related garbage collection function uses a threshold determined by the ratio of free blocks to total blocks in the storage system to control its activation and deactivation. Garbage collection will not be executed until the ratio reaches the threshold. Only after the threshold is reached will garbage collection be executed. The activation of garbage collection indicates the presence of a large number of non-free blocks. Therefore, processing these blocks consumes significant resources, reducing the resources available for front-end input / output (IO). This significantly reduces storage system performance after garbage collection is activated. During peak service hours, this garbage collection feature can significantly increase service latency, severely impacting user experience.

[0004] Therefore, how to alleviate the impact of the garbage collection function on storage system performance after it is enabled and reduce the impact on user services is a technical problem that people in this field urgently need to solve. Summary of the Invention

[0005] The present invention provides a garbage collection method and electronic device for a storage system, which at least solves the problem in the related art that after the garbage collection function is activated, the performance of the storage system will change dramatically; during the user's business peak period, the garbage collection function causes the user's business delay to increase significantly, seriously affecting user use.

[0006] The present invention provides a garbage collection method for a storage system, comprising: In response to the storage pool being created, starting a garbage collection function of the storage system; comparing a first relationship between task concurrency amounts in two adjacent cycles and a second relationship between the number of inputs and outputs processed by the storage system; When it is detected that the first relationship is that the task concurrency of the previous cycle of two adjacent cycles is less than the task concurrency of the next cycle, and if the second relationship is that the input and output quantity of the next cycle is less than the input and output quantity of the previous cycle, the storage system is controlled to perform garbage collection tasks in a cycle after the next cycle according to the task concurrency less than that of the next cycle; If the second relationship is that the input and output quantities of two adjacent cycles are the same, the storage system is controlled to execute the garbage collection task in a cycle after the latter cycle according to a task concurrency greater than that of the latter cycle.

[0007] The beneficial effects of the present invention are that, firstly, in this method, the concurrency of garbage collection tasks is controlled based on the relationship between the concurrency of garbage collection tasks and the number of inputs and outputs processed by the storage system, and the garbage collection of the storage system is controlled according to the determined concurrency of garbage collection tasks, thereby realizing garbage collection of the storage system; secondly, in the process of controlling the concurrency of garbage collection tasks based on the relationship between the concurrency of garbage collection tasks and the number of inputs and outputs processed by the storage system, when the concurrency of garbage collection tasks increases and the number of inputs and outputs processed by the storage system decreases, the concurrency of garbage collection tasks is reduced, thereby reducing the impact of garbage collection tasks on business peak periods and reducing the latency of user services; thirdly, when the concurrency of garbage collection tasks increases and the number of inputs and outputs processed by the storage system decreases, the concurrency of garbage collection tasks is reduced, thereby reducing the impact of garbage collection tasks on business peak periods and reducing the latency of user services; If the number of input and output processed by the system is the same, the concurrency of the garbage collection task is increased, and the garbage collection is controlled according to the increased concurrency of the garbage collection task, thereby realizing the reasonable utilization of the user business low period; in addition, compared with the method of controlling the start and stop of the garbage collection function based on the threshold determined by the ratio of the free blocks to the total blocks in the storage system, the startup of the garbage collection function indicates that there are a large number of non-free blocks, that is, the performance of the storage system changes drastically after the garbage collection function is started. In the method provided by the present invention, the garbage collection function of the storage system is started after the storage pool is established, which avoids the existence of a large number of non-free blocks waiting for recovery at the same time. Therefore, the impact of the garbage collection function on the storage system performance after the startup is alleviated, and the efficiency of garbage collection is improved.

[0008] The present invention also provides an electronic device, comprising: memory for storing computer programs; A processor is used to implement the steps of the garbage collection method of the above storage system when executing a computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 A flowchart of a garbage collection method for a storage system provided by an embodiment of the present invention; Figure 2 This is an overall flow chart of a garbage collection method for a storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0012] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0013] The related garbage collection function is mostly a background task, and its start and stop are controlled by a threshold. This threshold is generally a specific ratio of free blocks to total blocks. Generally speaking, the garbage collection background task will regularly monitor the threshold. Before the threshold is reached, no actual garbage collection work will be performed. Only after the threshold is reached will metadata queries and modifications be performed on valid data to achieve data migration or the recovery of invalid space. Under the action of garbage collection, the number of free blocks will generally increase. When the startup threshold is not met, garbage collection will stop working again until the next condition is met. This garbage collection function also brings a series of problems, including the need to retain reserved space, resulting in low utilization of users' actual physical space. Among these, performance issues are the most significant. During peak business hours, the related garbage collection function will cause a significant increase in user business latency and cannot be alleviated, seriously affecting user usage.

[0014] As described above, the garbage collection function only starts when the ratio of free blocks to total blocks reaches a certain threshold. This results in a sharp drop in performance around the threshold. This is because starting the garbage collection function consumes significant resources such as the Central Processing Unit (CPU) and memory, reducing the resources available for front-end I / O. Furthermore, the garbage collection function uses a single threshold not only as a reference for starting and stopping tasks but also as the sole reference for controlling task concurrency. A high number of free blocks results in low concurrency and slow collection speeds, while a low number of free blocks increases task concurrency and speeds up collection. This also means that to ensure a constant number of free blocks, the lower the number of free blocks, the more resources garbage collection preempts, resulting in poorer front-end performance. Consequently, in real-world production, the performance of the garbage collection function drops sharply as the amount of written data increases, and then gradually deteriorates until it reaches a steady state.

[0015] In order to alleviate the impact of the garbage collection function on performance, especially to reduce the impact on user service peak latency, an embodiment of the present invention provides a new garbage collection method for a storage system.

[0016] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1 A flowchart of a garbage collection method for a storage system provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: S11: In response to the storage pool being created, starting the garbage collection function of the storage system; S12: comparing a first relationship between task concurrency amounts in two adjacent cycles and a second relationship between the number of inputs and outputs processed by the storage system; S13: Execute the garbage collection task using different strategies according to the situation of the second relationship.

[0017] Specifically, in step S13, it includes: S130: When it is detected that the first relationship is that the task concurrency of the previous cycle of two adjacent cycles is less than the task concurrency of the next cycle, and if the second relationship is that the input and output quantity of the next cycle is less than the input and output quantity of the previous cycle, controlling the storage system to perform garbage collection tasks in a cycle after the next cycle according to the task concurrency less than that of the next cycle; S131: When it is detected that the first relationship is that the task concurrency of the previous cycle in two adjacent cycles is less than the task concurrency of the next cycle, if the second relationship is that the input and output numbers of the two adjacent cycles are the same, the storage system is controlled to perform garbage collection tasks in the cycle after the next cycle according to the task concurrency greater than that of the next cycle.

[0018] Before starting the garbage collection function of the storage system, check whether the storage system has a working mechanism that relies on a threshold to start and stop the garbage collection function. If so, first cancel the working mechanism that relies on the threshold to start and stop the garbage collection function, and then use the garbage collection method provided by the present invention to perform the garbage collection task.

[0019] The method of controlling the start and stop of the garbage collection function is based on a threshold value determined based on the ratio of free blocks to total blocks in the storage system. The start of the garbage collection function indicates that there are a large number of non-free blocks, that is, the performance of the storage system changes drastically after the garbage collection function is started. In the method provided by the present invention, the garbage collection function of the storage system is turned on after the storage pool is established, avoiding the existence of a large number of non-free blocks waiting for recovery at the same time. Therefore, the impact of the garbage collection function on the storage system performance after the startup is alleviated, and the efficiency of garbage collection is improved.

[0020] From the moment the storage pool is created, the all-flash pool module of the storage system controls the startup of the garbage collection function of the storage system. Set the statistical period for the input and output quantities of the storage system and the initial concurrency of the garbage collection task. There is no limit on the duration of the period and the initial concurrency of the garbage collection task, which are determined according to actual conditions. Generally, the initial concurrency of the garbage collection task is set to the minimum concurrency. The basic unit of garbage collection is grain. Recycling can be performed concurrently by multiple grains. The minimum concurrency can be an empirical value, such as one grain, or a measured value, such as the grain concurrency without affecting performance. The storage system can only start from the minimum concurrency when it starts. After the garbage collection function is started, the storage system is controlled to execute the garbage collection task according to the initial garbage collection concurrency. At the same time, the number of inputs and outputs processed by the storage system during the period is obtained. In implementation, obtaining the number of inputs and outputs processed by the storage system includes: Get the input and output stream processing path of the storage system; A statistical module for counting input and output quantities is set on the input and output flow processing path; Use the statistics module to obtain the number of input and output processed by the storage system.

[0021] By embedding statistical functions in the front-end input and output stream processing path, user business decoupling is achieved, avoiding the impact on the business.

[0022] To compare a first relationship between task concurrency amounts in two adjacent cycles and a second relationship between the number of inputs and outputs processed by the storage system, after enabling a garbage collection function of the storage system and before comparing the first relationship between task concurrency amounts in two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system, the method further includes: After detecting that the garbage collection function is enabled, the system executes garbage collection tasks according to the current task concurrency within the current cycle. The system also obtains the number of inputs and outputs processed by the storage system within the current cycle under the current task concurrency. Control the current task concurrency to increase according to the preset step size to obtain the new current task concurrency; In the next cycle after the current cycle, garbage collection tasks are executed according to the new current task concurrency. Under the new current task concurrency, the number of input and output processed by the storage system in the next cycle after the current cycle is obtained.

[0023] If the current cycle is the first cycle after the garbage collection function is started, the current task volume corresponding to the cycle is set to the minimum concurrency described above. In the first cycle, the garbage collection task is executed according to the minimum concurrency. At the same time, the number of inputs and outputs processed by the storage system in the first cycle is obtained at the minimum concurrency. In the second cycle, the concurrency of the garbage collection task is increased according to the preset step size. There is no limit on the preset step size. For example, if the preset step size is 1, the minimum concurrency is added by 1 in the second cycle as the concurrency of the garbage collection task in the second cycle. At the same time, the number of inputs and outputs processed by the storage system in the second cycle is obtained at this concurrency.

[0024] After obtaining the task concurrency corresponding to two adjacent cycles and the number of inputs and outputs processed by the storage system, the task concurrency of the two cycles can be compared to obtain the first relationship; and the number of inputs and outputs processed by the storage system in the two cycles can be compared to obtain the second relationship.

[0025] When it is detected that the first relationship is that the task concurrency of the previous cycle in two adjacent cycles is less than the task concurrency of the next cycle, if the second relationship is that the input and output number of the next cycle is less than the input and output number of the previous cycle, the storage system is controlled to perform garbage collection tasks in the cycle after the next cycle according to the task concurrency less than that of the next cycle.

[0026] If two adjacent cycles are the first cycle and the second cycle after the garbage collection function is started, if the task concurrency of the first cycle is less than the task concurrency of the second cycle, the number of inputs and outputs processed by the storage system in the second cycle is less than the number of inputs and outputs processed by the storage system in the first cycle, that is, it is determined that adding the garbage collection task affects the inputs and outputs processed by the storage system, that is, it affects the business. Therefore, in this case, in order to reduce the impact of the garbage collection task on the user's business, the storage system is controlled to perform the garbage collection task in the cycle after the second cycle according to the task concurrency that is less than the second cycle.

[0027] To improve the precision and accuracy of the determined concurrency, in implementation, controlling the storage system to perform garbage collection tasks in a cycle following a subsequent cycle according to a task concurrency smaller than that of the subsequent cycle includes: The task concurrency of the next cycle is used as the current task concurrency, and the current task concurrency is controlled to decrease according to the preset step size and a new current task concurrency is obtained to obtain a task concurrency smaller than the task concurrency of the next cycle to perform the garbage collection task; The storage system is controlled to execute the garbage collection task according to the new current task concurrency in the first cycle after the next cycle.

[0028] For example, the task concurrency of the second cycle is reduced according to a preset step size to obtain a new current task concurrency, and the new current task concurrency is used as the task concurrency of the third cycle. The storage system is controlled to perform garbage collection tasks according to the new current task concurrency in the third cycle.

[0029] After controlling the storage system to perform the garbage collection task according to the new current task concurrency in the first cycle after the next cycle, the method further includes: Set the first cycle after the next cycle as the current cycle, and return to the step of obtaining the number of inputs and outputs processed by the storage system in the cycle after the current cycle under the new current task concurrency; After comparing the first relationship between the task concurrency amounts of two adjacent cycles and the second relationship between the input and output quantities processed by the storage system, the method further includes: In the case where it is detected that the first relationship is that the task concurrency of the previous cycle is greater than the task concurrency of the next cycle between two adjacent cycles, if the second relationship is that the number of inputs and outputs of the next cycle is greater than the number of inputs and outputs of the previous cycle, then returning to the step of using the task concurrency of the next cycle as the current task concurrency, controlling the current task concurrency to decrease according to a preset step size, and obtaining a new current task concurrency; If the second relationship is that the input and output quantities of two adjacent cycles are the same, the storage system is controlled to perform the garbage collection task according to the task concurrency of the next cycle within the first preset time period.

[0030] In this method, the task concurrency of the next cycle is reduced according to a preset step size, and the reduced garbage collection concurrency is used as the garbage collection concurrency of the first cycle after the next cycle, and the first and second relationships of the two new adjacent cycles are continuously obtained. If the task concurrency of the previous cycle is still greater than the task concurrency of the next cycle in the two adjacent cycles, and the second relationship is that the input and output quantity of the next cycle is greater than the input and output quantity of the previous cycle, it means that the garbage collection task still affects the user business, then the task concurrency will continue to be reduced, and the first and second relationships of the two adjacent cycles will continue to be analyzed until it is detected that the input and output quantities of the two adjacent cycles are the same, then the storage system will be controlled to perform the garbage collection task according to the task concurrency of the next cycle within the first preset time length. There is no limit on the first preset time length, which is determined according to actual conditions.

[0031] Since user services vary in different periods, the number of input and output processed by the storage system varies. Therefore, in this embodiment, the previous concurrency value is not directly used. Instead, a new concurrency value is re-determined based on the concurrency of the storage system and the input and output volume processed, thereby improving the accuracy of the concurrency determination. Moreover, by controlling the reduced concurrency according to a preset step size, the accuracy of the concurrency determination is improved.

[0032] After detecting that the second relationship is that the input and output quantities of two adjacent cycles are the same, controlling the storage system to perform the garbage collection task according to the task concurrency of the subsequent cycle within the first preset duration, in order to adapt the garbage collection task concurrency to an actual business scenario, after controlling the storage system to perform the garbage collection task according to the task concurrency of the subsequent cycle within the first preset duration, the method further includes: Get the current task concurrency corresponding to the first preset duration; Control the current task concurrency to decrease according to a preset step size and obtain a new current task concurrency; return to the step of comparing the first relationship between the task concurrencies of two adjacent cycles and the second relationship between the input and output quantities processed by the storage system.

[0033] In this method, after executing for a period of time according to the determined concurrency, the concurrency of the garbage collection task continues to be controlled according to the first relationship and the second relationship, so that the concurrency of the garbage collection task can be adjusted according to the actual business scenario.

[0034] The above describes reducing the task concurrency according to a preset step size so that the number of inputs and outputs processed by the storage system in two adjacent cycles is the same. In practice, in order to improve the efficiency of ensuring that the number of inputs and outputs processed by the storage system in two adjacent cycles is the same, the method further includes: After controlling the storage system to perform the garbage collection task in a cycle following the next cycle according to a task concurrency smaller than that of the next cycle, and continuing to compare the first relationship between the task concurrencies of two adjacent cycles and the second relationship between the input and output quantities processed by the storage system, if it is detected that the second relationship is that the input and output quantities of the two adjacent cycles are the same, obtaining the first task concurrency corresponding to when the storage system is controlled to perform the garbage collection task in a cycle following the next cycle according to a task concurrency smaller than that of the next cycle; and obtaining the second task concurrency corresponding to when the second relationship is that the input and output quantities of the two adjacent cycles are the same; Obtaining the concurrency difference between the concurrency of the first task and the concurrency of the second task; Use multiple sets of first task concurrency and concurrency differences to train a deep learning model; Obtaining the current first task concurrency corresponding to when the control storage system executes the garbage collection task in a cycle after the next cycle according to the task concurrency smaller than the next cycle; Input the current first task concurrency into the trained deep learning model; Output the current concurrency difference through the trained deep learning model; Get the target concurrency obtained by subtracting the difference between the current concurrency of the first task and the current concurrency. Adjust the task concurrency to the target concurrency and execute garbage collection tasks according to the target concurrency.

[0035] In this method, a deep learning model is trained by obtaining the concurrency difference between multiple groups of first task concurrency and second task concurrency, so that the change in task concurrency when the input and output numbers of two adjacent cycles are the same is predicted by the deep learning model. Then, the concurrency of garbage collection is controlled according to the predicted concurrency, thereby improving the efficiency of making the input and output numbers processed by the storage system the same in two adjacent cycles and reducing the impact of garbage collection concurrency on user business.

[0036] The above describes a method for determining the task concurrency in the cycle after two adjacent cycles in a scenario where the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs of the next cycle is less than the number of inputs and outputs of the previous cycle. The following describes a method for determining the task concurrency in the cycle after two adjacent cycles in a scenario where it is detected that the first relationship is that the task concurrency of the previous cycle is less than the task concurrency of the next cycle, and the second relationship is that the number of inputs and outputs of the two adjacent cycles is the same. Specifically: If the first relationship is that the task concurrency of the first cycle is less than that of the second cycle, and the second relationship is that the input and output quantities of the two cycles are the same, the storage system is controlled to execute garbage collection tasks in the cycle after the second cycle at a task concurrency greater than that of the second cycle. If the task concurrency increases but the number of input and output processed by the storage system remains unchanged, the garbage collection task concurrency is increased, effectively utilizing business low periods.

[0037] During implementation, controlling the storage system to execute the garbage collection task in a cycle subsequent to the next cycle according to a task concurrency greater than that of the next cycle includes: The task concurrency of the next cycle is used as the current task concurrency, and the step of controlling the current task concurrency to increase according to the preset step size to obtain a new current task concurrency is returned, so as to obtain a task concurrency greater than the task concurrency of the next cycle to execute the garbage collection task; The storage system is controlled to execute the garbage collection task according to the new current task concurrency in the first cycle after the next cycle.

[0038] For example, the storage system is controlled to execute garbage collection tasks in the third cycle after the second cycle, i.e., at a task concurrency smaller than that of the second cycle. To improve the accuracy of the determined concurrency, the task concurrency of the subsequent cycle is increased by a preset step size to obtain a new current task concurrency. This new current task concurrency is used as the task concurrency of the third cycle, and the storage system is controlled to execute garbage collection tasks in the third cycle at the new current task concurrency.

[0039] After controlling the storage system to perform the garbage collection task according to the new current task concurrency in the first cycle after the next cycle, the method further includes: The first cycle after the next cycle is taken as the current cycle, and the step of obtaining the number of input and output processed by the storage system in the next cycle after the current cycle under the new current task concurrency is returned.

[0040] In this method, the task concurrency of the next cycle is increased according to a preset step size, and the increased garbage collection concurrency is used as the garbage collection concurrency of the first cycle after the next cycle, and the first relationship and second relationship of the two new adjacent cycles are continuously obtained. The concurrency of the garbage collection tasks is controlled according to the first relationship and the second relationship, so that the concurrency of the garbage collection tasks can be adjusted according to the actual business scenario.

[0041] In practice, business may sometimes stop. To further utilize the business downturn, the storage system's garbage collection method also includes the following: When it is detected that the input and output quantity of the storage system is equal to 0 for multiple consecutive cycles, the service is determined to be stopped, and the maximum concurrency among the preset task concurrency quantities is obtained; The storage system is controlled to execute garbage collection tasks according to the maximum concurrency within a second preset time period.

[0042] The second preset duration and maximum concurrency are not limited and are determined based on actual conditions. In this method, if the storage system's input and output counts are detected to be zero for multiple consecutive cycles, the front-end service is deemed to have stopped. After determining that the front-end service has stopped, this method executes garbage collection tasks at the maximum concurrency, utilizing the low-traffic period and providing more idle blocks for subsequent user service processing.

[0043] In practice, there may be a situation where the front-end business, due to the high concurrent recovery volume, causes the storage system to process zero input and output, which is mistakenly considered to be the front-end user business suspension. To avoid this situation, in practice, the business suspension is determined to include: When the input and output quantity of the storage system is detected to be equal to 0 for the first time, the garbage collection function of the storage system is controlled to be disabled; After the garbage collection function is disabled, if the number of input and output of the storage system is detected to be equal to 0 for multiple consecutive cycles, the service is determined to be stopped; After confirming that the business is stopped and before obtaining the maximum concurrency of the preset task concurrency, the following steps are also included: Controls the startup of the garbage collection function of the storage system.

[0044] In this method, when the number of garbage collection is counted to 0 for the first time, the garbage collection function is completely stopped. Then, the number of garbage collection is counted again after multiple cycles after the stop. If so, the service is determined to be stopped. This improves the accuracy of determining the service stop.

[0045] After executing the garbage collection task at the maximum concurrency, the user business may change. In order to reduce the impact of the garbage collection task on the user business and ensure the efficiency of garbage collection, in implementation, after controlling the storage system to execute the garbage collection task at the maximum concurrency within the second preset time, it also includes: The maximum concurrency is used as the current task concurrency; Starting from executing the garbage collection task with the maximum concurrency, the step of obtaining the number of inputs and outputs processed by the storage system within a third preset time period, and returning to the step of comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system.

[0046] The third preset duration is not limited and is determined based on actual conditions. In this method, after running at the maximum concurrency, the concurrency of the garbage collection task is re-determined after a period of time to reduce the impact of the garbage collection task on user services and ensure the efficiency of garbage collection.

[0047] In order to enable those skilled in the art to better understand the garbage collection method for a storage system provided by the present invention, the above process will be further described below with reference to the accompanying drawings and specific embodiments. Figure 2 The overall flow chart of a garbage collection method for a storage system provided by an embodiment of the present invention is as follows: Figure 2 As shown, the method includes: S14: The full flash pool is created successfully and the garbage collection function is started. S15: Garbage collection runs with minimum concurrency; S16: Counts the number of front-end inputs and outputs processed by the storage system after running for one cycle at the minimum concurrency. S17: Increase the concurrency of garbage collection tasks; S18: Count the number of front-end inputs and outputs after one cycle of concurrent operation. S19: Compare with the previous statistics to determine whether they are basically consistent with the previous statistics; if so, return to step S17; if not, proceed to step S20; S20: If it is detected that the number is less than the previous one, the concurrency of garbage collection tasks is reduced; S21: Uniformly reduce the number of front-end input and output after one cycle of concurrent operation; S22: Continue to compare with the previous statistics to determine whether it is basically consistent with the previous one; if not, return to step S20; if so, go to step S23; S23: After running for a period of time at the current concurrency, re-initiate the change in concurrency; return to step S17; S24: If the number of input and output processed by the storage system is 0 within multiple cycles, which meets the interruption criterion, garbage collection is performed at the highest concurrency.

[0048] In the implementation, (1) first, the working mechanism of starting and stopping the garbage collection function based on the threshold is cancelled, and the garbage collection function is started after the full flash pool is created; (2) Implement an IO statistics function on the front-end IO flow processing path to count the number of front-end IOs processed within a unit period, for example, the statistical period is 5 seconds; (3) The garbage collection function starts working at the lowest concurrency after startup and counts the number of front-end IOs processed per unit cycle at this concurrency intensity; (4) After a statistical cycle, the task concurrency of the garbage collection function is increased by 1, and the number of front-end IOs processed in the new cycle is continued to be counted; if the number of IOs counted after the end of the cycle is basically the same as that in (3), it means that the current garbage collection concurrency will not affect the business pressure of the front-end, and the task concurrency can continue to be increased; (5) Repeat step (4) and compare the IO quantity at the end of each statistical cycle with the IO quantity of the previous statistical cycle; (6) Until the number of IOs in the latest statistical cycle is less than the number of IOs in the previous one, it means that the increased concurrency of garbage collection tasks has affected the processing capacity of front-end IO. It is necessary to reduce the concurrency and continue to count the number of IOs in the cycle after the reduction; (7) Execute step (6) repeatedly until the number of IOs in the latest statistical period is basically the same as the previous one, then garbage collection can be continued for a period of time at this concurrency, for example, the duration is 1 minute; (8) After 1 minute of continuous execution, based on the existing task concurrency, there may be an increase in user load. Therefore, continue to reduce the concurrency and compare the number. If the IO number after the concurrency reduction is the same as before the reduction, you can loop through steps (4), (5), and (6). If the IO number after the concurrency reduction is higher than before the reduction, you need to loop through step (6). (9) If the number of IO counted during the period is 0, it is considered that the current front-end business has stopped, and garbage collection can be performed at the highest concurrency and re-tested after a period of execution.

[0049] The garbage collection method for the storage system provided by the present invention solves the impact on performance caused by using a single threshold as a monitoring condition in related technologies, especially the inability to identify actual scenarios and the impact on user business peak periods. By dynamically obtaining the front-end IO pressure and actively adjusting the concurrency control mechanism of the garbage collection tasks, it achieves the rational use of the low pressure period of user business and the staggered peak pressure period of user business. While ensuring the efficiency of garbage collection work, it greatly improves the system performance and alleviates the performance fluctuation problem during operation. By optimizing the startup timing of garbage collection, it alleviates the performance impact of the full-flash stack storage pool module after starting garbage collection.

[0050] Specifically, the present invention provides a method for dynamically regulating the concurrency of garbage collection tasks after starting garbage collection, determining the appropriate concurrency of garbage collection tasks based on the current business pressure and the amount of front-end IO processing under different concurrencies. By integrating multiple thresholds to determine the concurrency intensity of garbage collection tasks and optimizing the timing of starting and stopping garbage collection tasks, the impact of the garbage collection function on performance throughout the entire business operation period is ultimately optimized, particularly enabling adaptation and identification of different business scenarios, ensuring that the storage system can provide reliable performance regardless of business peak or low periods.

[0051] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0052] An embodiment of the present invention further provides a garbage collection device for a storage system, comprising: A startup module, configured to start a garbage collection function of the storage system in response to completion of storage pool creation; a comparison module, configured to compare a first relationship between task concurrency amounts in two adjacent cycles and a second relationship between input and output quantities processed by the storage system; A first control module is configured to, when detecting that the first relationship is that the task concurrency of a previous cycle of two adjacent cycles is less than the task concurrency of a subsequent cycle, and if the second relationship is that the number of inputs and outputs of the subsequent cycle is less than the number of inputs and outputs of the previous cycle, control the storage system to perform garbage collection tasks in a cycle after the subsequent cycle according to the task concurrency that is less than that of the subsequent cycle; The second control module is used to control the storage system to perform garbage collection tasks in a cycle after the next cycle according to a task concurrency greater than that of the next cycle if the second relationship is that the input and output quantities of two adjacent cycles are the same.

[0053] In some embodiments, the garbage collection device of the storage system further includes: The first acquisition module is configured to execute garbage collection tasks according to the current task concurrency within the current cycle after detecting that the garbage collection function is enabled, and to obtain the number of inputs and outputs processed by the storage system within the current cycle under the current task concurrency; The third control module is used to control the current task concurrency to increase according to a preset step size to obtain a new current task concurrency; The second acquisition module is used to execute the garbage collection task according to the new current task concurrency in the next cycle of the current cycle, and obtain the input and output quantity processed by the storage system in the next cycle of the current cycle under the new current task concurrency.

[0054] In some embodiments, the second control module includes: The first module is used to use the task concurrency of the next cycle as the current task concurrency, and return to trigger the third control module to obtain a task concurrency greater than the next cycle to perform garbage collection tasks; A fourth control module, configured to control the storage system to execute the garbage collection task according to the new current task concurrency in a first cycle following the next cycle; The storage system's garbage collection device also includes: The second module is used to take the first cycle after the next cycle as the current cycle and return to trigger the second acquisition module.

[0055] In some embodiments, the first control module includes: The third acquisition module is used to use the task concurrency of the next cycle as the current task concurrency, control the current task concurrency to decrease according to a preset step size and obtain a new current task concurrency, so as to obtain a task concurrency smaller than the task concurrency of the next cycle to perform the garbage collection task; a fifth control module, configured to control the storage system to execute the garbage collection task according to the new current task concurrency in a first cycle following the next cycle; The third module is used to set the first cycle after the next cycle as the current cycle and return to trigger the second acquisition module; The storage system's garbage collection device also includes: a detection module configured to, when detecting that the first relationship is that the task concurrency of the previous cycle of two adjacent cycles is greater than the task concurrency of the next cycle, and if the second relationship is that the number of inputs and outputs of the next cycle is greater than the number of inputs and outputs of the previous cycle, return to trigger the third acquisition module; The sixth control module is configured to control the storage system to execute the garbage collection task according to the task concurrency of the next cycle within the first preset time period if the second relationship is that the input and output quantities of two adjacent cycles are the same.

[0056] In some embodiments, the garbage collection device of the storage system further includes: A fourth acquisition module is used to obtain the current task concurrency corresponding to the first preset time length; The seventh control module is used to control the current task concurrency to decrease according to a preset step size and obtain a new current task concurrency; and return to trigger the comparison module.

[0057] In some embodiments, the garbage collection device of the storage system further includes: A determination and acquisition module, configured to determine that the service is stopped and obtain the maximum concurrency among the preset task concurrency when detecting that the input and output quantity of the storage system is equal to 0 for multiple consecutive cycles; An eighth control module is used to control the storage system to perform garbage collection tasks according to the maximum concurrency within a second preset time period.

[0058] In some embodiments, the garbage collection device of the storage system further includes: The fourth module is used to set the maximum concurrency as the current task concurrency; The fifth acquisition module is used to obtain the number of input and output processed by the storage system within a third preset time period starting from the execution of the garbage collection task at the maximum concurrency, and return it to trigger the comparison module.

[0059] For descriptions of features in the embodiments corresponding to the garbage collection device of the storage system, reference may be made to the relevant descriptions of the embodiments corresponding to the garbage collection method of the storage system, which will not be detailed here.

[0060] An embodiment of the present invention further provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps of any of the above-mentioned embodiments of the garbage collection method for the storage system.

[0061] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the garbage collection method for a storage system when running.

[0062] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0063] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned garbage collection method embodiments for a storage system are implemented.

[0064] An embodiment of the present invention also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned garbage collection method embodiments of the storage system.

[0065] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0066] The above is a detailed introduction to the garbage collection method and electronic device of a storage system provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only applicable to help understand the method of the present invention and its core ideas. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A garbage collection method for a storage system, characterized in that: include: In response to the storage pool being created, starting a garbage collection function of the storage system; comparing a first relationship between task concurrency amounts in two adjacent cycles and a second relationship between the number of inputs and outputs processed by the storage system; When it is detected that the first relationship is that the task concurrency of a previous cycle of two adjacent cycles is less than the task concurrency of a subsequent cycle, if the second relationship is that the number of inputs and outputs of the subsequent cycle is less than the number of inputs and outputs of the previous cycle, controlling the storage system to perform garbage collection tasks in a cycle after the subsequent cycle according to the task concurrency less than that of the subsequent cycle; If the second relationship is that the input and output quantities of two adjacent cycles are the same, the storage system is controlled to execute the garbage collection task in a cycle after the latter cycle according to a task concurrency greater than that of the latter cycle.

2. The garbage collection method of the storage system according to claim 1, characterized in that: After starting the garbage collection function of the storage system and before comparing the first relationship between the task concurrency amounts of two adjacent cycles and the second relationship between the input and output quantities processed by the storage system, the method further includes: After detecting that the garbage collection function is enabled, executing garbage collection tasks according to the current task concurrency in the current cycle, and obtaining the number of inputs and outputs processed by the storage system in the current cycle under the current task concurrency; Controlling the current task concurrency to increase according to a preset step size to obtain a new current task concurrency; In a cycle next to the current cycle, garbage collection tasks are executed according to the new current task concurrency, and under the new current task concurrency, the number of inputs and outputs processed by the storage system in the cycle next to the current cycle is obtained.

3. The garbage collection method of the storage system according to claim 2, characterized in that: Controlling the storage system to perform garbage collection tasks in a period after the latter period according to a task concurrency greater than that of the latter period includes: Taking the task concurrency of the subsequent cycle as the current task concurrency, returning to the step of controlling the current task concurrency to increase according to a preset step size to obtain a new current task concurrency, so as to obtain a task concurrency greater than that of the subsequent cycle to execute a garbage collection task; Controlling the storage system to perform garbage collection tasks according to the new current task concurrency in a first cycle after the latter cycle; After the control storage system performs the garbage collection task according to the new current task concurrency in a first cycle after the latter cycle, the method further includes: The first cycle after the next cycle is used as the current cycle, and the process returns to the step of obtaining the number of inputs and outputs processed by the storage system in the next cycle after the current cycle under the new current task concurrency.

4. The garbage collection method of the storage system according to claim 2, characterized in that: The controlling storage system to perform the garbage collection task in a period after the latter period according to a task concurrency smaller than that of the latter period includes: Taking the task concurrency of the subsequent cycle as the current task concurrency, controlling the current task concurrency to decrease according to a preset step size and obtaining a new current task concurrency, so as to obtain a task concurrency smaller than the task concurrency of the subsequent cycle to execute the garbage collection task; Controlling the storage system to perform garbage collection tasks according to the new current task concurrency in a first cycle after the latter cycle; After the control storage system performs the garbage collection task according to the new current task concurrency in a first cycle after the latter cycle, the method further includes: Taking the first cycle after the next cycle as the current cycle, and returning to the step of obtaining the number of inputs and outputs processed by the storage system in the cycle next to the current cycle under the new current task concurrency; After comparing the first relationship between the task concurrency amounts of two adjacent cycles and the second relationship between the input and output quantities processed by the storage system, the method further includes: In the case where it is detected that the first relationship is that the task concurrency of the previous cycle is greater than the task concurrency of the next cycle between two adjacent cycles, if the second relationship is that the number of inputs and outputs of the next cycle is greater than the number of inputs and outputs of the previous cycle, returning to the step of using the task concurrency of the next cycle as the current task concurrency, controlling the current task concurrency to decrease according to a preset step size, and obtaining a new current task concurrency; If the second relationship is that the input and output quantities of two adjacent cycles are the same, the storage system is controlled to perform the garbage collection task according to the task concurrency of the subsequent cycle within a first preset time period.

5. The garbage collection method of the storage system according to claim 4, characterized in that: After controlling the storage system to perform the garbage collection task according to the task concurrency of the subsequent cycle within the first preset time period, the method further includes: Obtain the current task concurrency corresponding to the first preset time length; Control the current task concurrency to decrease according to the preset step size and obtain a new current task concurrency; return to the step of comparing the first relationship between the task concurrencies of two adjacent cycles and the second relationship between the input and output quantities processed by the storage system.

6. The garbage collection method of the storage system according to claim 5, characterized in that: Also includes: When it is detected that the input and output quantity of the storage system is equal to 0 for multiple consecutive cycles, the service is determined to be stopped, and the maximum concurrency among the preset task concurrency quantities is obtained; The storage system is controlled to execute the garbage collection task according to the maximum concurrency within a second preset time period.

7. The garbage collection method of the storage system according to claim 6, characterized in that: Determining business cessation includes: When the input and output quantity of the storage system is detected to be equal to 0 for the first time, the garbage collection function of the storage system is controlled to be disabled; After the garbage collection function is disabled, if the number of input and output of the storage system is detected to be equal to 0 for multiple consecutive cycles, the service is determined to be stopped; After determining that the service is stopped, and before obtaining the maximum concurrency in the preset task concurrency, the method further includes: Controls the startup of the garbage collection function of the storage system.

8. The garbage collection method for a storage system according to claim 6 or 7, characterized in that: Obtaining the number of I / O processed by the storage system includes: Get the input and output stream processing path of the storage system; A statistical module for counting input and output quantities is set on the input and output flow processing path; The number of inputs and outputs processed by the storage system is obtained through the statistics module.

9. The garbage collection method of the storage system according to claim 8, characterized in that: After controlling the storage system to execute the garbage collection task according to the maximum concurrency within the second preset time period, the method further includes: Using the maximum concurrency as the current task concurrency; Starting from executing the garbage collection task with the maximum concurrency, the number of inputs and outputs processed by the storage system is obtained within a third preset time period, and the step of comparing the first relationship between the task concurrency of two adjacent cycles and the second relationship between the number of inputs and outputs processed by the storage system is returned.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the garbage collection method for a storage system according to any one of claims 1 to 9 when executing the computer program.

Images