Method, apparatus, and computer program product for managing a storage system

Abstract

Embodiments of the present disclosure relate to a method, apparatus and computer program product for managing a storage system. The method includes determining, based on respective current resource usage of a plurality of storage tiers in the storage system, usage levels of task types of a plurality of task types associated with respective ones of the plurality of storage tiers, the plurality of storage tiers sharing physical resources of the storage system, tasks being executed on respective storage tiers according to the task types to which the tasks belong; determining, based on respective historical resource usage of the plurality of storage tiers, priority levels of the plurality of task types respectively; and selecting, based on the usage levels and the priority levels of the plurality of task types and the task types to which a plurality of tasks to be executed belong, a set of tasks from the plurality of tasks for execution, a number of tasks belonging to each of the task types in the set of tasks being determined by the usage level of the respective task type, the set of tasks being ordered according to the respective priority levels of the respective task types.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to the field of data storage, and more specifically to methods, apparatus, and computer program products for managing storage systems. Background Technology

[0002] In storage systems, physical storage resources can be categorized by various hardware interfaces and media types, such as Non-Volatile Memory Host Controller Interface (NVMe), Hard Disk Drives (HDDs), and Solid State Drives (SSDs). At the software level, these storage resources can be virtualized into multiple virtual resources or virtual storage layers, such as data tiering capabilities, storage pool space, and free data windows. These different types of virtual resources share the physical storage resources of the storage system. Different types of tasks are typically executed on different virtual resources. Therefore, better resource scheduling among the various types of tasks executed on different virtual resources is a significant challenge. Summary of the Invention

[0003] Embodiments of this disclosure provide methods, apparatus, and computer program products for managing storage systems.

[0004] In a first aspect of this disclosure, a method for managing a storage system is provided. The method includes: determining a usage level for a plurality of task types associated with corresponding storage layers among the plurality of storage layers, based on the current resource usage of each of the plurality of storage layers in the storage system; the plurality of storage layers sharing physical resources of the storage system; tasks being executed on corresponding storage layers according to their respective task types; determining a priority level for each of the plurality of task types based on the historical resource usage of each of the plurality of storage layers; and selecting a set of tasks for execution from among the plurality of tasks based on the usage level and priority level of the plurality of task types and the task types to which the plurality of tasks to be executed belong, wherein the number of tasks belonging to each task type in the set of tasks is determined by the usage level of the corresponding task type, and the set of tasks is ordered according to the corresponding priority level of each task type.

[0005] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processing unit and at least one memory. The at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform actions including: determining a usage level of a plurality of task types associated with corresponding storage layers in a storage system based on the current resource usage of each of the plurality of storage layers, wherein the plurality of storage layers share physical resources of the storage system, and tasks are executed on corresponding storage layers according to their respective task types; determining a priority level for each of the plurality of task types based on the historical resource usage of each of the plurality of storage layers; and selecting a set of tasks for execution from the plurality of tasks based on the usage level and priority level of the plurality of task types and the task types to which the plurality of tasks to be executed belong, wherein the number of tasks belonging to each task type in the set of tasks is determined by the usage level of the corresponding task type, and the set of tasks is ordered according to the corresponding priority level of each task type.

[0006] In a third aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer storage medium and includes machine-executable instructions. When executed by a device, the machine-executable instructions cause the device to perform any step of the method described in the first aspect of this disclosure.

[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0008] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0009] Figure 1 A schematic diagram of an example system that can be implemented therein according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A schematic block diagram of task resource scheduling according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart is shown illustrating an example method for task resource scheduling according to some embodiments of this disclosure;

[0012] Figure 4A schematic diagram illustrating historical resource usage according to some embodiments of this disclosure is shown;

[0013] Figure 5 A schematic diagram of task resource scheduling according to some embodiments of the present disclosure is shown;

[0014] Figure 6 Another schematic diagram of task resource scheduling according to some embodiments of the present disclosure is shown;

[0015] Figure 7 Another schematic diagram of task resource scheduling according to some embodiments of the present disclosure is shown;

[0016] Figure 8 A schematic diagram illustrating resource usage according to some embodiments of the present disclosure and resource usage in a conventional scheme is shown; and

[0017] Figure 9 A schematic block diagram of an example device that can be used to implement embodiments of the present disclosure is shown.

[0018] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0019] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0020] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] Figure 1A schematic diagram of a storage system 100 in which embodiments of the present disclosure may be implemented is shown. In the storage system 100, physical storage resources can be provided and categorized through various hardware interfaces and media types, such as NVMe, Serial Attached Small Computer System Interface (SAS) flash memory, HDD, SSD, and nearline (NL)-SAS. The storage system 100 is used to provide storage resource-related tasks, including tasks such as defrauding, idle resource reclamation (also known as resource reclamation), and data tiering. It should be understood that... Figure 1 The system shown is merely an example. In real-world applications, storage system 100 may contain many other devices and / or components within those devices, or the devices and / or components shown may be arranged in other ways.

[0022] like Figure 1 As shown, storage system 100 includes multiple (e.g., N) storage tiers 110-1, 110-2, ..., 110-N, where N is an integer greater than 1. In the following text, for ease of discussion, storage tiers 110-1, 110-2, ..., 110-N are sometimes collectively referred to as storage tier 110 or simply as storage tier 110. In this document, storage tiers are also referred to as virtual storage tiers or virtual resource tiers. Examples of storage tiers 110 include, but are not limited to, data windows, storage pools, or storage tiers. Each storage tier 110 of storage system 100 shares the physical storage resources of storage system 100. For example, each storage tier 110 shares a segment of input / output (I / O) bandwidth on storage system 100.

[0023] Each storage layer 110 is associated with a different task type 120. For example, storage layer 110-1 is associated with task type 120-1; storage layer 110-2 is associated with task type 120-2; ... storage layer 110-N is associated with task type 120-N. In the following discussion, task types 120-1, 120-2, ... 120-N are sometimes collectively referred to or individually as task type 120.

[0024] Each task in storage system 100 is executed on the corresponding storage layer 110 according to its task type 120. For example, a task belonging to task type 120-1 is executed on storage layer 110-1. In other words, a task belonging to task type 120-1 is a resource on storage layer 110-1. In this document, a task is also referred to as a job or a request. For example, if storage layer 110-1 is a data window (also referred to as a data window allocator), then task type 120-1 could be defraction. Defraction tasks are used to organize free data windows to support sequential data write capabilities.

[0025] If storage layer 110-2 is a storage pool, then task type 120-2 can be idle resource reclamation. Idle resource reclamation tasks are used to return idle space to the storage pool. Using idle resource reclamation tasks ensures that there are sufficient resources available in the storage pool.

[0026] If storage tiers 110-N are storage tiers, then task types 120-N can be data tiers. Data tiering tasks are used to reallocate hot data (i.e., data that is frequently accessed) from low-performance storage media, such as HDDs, to high-performance storage media, such as SAS flash, NVMe, etc. Data tiering tasks provide optimized IO performance for hybrid storage systems.

[0027] It should be understood that, in addition to the examples of storage layer 110 and its associated task type 120 listed above, other storage layers 110 and associated task types 120 may exist. It should be understood that the number of storage layers 110 in the storage system 100 can be arbitrary, and the number of task types executed on the storage layers 110 can also be arbitrary. It should be understood that... Figure 1 The storage system 100 shown is merely exemplary and not limiting. The storage system according to this disclosure may also have other forms or structures.

[0028] Typically, the various task types mentioned above, such as defraction, idle resource reclamation, and data tiering, all involve the use of shared physical storage resources, such as I / O bandwidth, to move data. When these tasks are executed simultaneously, they compete for limited physical storage resources, such as I / O bandwidth. However, in conventional solutions, it is difficult to rationally allocate or schedule resources among these various task types for task execution.

[0029] A common approach to resource scheduling prioritizes tasks based on their type. For example, data stratification is typically given the highest priority, defraction the medium priority, and idle resource reclamation the lowest priority. However, this approach can cause several problems. It tends to prioritize high-priority tasks, potentially disrupting or even interrupting lower-priority tasks. For instance, defraction tasks might consistently interrupt idle resource reclamation, leading to storage pool exhaustion.

[0030] Embodiments of this disclosure propose a scheme for managing a storage system to address one or more of the aforementioned problems and other potential problems. In this scheme, a usage level for a task type associated with a corresponding storage layer is determined among multiple task types based on the current resource usage of each of the multiple storage layers. The scheme further includes determining a priority level for each of the multiple task types based on the historical resource usage of each of the multiple storage layers. The scheme also includes selecting a set of tasks for execution from among the multiple tasks based on the usage level and priority levels of the multiple task types and the task types to which the multiple tasks to be executed belong. The number of tasks of different task types in the selected set of tasks is determined by the usage level of that task type. Each task in the selected set of tasks is ordered based on the priority level of its respective task type.

[0031] In this way, the usage level and priority of each task type can be determined based on the current and historical resource usage of different storage layers. This allows for the rational allocation of resources and the rational scheduling of tasks based on the current and historical resource usage of different storage layers. The solution disclosed herein can allocate and utilize resources more rationally, thereby improving the overall performance of the storage system.

[0032] The basic principles and several exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0033] Figure 2 A schematic block diagram 200 for task resource scheduling according to some embodiments of the present disclosure is shown. For example... Figure 2 As shown, multiple tasks 210 to be executed can be input into the task scheduling submodule 240 in the scheduling module 220. Figure 2 The diagram also shows the usage level determination submodule 230 and priority level determination submodule 250 in the scheduling module 220. The usage level determination submodule 230 and priority level determination submodule 250 provide the determined usage level and priority level to the task scheduling submodule 240, respectively. The task scheduling submodule 240 can determine and output multiple scheduled tasks 260 based on the multiple tasks 210 to be executed and the received usage level and priority level.

[0034] The various modules and submodules in block diagram 200 can be implemented by devices of storage system 100. It should be understood that the various modules and submodules in block diagram 200 can also be implemented by other suitable devices or apparatuses. Block diagram 200 may include additional modules not shown and / or modules shown may be omitted; the scope of this disclosure is not limited in this respect. For ease of explanation, reference will be made below. Figure 3 Let us describe in detail the various modules and submodules in block diagram 200.

[0035] The following will refer to Figure 3 This document describes a flowchart of a method 300 for scheduling task resources according to some embodiments of the present application. Method 300 may be implemented by a device of the storage system 100. For example, the various steps in method 300 may be implemented by… Figure 2 The method is implemented using the modules and / or submodules shown. It should be understood that method 300 can also be performed by other suitable devices or apparatuses. Method 300 may include additional actions not shown and / or the actions shown may be omitted; the scope of this disclosure is not limited in this respect. For ease of explanation, reference will be made to... Figure 1 and Figure 2 To describe process 300.

[0036] like Figure 3 As shown, at 310, scheduling module 220 determines the usage level of each of the multiple task types. For example, it can use... Figure 2 The usage level determination submodule 230 determines the usage level of the task type among the multiple task types 120 that is associated with the corresponding storage layer among the multiple storage layers 110, based on the current resource usage of each of the multiple storage layers 110. The multiple storage layers 110 share the physical resources of the storage system 100, and the tasks in the storage system 100 are executed on the corresponding storage layer 110 according to their respective task types 120.

[0037] In some embodiments, the usage level of multiple task types can be determined based on the current resource usage of each storage tier, as shown in Table 1 below. As shown in Table 1, for a storage pool, the usage level of the associated idle resource reclamation task type can be determined based on the storage pool's free space rate. For a data window, the usage level of the associated defrauding task type can be determined based on the idle data window count. For a storage tier, the usage level of the associated data tiering task type can be determined based on the data slice temperature.

[0038] Table 1 Examples of Usage Levels for Task Types

[0039]

[0040] It should be understood that the various storage layer resource usage conditions described above, such as storage pool free space rate, free data window count, and data slice temperature, can all be determined and recorded by the storage system 100. In some embodiments, other types of storage layers and other types of storage layer resource usage conditions may also be included.

[0041] The usage level determination methods shown in Table 1 are merely exemplary and not restrictive. In some embodiments, more usage levels may be set. For example, usage levels may be set to 1-8 or other appropriate ranges.

[0042] In this way, the storage resource usage of different storage layers, represented in different ways, can be normalized. By using usage levels like 1, 2, and 3 in Table 1, usage levels can be represented intuitively and simply, facilitating subsequent task resource scheduling.

[0043] Additionally or alternatively, in some embodiments, the usage level of each task type 120 can be determined based on the average resource usage of each of the multiple storage layers 110 within the current first time period. For example, in some embodiments, the current scheduling period (i.e., the first time period) can be divided into M (e.g., 5 or other suitable natural numbers) observation windows. In each observation window, the usage level of each task type 120 is determined once based on the resource usage of each storage layer 110. For each task type 120, the average value can be determined as the usage level of that task type 120 by averaging these M determined usage levels. It should be understood that the length of the current scheduling period can be a time length determined empirically or a time length determined based on the current status of tasks to be executed.

[0044] In some embodiments, other methods can also be used to determine the usage level of each task type 120 based on the average resource usage of each of the multiple storage layers 110 during the current first time period. For example, a weight HC can be determined based on the average resource usage of each of the multiple storage layers 110 during the current first time period. Based on the weight HC and the resource usage of each storage layer 110 at the current moment, the usage level of each task type 120 can be determined.

[0045] The weight HC can be determined through the following process: For each storage layer 110 (i.e., for each task type 120), the usage level is determined at each of the M observation windows in the current scheduling cycle. The number of values ​​for each usage level is counted, and this count is determined as the weight HC. For example, if in M ​​determination processes, say 5, a usage level with a value of 1 is determined once, a usage level with a value of 2 three times, and a usage level with a value of 3 once, then in this example, the weight HCs for usage levels with values ​​of 1, 2, and 3 for storage layer 110 (i.e., for each task type 120) are 1, 3, and 1, respectively.

[0046] In some embodiments, only the weight Max HC with the highest value can be retained, i.e., other usage levels with lower statistical counts can be filtered out. In this example, the weights HC for usage levels with values ​​of 1, 2, and 3 for storage layer 110 (i.e., for each task type 120) are 0, 3, and 0, respectively. In this way, the usage level adjusted for task type 120 according to the weights can be calculated as 2*3=6.

[0047] This approach allows for the consideration of resource usage across various storage layers over a short period, preventing inaccurate usage level determination due to sudden changes in resource usage. This results in a more accurate determination of the usage level.

[0048] Additionally or alternatively, in some embodiments, the usage levels of multiple task types 120 can be updated based on the average historical resource usage of each of the multiple storage layers 110 over a second time period preceding the current time. The second time period is a longer period than the current scheduling cycle or the first time period. For example, K (e.g., 5 or other suitable natural numbers) scheduling cycles preceding the current time can be defined as the second time period.

[0049] You can retrieve the usage level records for the first K scheduling cycles, or the Max HC records for the first K scheduling cycles. Table 2 below shows the Max HC records for the first K scheduling cycles. In Table 2, the value of K is 5, and the usage level value is 1, 2, or 3.

[0050] Table 2. Record of Max HC for K scheduling cycles

[0051] Period 1 0 3 0 Period 2 0 0 4 Period 3 2 2 0 Period 4 0 2 2 Period 5 0 0 3

[0052] Based on Table 2, the miss count MC, normalized miss count NMC, and reverse miss count IMC can be calculated. The miss count MC represents the count of misses for each usage level that do not belong to K scheduling cycles. For example, in Table 2, usage level 1 only has a non-zero Max HC in cycle 3, so the MC for usage level 1 is 4. NMC = MC / K, which normalizes MC. IMC = 1 / NMC = K / MC, which is the reciprocal of NMC. Table 3 below shows the values ​​of MC, NMC, and IMC calculated based on the example Max HC in Table 2.

[0053] Table 3 Example MC, NMC and IMC values

[0054] Period 1 0 3 0 Period 2 0 0 4 Period 3 2 2 0 Period 4 0 2 2 Period 5 0 0 3 MC 4 2 2 NMC 4 / 5 2 / 5 2 / 5 IMC 1.25 2.5 2.5

[0055] It should be understood that a higher NMC value indicates that the usage level is less likely to be the dominant usage level during the observation period (i.e., K scheduling periods). Conversely, a higher IMC value indicates that the level is likely to be the dominant usage level. If the MC value is 0, the IMC can be set to a preset maximum value, IMCMAX, such as 10 or another appropriate value greater than other IMC values.

[0056] Using IMC instead of HC to represent long-term usage levels avoids the gradual thinning of the HC matrix as the observation period increases. Using IMC provides more accurate results for long-term usage level assessment compared to HC.

[0057] Based on the assessment of long-term use levels according to IMC described above, the use levels for each task type 120 can be updated. For example, the adjusted final use level can be calculated using the following formula (1).

[0058]

[0059] In Equation (1), hc-imc weight represents the usage level based on short-term and long-term factors (i.e., considering the usage level of the current first time period and the historical second time period). Level represents the usage level, and the value of Level is a natural number from 1 to N. N can be any natural number greater than 1, for example, N is 3 in the examples in Tables 2 and 3. Examples of adjusted usage levels obtained according to Equation (1) are shown below in Tables 4-6.

[0060] Table 4 Examples of Defragmentation Usage Levels

[0061] <![CDATA[L3]]> 3 2 <![CDATA[L2]]> 1 2 <![CDATA[L1]]> 1 4 hc-imc weight 26

[0062] Table 5 Examples of Idle Resource Recycling and Utilization Levels

[0063] <![CDATA[L3]]> 1 2 <![CDATA[L2]]> 1 2 <![CDATA[L1]]> 3 4 hc-imc weight 22

[0064] Table 6 Examples of Idle Resource Recycling and Utilization Levels

[0065] <![CDATA[L3]]> 3 4 <![CDATA[L2]]> 1 2 <![CDATA[L1]]> 1 2 hc-imc weight 42

[0066] As shown in Tables 4-6 above, the HC (Household Cost) for defraction and data stratification tasks is the same, meaning their usage levels are similar in the current period. However, the IMC (Integrated Management Cost) for usage level 3 of the data stratification task type is 4, while the IMC for usage level 3 of the data stratification task type is 2. This indicates that, based on long-term historical resource usage, data stratification has a higher usage level.

[0067] The hc-imc weight obtained from equation (1) can reflect the higher usage level of the data layer. In this way, the data layer can obtain a higher usage level, thereby obtaining more IO bandwidth. This method of determining the usage level by considering both short-term and long-term resource usage is particularly suitable for task resource scheduling of multiple task types with similar short-term resource usage.

[0068] Next, at 320, scheduling module 220 determines the priority level of each of the multiple task types 120. For example, priority level determination submodule 250 can determine the priority level of each of the multiple task types 120 based on the historical resource usage of each of the multiple storage layers 110. In some embodiments, the scheduling interval ratio between the various task types 120 can be determined according to the priority level of each task type 120. In this document, priority level is also referred to as interval level.

[0069] In some embodiments, the task types 120 used in the storage system 100 can be determined based on the historical resource usage of each of the multiple storage layers 110. The priority of the used task types 120 is set to a larger value, such as 1, while the priority of the unused task types 120 is set to a smaller value, such as 0.

[0070] Alternatively, or as an alternative, the priority of multiple task types 120 can be determined based on the historical resource usage changes of each of the multiple storage layers 110. For example, the priority of each task type 120 can be determined based on its historical resource usage changes over a third time period prior to the current moment. The third time period can be the same as the second time period described above, or it can be selected as another time length.

[0071] Taking Table 2 above as an example, the average dominant usage level can be calculated for each task category 120 in each scheduling cycle. It should be understood that when the Max HC of two usage levels is the same, they can be averaged to obtain the average dominant usage level.

[0072] Table 7 Examples of Average Usage Levels

[0073] Usage Level (N) 2 3 1.5＝(1+2) / 2 2.5＝(2+3) / 2 3

[0074] Table 8 shows examples of average usage levels for various task types. From Table 8, we can see how the usage level of each task level 120 changes during the third time period.

[0075] Table 8 Examples of Average Usage Levels for Different Task Types

[0076]

[0077]

[0078] Figure 4 A schematic diagram illustrating historical resource usage according to some embodiments of this disclosure is shown. Figure 4 Curve 410 in the table is plotted from the average usage level of defraction in Table 8. Curve 420 is plotted from the average usage level of data stratification in Table 8. Curve 430 is plotted from the average usage level of idle resource reclamation in Table 8. From Figure 4 As can be seen, the average usage level of defraction gradually increases, therefore its priority can be set to the highest, such as 3. The average usage level of idle resource reclamation tends to decrease, therefore its priority can be set to the lowest, such as 1. The priority of data stratification can be set to a moderate level, such as 2. It should be understood that the priority values ​​1, 2, and 3 described above are only illustrative and not restrictive. Other appropriate values ​​can also be set for the priority levels.

[0079] For additional or alternative sites, priority levels can be calculated using equation (2) or equation (3).

[0080]

[0081]

[0082] The index level in equations (2) and (3) i and index level i-1 Let i and i-1 represent the average usage level, respectively. Δ(·) represents the difference function. i is a value greater than or equal to 2 and less than or equal to the highest usage level N (e.g., 3). index level increment′ represents the calculated average usage level increment between -N and N. index level increment represents the average usage level increment as an increment value between 1 and N+1. Using a positive value to represent the offset by adding an offset of N+1 makes calculation easier. Table 9 shows the offset average usage level increments determined according to the example in Table 8.

[0083] Table 9 Examples of Average Usage Level Increments After Offset

[0084]

[0085] According to Table 9, the priority levels for each task type 120 can be set to the values ​​of the offset average usage level increments. For example, the priority level for defrauding can be set to 5, the priority level for idle resource reclamation can be set to 2, and the priority level for data stratification can be set to 4.

[0086] In this way, higher priority can be assigned to task types 120 with progressively increasing usage levels, ensuring they are prioritized during task scheduling. This allows for prediction of future usage levels based on changes in usage levels (i.e., changes in historical usage).

[0087] Return to Figure 3 At 330, scheduling module 220 selects a set of tasks for execution from multiple tasks based on the usage level and priority level of multiple task types 120 and the task type 120 to which the multiple tasks 210 to be executed belong. For example, task scheduling submodule 240 can select a set of tasks from the multiple tasks 210 to be executed for execution in the current scheduling loop based on the usage level determined by usage level determination submodule 230, the priority level determined by priority level determination submodule 250, and the multiple tasks 210 to be executed. The remaining tasks can be executed in the next or subsequent scheduling loops.

[0088] The number of tasks belonging to a specific task type 120 within this set of tasks is determined by the usage level of that task type. The tasks in this set are sorted according to the priority of their respective task types 120. The remaining tasks can be selected and sorted in a similar manner. Scheduling all tasks 210 to be executed, including selection and sorting, yields multiple scheduled tasks 260.

[0089] In some embodiments, a set of tasks can be selected from the multiple tasks 210 based on the usage level, priority level, and task type to which the multiple tasks 210 to be executed belong, using a weighted load balancing (WRR) algorithm or an interval ratio (IWRR) algorithm.

[0090] Alternatively, a set of tasks can be selected from the multiple tasks 210 to be executed based on the usage levels of multiple task types 120. The number of tasks belonging to the corresponding task type in the selected set of tasks does not exceed the number indicated by the usage level of the corresponding task type 120. This set of tasks can be sorted by task type in an alternating manner according to the number indicated by the priority level of each task type 120.

[0091] Figure 5 and Figure 6 Two schematic diagrams illustrating task resource scheduling according to some embodiments of the present disclosure are shown. For example... Figure 5As shown, the multiple tasks 210 to be executed include a first task sequence 510 belonging to task type 120-1 and a second task sequence 520 belonging to task type 120-2. The first task sequence 510 and the second task sequence 520 can be in the form of a queue or other suitable sequence form. The first task sequence 510 includes tasks A, B, C, D, E, F, and G. The second task sequence 520 includes tasks U, V, W, X, Y, and Z. It should be understood that the above-mentioned letter symbols, such as A and U, are only illustrative and do not represent the content of the tasks they represent.

[0092] The usage level sequence 530 can be determined by the usage level determination submodule 230, in Figure 5 In this context, the priority sequence 530 is (5, 3). The priority sequence 540 can be determined by the priority determination submodule 250. Figure 5 In this context, the priority sequence 530 is (1, 1). The task scheduling submodule 240 can obtain multiple scheduled tasks 250 based on the multiple tasks 210 to be executed, using the priority sequence 530 and priority level 540. For example... Figure 5 As shown, the scheduled tasks 250 include two task sequences 550 and task sequence 560. Task sequence 550 can be executed in the current scheduling cycle, and task sequence 560 can be executed in the next scheduling cycle.

[0093] like Figure 5 As shown, task sequence 550 can sequentially include tasks A, U, B, V, C, W, D, and E. It can be seen that there are 5 tasks belonging to task type 120-1 in task sequence 550, and 3 tasks belonging to task type 120-2. Tasks belonging to different task types in task sequence 550 are arranged alternately at a 1:1 interval ratio.

[0094] Figure 6 and Figure 5 Similarly, there are two task sequences 510 and 520 with the same multiple tasks 210 to be executed, and the same use level sequence 530. Figure 6 Examples and Figure 5 The difference lies in the priority sequence 610, which is (3, 1). From Figure 6 It can be seen that, due to the different priority sequence 610, the task sequence 620 and task sequence 630 among the multiple scheduled tasks 250 determined by the task scheduling submodule 240 are different from those of the task sequence 610. Figure 5 Task sequence 550 and task sequence 560 are different.

[0095] like Figure 6As shown, task sequence 620 includes tasks A, B, C, U, D, E, V, and W. Similar to task sequence 550, task sequence 620 also has 5 tasks belonging to task type 120-1 and 3 tasks belonging to task type 120-2. The difference lies in the order in which the tasks in task sequence 620 are arranged, which is different from that in task sequence 550.

[0096] based on Figure 6 The task sequence 620 obtained by sorting the priority sequence 610 can prioritize the execution of tasks belonging to task type 120-1. As the historical usage level of task type 120-1 gradually increases, this task sorting method can achieve better resource scheduling results.

[0097] Figure 7 Another schematic diagram of task resource scheduling according to some embodiments of the present disclosure is shown. Figure 7 The usage level sequence 740 can be determined by the usage level determination submodule 230 according to the process described above. Figure 7 The priority sequence 750 can be determined by the priority determination submodule 250 according to Table 9.

[0098] Figure 7 The multiple tasks 210 to be executed include three task sequences 710, 720, and 730. Task sequence 710 is for defraction. Task sequence 720 is for idle resource reclamation. Task sequence 730 is for data layering.

[0099] The task scheduling submodule 240, based on the usage level sequence 740 and priority level sequence 750, can schedule the three task sequences 710, 720, and 730 of the multiple tasks 210 to be executed, resulting in a scheduled task sequence 760 to be executed in the current scheduling cycle and a task sequence 770 to be executed in the next scheduling cycle. It should be understood that the maximum number of tasks that can be executed in each scheduling cycle can be determined by the sum of the usage levels of each task type. For example... Figure 7 As shown, the task sequence 760 executed in the current scheduling cycle includes tasks A, B, C, D, E, M, N, U, V, W, X, F, O, P, and Y. The task sequence 770 executed in the next scheduling cycle includes tasks G, Q, and Z.

[0100] Figure 8A schematic diagram illustrating resource usage 850 according to some embodiments of the present disclosure and resource usage 800 in a conventional scheme is shown. In the resource usage 800 of the conventional scheme, curve 810 represents the usage level change curve for defraction task types. Curve 820 represents the usage level change curve for idle resource reclamation task types. Curve 830 shows the proportion of defraction tasks to idle resource reclamation tasks. Defraction tasks are above curve 830, and idle resource reclamation tasks are below curve 830.

[0101] In the resource usage data 850 of this disclosure, curve 860 represents the usage level change curve for the defraction task type. Curve 870 represents the usage level change curve for the idle resource reclamation task type. Curve 880 shows the ratio of defraction tasks to idle resource reclamation tasks. Defraction tasks are above curve 880, and idle resource reclamation tasks are below curve 880.

[0102] A comparison of usage scenarios 800 and 850 reveals that in usage scenario 800, the usage level of defractionation tasks drops rapidly between scheduling cycles 80 and 100. This leads to a decrease in IO performance. Similarly, in usage scenario 800, after 100 scheduling cycles, the usage level of idle resource reclamation tasks drops rapidly, affecting their performance. In contrast, in usage scenario 850, the usage level of each task type decreases gradually, without performance degradation due to competition between them.

[0103] The above combination Figures 1-8 Examples of task resource scheduling according to some embodiments of the present disclosure are shown. The scheme of the present disclosure enables the scheduling of multiple tasks to be executed based on the current and historical resource usage of different storage tiers, thereby enabling a more rational allocation of resources occupied by each task type. In this way, the utilization of I / O bandwidth can be improved, thereby improving the performance of the storage system.

[0104] Figure 9 A schematic block diagram of an example device 900 that can be used to implement embodiments of the present disclosure is shown. For example, such as Figure 1 The storage system 100 shown can be implemented by device 900. For example... Figure 9As shown, device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0105] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0106] The various processes and handling described above, such as method 300, can be executed by processing unit 901. For example, in some embodiments, method 300 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of method 300 described above can be performed.

[0107] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0108] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0109] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0110] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0111] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0112] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0113] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0115] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for managing a storage system, comprising: By normalizing the current resource usage of each of the multiple storage layers in the storage system, the usage level of the task type associated with the corresponding storage layer among the multiple task types is determined. The multiple storage layers share the physical resources of the storage system, and the tasks in the storage system are executed on the corresponding storage layer according to their task type. Based on the historical resource usage of each of the multiple storage layers, the priority level of each of the multiple task types is determined; as well as Based on the usage level and priority level of the multiple task types and the task types to which the multiple tasks to be executed belong, a set of tasks is selected from the multiple tasks for execution. The number of tasks in the set belonging to the corresponding task type is determined by the usage level of the corresponding task type. The set of tasks is sorted according to the priority level of their respective task types.

2. The method of claim 1, wherein determining the usage level of the plurality of task types includes: Based on the average resource usage of each of the plurality of storage layers during the current first time period, the corresponding usage level of the task type associated with the corresponding storage layer among the plurality of storage layers is determined.

3. The method according to claim 2, further comprising: The usage level of each of the multiple task types is updated based on the average historical resource usage of each of the multiple storage layers during a second time period prior to the current time, wherein the second time period is longer than the first time period.

4. The method of claim 1, wherein determining the priority level of the plurality of task types comprises: Based on the changes in historical resource usage of each of the plurality of storage layers during a third time period prior to the current moment, the corresponding priority level of the task type associated with the corresponding storage layer among the plurality of storage layers is determined, wherein the task type associated with the storage layer with gradually increasing historical resource usage corresponds to a higher priority level.

5. The method of claim 1, wherein selecting the set of tasks from the plurality of tasks comprises: Based on the usage level, priority level, and task type to which the multiple tasks to be executed belong, a weighted load balancing (WRR) algorithm is used to select a set of tasks from the multiple tasks.

6. The method of claim 1, wherein selecting the set of tasks from the plurality of tasks comprises: Based on the usage level of the plurality of task types, a set of tasks is selected from the plurality of tasks, wherein the number of tasks belonging to the corresponding task type in the set of tasks does not exceed the number indicated by the usage level of the corresponding task type.

7. The method of claim 1, wherein the set of tasks is alternately ordered by task type according to the number indicated by the priority level of each task type.

8. The method of claim 1, wherein the plurality of task types includes at least two of the following: Defragmentation Idle resource recycling, or Data layering.

9. An electronic device, comprising: At least one processor; as well as At least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured, together with the at least one processor, to cause the electronic device to perform actions, the actions including: By normalizing the current resource usage of each of the multiple storage layers in the storage system, the usage level of the task type associated with the corresponding storage layer among the multiple task types is determined. The multiple storage layers share the physical resources of the storage system, and the tasks in the storage system are executed on the corresponding storage layer according to their task type. Based on the historical resource usage of each of the multiple storage layers, the priority level of each of the multiple task types is determined; and Based on the usage level and priority level of the multiple task types and the task types to which the multiple tasks to be executed belong, a set of tasks is selected from the multiple tasks for execution. The number of tasks in the set belonging to the corresponding task type is determined by the usage level of the corresponding task type. The set of tasks is sorted according to the priority level of their respective task types.

10. The electronic device of claim 9, wherein determining the usage level of the plurality of task types includes: Based on the average resource usage of each of the plurality of storage layers during the current first time period, the corresponding usage level of the task type associated with the corresponding storage layer among the plurality of storage layers is determined.

11. The electronic device of claim 10, wherein the action further comprises: The usage level of each of the multiple task types is updated based on the average historical resource usage of each of the multiple storage layers during a second time period prior to the current time, wherein the second time period is longer than the first time period.

12. The electronic device of claim 9, wherein determining the priority level of the plurality of task types includes: Based on the changes in historical resource usage of each of the plurality of storage layers during a third time period prior to the current moment, the corresponding priority level of the task type associated with the corresponding storage layer among the plurality of storage layers is determined, wherein the task type associated with the storage layer with gradually increasing historical resource usage corresponds to a higher priority level.

13. The electronic device of claim 9, wherein selecting the set of tasks from the plurality of tasks comprises: Based on the usage level, priority level, and task type to which the multiple tasks to be executed belong, a weighted load balancing (WRR) algorithm is used to select a set of tasks from the multiple tasks.

14. The electronic device of claim 9, wherein selecting the set of tasks from the plurality of tasks comprises: Based on the usage level of the plurality of task types, a set of tasks is selected from the plurality of tasks, wherein the number of tasks belonging to the corresponding task type in the set of tasks does not exceed the number indicated by the usage level of the corresponding task type.

15. The electronic device of claim 9, wherein the set of tasks is alternately ordered by task type according to the number indicated by the priority level of each task type.

16. The electronic device of claim 9, wherein the plurality of task types includes at least two of the following: Defragmentation Idle resource recycling, or Data layering.

17. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising machine-executable instructions that, when executed, cause a device to perform the method according to any one of claims 1-8.

Images