Storage cluster load balancing based on predicted performance indicators

Abstract

An apparatus includes a processing device configured to initiate load balancing for a storage cluster comprising two or more storage nodes, predict performance metrics for the storage nodes at two or more points in time within a specified future time period, and based at least in part on the predicted performance metrics, select a first one of the storage nodes as a source storage node and a second one of the storage nodes as a target storage node. The processing device is further configured to determine at least one storage object residing on the source storage node and to perform load balancing for the storage cluster by migrating the at least one storage object from the source storage node to the target storage node, the at least one storage node reducing a rate of performance imbalance for the storage cluster within the specified future time period when migrated to the target storage node.

Description

Technical Field

[0001] This field relates generally to information processing, and more specifically to storage in information processing systems. Background Technology

[0002] Storage arrays and other types of storage systems are typically shared by multiple host devices over a network. Applications running on these host devices each consist of one or more processes that perform application functionality. These processes issue input-output (I / O) operation requests to the storage system. The storage system's storage controller serves these I / O operation requests. In some information processing systems, multiple storage systems can be used to form a storage cluster. Summary of the Invention

[0003] The illustrative embodiments of this disclosure provide techniques for storage cluster load balancing based on predicted performance metrics.

[0004] In one embodiment, an apparatus includes at least one processing means, the at least one processing means including a processor coupled to memory. The at least one processing means is configured to perform the following steps: initiating load balancing for a storage cluster comprising two or more storage nodes; predicting performance metrics of the two or more storage nodes of the storage cluster at two or more time points within a specified future time period; and selecting a first storage node of the two or more storage nodes of the storage cluster as a source storage node and a second storage node of the two or more storage nodes of the storage cluster as a target storage node, based at least in part on the predicted performance metrics of the two or more storage nodes of the storage cluster at the two or more time points within the specified future time period. The at least one processing means is further configured to perform the following steps: identifying at least one storage object residing on the source storage node, the at least one storage object reducing the performance imbalance rate of the storage cluster within at least the specified future time period when migrated to the target storage node; and performing load balancing for the storage cluster by migrating the at least one storage object from the source storage node to the target storage node.

[0005] These and other illustrative embodiments include, but are not limited to, methods, devices, networks, systems, and processor-readable storage media. Attached Figure Description

[0006] Figure 1 This is a block diagram of an information processing system configured for load balancing of a storage cluster based on predicted performance metrics, as described in the illustrative implementation scheme.

[0007] Figure 2This is a flowchart of an exemplary process for load balancing of a storage cluster based on predicted performance metrics in an illustrative implementation.

[0008] Figure 3 The illustration shows storage objects stored on storage nodes of a storage cluster in an illustrative implementation.

[0009] Figure 4 The diagram illustrates the storage node processing load before and after a storage object rebalancing operation in an illustrative implementation.

[0010] Figure 5 A graph showing the trends of increasing and decreasing storage access data patterns in the illustrative implementation is provided.

[0011] Figure 6 A graph showing the trend of circular storage access data patterns in an illustrative implementation is provided.

[0012] Figure 7 A graph showing the trend of irregular storage access data patterns in an illustrative implementation is provided.

[0013] Figure 8 An illustrative implementation of a process flow for optimizing storage cluster performance by moving data across storage nodes in a storage cluster is shown.

[0014] Figure 9 The diagram illustrates the processing load of the smart storage nodes before and after the storage object rebalancing operation in the illustrative implementation.

[0015] Figure 10 and Figure 11 An example of a processing platform, which can be used to implement at least a portion of an information processing system, is shown in an illustrative embodiment. Detailed Implementation

[0016] This document describes illustrative embodiments with reference to exemplary information processing systems and associated computers, servers, storage devices, and other processing apparatuses. However, it should be understood that the embodiments are not limited to use with the specific illustrative system and apparatus configurations shown. Therefore, the term "information processing system" as used herein is intended to be broadly interpreted to encompass, for example, processing systems including cloud computing and storage systems, as well as other types of processing systems including various combinations of physical and virtual processing resources. Thus, an information processing system may include, for example, at least one data center or other type of cloud-based system that includes one or more clouds hosting tenants accessing cloud resources.

[0017] Figure 1An information processing system 100 is illustrated, configured according to an illustrative embodiment to provide functionality for storage cluster load balancing based on predicted performance metrics. The information processing system 100 includes one or more host devices 102-1, 102-2, ... 102-M (collectively referred to as storage arrays 106) communicating via a network 104 with one or more storage arrays 106-1, 106-2, ... 106-M (collectively referred to as storage arrays 106). The network 104 may include a storage area network (SAN).

[0018] Storage array 106-1 (e.g.) Figure 1 The storage array 106-1 (shown) includes multiple storage devices 108, each storing data utilized by one or more applications running on the host device 102. The storage devices 108 are illustratively arranged in one or more storage pools. The storage array 106-1 also includes one or more storage controllers 110 that facilitate I / O processing of the storage devices 108. The storage array 106-1 and its associated storage devices 108 are examples of what is more generally referred to herein as a “storage system.” Such a storage system in this embodiment is shared by the host device 102 and is therefore also referred to herein as a “shared storage system.” In embodiments where only a single host device 102 exists, the host device 102 can be configured to exclusively use the storage system.

[0019] Host device 102 illustratively includes a corresponding computer, server, or other type of processing device capable of communicating with storage array 106 via network 104. For example, at least a subset of host device 102 may be implemented as a corresponding virtual machine of a computing service platform or other type of processing platform. In such an arrangement, host device 102 illustratively provides computing services, such as executing one or more applications on behalf of each of one or more users associated with a corresponding host device in host device 102.

[0020] The term “user” in this article is intended to be interpreted broadly as encompassing numerous arrangements of human, hardware, software, or firmware entities, and combinations thereof.

[0021] Computing and / or storage services can be provided to users based on Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and / or Function as a Service (FaaS) models; however, it should be understood that numerous other cloud infrastructure deployments can be used. Furthermore, illustrative implementations can be implemented outside the context of cloud infrastructure, such as in the case of stand-alone computing and storage systems implemented within a given enterprise.

[0022] Storage device 108 of storage array 106-1 may implement logical units (LUNs) configured to store user objects associated with host device 102. These objects may include files, blocks, or other types of objects. Host device 102 interacts with storage array 106-1 using read / write commands and other types of commands transmitted over network 104. In some embodiments, such commands more specifically include Small Computer System Interface (SCSI) commands, but other types of commands may be used in other embodiments. The terminology used extensively herein for a given I / O operation descriptively includes one or more such commands. References to terms such as “input-output” and “IO” herein should be understood to refer to input and / or output. Therefore, an I / O operation involves at least one of input and output.

[0023] Furthermore, as used herein, the term "storage device" is intended to be broadly interpreted to encompass, for example, logical storage devices, such as LUNs or other logical storage volumes. A logical storage device can be defined in storage array 106-1 as a distinct portion comprising one or more physical storage devices. Therefore, storage device 108 can be considered to include a corresponding LUN or other logical storage volume.

[0024] The storage device 108 of storage array 106-1 can be implemented using a solid-state drive (SSD). Such SSDs are implemented using non-volatile memory (NVM) devices such as flash memory. Other types of NVM devices that can be used to implement at least a portion of storage device 108 include non-volatile random access memory (NVRAM), phase-change RAM (PC-RAM), and magnetic RAM (MRAM). Various types of NVM devices or other storage devices, as well as various combinations thereof, can also be used. For example, a hard disk drive (HDD) can be used in conjunction with or in place of an SSD or other type of NVM device. Therefore, at least a subset of storage device 108 can be implemented using a wide variety of other types of electronic or magnetic media.

[0025] exist Figure 1In the information processing system 100, it is assumed that storage array 106 is part of storage cluster 105 (e.g., where storage array 106 may be used to implement one or more storage nodes in a clustered storage system including multiple storage nodes interconnected by one or more networks), and it is assumed that host device 102 submits IO operations to be processed by storage cluster 105. It is assumed that at least one of the storage controllers of storage array 106 (e.g., storage controller 110 of storage array 106-1) implements intelligent data movement functionality across storage devices 108 of storage array 106-1 (e.g., between different storage devices in storage device 108 or portions thereof providing different storage tiers in storage cluster 105) and between storage array 106-1 and one or more other storage arrays among storage arrays 106-2 to 106-M. This intelligent data movement functionality is provided via storage node and storage object performance metric prediction module 112 (also referred to as performance metric prediction module 112) and storage object movement module 114.

[0026] The intelligent data movement functionality serves as part of the storage cluster-level load balancing operation of storage cluster 105. Load balancing of storage cluster 105 can be initiated in response to various conditions, such as user requests, determination that at least a threshold amount of time has elapsed since the last load balancing, determination that the current performance imbalance rate of the storage cluster exceeds an acceptable imbalance rate threshold, etc. Once load balancing is initiated, the performance metric prediction module 112 predicts the performance metrics of two or more storage nodes (e.g., storage array 106) of storage cluster 105 at two or more points in time within a specified future time period. The performance metric prediction module 112 then selects a first storage node from the two or more storage nodes of the storage cluster as the source storage node and a second storage node from the two or more storage nodes of the storage cluster as the target storage node, based at least in part on the predicted performance metrics of the two or more storage nodes of the storage cluster at the two or more points in time within the specified future time period.

[0027] The performance metric prediction module 112 is also configured to determine at least one storage object residing on a source storage node that, upon migration to a target storage node, reduces the performance imbalance rate of the storage cluster over at least a specified future time period. This determination may be based, at least in part, on predicted performance metrics of two or more storage objects residing on the source storage node at two or more points in time within the specified future time period. The storage object migration module 114 is configured to perform load balancing for the storage cluster 105 by migrating at least one storage object from the source storage node to the target storage node.

[0028] As described above, in some implementations, it is assumed Figure 1Storage array 106 in the embodiment is part of storage cluster 105. It is assumed that storage cluster 105 provides or implements multiple different storage tiers of a tiered storage system. For example, a given tiered storage system may include a speed or performance tier implemented using flash storage devices or other types of SSDs and a capacity tier implemented using HDDs, where one or more of these tiers may be server-based. It will be apparent to those skilled in the art that a variety of other types of storage devices and tiered storage systems can be used in other embodiments. The specific storage device used in a given storage tier may vary depending on the specific needs of a given embodiment, and a variety of different storage device types may be used in a single storage tier. As previously indicated, the term “storage device” as used herein is intended to be interpreted broadly and may therefore encompass, for example, SSDs, HDDs, flash drives, hybrid drives, or other types of storage products and devices or portions thereof, and illustratively includes logical storage devices such as LUNs.

[0029] It should be understood that a tiered storage system may include more than two storage tiers, such as one or more "performance" tiers and one or more "capacity" tiers, wherein the performance tiers illustratively provide increased IO performance characteristics relative to the capacity tiers, and the capacity tiers illustratively use storage that is relatively less expensive than the performance tiers. There may also be multiple performance tiers, each providing a different level of service or performance as needed, or there may be multiple capacity tiers.

[0030] Despite Figure 1 In one implementation, the performance metric prediction module 112 and the storage object movement module 114 are shown as being implemented inside the storage array 106-1 and outside the storage controller 110. However, in other implementations, one or both of the performance metric prediction module 112 and the storage object movement module 114 may be implemented at least partially inside the storage controller 110, or at least partially outside the storage array 106-1, such as being implemented in one of the host devices 102, on one or more other storage arrays 106-2 to 106-M, or on one or more servers outside the host device 102 and storage array 106 (e.g., including implementation on a cloud computing platform or other type of information technology (IT) infrastructure). Furthermore, although... Figure 1 Not shown, but other storage arrays in storage arrays 106-2 to 106-M may implement corresponding instances of the performance metric prediction module 112 and the storage object movement module 114.

[0031] At least some of the functionality of the performance metric prediction module 112 and the storage object movement module 114 can be implemented, at least in part, in the form of software stored in memory and executed by the processor.

[0032] Figure 1 In the embodiments, the host device 102 and storage array 106 are assumed to be implemented using at least one processing platform, wherein each processing platform includes one or more processing units, each processing unit having a processor coupled to memory. Such processing units may illustratively include specific arrangements of computing, storage, and networking resources. For example, in some embodiments, the processing units are implemented at least in part using virtual resources such as VMs or Linux containers (LXC) or a combination of both, such as in an arrangement in which Docker containers or other types of LXC are configured to run on virtual machines (VMs).

[0033] Although host device 102 and storage array 106 can be implemented on respective different processing platforms, numerous other arrangements are possible. For example, in some embodiments, one or more of host devices 102 and at least some portions of one or more of storage arrays 106 are implemented on the same processing platform. One or more of the storage arrays 106 can therefore be implemented at least partially within at least one processing platform implementing at least one subset of the host devices 102.

[0034] Network 104 can be implemented using a variety of different types of networks to interconnect storage system components. For example, network 104 may include a SAN as part of a global computer network such as the Internet, but other types of networks may be part of a SAN, including wide area networks (WANs), local area networks (LANs), satellite networks, telephone or wired networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or various parts or combinations of these and other types of networks. Therefore, in some embodiments, network 104 includes a combination of a variety of different types of networks, each including processing means configured to communicate using Internet Protocol (IP) or other relevant communication protocols.

[0035] As a more specific example, some implementations may utilize one or more high-speed local area networks, in which associated processing devices communicate with each other using peripheral interconnect high-speed (PCIe) cards and networking protocols such as InfiniBand, Gigabit Ethernet, or Fibre Channel. As those skilled in the art will understand, numerous alternative networking arrangements are possible in a given implementation.

[0036] While in some embodiments certain commands used by host device 102 to communicate with storage array 106 illustratively include SCSI commands, other types of commands and command formats may be used in other embodiments. For example, some embodiments may utilize command features and functionality associated with NVM Express (NVMe) as described in NVMe specification revision 1.3 of May 2017, which is incorporated herein by reference. Other storage protocols of this type that may be utilized in the illustrative embodiments disclosed herein include: architecture-based NVMe, also known as NVMoF; and Transmission Control Protocol (TCP)-based NVMe, also known as NVMe / TCP.

[0037] Assuming that the memory array 106-1 in this embodiment includes persistent memory implemented using flash memory or other types of non-volatile memory, more specific examples include NAND-based flash memory or other types of non-volatile memory, such as resistive RAM, phase-change memory, spin torque transfer magnetoresistive RAM (STT-MRAM), and 3D XPoint-based... TM Intel Optane memory TM Device. Further assuming that the persistent memory is separate from the storage device 108 of storage array 106-1, however, in other embodiments, the persistent memory may be implemented as one or more designated portions of one or more of the storage devices 108. For example, in some embodiments, such as those involving an all-flash memory array, storage device 108 may include a flash-based storage device, or may be implemented wholly or partially using other types of non-volatile memory.

[0038] As mentioned above, communication between host device 102 and storage array 106 can utilize PCIe connections or other types of connections implemented on one or more networks. For example, illustrative embodiments may use interfaces such as Internet SCSI (iSCSI), Serial Attached SCSI (SAS), and Serial ATA (SATA). In other embodiments, numerous other interfaces and associated communication protocols may be used.

[0039] In some implementations, storage array 106 can be implemented as part of a cloud-based system.

[0040] Therefore, it should be apparent that the term “memory array” as used herein is intended to be interpreted broadly and may encompass several different instances of commercially available memory arrays.

[0041] Other types of storage products that can be used to implement a given storage system in the illustrative embodiments include software-defined storage, cloud storage, object-based storage, and scale-out storage. In the illustrative embodiments, combinations of these and other storage types can also be used to implement a given storage system.

[0042] In some implementations, the storage system includes a first storage array and a second storage array arranged in an active-active configuration. For example, such an arrangement can be used to ensure that data stored in one storage array is replicated to the other using a synchronous replication process. This data replication across multiple storage arrays can be used to facilitate fault recovery in system 100. Thus, one of the storage arrays can act as a production storage array relative to another storage array that serves as a backup or recovery storage array.

[0043] However, it should be understood that the embodiments disclosed herein are not limited to active-active configuration or any other particular storage system arrangement. Therefore, the illustrative embodiments in this document can be configured using a variety of other arrangements, including, for example, active-passive arrangements, active-active asymmetric logical unit access (ALUA) arrangements, and other types of ALUA arrangements.

[0044] These and other storage systems may be part of what is more generally referred to herein as a processing platform, which includes one or more processing units, each including a processor coupled to memory. A given such processing unit may correspond to one or more virtual machines or other types of virtualization infrastructure, such as Docker containers or other types of LXC. As indicated above, communication between such elements of system 100 may take place on one or more networks.

[0045] As used herein, the term "processing platform" is intended to be interpreted broadly to include, for example, but not limited to, multiple sets of processing devices and one or more associated storage systems configured to communicate over one or more networks. For example, a distributed implementation of host device 102 is possible, wherein some host devices of host device 102 reside in a data center located in a first geographic location, while other host devices of host device 102 reside in one or more other data centers located in one or more other geographic locations that may be far from the first geographic location. Storage array 106 may be implemented at least partially in the first geographic location, the second geographic location, and one or more other geographic locations. Thus, in some implementations of system 100, different host devices of host device 102 and storage array 106 may reside in different data centers.

[0046] Numerous other distributed implementations of host device 102 and storage array 106 are possible. Therefore, host device 102 and storage array 106 can also be implemented in a distributed manner across multiple data centers.

[0047] The following will combine Figure 10 and Figure 11 Additional examples of the processing platform used to implement part of system 100 in the illustrative embodiments are described in more detail.

[0048] It should be understood that Figure 1 The specific set of components shown for storage cluster load balancing based on predicted performance metrics is presented by way of illustrative example only, and in other embodiments, additional or alternative components may be used. Therefore, another implementation may include additional or alternative systems, devices, and other network entities, as well as different arrangements of modules and other components.

[0049] It should be understood that these and other features of the illustrative implementation are presented by way of example only and should not be construed as restrictive in any way.

[0050] Now refer to Figure 2 The flowchart below describes in more detail an exemplary process for storage cluster load balancing based on predicted performance metrics. It should be understood that this particular process is merely an example, and additional or alternative processes for storage cluster load balancing based on predicted performance metrics may be used in other implementations.

[0051] In this implementation, the process includes steps 200 to 208. These steps are assumed to be performed by the performance metric prediction module 112 and the storage object movement module 114. The process begins at step 200: initiating load balancing for a storage cluster comprising two or more storage nodes.

[0052] In step 202, performance metrics for two or more storage nodes in the storage cluster are predicted at two or more time points within a specified future time period. Step 202 may include, for a given storage node among the two or more storage nodes in the storage cluster, determining the access frequency trend pattern of the given storage node, and using the access frequency trend pattern to calculate the predicted performance metrics for the given storage node at each of the two or more time points within the specified future time period. Determining the access frequency trend pattern of the given storage node may include generating a trend function to predict the total amount of data accessed by the given storage node at two or more time points within the specified future time period. The access frequency trend pattern of the given storage node may include one of an increasing access frequency trend pattern and a decreasing access frequency trend pattern, and the prediction function may be generated using a least squares algorithm. The access frequency trend pattern may include a cyclic access frequency trend pattern, and the prediction function may be generated using at least one of an autocorrelation algorithm and a discrete Fourier transform algorithm. The access frequency trend pattern may include an irregular access frequency pattern, and the prediction function may be generated using the average historical data access of the given storage object over previous time periods.

[0053] Figure 2 The process continues to step 204: Based at least in part on the predicted performance metrics of two or more storage nodes in the storage cluster at two or more points in time within a specified future time period, a first storage node in the storage cluster is selected as the source storage node, and a second storage node in the storage cluster is selected as the target storage node. The sum of the predicted performance metrics of the first storage node selected as the source storage node in the storage cluster at two or more points in time within the specified future time period is higher than that of the second storage node selected as the target storage node in the storage cluster.

[0054] Step 206 includes identifying at least one storage object residing on the source storage node, which, upon migration to the target storage node, reduces the performance imbalance rate of the storage cluster over at least a specified future time period. In some implementations, the identification in step 206 is based at least in part on predicting performance metrics of two or more storage objects residing on the source storage node at two or more points in time within the specified future time period. Predicting performance metrics of two or more storage objects residing on the source storage node at two or more points in time within the specified future time period may include: identifying access frequency trend patterns of the two or more storage objects residing on the source storage node; using the identified access frequency trend patterns to calculate predicted performance metrics for each of the two or more storage objects residing on the source storage node at two or more points in time within the specified future time period; and selecting a given storage object residing on the source storage node as at least one storage object from the two or more storage objects residing on the source storage node, based at least in part on a comparison of the sum of the predicted performance metrics of the two or more storage objects residing on the source storage node at two or more points in time within the specified future time period. Selecting a given storage object residing on the source storage node from two or more storage objects as at least one storage object may be further based, at least in part, on the type of access frequency trend pattern associated with the two or more storage objects residing on the source storage node. Selecting a given storage object residing on the source storage node from two or more storage objects as at least one storage object may further, or alternatively, be based, at least in part, on the confidence level of the predicted performance metrics of the two or more storage objects residing on the source storage node.

[0055] In step 208, load balancing is performed on the storage cluster by migrating at least one storage object from the source storage node to the target storage node. In some implementations, initiating load balancing for the storage cluster in step 200 is in response to detecting that the current performance imbalance rate of the storage cluster exceeds a first specified threshold imbalance rate. Steps 202 through 208 may be repeated until the current performance imbalance rate of the storage cluster is lower than a second specified threshold imbalance rate. The second specified threshold imbalance rate may be less than the first specified threshold imbalance rate.

[0056] Storage object load balancing is a feature that allows for the optimization of storage resource utilization in a storage cluster. Storage object load balancing functionally identifies overused storage nodes in a storage cluster and migrates storage objects (e.g., LUNs, file systems, data stores, files, etc.) from overused storage nodes to inactive storage nodes in the storage cluster in real time. Illustrative implementations provide a novel performance balancing mechanism for storage clusters. In some implementations, performance trends are learned from historical data for each storage node in the storage cluster and for each storage object stored on those nodes. These learned performance trends for storage nodes and storage objects are then used to perform intelligent storage cluster-level performance balancing operations. Intelligent storage cluster-level performance balancing is performed by balancing predictions of storage node and storage object performance over a future time period (e.g., 10 days). Therefore, performing storage cluster-level performance balancing at a given time will keep the performance distribution of the entire storage cluster balanced over a relatively long period (e.g., a future time period during which storage node and storage object performance predictions are made). It should be noted that storage cluster-level performance balancing does not necessarily provide the best balance for the storage cluster at the moment (e.g., due to performance predictions over a specified future time period), but rather a balance over a specified future time period during which performance is predicted during the overall improvement period. Exemplary implementations thus improve cross-storage cluster performance balancing by maintaining the distribution of storage objects across storage nodes as evenly as possible over a longer period than using conventional methods.

[0057] A storage cluster (e.g., storage cluster 105) is a configuration of multiple storage nodes (e.g., storage array 106) whose resources are pooled together as a resource pool contributed to the storage cluster. Resources may include processing resources (e.g., CPU or other computing resources), memory resources, network resources, and storage resources. Each storage node in the storage cluster contributes a set of such resources. In a storage cluster, it is important to balance the workload across each storage node to mitigate the risk of a negative impact on performance from an ever-increasing workload.

[0058] Figure 3Storage cluster 305 is illustrated, comprising multiple storage nodes 301-1, 301-2, ... 301-S (collectively referred to as storage nodes 301). Each of the storage nodes 301 stores a corresponding set of storage objects: storage node 301-1 stores storage objects 310-1-1, 310-1-2, ... 310-1-O (collectively referred to as storage objects 310-1), storage node 301-2 stores storage objects 310-2-1, 310-2-2, ... 310-2-O (collectively referred to as storage objects 310-2), and storage node 301-S stores storage objects 310-S-1, 310-S-2, ... 310-SO (collectively referred to as storage objects 310-S). Storage objects 310-1, 310-2, ... 310-S are collectively referred to as storage objects 310. It should be understood that the specific number "O" of storage objects on each of the storage nodes 301 can be different. For example, the "O" value of storage node 301-1 can be different from the "O" value of storage node 301-2.

[0059] Conventional storage cluster load balancing mechanisms typically focus on balancing the load of storage objects at a single point in time. Therefore, such conventional load balancing mechanisms can only achieve relatively short-term performance balancing. Because storage object load varies over time, the load balancing achieved through conventional mechanisms may be disrupted relatively quickly after the initial load balancing operation. Consequently, to maintain load balancing across storage nodes in the cluster, load balancing operations need to be performed repeatedly. Furthermore, frequent storage object migration operations themselves increase the load on the storage nodes, leading to inefficiency.

[0060] Figure 4 The graph 400 shows the storage node's processing load versus time. Figure 4 In the example, the storage cluster has six storage nodes. A regular load balancing operation is performed at a first time point 401 (e.g., time t1), where the regular load balancing operation only considers the current storage node performance at one point in time (e.g., time t1). Therefore, as can be seen from graph 400, the load balancing effect is temporary, and the storage node processing load becomes unbalanced again at a second time point 402 (e.g., before time t5). In some cases, the performance distribution of storage nodes may deteriorate rapidly after the load balancing operation is performed at time t1. Therefore, additional load balancing operations using such regular methods are often necessary.

[0061] In the illustrative implementation, a storage cluster performance balancing mechanism is used, which learns the performance trends of each storage node and storage object from historical data and leverages these performance trends to perform storage cluster-level balancing by predicting the performance of storage nodes and storage objects over a future time period (e.g., 10 days). Therefore, it is expected that each load balancing operation will maintain a balanced performance distribution across the storage cluster over a relatively long period (such as at least a future time period (e.g., 10 days) during which the performance of storage nodes and storage objects is predicted). In this way, the implementation can effectively improve storage cluster performance balancing by using relatively few rebalancing operations (e.g., and therefore relatively few storage object movement operations) to maintain the most balanced distribution of storage objects possible.

[0062] In storage systems, most data (e.g., stored objects) exhibits access frequency patterns that vary over time. These access frequency patterns can vary depending on how end-users or customers utilize the storage system (e.g., access frequency patterns specific to a customer's business). Non-stationary time series data is of particular interest because it is a key area for valuable forecasting. Non-stationary time series exhibit several patterns, including but not limited to increasing or decreasing trend patterns, cyclical trend patterns, and irregular trend patterns.

[0063] An increasing or decreasing trend pattern refers to a long-term increase or decrease in access frequency. Figure 5 A graph 500 illustrates the patterns of decreasing and increasing loss levels. The increasing and decreasing trend patterns can be linear or non-linear (e.g., exponential). In some implementations, the increasing and decreasing trend patterns are determined using least squares or other regression analyses. Least squares is a method used in regression analysis and has important applications in data fitting. Least squares problems typically fall into one of two categories: linear or ordinary least squares; and non-linear least squares.

[0064] Cyclical trend patterns refer to a frequency of visits that rises and falls in a certain regularity. A common example is seasonal data, whose time series are affected by seasonal factors, and the data has a fixed or known pattern (e.g., daily, weekly, monthly, yearly, etc.). Figure 6 A graph 600 illustrates the cyclical loss level access pattern. To determine whether the data access frequency is cyclical, autocorrelation and discrete Fourier transform methods can be used to detect periodicity and further determine the period or frequency of the cyclical or seasonal time series. Figure 6 In the curve 600, the loss level shows a quarterly cyclical trend pattern.

[0065] Irregular trend patterns refer to random changes or unpredictable access frequencies within a specified time range. Figure 7A graph 700 illustrating an irregular loss level access pattern is shown. In some implementations, data is classified as having an irregular trend pattern if it does not follow an increasing trend pattern, a decreasing trend pattern, or a cyclical trend pattern.

[0066] It should be understood that in other implementations, various other access frequency trend patterns may be used, and various other methods may be used to identify whether data access exhibits different access frequency trend patterns.

[0067] When triggering a smart rebalancing operation, it is first necessary to predict the performance of storage objects over a future time period. In some implementations, the future time period begins simultaneously with or close to the time when the smart rebalancing operation is triggered. Various data pattern detection and statistical analysis / data fitting models (e.g., least squares, autocorrelation, discrete Fourier transform, etc.) can be used to determine the trend function of storage node and storage object performance over time. Storage Node At the point of time Performance and storage objects At the point of time The performance can be calculated using a trend function as follows:

[0068]

[0069] In the equation above, This represents the trend function determined using data pattern detection and statistical analysis / data fitting models (such as least squares, autocorrelation, discrete Fourier transform, etc.).

[0070] Assume there is A number of periodic sampling points are used for the current rebalancing operation. A specific quantity The sampling period can be selected by the end user, who determines an appropriate and reasonable sampling period based on the actual usage scenario. Storage node for each periodic sampling point The sum of the predicted performance is calculated using the following formula:

[0071]

[0072] Cross Storage object of sampling points in each period The sum of the predicted performance is calculated using the following formula:

[0073]

[0074] Then, a set of rules is used to rank the storage nodes and storage objects to determine which of them are most likely to experience poor performance (e.g., high workload) in the future. The first step is to periodically sample each target (e.g., a storage node or storage object) by predicting its performance at certain points in the future time period. The next step is to calculate the sum of the predicted performance at different sampling points for each target (e.g., a storage node or storage object). The targets (e.g., storage nodes or storage objects) are then ranked according to the sum of their respective predicted performance values.

[0075] Based on a newly introduced performance prediction method for storage nodes and storage objects, a novel cluster storage object distribution rebalancing algorithm is implemented. First, the current performance imbalance rate of the storage cluster is periodically evaluated according to the following equation, expressed as: :

[0076]

[0077] In the above equations, Indicates storage node The performance, and This indicates the number of storage arrays or storage nodes in the storage cluster. The larger the value, the more storage nodes The greater the workload, the more storage nodes... The worse the performance, the worse the quality.

[0078] At the current imbalance rate When the threshold is exceeded, the storage object relocation algorithm will be triggered. Figure 8 The process flow 800 of the storage object relocation algorithm is shown, which begins at step 801. In step 803, the imbalance value of the storage cluster is calculated. In step 805, the current imbalance rate is determined. Does it exceed an acceptable threshold? If the result determined in step 805 is no, then process flow 800 ends in step 817. If the result determined in step 805 is yes, then process flow 800 proceeds to step 807. In step 807, for each storage node in the storage cluster... calculate Among them, the one with the largest The storage node was selected as the source storage node, while the one with the smallest... The selected storage node is chosen as the destination storage node. It should be understood that in some implementations, step 807 may include calculating the destination storage node only for a subset of the storage nodes in the storage cluster, rather than for all storage nodes in the storage cluster. The specific number of storage nodes in a subset can be user-configurable or based on some other factor. As an example, this can be calculated for different storage nodes in a storage cluster. Value, until the maximum value calculated is determined. The minimum value and calculation There exists at least a threshold difference between the values, such that in the case of the maximum value calculated... The value storage node and the minimum computational value Moving storage objects between storage nodes has at least a threshold benefit.

[0079] In step 809, for each storage object residing in the source storage node calculate And select the one with the largest The storage object is used as the target storage object. It should be understood that in some implementations, step 809 may include calculating only a subset of the storage objects residing in the source storage node, rather than all storage objects residing on the source storage node. The specific number of storage objects in a subset can be user-configurable or based on some other factor. As an example, this can be calculated for different storage objects residing in the source storage node. Value, until the maximum value calculated is determined. The minimum value and calculation There exists at least a threshold difference between the values, such that the maximum value calculated is obtained by... The value storage object is moved to the minimum value with computation. There is at least a threshold benefit to the value storage object.

[0080] In step 811, the target storage object is moved from the source storage node to the destination storage node. In step 813, the current imbalance rate is recalculated after the target storage object is relocated. In step 815, determine whether... Here, Φ This is the expected performance imbalance rate of the storage cluster that the end user wants to achieve. (Selection) Φ The value is set to avoid excessive storage object migration operations, which could otherwise lead to resource contention. If the result of step 815 is yes, then process flow 800 ends in step 817. If the result of step 815 is no, then the process flow returns to step 807.

[0081] In process flow 800, the performance of storage nodes and storage objects is predicted by sampling over a specified future time period, and the most "valuable" storage object (e.g., the one with the highest value) residing in the storage cluster among the higher-loaded storage nodes (e.g., the source storage node) is selected. Move from a high-load storage node in the storage cluster to the best low-load storage node in the storage cluster (e.g., the target storage node).

[0082] In some implementations, additional characteristics or factors are considered when selecting source and target storage nodes, as well as the most “valuable” storage objects to be moved from the source node to the target node within the storage cluster. Such characteristics and factors may include the type of access frequency trend pattern predicted for a given storage object. As an example, in some implementations, storage objects predicted to have irregular access frequency trend patterns over future time periods may be considered poor candidates for movement as part of cluster-level load balancing (e.g., because the associated access frequency of such storage objects is difficult to predict over future time periods).

[0083] Such characteristics and factors may also include, or alternatively, the confidence level of the predicted performance of the source storage node, the target storage node, and the storage object. For example, if the predicted access frequency trend pattern of a given storage object is below a certain specified confidence threshold, this may indicate that the given storage object is a poor candidate for being moved as part of a cluster-level load balancer (e.g., because the predicted access frequency trend pattern may be incorrect due to the low confidence level of the prediction).

[0084] Additional characteristics and factors may include user-specified rules for the following: certain storage objects that should not be moved between storage nodes; whether different storage objects should or should not coexist on the same storage node in the storage cluster; etc. Various other characteristics and factors may be considered when selecting source storage nodes, target storage nodes, and one or more storage objects to be moved from the source storage node to the target storage node.

[0085] Figure 9 The graph 900, showing the storage node's processing load versus time, is similar to... Figure 4 The curve 400 shows the load balancing, but process flow 800 is used for storage cluster-level load balancing instead of conventional load balancing methods. Figure 9 In the example, the storage cluster has six storage nodes. Process flow 800 is executed at a first time point 901 (e.g., time t1), which takes into account predictions of the performance of storage nodes and storage objects over multiple time points (e.g., times t1 to t5). Therefore, it can be seen from graph 900 that the balancing effect is longer-lasting (e.g., compared to...). Figure 4Compared to the conventional load balancing performed in the example, the storage node processing load remains balanced over a longer period, until at least time 903 (e.g., until at least time t5), corresponding to at least the predicted performance of the storage nodes and storage objects. Therefore, the frequency of load balancing operations can be advantageously reduced. Rebalancing is based on a comprehensive performance analysis of the storage objects and storage nodes, enabling the storage cluster to achieve fully balanced performance without needing to perform rebalancing again at time t5. This not only improves the efficiency of rebalancing but also reduces the consumption of resources in the storage cluster (e.g., processing, memory, storage, and network resources of the storage nodes) and reduces service reliability issues caused by frequent storage object movements.

[0086] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and are not required to exist in other embodiments. Furthermore, the specific types of information processing system features and functionalities shown in the accompanying drawings and as described above are merely exemplary, and numerous other arrangements may be used in other embodiments.

[0087] Now refer to Figure 10 and Figure 11 An illustrative embodiment of a processing platform is described in more detail, which is used to implement the functionality of storage cluster load balancing for prediction-based performance metrics. Although described in the context of system 100, in other embodiments, these platforms may also be used to implement at least a portion of other information processing systems.

[0088] Figure 10 An exemplary processing platform including cloud infrastructure 1000 is shown. Cloud infrastructure 1000 includes components that can be used to implement… Figure 1 The information processing system 100 comprises at least a portion of the physical and virtual processing resources. The cloud infrastructure 1000 includes multiple virtual machines (VMs) and / or container sets 1002-1, 1002-2, ... 1002-L implemented using virtualization infrastructure 1004. Virtualization infrastructure 1004 runs on physical infrastructure 1005 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructure illustratively includes the kernel control group of a Linux operating system or other types of operating systems.

[0089] The cloud infrastructure 1000 also includes a collection of applications 1010-1, 1010-2, ... 1010-L running on corresponding collections of VMs / containers in the VM / container collections 1002-1, 1002-2, ... 1002-L, under the control of the virtualization infrastructure 1004. The VM / container collection 1002 may include a corresponding VM, a corresponding collection of one or more containers, or a corresponding collection of one or more containers running in a VM.

[0090] exist Figure 10 In some implementations of the scheme, the VM / container set 1002 includes corresponding VMs implemented using a virtualization infrastructure 1004 including at least one hypervisor. A hypervisor platform may be used to implement the hypervisor within the virtualization infrastructure 1004, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines may include one or more distributed processing platforms, which include one or more storage systems.

[0091] exist Figure 10 In other implementations of the scheme, the VM / container set 1002 includes corresponding containers implemented using virtualization infrastructure 1004 that provides operating system-level virtualization functionality, such as support for Docker containers running on bare metal hosts or running on VMs. The containers are implemented illustratively using the corresponding kernel control group of the operating system.

[0092] As is evident from the above, one or more of the processing modules or other components of system 100 may each run on a computer, server, storage device, or other processing platform element. Such an element may be considered as an example of what is more generally referred to herein as a "processing device". Figure 10 The cloud infrastructure 1000 shown can represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 11 The processing platform 1100 shown in the figure.

[0093] In this embodiment, the processing platform 1100 includes a portion of the system 100 and includes a plurality of processing devices denoted as 1102-1, 1102-2, 1102-3, ... 1102-K, which communicate with each other via a network 1104.

[0094] Network 1104 may include any type of network, such as global computer networks (such as the Internet), WANs, LANs, satellite networks, telephone or wired networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or portions or combinations of these and other types of networks.

[0095] The processing device 1102-1 in the processing platform 1100 includes a processor 1110 coupled to a memory 1112.

[0096] Processor 1110 may include a microprocessor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), video processing unit (VPU) or other types of processing circuitry, as well as portions or combinations of such circuitry elements.

[0097] Memory 1112 may take any combination of forms including random access memory (RAM), read-only memory (ROM), flash memory, or other types of memory. Memory 1112 and other memories disclosed herein should be considered as illustrative examples of what is more generally referred to as a “processor-readable storage medium” storing executable program code of one or more software programs.

[0098] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. A given such article of manufacture may include, for example, a storage array, a storage disk, or an integrated circuit comprising RAM, ROM, flash memory, or other electronic memory, or any of a variety of other types of computer program products. As used herein, the term "article of manufacture" should be understood to exclude transient propagated signals. Numerous other types of computer program products including processor-readable storage media may be used.

[0099] The processing device 1102-1 also includes a network interface circuit 1114 for interfacing the processing device with the network 1104 and other system components, and may include a conventional transceiver.

[0100] The other processing devices 1102 of the processing platform 1100 are assumed to be configured in a manner similar to that shown for the processing device 1102-1 in the figure.

[0101] Furthermore, the specific processing platform 1100 shown in the figure is presented only by way of example, and the system 100 may include additional or alternative processing platforms, as well as a number of different processing platforms in any combination, wherein each such platform includes one or more computers, servers, storage devices or other processing devices.

[0102] For example, other processing platforms used to implement illustrative implementation schemes may include converged infrastructure.

[0103] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0104] As previously indicated, components of the information processing system disclosed herein can be implemented, at least in part, in the form of one or more software programs stored in memory and executed by a processor of a processing device. For example, at least a portion of the functionality disclosed herein for storage cluster load balancing based on predicted performance metrics is illustratively implemented as software running on one or more processing devices.

[0105] It should be emphasized again that the above embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments can be used. For example, the disclosed technology is applicable to a variety of other types of information processing systems, storage systems, storage clusters, etc. Moreover, the specific configurations of the system and apparatus elements illustratively shown in the accompanying drawings, and the associated processing operations, may change in other embodiments. Furthermore, the various assumptions made above in describing the illustrative embodiments should be considered exemplary and not as requirements or limitations of this disclosure. Numerous other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. A device for load balancing in a storage cluster, comprising: At least one processing device, the at least one processing device including a processor coupled to a memory; The at least one processing device is configured to perform the following steps: Initiate load balancing for a storage cluster consisting of two or more storage nodes; Predict the performance metrics of the two or more storage nodes of the storage cluster at two or more points in time within a specified future time period; Based at least in part on the predicted performance metrics of the two or more storage nodes of the storage cluster at two or more time points within the specified future time period, a first storage node of the two or more storage nodes of the storage cluster is selected as the source storage node, and a second storage node of the two or more storage nodes of the storage cluster is selected as the target storage node. Identify at least one storage object residing on the source storage node, and when the at least one storage object is migrated to the target storage node, reduce the performance imbalance rate of the storage cluster over at least the specified future time period; as well as Load balancing is performed on the storage cluster by migrating at least one storage object from the source storage node to the target storage node.

2. The device of claim 1, wherein initiating load balancing for the storage cluster is in response to detecting that the current performance imbalance rate of the storage cluster exceeds a first specified threshold imbalance rate.

3. The device of claim 2, wherein the prediction, selection, determination and execution steps are repeated until the current performance imbalance rate of the storage cluster is lower than a second specified threshold imbalance rate.

4. The device according to claim 3, wherein the second specified threshold imbalance rate is less than the first specified threshold imbalance rate.

5. The device of claim 1, wherein predicting the performance metrics of a given node among the two or more storage nodes of the storage cluster at the two or more time points within the specified future time period comprises: Determine the access frequency trend pattern of the given storage node; as well as The access frequency trend pattern is used to calculate the predicted performance metrics of the given storage node at each of the two or more time points within the specified future time period.

6. The device of claim 5, wherein determining the access frequency trend pattern of the given storage node comprises generating a trend function to predict the total amount of data accessed by the given storage node at two or more time points within the specified future time period.

7. The device of claim 6, wherein the access frequency trend pattern of the given storage node includes one of an increasing access frequency trend pattern and a decreasing access frequency trend pattern, and wherein the trend function is generated using a least squares algorithm.

8. The device of claim 6, wherein the access frequency trend pattern includes a cyclic access frequency trend pattern, and wherein the trend function is generated using at least one of an autocorrelation algorithm and a discrete Fourier transform algorithm.

9. The device of claim 6, wherein the access frequency trend pattern includes an irregular access pattern, and wherein the trend function is generated using the average value of historical data accesses of the given storage node over a previous time period.

10. The device of claim 1, wherein the sum of the predicted performance metrics of the first storage node selected as the source storage node from among the two or more storage nodes of the storage cluster at the two or more time points within the specified future time period is higher than that of the second storage node selected as the target storage node from among the two or more storage nodes of the storage cluster.

11. The device of claim 1, wherein determining the at least one storage object residing on the source storage node is based at least in part on predicting performance metrics of two or more storage objects residing on the source storage node at two or more points in time within the specified future time period, wherein the at least one storage object reduces the performance imbalance rate of the storage cluster during at least the specified future time period when it is migrated to the target storage node.

12. The apparatus of claim 11, wherein predicting the performance metrics of the two or more storage objects residing on the source storage node at the two or more time points within the specified future time period comprises: Determine the access frequency trend pattern of the two or more storage objects residing on the source storage node; The determined access frequency trend pattern is used to calculate the predicted performance metrics for each of the two or more storage objects residing on the source storage node at each of the two or more time points within the specified future time period. as well as The at least one storage object is selected from the two or more storage objects residing on the source storage node by comparing the sum of the predicted performance metrics of the two or more storage objects residing on the source storage node at two or more time points within the specified future time period.

13. The device of claim 12, wherein selecting the given storage object residing on the source storage node from the two or more storage objects as the at least one storage object is further based at least in part on the type of access frequency trend pattern associated with the two or more storage objects residing on the source storage node.

14. The device of claim 12, wherein selecting the given storage object residing on the source storage node from the two or more storage objects as the at least one storage object is further based at least in part on the confidence level of the predicted performance metric of the two or more storage objects residing on the source storage node.

15. A computer program product comprising a non-transitory processor-readable storage medium in which program code of one or more software programs is stored, wherein the program code, when executed by at least one processing device, causes the at least one processing device to perform the following steps: Initiate load balancing for a storage cluster consisting of two or more storage nodes; Predict the performance metrics of the two or more storage nodes of the storage cluster at two or more points in time within a specified future time period; Based at least in part on the predicted performance metrics of the two or more storage nodes of the storage cluster at two or more time points within the specified future time period, a first storage node of the two or more storage nodes of the storage cluster is selected as the source storage node, and a second storage node of the two or more storage nodes of the storage cluster is selected as the target storage node. Identify at least one storage object residing on the source storage node, and when the at least one storage object is migrated to the target storage node, reduce the performance imbalance rate of the storage cluster over at least the specified future time period; as well as Load balancing is performed on the storage cluster by migrating at least one storage object from the source storage node to the target storage node.

16. The computer program product of claim 15, wherein the sum of the predicted performance metrics of the first storage node selected as the source storage node among the two or more storage nodes of the storage cluster at the two or more time points within the specified future time period is higher than that of the second storage node selected as the target storage node among the two or more storage nodes of the storage cluster.

17. The computer program product of claim 15, wherein determining the at least one storage object residing on the source storage node is based at least in part on predicting performance metrics of two or more storage objects residing on the source storage node at two or more points in time within the specified future time period, wherein the at least one storage object reduces the performance imbalance rate of the storage cluster during at least the specified future time period when it is migrated to the target storage node.

18. A method for load balancing in a storage cluster, comprising: Initiate load balancing for a storage cluster consisting of two or more storage nodes; Predict the performance metrics of the two or more storage nodes of the storage cluster at two or more points in time within a specified future time period; Based at least in part on the predicted performance metrics of the two or more storage nodes of the storage cluster at two or more time points within the specified future time period, a first storage node of the two or more storage nodes of the storage cluster is selected as the source storage node, and a second storage node of the two or more storage nodes of the storage cluster is selected as the target storage node. Identify at least one storage object residing on the source storage node, and when the at least one storage object is migrated to the target storage node, reduce the performance imbalance rate of the storage cluster over at least the specified future time period; as well as Load balancing is performed on the storage cluster by migrating at least one storage object from the source storage node to the target storage node; The method is performed by at least one processing device, the processing device including a processor coupled to a memory.

19. The method of claim 18, wherein the sum of the predicted performance metrics of the first storage node selected as the source storage node from the two or more storage nodes of the storage cluster at the two or more time points within the specified future time period is higher than that of the second storage node selected as the target storage node from the two or more storage nodes of the storage cluster.

20. The method of claim 18, wherein determining the at least one storage object residing on the source storage node is based at least in part on predicting performance metrics of two or more storage objects residing on the source storage node at two or more points in time within the specified future time period, wherein the at least one storage object reduces the performance imbalance rate of the storage cluster during at least the specified future time period when it is migrated to the target storage node.

Images