Automatic file system capacity management using file system utilization prediction

Abstract

Automated file system capacity management techniques are provided using file system utilization forecasting. One method includes obtaining input data representing utilization of a storage capacity of a file system of a given storage system, forecasting future utilization of the storage capacity of the file system based on a portion of the obtained input data, and automatically adjusting the storage capacity of the file system based at least in part on a result of a comparison of the forecasted utilization of the storage capacity with a current utilization of the storage capacity. The comparison of the forecasted utilization of the storage capacity with a current utilization can include comparing the current utilization of the storage capacity with forecasted utilization of the storage capacity over at least a first time period and a second time period following a current time period to determine a trend in the utilization of the storage capacity.

Description

Technical Field

[0001] This field generally relates to information processing technologies, and more specifically to information storage. Background Technology

[0002] Storage arrays and other types of storage systems are typically shared by multiple host devices over a network. Applications running on these host devices issue input-output (I / O) operations to the storage system. File systems can be implemented within the storage system to organize and retrieve data stored there. For example, data can be stored in file formats and accessed using filenames and / or addresses. Summary of the Invention

[0003] In one implementation, a method includes: obtaining input data representing the utilization rate of storage capacity of a file system of a given storage system; predicting, at least in part, the future utilization rate of the storage capacity of the file system of the given storage system over one or more time periods based on at least a portion of the obtained input data; and automatically adjusting the storage capacity of the file system of the given storage system based, at least in part, the result of a comparison between the predicted utilization rate of the storage capacity and the current utilization rate of the storage capacity.

[0004] In some implementations, the one or more time periods correspond to multi-step long-term windows. The comparison of the predicted utilization of the storage capacity with the current utilization of the storage capacity may include comparing the current utilization of the storage capacity with the predicted utilization of the storage capacity over at least a first time period and a second time period after the current time period to determine a trend in the utilization of the storage capacity of the file system.

[0005] Other illustrative embodiments include, but are not limited to, devices, systems, methods, and computer program products including processor-readable storage media. Attached Figure Description

[0006] Figure 1 An information processing system according to an illustrative embodiment is shown, the information processing system being configured for automatic file system capacity management using file system utilization prediction;

[0007] Figure 2 The determination of one or more storage capacity adjustments for a file system based on multi-step prediction of file system utilization according to an illustrative implementation scheme is shown.

[0008] Figure 3 This is a sample table showing a comparison of the performance of the initial model and a tuned model with hyperparameters tuned according to the illustrative implementation scheme;

[0009] Figures 4A to 4C A number of different file system utilization conditions for performing automatic file system expansion are shown according to an illustrative implementation scheme;

[0010] Figure 5 This illustrates a method for using based on an illustrative embodiment. Figures 4A to 4C A sample table of many exemplary automatic file system extension classification rules for performing automatic file system extensions under different file system utilization conditions;

[0011] Figure 6A Exemplary pseudocode for performing an automatic file system reduction process according to an illustrative embodiment is shown;

[0012] Figure 6B Exemplary pseudocode is shown for a process of determining the watermark level of a dynamic storage pool according to an illustrative embodiment;

[0013] Figures 7A to 7D A number of different file system utilization conditions for performing automatic file system reduction are shown according to the illustrative implementation scheme;

[0014] Figure 8 This illustrates a method for using a method based on an illustrative embodiment. Figures 7A to 7D A sample table of many exemplary automatic file system reduction classification rules for performing automatic file system reduction under different file system utilization conditions;

[0015] Figure 9 This is a flowchart illustrating an exemplary implementation of a process for automatic file system capacity management using file system utilization prediction, according to an illustrative embodiment.

[0016] Figure 10 An exemplary processing platform is shown that can be used to implement at least a portion of one or more embodiments of this disclosure, the exemplary processing platform including cloud infrastructure; and

[0017] Figure 11 Another exemplary processing platform is shown that can be used to implement at least a portion of one or more embodiments of this disclosure. Detailed Implementation

[0018] This document will describe illustrative embodiments of the present disclosure with reference to exemplary communication devices, storage devices, and processing devices. However, it should be understood that the present disclosure is not limited to use with the specific illustrative configurations shown. One or more embodiments of the present disclosure provide methods, apparatus, and computer program products for automatic file system capacity management using file system utilization prediction.

[0019] As mentioned above, file systems can be implemented in storage systems to organize and retrieve data stored within them. As the capacity utilization of the file system changes, the storage space allocated to it can also change. Several technical issues arise regarding when to perform such capacity management tasks and how such changes in storage space allocation occur. For example, it is generally undesirable to expand and / or shrink the allocated storage space too rapidly (this may be referred to as storage space "thrashing"). Furthermore, capacity management tasks can also degrade the performance of the storage system by consuming its resources.

[0020] In one or more embodiments, techniques for automatic file system capacity management using file system utilization prediction are provided. The future utilization of the file system's storage capacity over one or more time periods can be predicted based on real-time (e.g., current) storage capacity utilization data, as discussed below. Storage capacity can be automatically adjusted based on a comparison of the predicted utilization with the current utilization of the storage capacity.

[0021] Typically, when configuring a file system, a small amount of storage capacity is initially allocated to the file system, and then the amount of allocated storage space is adjusted based on the utilization rate of the allocated storage space by clients. To improve client storage efficiency, it is generally important that the amount of allocated storage space is reasonably consistent with the utilization rate of the allocated storage space. Existing file system space management implementations can provide functionality for automatic file system expansion (sometimes called auto-expansion) and / or automatic file system reduction (sometimes called auto-compression), thereby automatically expanding or shrinking the amount of allocated storage space based on capacity requirements (e.g., based on the high watermark threshold (HWM) for automatic file system expansion and the low watermark threshold (LWM) for automatic file system reduction). For example, the file system capacity can be automatically expanded when the utilization rate of the allocated storage space exceeds and remains above 75% of the allocated storage system. Conversely, the file system capacity can be automatically reduced, for example, when the utilization rate of the allocated storage space falls below 70% of the allocated storage system. In some implementations, dynamic high watermark and low watermark thresholds can be employed.

[0022] Performing file system capacity management tasks in a storage system can present numerous performance challenges. For instance, when an automatic file system shrinks and reclaims portions of the storage system, data may be temporarily unavailable during the reclamation and deallocation process (e.g., due to the large number of internal I / O operations typically performed to reduce the amount of storage space allocated to the file system). Additionally, automatic file system shrinking can be time-consuming, potentially reducing overall system availability, as additional locks can be used to prevent conflicts with other space management processes.

[0023] Figure 1 An information processing system 100 configured according to an illustrative embodiment is shown. The information processing system 100 includes: a plurality of host devices 102-1, 102-2, ..., 102-N, collectively referred to herein as host devices 102; and a plurality of storage arrays 106-1, 106-2, ..., 106-M, collectively referred to herein as host storage array 106. The host devices 102 are configured to communicate with the storage system 106 via a network 104. The network 104 may include a storage area network (SAN). Similarly, one or more file system utilization prediction servers 120 are coupled to the network 104, as discussed below.

[0024] Host device 102 illustratively includes a server, or other type of computer, such as an enterprise computer system, a cloud-based computer system, or other arrangement of multiple computing nodes associated with a corresponding user. Host device 102-1 represents one or more of host devices 102. Figure 1 As shown, a representative host device 102-1 executes one or more applications 103-1 to 103-0 (collectively referred to herein as application 103).

[0025] For example, in some embodiments, host device 102 illustratively provides computing services, such as executing one or more applications 103 on behalf of each of one or more users associated with a corresponding host device in the host device. Such applications 103 illustratively generate I / O operations processed by storage array 106. As used herein, the term "input / output" means at least one of an input and an output. For example, an I / O operation may include write requests and / or read requests to a logical address of a particular storage array 106. These and other types of I / O operations are also generally referred to herein as I / O requests.

[0026] Host device 102 and / or application 103 are configured to interact with storage array 106 via network 104. Such interaction illustratively includes generating I / O operations, such as write requests and read requests; and sending such requests via network 104 for processing by storage array 106.

[0027] like Figure 1The exemplary storage array 106-1 shown includes a plurality of storage devices 108-1, ..., 108-P (collectively referred to as storage devices 108), each storage device storing data utilized by one or more applications 103 running on one or more of the host devices 102. The storage devices 108 are illustratively arranged in one or more storage pools. The storage array 106-1 also includes one or more storage controllers 110 facilitating I / O processing of the storage devices 108. The storage array 106-1 and its associated storage devices 108 are examples of what is more generally referred to herein as a “storage system.” Such a storage system in this embodiment is shared by the host devices 102 and is therefore also referred to herein as a “shared storage system.” In embodiments where only a single host device 102 exists, the host device 102 may be configured to exclusively use the storage system.

[0028] Storage array 106-1 also includes a file system 112 for organizing and retrieving data stored in storage device 108. Although in Figure 1 In this example, it is referred to as a file system, but it should be understood that data can be stored in various formats according to the file system 112, including but not limited to files, blocks, and / or objects. Furthermore, although the file system 112 is shown as a separate block in the storage array 106-1, it should be understood that all or part of the file system 112 may be implemented as part of the storage controller 110 and / or storage device 108.

[0029] Storage array 106-1 implements a file system capacity adjustment module 114, which is configured to perform intelligent automatic file system expansion and / or automatic file system shrinkage functionality on file system 112 using file system utilization prediction, as described herein. Although not explicitly stated herein for clarity... Figure 1 It is explicitly stated, but in some implementations it is assumed that other storage arrays in storage arrays 106-2 to 106-M implement corresponding additional instances of file system 112 and / or file system capacity adjustment module 114, as well as a corresponding set of storage devices and storage controllers implemented in a manner similar to that described herein with respect to storage array 106-1.

[0030] In addition, although Figure 1While shown external to the storage controller 110 and storage device 108 of storage array 106-1, in other embodiments, the file system capacity adjustment module 114 may be implemented at least partially within one or more of the storage controller 110 and / or storage device 108. For example, at least one of the storage controllers 110 may be implemented as a virtual storage controller running on an embedded hypervisor of storage array 106-1. The file system capacity adjustment module 114 may be part of such a virtual storage controller or may be implemented separately from the virtual storage controller on such an embedded hypervisor. Various other arrangements are possible, including implementing at least a portion of the functionality of the file system capacity adjustment module 114 external to storage array 106-1 (e.g., on one or more of host devices 102, on a separate server, or on a cloud computing infrastructure). In alternative or additional implementations, a single file system 112 may be used to organize and retrieve data from storage devices 108 on all or a subset of storage arrays 106-1, 106-2, ..., 106-M, and a single file system capacity adjustment module 114 may be used to manage the file system capacity across all or a subset of storage arrays 106-1, 106-2, ..., 106-M.

[0031] At least some of the functionality of the file system capacity adjustment module 114 can be implemented, at least in part, in the form of software stored in memory and executed by a processor.

[0032] Assuming that the storage array 106-1 in this embodiment includes persistent memory implemented using flash memory or other types of non-volatile memory (NVM) of the storage array 106-1. Other types of NVM devices that can be used to implement at least a portion of the storage device 108 include non-volatile RAM (NVRAM), phase-change RAM (PC-RAM), magnetic RAM (MRAM), resistive RAM, spin-transfer torque magnetoresistive RAM (STT-MRAM), and 3D XPoint-based... TM Intel Optane memory TM Device. Further assuming that the persistent memory is separate from the storage device 108 of storage array 106-1, however, in other embodiments, the persistent memory may be implemented as one or more designated portions of one or more of the storage devices 108. For example, in some embodiments, such as those involving all-flash storage arrays, storage device 108 may include flash-based storage devices, or may be implemented wholly or partially using other types of non-volatile memory.

[0033] These and various combinations of many different types of NVM devices can also be used. For example, a hard disk drive (HDD) can be used in conjunction with or in place of a solid-state drive (SSD) or other type of NVM device in storage array 106-1. Therefore, at least a subset of storage device 108 can be implemented using a wide variety of other types of electronic or magnetic media.

[0034] Therefore, it should be understood that in other embodiments, many different types of storage devices 108 may be used in storage array 106-1. For example, the term "storage system" as used herein in a broad sense may include a combination of different types of storage devices, such as in the case of a multi-tiered storage system including a flash-based fast tier and a disk-based capacity tier. In such embodiments, each of the fast tiers and capacity tiers of the multi-tiered storage system includes multiple storage devices, wherein different types of storage devices are used in different storage tiers within the storage tiers. For example, a fast tier may include flash drives, while a capacity tier may include HDDs. The specific storage devices used in a given storage tier may be different in embodiments, and multiple different types of storage devices may be used in a single storage tier. As used herein, the term "storage device" is intended to be interpreted broadly to cover, for example, SSDs, HDDs, flash drives, hybrid drives, or other types of storage devices.

[0035] In some implementations, storage array 106 can be implemented as part of a cloud-based system.

[0036] In some implementations, at least one of the storage arrays 106 illustratively includes one or more Unity... TM Or PowerMax TM Storage arrays, as well as other commercially available storage arrays from Dell Technologies.

[0037] Therefore, it should be apparent that the term "storage array" as used herein is intended to be interpreted broadly and may encompass many different instances of commercially available storage arrays. Other types of storage products that may be used to implement a given storage system in the illustrative embodiments include software-defined storage, cloud storage, object-based storage, and / or scale-out storage. In the illustrative embodiments, combinations of these and other storage types may also be used to implement a given storage system.

[0038] Storage device 108 of storage array 106-1 may implement logical units (LUNs) configured to store objects of a user associated with host device 102. These objects may include files, blocks, or other types of objects. Host device 102 interacts with storage array 106-1 using read and / or write commands, as well as other types of commands transmitted over network 104. Such commands more specifically include Small Computer System Interface (SCSI) commands or Internet SCSI (iSCSI) commands in some embodiments, but other types of commands may be used in other embodiments, including commands as part of a standard command set, or custom commands, such as “vendor-unique commands” or VU commands that are not part of a standard command set. Therefore, the term “command” as used herein is intended to be interpreted broadly to encompass, for example, a combination of multiple individual commands. In other embodiments, numerous other commands may be used.

[0039] While in some implementations, certain commands used by host device 102 to communicate with storage array 106-1 illustratively include SCSI or iSCSI commands, other implementations may utilize the command features and functionality associated with NVM Express (NVMe) to perform I / O operations. Other storage protocols of this type that may be utilized in the illustrative implementations disclosed herein include: architecture-based NVMe, also known as NVMoF; and Transmission Control Protocol (TCP)-based NVMe, also known as NVMe / TCP.

[0040] Furthermore, as used herein, the term "storage device" is intended to be broadly interpreted to encompass, for example, logical storage devices, such as LUNs or other logical storage volumes. A logical storage device can be defined in storage array 106-1 as a distinct portion comprising one or more physical storage devices. Therefore, storage device 108 can be considered to include a corresponding LUN or other logical storage volume.

[0041] The file system utilization prediction server 120 can be implemented, for example, in the cloud or at the site of an enterprise or another entity. In some implementations, the file system utilization prediction server 120 or a portion thereof can be implemented as part of a host device separate from the storage system (e.g., to prevent the disclosed techniques used for automatic file system capacity management using file system utilization prediction from impairing the performance of the storage system) or as part of the storage system. Moreover, as... Figure 1As described, the file system utilization prediction server 120 also includes a Long Short-Term Memory (LSTM) network training module 122, an LSTM hyperparameter tuner 124, and at least one trained LSTM model 126. In some implementations, the LSTM training module 122 trains one or more LSTM models to generate at least one trained LSTM model 126, as illustrated below in conjunction with examples... Figure 2 Further discussion. The LSTM hyperparameter tuner 124 can optimize the hyperparameters, for example, by performing Bayesian optimization to tune one or more hyperparameters of at least one trained LSTM model 126 to obtain accurate and stable prediction data. Bayesian optimization can specify a range of values ​​for each hyperparameter, select a metric to optimize (e.g., root mean square error or another error metric), and search for specific combinations of hyperparameter values ​​that optimize the selected metric.

[0042] One or more aspects of this disclosure recognize that LSTM models perform well in high-precision time series forecasting. It has been found that when an LSTM model is trained on diverse file system utilization data (sometimes referred to as file system usage data), the trained LSTM model reliably generates multi-step predictions of the utilization data from real utilization data of varying ranges and patterns. In some implementations, file system utilization forecasting is treated as a univariate time series forecasting modeling problem, where historical file system storage utilization data provides the input variable, and future predictions of storage utilization data provide the output variable. The LSTM model automatically learns one or more features associated with the file system utilization data and outputs a variable-length sequence that can be used for multi-step forecasting.

[0043] It should be understood that, Figure 1 The particular arrangement of elements 122, 124, and 126 shown in the file system utilization prediction server 120 of the embodiment is presented by way of example only, and alternative arrangements may be used in other embodiments. For example, in other embodiments, the functionality associated with elements 122, 124, and 126 may be combined into a single element or separated across more elements. As another example, multiple different processors may be used to implement different elements or portions thereof in elements 122, 124, and 126.

[0044] At least portions of elements 122, 124, and 126 may be implemented, at least partially, in the form of software stored in memory and executed by a processor. (Refer to, for example...) Figures 2 to 9 An exemplary process for using elements 122, 124, and 126 of an exemplary file system utilization prediction server 120 in information processing system 100 is described in more detail.

[0045] Additionally, the file system utilization prediction server 120 may have an associated database 130 configured to store file system utilization data 140, for example, related to one or more file systems (such as file system 112). For instance, the file system utilization data 140 related to one or more file systems may include storage utilization data of file system 112 at defined time steps (e.g., every minute), which can be used by the LSTM training module 122 of the file system utilization prediction server 120 to train an LSTM model (e.g., training every 8 hours). The collected data will be transferred to the file system utilization prediction server 120 for LSTM training.

[0046] In this embodiment, database 130 is implemented using one or more storage systems associated with file system utilization prediction server 120. Such storage systems may include any of a variety of different types of storage devices, such as network attached storage (NAS), SAN, direct attached storage (DAS), and distributed DAS, as well as combinations of these and other storage types (including software-defined storage).

[0047] Figure 1 In the implementation scheme, host device 102, storage array 106, and / or file system utilization prediction server 120 are assumed to be implemented using at least one processing platform, wherein each processing platform includes one or more processing units, each processing unit having a processor coupled to memory. Such processing units may illustratively include specific arrangements of computing, storage, and networking resources. For example, in some implementation schemes, the processing units are implemented using at least in part virtual resources such as virtual machines (VMs) or Linux containers (LXCs) or a combination of both, such as in an arrangement in which Docker containers or other types of LXCs are configured to run on VMs.

[0048] Host device 102, storage array 106, and / or file system utilization prediction server 120 may be implemented on respective different processing platforms, but numerous other arrangements are also possible. For example, in some embodiments, at least portions of host device 102, storage array 106, and / or file system utilization prediction server 120 are implemented on the same processing platform. For example, one or more of storage arrays 106 may therefore be implemented at least partially within at least one processing platform implementing at least one subset of host device 102 and / or file system utilization prediction server 120.

[0049] As used herein, the term "processing platform" is intended to be interpreted broadly to include, for example, but not limited to, multiple sets of processing devices and associated storage systems configured to communicate over one or more networks. For example, a distributed implementation of system 100 is possible, wherein some components of the system reside in a data center located in a first geographical location, while other components reside in one or more other data centers located in one or more other geographical locations that may be far from the first geographical location. Therefore, in some implementations of system 100, host device 102, storage array 106, and / or file system utilization prediction server 120 may reside in different data centers. Numerous other distributed implementations of host device and storage array 106-1 are possible.

[0050] The following will combine Figure 10 and Figure 11 Additional examples of the processing platform used to implement host device 102, storage array 106, and / or file system utilization prediction server 120 are described in more detail in the illustrative implementation.

[0051] In one or more embodiments, the disclosed techniques for automatic file system capacity management using file system utilization prediction are illustratively performed, at least in part, by storage controller 110, file system capacity adjustment module 114, and / or file system utilization prediction server 120 in a manner described in more detail elsewhere herein.

[0052] The term “user” in this article is intended to be interpreted broadly as encompassing numerous arrangements of human, hardware, software, or firmware entities, and combinations thereof.

[0053] This provides computing and / or storage services to users under Platform as a Service (PaaS), Infrastructure as a Service (IaaS), Storage as a Service (STaaS), and / or Function as a Service (FaaS) models. However, it should be understood that numerous other cloud infrastructure deployments can also be used. Moreover, illustrative implementations can be implemented outside the context of cloud infrastructure, such as in the case of stand-alone computing and storage systems implemented within a given enterprise.

[0054] Network 104 can be implemented using various different types of networks to interconnect storage system components. Assuming network 104 constitutes part of a global computer network such as the Internet, other types of networks can be part of network 104, including wide area networks (WANs), local area networks (LANs), satellite networks, telephone or cable networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or portions or combinations of these and other types of networks. Therefore, in some embodiments, network 104 includes a combination of various different types of networks, each including processing means configured to communicate using Internet Protocol (IP) or other communication protocols.

[0055] As a more specific example, some implementations may utilize one or more high-speed local area networks, in which associated processing devices communicate with each other using peripheral interconnect high-speed (PCIe) cards and networking protocols such as InfiniBand, Gigabit Ethernet, or Fibre Channel. As those skilled in the art will understand, numerous alternative networking arrangements are possible in a given implementation.

[0056] Other types of storage products that can be used to implement a given storage system in the illustrative embodiments include software-defined storage, cloud storage, object-based storage, clustered storage systems, and scale-out storage. In the illustrative embodiments, combinations of these and other storage types can also be used to implement a given storage system.

[0057] Therefore, the term "storage system" as used herein is intended to be interpreted broadly and should not be considered limited to a specific type of storage system, such as CAS systems, distributed storage systems, or storage systems based on flash memory or other types of NVM storage devices. The term "storage system" as used broadly herein can include, for example, any type of system, which includes multiple storage devices such as NAS, SAN, DAS, and distributed DAS, as well as combinations of these and other storage types (including software-defined storage).

[0058] It should be understood that these and other features of the illustrative implementation are presented by way of example only and should not be construed as restrictive in any way.

[0059] Therefore, different numbers, types and arrangements of system components, such as host device 102, storage array 106, network 104, file system utilization prediction server 120 and database 130 or parts thereof, can be used in other implementations.

[0060] It should be understood that, in cases such as Figure 1The specific sets of modules, elements, and other components implemented in the illustrated system 100 are presented by way of example only. In other embodiments, only a subset of these components, or additional or alternative sets of components, may be used, and such components may exhibit alternative functionality and configuration.

[0061] Figure 2 The illustration shows the determination of one or more storage capacity adjustments 295 for the file system using multi-step file system utilization prediction 270 according to the illustrative implementation scheme. Figure 2 In the example, file system utilization data 210 (e.g., historical file system utilization data 210) is applied to the data preparation and scaling module 230 during the training phase. The data preparation and scaling module preprocesses the file system utilization data 210 (e.g., historical utilization data) to make the file system utilization data 210 suitable for training one or more LSTM models.

[0062] As mentioned above, file system utilization data 210 may include, for example, storage utilization data of file system 112 collected at defined time steps (e.g., every minute) (e.g., file system utilization data 140 from database 130). Data preparation and scaling module 230 separates the preprocessed file system utilization data 210 to generate (i) training and validation data 235, which is applied to one or more initial LSTM models 250 for training and validation of the one or more initial LSTM models 250 respectively, and (ii) test data 240, which is applied to one or more trained LSTM models 260 generated from the training phase. For example, the training dataset may comprise 80% of the file system utilization dataset. Additionally, data preparation and scaling module 230 may, for example, use a min-max scaler (sometimes called a Minmax scaler) to scale the values ​​in the range of 0 to 1. In at least one embodiment, this can be achieved, for example, by... Figure 1 The LSTM training module 122 of the file system utilization prediction server 120 is trained and updated every eight hours.

[0063] exist Figure 2 In the implementation scheme, the LSTM model parameter tuner 245 (e.g., a hyperparameter tuner) tunes one or more hyperparameters of one or more initial LSTM models 250 during the training phase, for example, using Bayesian optimization, as described below. Figure 3 Further discussion.

[0064] In the prediction phase, real-time file system utilization data 290 from storage array 280 is applied to one or more exemplary trained LSTM models 260 to generate multi-step file system utilization predictions 270, as described below. Figures 4A to 4C Further discussion. Real-time file system utilization data 290 may include, for example, file system utilization data from the most recent time window and / or file system utilization data prior to one or more time windows where file system utilization prediction is required. In some embodiments, real-time file system utilization data 290 may also be applied during the prediction phase to a data preparation and scaling module 230 (or another preprocessing module) that preprocesses the real-time file system utilization data 290 to make it suitable for processing by the trained LSTM model 260. Therefore, in some embodiments, file system utilization data 210 may include real-time file system utilization data 290.

[0065] As mentioned above, the multi-step file system utilization prediction 270 can be stored in the database 130 and accessed via the file system capacity adjustment module 285 of the storage array 280 to perform the disclosed prediction-based file system capacity management techniques. In at least some embodiments, the multi-step file system utilization prediction 270 includes two-step utilization predictions, and the file system capacity adjustment module 285 compares the two-step utilization predictions generated by one or more exemplary trained LSTM models 260 with real-time file system utilization data 290 to determine one or more file system storage capacity adjustments 295 (e.g., by evaluating trends in file system utilization), as described below, for example, in combination with... Figures 4A to 4C and Figures 7A to 7D Further discussion is needed.

[0066] As discussed further below, the file system capacity adjustment module 285 implements the disclosed automatic file system capacity management techniques to determine, in some embodiments, the degree of adjustment (if any) to the file system storage capacity in each time step using real-time file system utilization data 290 within the current time step and two-step utilization predictions generated by one or more exemplary trained LSTM models 260 within two time steps after the current time step. As used herein, the current utilization of storage capacity can be the utilization of storage capacity in the current time period or the utilization of storage capacity in the current time period of interest (not necessarily the current time) (such as the time period currently being processed), which will be apparent to those skilled in the art.

[0067] Among other benefits, LSTM models are trained and used to achieve, for example... Figure 2 The example uses a separate file system utilization prediction server 220 instead of a multi-step prediction 270 that generates file system utilization via storage array 280, so that the performance of storage array 280 is not compromised.

[0068] Figure 3 This is a sample table 300 showing a comparison of the performance of the initial LSTM model and a tuned LSTM model with hyperparameters tuned according to the illustrative implementation. Figure 3 In the example, exemplary table 300 includes a file system identifier, input shape, initial RMSE (root mean square error) of the initial LSTM model, tuned RMSE of the tuned LSTM model, initial hyperparameter set of the initial LSTM model, and tuned hyperparameter set of the tuned LSTM model for each file system.

[0069] As mentioned above, Figure 2 The file system utilization prediction server 220 trains one or more initial LSTM models 250 to generate at least one trained LSTM model 260. The LSTM model parameter tuner 245 can optimize the hyperparameters, for example, by performing Bayesian optimization to tune one or more hyperparameters of at least one trained LSTM model 260 to obtain accurate and stable prediction data.

[0070] In at least some implementations, the LSTM model processes a three-dimensional input. The three dimensions of the input include: (i) a data sample (e.g., file system utilization data, where a sequence is a sample and a batch consists of one or more samples); (ii) a time step (where a time step is an observation point in the data sample); and (iii) a feature (where a feature is an observation value at a time step). Figure 3 In the first line of the example, for instance, the input shape “9884,100,1” indicates that 9884 samples were collected over 100 time steps in the input layer of the LSTM model, with only one feature (e.g., file system utilization data) in each time step.

[0071] exist Figure 3In the example, the set of hyperparameters to be tuned includes the number of epochs; batch size; number of neurons; random dropout; activation and learning rate. The number of epochs indicates the number of times training data is applied to the LSTM network during training. The batch size is the number of subsamples given to the LSTM network before parameter updates are performed. The number of neurons indicates the number of neurons in each hidden layer of the LSTM network. In at least one implementation, the number of neurons in each layer is set to be the same (but may differ). It will be apparent to those skilled in the art that the number of neurons should be adjusted according to the complexity of the solution (e.g., prediction tasks with higher levels of complexity require additional neurons). The number of neurons can be set in the range of 10 to 100. The learning rate defines how quickly the LSTM network updates its parameters (e.g., a low learning rate slows down the learning process but results in smooth convergence, while a larger learning rate speeds up learning but may not converge). Random dropout is a regularization technique to avoid overfitting the LSTM model (e.g., improving validation accuracy), thus improving generalization ability. Activation functions can be used to introduce nonlinearity into the LSTM model, which allows the deep learning model to learn nonlinear prediction boundaries.

[0072] Figures 4A to 4C Several different file system utilization conditions 400, 430, and 450 for performing automatic file system expansion are shown according to illustrative embodiments. As mentioned above, in some embodiments, in Figure 2 In the example, the multi-step file system utilization prediction 270 generated by one or more trained LSTM models 260 includes two-step file system utilization predictions (U1 and U2), and the file system capacity management module 285 compares the two-step utilization predictions (e.g., within two future time steps) generated by one or more exemplary trained LSTM models 260 with real-time file system utilization data 290 (U0) within the current time step to determine one or more file system storage capacity adjustments 295 (e.g., by evaluating the trend of file system utilization).

[0073] exist Figure 4A In the example, the current file system utilization U0 is lower than the first-step utilization prediction U1, which is also lower than the second-step utilization prediction U2, indicating that the file system utilization is predicted to be expanding. Figure 4B In the example, the current file system utilization U0 is lower than the first-step utilization prediction U1, which is greater than the second-step utilization prediction U2. This indicates that the file system utilization is predicted to expand but is expected to decrease. Figure 4CIn the example, the current file system utilization U0 is greater than the first-step utilization prediction U1, which indicates that the file system utilization is predicted to decrease (e.g., shrink).

[0074] Figure 5 which is a sample table 500 showing many exemplary automatic file system expansion classification rules for performing automatic file system expansion based on Figures 4A to 4C different file system utilization conditions 400, 430, 450 according to an illustrative embodiment. In Figure 5 the example, the exemplary automatic file system expansion classification rule adopts: (i) a gap threshold based on the gap (e.g., difference) between the allocated storage space and the utilized storage space for controlling the target size of file system capacity management (G au ) (e.g., G au the default value may be 10%); and (ii) a minimum gap threshold between the allocated storage space and the utilized storage space (Min g ) (e.g., Min g the default value may be 10GBs).

[0075] The first automatic file system expansion classification rule corresponds to the Figure 4A associated file system condition, wherein the file system utilization is predicted to expand. As shown by the corresponding condition, when U0<U1<U2, the file system utilization is predicted to expand. In this case, when G au and Min g meet the following conditions, the automatic expansion of the file system is skipped:

[0076] and Min g <A0-U1,

[0077] wherein A i is the allocated size of the file system at time step i.

[0078] Otherwise, when G au exceeds the difference between A0 (current allocated storage space) and U1 or when the difference between A0 and U1 exceeds Min g , the file system capacity adjustment module 114 will evaluate the multi-step file system utilization prediction 270 to determine the target size expansion amount for automatic expansion of the file system as follows:

[0079]

[0080] The second automatic file system expansion classification rule corresponds to the Figure 4BAssociated file system conditions, wherein the file system utilization is predicted to expand but is expected to decrease subsequently. As shown in the corresponding condition, when U0<U1 but U2<U1, the file system utilization is predicted to expand and then decrease. In this case, automatic expansion of the file system is performed based on the following target expansion size:

[0081] Target_Size_Expansion = MIN(U1×(1+G au ),U1+Min g ).

[0082] The third automatic file system expansion classification rule corresponds to Figure 4C associated file system conditions, wherein the file system utilization is not predicted to expand. As shown in the corresponding condition, when U0>U1, the file system utilization is not predicted to expand. In this case, automatic expansion of the file system is skipped.

[0083] As mentioned above, for example, a watchdog may be enabled to check the utilization of file system space every two minutes to determine whether automatic expansion of the file system should be triggered.

[0084] Figure 6A illustrates an exemplary pseudocode of a process for performing automatic file system reduction according to an illustrative embodiment. In Figure 6A the example, the automatic file system reduction process 600 compares the current utilization P of the storage pool 利用率水平 with a storage pool watermark PWM determined, for example, by a dynamic storage pool watermark determination process 650, as will be discussed in further detail below in conjunction with Figure 6B further discussion below. The automatic file system reduction process 600 uses the comparison result to determine whether the allocated size of the storage pool should be reduced.

[0085] When P 利用率水平 is less than or equal to PWM, the automatic file system reduction process 600 disables automatic file system reduction for all file systems in the storage pool.

[0086] When P 利用率水平 is greater than PWM, the automatic file system reduction process 600 determines a target file system reduction size by comparing the current utilization of the storage capacity of the file system with the predicted utilization of the storage capacity of the file system, as will be discussed in further detail below in conjunction with Figures 7A to 7D and Figure 8 further discussion below.

[0087] Figure 6B illustrates an exemplary pseudocode of a process for determining a dynamic storage pool watermark level according to an illustrative embodiment. In Figure 6B the example, when P 大小When the size is less than or equal to a threshold (such as 100 trillion bytes), the dynamic storage pool level determination process 650 sets the PWM to the storage pool size P. 大小 75% of the value. When P 大小 When the threshold is exceeded (e.g., 100 trillion bytes), the dynamic storage pool level determination process 650 sets the PWM to the following values:

[0088]

[0089] However, due to Figure 6B The final rule shown, the exemplary dynamic storage pool level determination process 650, will not set the PWM to be greater than P. 大小 90% of the value. Specific percentage amounts (e.g., 75% and 90%), storage thresholds (e.g., 100 trillion bytes), and Figure 6B The PWM upper limit specified in the example may be configurable in some implementations and may have different values ​​for different file systems in other implementations, as will be apparent to those skilled in the art.

[0090] Figures 7A to 7D A number of different file system utilization conditions 700, 730, 750, and 770 for performing automatic file system reduction according to illustrative embodiments are shown respectively. In some embodiments, the disclosed techniques for automatic file system reduction attempt to avoid unnecessary space reclamation assessments checking the state of the storage pool relative to the predicted space utilization over two time steps.

[0091] As mentioned above, in some implementation schemes, by Figure 2 The multi-step file system utilization prediction 270 generated by one or more trained LSTM models 260 includes two-step utilization predictions (U1 and U2), and the file system capacity adjustment module 285 compares the two-step utilization predictions generated by one or more exemplary trained LSTM models 260 with real-time file system utilization data 290 (U0) at the current time to determine one or more file system storage capacity adjustments 295 (e.g., by evaluating the trend of file system utilization).

[0092] exist Figure 7A In the example, the current file system utilization U0 is greater than the first-step utilization prediction U1, which is also greater than the second-step utilization prediction U2, indicating that the file system utilization is predicted to decrease (e.g., shrink). Figure 7BIn the example, the current file system utilization U0 is greater than the first-step utilization prediction U1, which is less than the second-step utilization prediction U2. This indicates that the file system utilization is predicted to decrease, but is expected to be lower than the current utilization expansion. Figure 7C In the example, the current file system utilization U0 is greater than the first-step utilization prediction U1, but the second-step utilization prediction U2 is greater than the current file system utilization U0. This indicates that the file system utilization is predicted to decrease and is expected to expand beyond the current utilization. Figure 7D In the example, the current file system utilization U0 is less than the utilization prediction U1 in the first step, which indicates that the file system utilization is predicted to increase (e.g., expand).

[0093] Figure 8 These are examples illustrating the use of illustrative embodiments for... Figures 7A to 7D Table 800 provides a sample of many exemplary automatic file system reduction classification rules for performing automatic file system reduction under different file system utilization conditions 700, 730, 750, and 770. Figure 8 In the example, the exemplary automatic file system shrinking rule is similar to Figure 5 The automatic file system extension classification rules adopt (i)G au and (ii)Min g .

[0094] The first automatic file system shrinkage classification rule corresponds to... Figure 7A The associated file system conditions indicate that file system utilization is predicted to decrease. As shown in the corresponding conditions, file system utilization is predicted to decrease when U0 > U1 > U2. In this case, free space will be reclaimed to the extent possible by reducing the target file system size to the following reduction target value:

[0095] Target_size_reduction = MIN(U1×(1+G) au ),U1+Min g ).

[0096] The second automatic file system shrinkage classification rule corresponds to... Figure 7B The associated file system conditions, where file system utilization is predicted to decrease but is expected to exceed the current utilization by an extent. As shown in the corresponding conditions, when U2 > U1 and U0 > U2, file system utilization is predicted to decrease but is expected to be below the current utilization by an extent. In this case, automatic file system reduction is performed based on the following file system target size reduction (e.g., to avoid jitter):

[0097] Target size reduction = MIN(U2×(1+G)) au ),U2+Ming ).

[0098] The third automatic file system shrink classification rule corresponds to Figure 7C associated file system conditions, wherein file system utilization is predicted to decrease but an expansion exceeding the current utilization is expected. As shown in the corresponding condition, when U0<U2, file system utilization is predicted to decrease but an expansion exceeding the current utilization is expected. In this case, the automatic file system shrink of the file system is skipped.

[0099] The fourth automatic file system shrink classification rule corresponds to Figure 7D associated file system conditions, wherein file system utilization is predicted to increase. As shown in the corresponding condition, when U0<U1, file system utilization is predicted to decrease but an expansion exceeding the current utilization is expected. In this case, the automatic shrink of the file system is also skipped.

[0100] Figure 9 is a flowchart illustrating an exemplary implementation of process 900 for automatic file system capacity management using file system utilization prediction in accordance with an illustrative embodiment. In Figure 9 the example, process 900 initially obtains input data in step 902, said input data representing the utilization of storage capacity of a file system of a given storage system.

[0101] In step 904, future utilization of the storage capacity of the file system of the given storage system over one or more time periods is predicted based at least in part on at least a portion of the obtained input data. In step 906, the storage capacity of the file system of the given storage system is automatically adjusted based at least in part on a comparison result between the predicted utilization of the storage capacity and the current utilization of the storage capacity.

[0102] In one or more embodiments, the prediction is performed using a trained long short-term memory network. Bayesian optimization may be performed to tune one or more hyperparameters of the long short-term memory network to obtain the trained long short-term memory network.

[0103] In some embodiments, the one or more time periods correspond to a multi-step time window. For example, the comparison between the predicted utilization of the storage capacity and the current utilization of the storage capacity may include comparing the current utilization of the storage capacity with the predicted utilization of the storage capacity in at least a first time period and a second time period after the current time period, to determine a trend of the utilization of the storage capacity of the file system.

[0104] In response to (i) the predicted utilization of storage capacity in the second time period exceeding the predicted utilization of storage capacity in the first time period and (ii) the predicted utilization of storage capacity in the first time period exceeding the current utilization of storage capacity in the current time period, the size of the file system's storage capacity can be expanded by a first expansion amount. In response to (i) the predicted utilization of storage capacity in the first time period exceeding the current utilization of storage capacity in the current time period and (ii) the predicted utilization of storage capacity in the first time period exceeding the predicted utilization of storage capacity in the second time period, the size of the file system's storage capacity can also be expanded by a second expansion amount, wherein the second expansion amount is smaller than the first expansion amount.

[0105] The storage capacity of the file system may be prevented from being expanded by a first expansion amount in response to: (a)(i) the difference between the currently allocated storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period is less than a first gap threshold; and (b)(i) the difference between the currently allocated storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period is higher than a second gap threshold.

[0106] In response to (i) the current utilization of storage capacity in the current time period exceeding the predicted utilization of storage capacity in the first time period and (ii) the predicted utilization of storage capacity in the first time period exceeding the predicted utilization of storage capacity in the second time period, the size of the file system's storage capacity can be reduced by a first reduction amount. In response to (i) the predicted utilization of storage capacity in the second time period exceeding the predicted utilization of storage capacity in the first time period and (ii) the current utilization of storage capacity in the current time period exceeding the predicted utilization of storage capacity in the second time period, the size of the file system's storage capacity can also be reduced by a second reduction amount, wherein the second reduction amount is less than the first reduction amount.

[0107] For example, combining Figure 6A , Figure 6B pseudocode and Figure 9 The specific processing operations and other network functionalities described in the flowcharts are presented as illustrative examples only and should not be construed as limiting the scope of this disclosure in any way. Alternative embodiments may use other types of processing operations to automatically manage file system capacity using file system utilization prediction. For example, the order of processing steps may be changed in other embodiments, or certain steps may be performed simultaneously rather than sequentially. On one hand, the process may skip one or more actions. On the other hand, one or more actions may be performed simultaneously. On some other hand, additional actions may be performed.

[0108] Additionally, while one or more exemplary embodiments implement at least a portion of the disclosed prediction-based file system capacity management functionality in storage array 106-1, in one or more alternative embodiments, at least a portion of the disclosed prediction-based file system capacity management functionality may be implemented on one or more of the host devices 102 or another dedicated device. For example, as mentioned above, in at least some embodiments, the training, maintenance (e.g., updated training), and use of the LSTM model may be performed in the file system utilization prediction server 120 (thus reducing or avoiding any performance impairment to the storage array due to the disclosed prediction-based file system capacity management functionality).

[0109] The disclosed techniques for automated file system capacity management using file system utilization prediction improve storage efficiency by allocating reasonable target sizes for automated file system capacity reduction and expansion. Additionally, some implementations evaluate storage pool utilization to further improve file system space capacity management functionality. For example, jitter behavior between automated file system capacity reduction and expansion can be reduced by skipping unnecessary file system capacity adjustments. As mentioned above, LSTM models used in at least some implementations have been found to provide high-accuracy utilization predictions.

[0110] It has been found that the disclosed techniques for automatic file system capacity management using file system utilization prediction demonstrate a reduced gap between the allocated size of the file system and the target size of the updated file system. Furthermore, it largely avoids the jitter effect in future time periods (e.g., time period 5).

[0111] It should also be understood that the prediction-based file system capacity management techniques disclosed herein can be implemented at least in part as one or more software programs stored in memory and executed by a processor of a processing device such as a computer. As previously mentioned, the memory or other storage device embodying such program code is an example of what is more generally referred to herein as a “computer program product.”

[0112] The disclosed techniques for automated file system capacity management using file system utilization prediction can be implemented using one or more processing platforms. Therefore, one or more of the processing modules or other components can each run on a computer, storage device, or other processing platform element. Such elements can be considered as examples of what is more generally referred to herein as a "processing device".

[0113] As mentioned above, the illustrative embodiments disclosed herein offer several significant advantages over conventional arrangements. It should be understood that the specific advantages described above and elsewhere herein are associated with the specific illustrative embodiment and do not need to exist in other embodiments. Furthermore, the specific types of information processing system features and functions shown and illustrated herein are merely exemplary, and numerous other arrangements may be used in other embodiments.

[0114] In these and other implementations, computing services can be provided as PaaS services to cloud infrastructure tenants or other system users, but numerous alternative arrangements are also possible.

[0115] Some illustrative implementations of a processing platform that can be used to implement at least a portion of an information processing system include a cloud infrastructure comprising virtual machines implemented using a hypervisor running on a physical infrastructure. The cloud infrastructure also includes multiple sets of applications running on corresponding virtual machines within the virtual machines under the control of the hypervisor. Multiple hypervisors may also be used, each providing a set of virtual machines using at least one underlying physical machine. Different sets of virtual machines provided by one or more hypervisors can be used to configure multiple instances of various components of the system.

[0116] These and other types of cloud infrastructure can be used to provide content also referred to herein as a multi-tenant environment. One or more system components (such as a cloud-based predictive file system capacity management engine or parts thereof) are illustratively implemented for use by tenants in such a multi-tenant environment.

[0117] The cloud infrastructure disclosed herein may include cloud-based systems such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. In illustrative implementations, virtual machines provided in such systems may be used to implement at least portions of a cloud-based, predictive file system capacity management platform. Cloud-based systems may include object storage areas such as Amazon S3, GCP cloud storage, and Microsoft Azure Blob storage.

[0118] In some implementations, the cloud infrastructure additionally or alternatively includes multiple containers implemented using container host devices. For example, a given container in the cloud infrastructure illustratively includes Docker containers or other types of Linux containers. Containers can run on virtual machines in a multi-tenant environment, but other arrangements are also possible. Containers can be used to implement various different types of functionality within storage devices. For example, containers can be used to implement corresponding processing units that provide computing services for cloud-based systems. Furthermore, containers can be used in conjunction with other virtualization infrastructure, such as virtual machines implemented using hypervisors.

[0119] Now refer to Figure 10 and Figure 11 An illustrative implementation of the processing platform is described in more detail. In other implementations, these platforms may also be used to implement at least parts of other information processing systems.

[0120] Figure 10 An exemplary processing platform including cloud infrastructure 1000 is illustrated. Cloud infrastructure 1000 includes a combination of physical and virtual processing resources that can be used to implement at least a portion of information processing system 100. Cloud infrastructure 1000 includes multiple VM and / or container sets 1002-1, 1002-2, ..., 1002-L implemented using virtualization infrastructure 1004. Virtualization infrastructure 1004 runs on physical infrastructure 1005 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructure illustratively includes a kernel control group of a Linux operating system or other types of operating system.

[0121] The cloud infrastructure 1000 further includes a collection of applications 1010-1, 1010-2, ..., 1010-L running on corresponding collections of VMs / containers in VM / container collections 1002-1, 1002-2, ..., 1002-L, under the control of the virtualization infrastructure 1004. The VM / container collection 1002 may include a corresponding VM, a corresponding collection of one or more containers, or a corresponding collection of one or more containers running in a VM.

[0122] exist Figure 10 In some implementations of the scheme, the VM / container set 1002 includes respective VMs implemented using a virtualization infrastructure 1004 including at least one hypervisor. Such implementations can provide predictive file system capacity management functionality of the type described above for one or more processes running on a given VM. For example, each of the VMs may implement predictive file system capacity management control logic and associated file system utilization monitoring to provide predictive file system capacity management functionality for one or more processes running on that particular VM.

[0123] An example of a hypervisor platform that can be used to implement hypervisors in virtualization infrastructure 1004 is... It can have an associated virtual infrastructure management system, such as vCenter TM The underlying physical machine may include one or more distributed processing platforms, and the distributed processing platform may include one or more storage systems.

[0124] exist Figure 10 In other implementations of the scheme, the VM / container set 1002 includes corresponding containers implemented using virtualization infrastructure 1004 that provides operating system-level virtualization functionality, such as support for Docker containers running on a bare metal host or running on a VM. The containers are implemented illustratively using the corresponding kernel control group of the operating system. Such implementations can provide predictive file system capacity management functionality of the type described above for one or more processes running on different containers within a container. For example, a container host apparatus supporting multiple containers in one or more container sets can implement one or more instances of predictive file system capacity management control logic and associated file system utilization monitoring to provide predictive file system capacity management functionality.

[0125] As is evident from the above, one or more of the processing modules or other components of system 100 may each run on a computer, server, storage device, or other processing platform element. Such a given element may be considered as an example of what is more generally referred to herein as a "processing device". Figure 10 The cloud infrastructure 1000 shown can represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 11 The processing platform 1100 shown in the figure.

[0126] In this embodiment, processing platform 1100 includes at least a portion of a given system and includes a plurality of processing devices, denoted as 1102-1, 1102-2, 1102-3, ..., 1102-K, which communicate with each other via network 1104. Network 1104 may include any type of network, such as WAN, LAN, satellite network, telephone or wired network, cellular network, wireless network (such as WiFi or WiMAX), or portions or combinations of these and other types of networks.

[0127] Processing device 1102-1 in processing platform 1100 includes processor 1110 coupled to memory 1112. Processor 1110 may include a microprocessor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other types of processing circuitry and portions or combinations of such circuitry elements, and memory 1112, which may be considered as an example of a "processor-readable storage medium" storing executable program code of one or more software programs.

[0128] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. A given such article of manufacture may include, for example, a storage array, a storage disk, or an integrated circuit containing RAM, ROM, or other electronic memory, or any of a variety of other types of computer program products. As used herein, the term "article of manufacture" should be understood to exclude transient propagating signals. Numerous other types of computer program products including processor-readable storage media may be used.

[0129] The processing device 1102-1 also includes a network interface circuit 1114 for interfacing the processing device with the network 1104 and other system components, and may include a conventional transceiver.

[0130] The other processing devices 1102 of the processing platform 1100 are assumed to be configured in a manner similar to that shown for the processing device 1102-1 in the figure.

[0131] Moreover, the particular processing platform 1100 shown in the figure is presented only by way of example, and a given system may include additional or alternative processing platforms, as well as a number of different processing platforms in any combination, wherein each such platform includes one or more computers, storage devices or other processing devices.

[0132] Multiple components of an information processing system can Figure 10 or Figure 11 The types shown can be implemented together on a common processing platform, or each such component can be implemented on a separate processing platform.

[0133] For example, other processing platforms used to implement the illustrative implementation scheme may include different types of virtualization infrastructure as an alternative to or supplement to virtualization infrastructure that includes virtual machines. Such virtualization infrastructure illustratively includes container-based virtualization infrastructure configured to provide Docker containers or other types of LXC.

[0134] As another example, in some implementations, the various parts of a given processing platform may include a converged infrastructure.

[0135] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0136] Furthermore, many other arrangements of computers, servers, storage devices, or other components are possible in an information processing system. These components can communicate with other elements of the information processing system via any type of network or other communication medium.

[0137] As previously indicated, components of the information processing system disclosed herein can be implemented, at least in part, in the form of one or more software programs stored in memory and executed by a processor of a processing device. For example, illustratively, at least portions of the functionality shown in one or more of the figures are implemented in the form of software running on one or more processing devices.

[0138] It should be emphasized again that the above embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments can be used. For example, the disclosed technology is applicable to a wide variety of other types of information processing systems. Moreover, in other embodiments, the specific configurations of the system and apparatus elements illustratively shown in the drawings and the associated processing operations can be changed. Furthermore, the various assumptions made above in describing the illustrative embodiments should also be considered exemplary and not as requirements or limitations of this disclosure. Numerous other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. A method for automatic file system capacity management using file system utilization prediction, comprising: Obtain input data, which represents the utilization rate of the storage capacity of the file system of a given storage system; The future utilization rate of the storage capacity of the file system of the given storage system over one or more time periods is predicted based at least in part on at least a portion of the obtained input data. The storage capacity of the file system of the given storage system is automatically adjusted, at least in part, based on a comparison between the predicted utilization of the storage capacity and the current utilization of the storage capacity. Specifically, in response to the predicted utilization rate of the storage capacity in the second time period exceeding the predicted utilization rate of the storage capacity in the first time period, and the predicted utilization rate of the storage capacity in the first time period exceeding the current utilization rate of the storage capacity in the current time period, the size of the storage capacity of the file system is expanded by a first expansion amount; and During at least one of the one or more time periods, one or more objects are stored in the file system of the given storage system using the adjusted storage capacity; wherein the method is performed by at least one processing device, the at least one processing device including a processor coupled to the memory.

2. The method of claim 1, wherein the prediction is performed using a trained long short-term memory network.

3. The method of claim 2, wherein Bayesian optimization is performed to tune one or more hyperparameters of the long short-term memory network to obtain the trained long short-term memory network.

4. The method according to claim 1, wherein the one or more time periods correspond to a multi-step long-term window.

5. The method of claim 1, wherein the first time period and the second time period are after the current time period, and wherein the comparison determines the trend of the utilization rate of the storage capacity of the file system.

6. The method of claim 1, wherein the expansion of the storage capacity of the file system by the first expansion amount is not performed in response to: (a)(i) the difference between the currently allocated storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period is less than a first gap threshold; and (b)(i) the difference between the currently allocated storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period is greater than a second gap threshold.

7. The method of claim 1, wherein in response to (i) the predicted utilization of the storage capacity in the first time period exceeding the current utilization of the storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period exceeding the predicted utilization of the storage capacity in the second time period, the size of the storage capacity of the file system is expanded by a second expansion amount, wherein the second expansion amount is less than the first expansion amount.

8. The method of claim 5, wherein the size of the storage capacity of the file system is reduced by a first reduction amount in response to (i) the current utilization of the storage capacity in the current time period exceeding the predicted utilization of the storage capacity in the first time period and (ii) the predicted utilization of the storage capacity in the first time period exceeding the predicted utilization of the storage capacity in the second time period.

9. The method of claim 8, wherein in response to (i) the predicted utilization of the storage capacity in the second time period exceeding the predicted utilization of the storage capacity in the first time period and (ii) the current utilization of the storage capacity in the current time period exceeding the predicted utilization of the storage capacity in the second time period, the size of the storage capacity of the file system is reduced by a second reduction amount, wherein the second reduction amount is less than the first reduction amount.

10. A device for automatic file system capacity management using file system utilization prediction, comprising: At least one processing device, the at least one processing device including a processor coupled to a memory; The at least one processing device is configured to perform the following steps: Obtain input data, which represents the utilization rate of the storage capacity of the file system of a given storage system; The future utilization rate of the storage capacity of the file system of the given storage system over one or more time periods is predicted, based at least in part on at least a portion of the obtained input data; and The storage capacity of the file system of the given storage system is automatically adjusted, at least in part, based on a comparison between the predicted utilization of the storage capacity and the current utilization of the storage capacity. Specifically, in response to the predicted utilization rate of the storage capacity in a second time period exceeding the predicted utilization rate of the storage capacity in a first time period, and the predicted utilization rate of the storage capacity in the first time period exceeding the current utilization rate of the storage capacity in the current time period, the size of the storage capacity of the file system is expanded by a first expansion amount; and During at least one of the one or more time periods, one or more objects are stored in the file system of the given storage system using the adjusted storage capacity.

11. The device of claim 10, wherein the prediction is performed using a trained long short-term memory network, and wherein Bayesian optimization is performed to tune one or more hyperparameters of the long short-term memory network to obtain the trained long short-term memory network.

12. The device of claim 10, wherein the first time period and the second time period are after the current time period, and wherein the comparison determines a trend in the utilization rate of the storage capacity of the file system.

13. The device of claim 12, wherein in response to (i) the predicted utilization of the storage capacity in the first time period exceeding the current utilization of the storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period exceeding the predicted utilization of the storage capacity in the second time period, the size of the storage capacity of the file system is expanded by a second expansion amount, wherein the second expansion amount is less than the first expansion amount.

14. The device of claim 12, wherein the size of the storage capacity of the file system is reduced by a first reduction amount in response to (i) the current utilization of the storage capacity in the current time period exceeding the predicted utilization of the storage capacity in the first time period and (ii) the predicted utilization of the storage capacity in the first time period exceeding the predicted utilization of the storage capacity in the second time period, and wherein the size of the storage capacity of the file system is reduced by a second reduction amount in response to (i) the predicted utilization of the storage capacity in the second time period exceeding the predicted utilization of the storage capacity in the first time period and (ii) the current utilization of the storage capacity in the current time period exceeding the predicted utilization of the storage capacity in the second time period, wherein the second reduction amount is less than the first reduction amount.

15. A non-transitory processor-readable storage medium storing program code for one or more software programs, wherein the program code, when executed by at least one processing device, causes the at least one processing device to perform the following steps: Obtain input data, which represents the utilization rate of the storage capacity of the file system of a given storage system; The future utilization rate of the storage capacity of the file system of the given storage system over one or more time periods is predicted based at least in part on at least a portion of the obtained input data. as well as The storage capacity of the file system of the given storage system is automatically adjusted, at least in part, based on a comparison between the predicted utilization of the storage capacity and the current utilization of the storage capacity. Wherein, in response to the predicted utilization rate of the storage capacity in the second time period exceeding the predicted utilization rate of the storage capacity in the first time period and the predicted utilization rate of the storage capacity in the first time period exceeding the current utilization rate of the storage capacity in the current time period, the size of the storage capacity of the file system is expanded by a first expansion amount. as well as During at least one of the one or more time periods, one or more objects are stored in the file system of the given storage system using the adjusted storage capacity.

16. The non-transitory processor-readable storage medium of claim 15, wherein the prediction is performed using a trained long short-term memory network, and wherein Bayesian optimization is performed to tune one or more hyperparameters of the long short-term memory network to obtain the trained long short-term memory network.

17. The non-transitory processor-readable storage medium of claim 15, wherein the first time period and the second time period are after the current time period, and wherein the comparison determines the trend of the utilization rate of the storage capacity of the file system.

18. The non-transitory processor-readable storage medium of claim 17, wherein the size of the storage capacity of the file system is expanded by a second expansion amount in response to (i) the predicted utilization of the storage capacity in the first time period exceeding the current utilization of the storage capacity in the current time period and (ii) the predicted utilization of the storage capacity in the first time period exceeding the predicted utilization of the storage capacity in the second time period, wherein the second expansion amount is less than the first expansion amount.

19. The non-transitory processor-readable storage medium of claim 17, wherein the size of the storage capacity of the file system is reduced by a first reduction amount in response to (i) the current utilization of the storage capacity in the current time period exceeding the predicted utilization of the storage capacity in the first time period and (ii) the predicted utilization of the storage capacity in the first time period exceeding the predicted utilization of the storage capacity in the second time period, and wherein the size of the storage capacity of the file system is reduced by a second reduction amount in response to (i) the predicted utilization of the storage capacity in the second time period exceeding the predicted utilization of the storage capacity in the first time period and (ii) the current utilization of the storage capacity in the current time period exceeding the predicted utilization of the storage capacity in the second time period, wherein the second reduction amount is less than the first reduction amount.

Images