Reducing emergency capacity reservation for storage systems

Abstract

Aspects of the present disclosure relate to reducing or eliminating emergency capacity usage in a storage system by including in-process space recovery and / or one or more mitigation mechanisms. In addition to background space recovery, in-process space recovery can also be realized. One or more mitigation mechanisms may be associated with a basis for the write request. In-process space recovery and / or mitigation mechanisms allow for reduced emergency capacity reservation, thus saving storage costs.

Description

Background Art

[0001] Storage systems may include mechanisms for increasing storage space, such as space reclamation and / or thin provisioning, but these mechanisms create the risk of exhausting storage capacity if any of the mechanisms fail. To avoid this risk, storage systems may provision some storage capacity for emergency situations to keep the storage system running until the underlying problem is resolved or mitigated. Even though this emergency storage may be rarely used, it may still occupy a large portion of the total storage capacity, resulting in wasted storage space. Summary of the invention

[0002] Various aspects of the present disclosure relate to reducing or eliminating emergency capacity usage in a storage system by including in-process space reclamation and / or one or more mitigation mechanisms. In addition to background space reclamation, in-process space reclamation can also be implemented. One or more mitigation mechanisms can be associated with a write request basis. In-process space reclamation and / or mitigation mechanisms allow for reduction of emergency capacity reservations, thereby saving storage costs.

[0003] One aspect of the present disclosure provides a method for reducing the emergency storage capacity of a storage system, the method comprising: monitoring, by one or more processors, the amount of time since background space reclamation has been performed; determining, by one or more processors, that the amount of time meets or exceeds an idle threshold; and in response to determining that the amount of time meets or exceeds the idle threshold, triggering, by one or more processors, in-process space reclamation, wherein the in-process space reclamation is bound to one or more write processes of the storage system.

[0004] In an example, in-process space reclamation is bound to one or more writer processes by running on the same hardware and having the same software package as the one or more writer processes.

[0005] In another example, the method further includes: monitoring, by one or more processors, file density of the storage system; determining, by one or more processors, that the file density meets or falls below a density threshold; and in response to determining that the file density meets or falls below the density threshold, triggering, by one or more processors, in-process space reclamation. In yet another example, the method further includes: monitoring, by one or more processors, storage usage of the storage system; determining, by one or more processors, that the storage usage meets or exceeds a storage usage threshold; and in response to determining that the storage usage meets or exceeds the storage usage threshold, triggering, by one or more processors, in-process space reclamation.

[0006] In yet another example, the method further includes, in response to determining that storage usage meets or exceeds a storage usage threshold, adjusting, by one or more processors, a basis for adjusting the write process. In yet another example, the basis for adjusting the write process further includes routing incoming traffic or a portion of the incoming traffic to a hard disk drive (HDD) if the storage system is a solid state drive (SSD), or routing to the SSD if the storage system is a HDD. In yet another example, the basis for adjusting the write process further includes transcoding the data into a more efficient storage encoding. In yet another example, the basis for adjusting the write process further includes moving the data to at least one of a wider redundant array of independent disks (RAID), a remote storage device, or a storage device at a different geographical location. In yet another example, the basis for adjusting the write process further includes initiating compression of the incoming traffic or increasing the compression rate of the incoming traffic.

[0007] In yet another example, the storage usage threshold includes a plurality of threshold levels, and a different mitigation mechanism is performed at each of the plurality of threshold levels.

[0008] In yet another example, the method further includes, in response to determining that the amount of time meets or exceeds the idle threshold, triggering, by the one or more processors, space reclamation running on older software than the in-process space reclamation.

[0009] Another aspect of the present disclosure provides a system comprising: one or more processors; and one or more storage devices, wherein the one or more storage devices are coupled to the one or more processors and store instructions, which when executed by the one or more processors cause the one or more processors to perform operations for reducing emergency storage capacity, the operations comprising: monitoring the amount of time since background space reclamation has been performed; determining that the amount of time meets or exceeds an idle threshold; and in response to determining that the amount of time meets or exceeds the idle threshold, triggering in-process space reclamation, wherein the in-process space reclamation is bound to one or more write processes of the storage system.

[0010] In an example, in-process space reclamation is bound to one or more writer processes by running on the same hardware and having the same software package as the one or more writer processes.

[0011] In another example, the operations further include: monitoring file density of the storage system; determining that the file density meets or is below a density threshold; and in response to determining that the file density meets or is below the density threshold, triggering in-process space reclamation. In yet another example, the operations further include: monitoring storage usage of the storage system; determining that the storage usage meets or exceeds a storage usage threshold; and in response to determining that the storage usage meets or exceeds the storage usage threshold, triggering in-process space reclamation.

[0012] In yet another example, the operations further include, in response to determining that the storage usage meets or exceeds the storage usage threshold, adjusting a basis for the write process.

[0013] Yet another aspect of the present disclosure provides a non-temporary computer-readable medium for storing instructions, which, when executed by one or more processors, cause the one or more processors to perform operations for reducing emergency storage capacity, the operations comprising: monitoring the amount of time since background space reclamation has been performed; determining that the amount of time meets or exceeds an idle threshold; and in response to determining that the amount of time meets or exceeds the idle threshold, triggering in-process space reclamation, wherein the in-process space reclamation is bound to one or more write processes of the storage system.

[0014] In an example, the operations further include: monitoring file density of the storage system; determining that the file density meets or is below a density threshold; and in response to determining that the file density meets or is below the density threshold, triggering in-process space reclamation. In another example, the operations further include: monitoring storage usage of the storage system; determining that the storage usage meets or exceeds a storage usage threshold; and in response to determining that the storage usage meets or exceeds the storage usage threshold, triggering in-process space reclamation.

[0015] In yet another example, the operations further include, in response to determining that the storage usage meets or exceeds the storage usage threshold, adjusting a basis for the write process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A block diagram of an example storage system is depicted in accordance with aspects of the present disclosure.

[0017] Figure 2 Depicted is a block diagram of an example storage capacity management system for one or more storage systems in accordance with aspects of the present disclosure.

[0018] Figure 3 Depicted is a block diagram of an example environment for implementing a storage capacity management system in accordance with aspects of the present disclosure.

[0019] Figure 4 Depicted is a flow diagram of an example process for managing storage capacity in one or more storage systems in accordance with aspects of the present disclosure.

[0020] Figure 5 Depicted is a flow diagram of an example process for managing storage capacity in one or more storage systems using different thresholds in accordance with aspects of the present disclosure.

[0021] Figure 6 Depicted is a block diagram of an example cloud-based platform to implement a storage capacity management system in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0022] The present technology generally relates to fallback mechanisms that reduce or eliminate emergency capacity usage in storage systems. The fallback mechanism can include in-process space reclamation implemented in addition to background space reclamation. The fallback mechanism can also include monitoring storage usage and implementing a mitigation mechanism in response to storage usage meeting a threshold. In-process space reclamation and / or monitoring of storage usage allows for a reduction in emergency capacity reservation, thereby saving storage costs.

[0023] The storage system may include a plurality of storage devices and logical volumes associated with the storage devices. The storage device may include a standard buffer and an emergency buffer. The storage system may further include a processing component configured to receive a write request. In response to the write request, the processing component may map the capacity from the standard buffer to one or more of the logical volumes or map the capacity from the emergency buffer to one or more of the logical volumes when the capacity of the standard buffer is insufficient. The processing component may route the write request to the standard buffer or the emergency buffer based on the mapping.

[0024] In order to optimize or generally improve storage space utilization, the storage system can utilize space reclamation (also known as garbage collection) and / or thin provisioning. Space reclamation can correspond to reclaiming storage space occupied by data that is no longer useful (such as data occupied by deleted files or objects, or data occupied by blocks that are later overwritten). For example, a log structured storage system appends new data to the end of a log file and later reclaims the overwritten space by compaction. Space reclamation can be a background process separate from the process responsible for writing data and run by a separate computing resource pool. Thin provisioning can correspond to overuse of storage resources in a shared multi-tenant storage environment to optimize or improve storage usage. For example, the capacity of a logical volume can exceed the capacity of a storage device. Overuse can be based on a statistical model that uses historical data to predict actual storage usage.

[0025] Space reclamation and thin provisioning allow for the addition of storage space in a storage system, but there is a risk of running out of storage capacity if either mechanism fails, such as if space reclamation is interrupted or runs at a degraded capacity due to a software glitch or hardware problem or if thin provisioning miscalculates actual storage usage. To avoid this risk, a storage system may provision a certain amount of storage capacity for emergency situations (referred to as an emergency buffer) to keep the storage system running until the underlying problem is resolved or mitigated, such as fixing the space reclamation or acquiring new storage capacity. Even though the emergency buffer may be used very rarely, e.g., less than one percent of the percentage of time the storage system is in use, the emergency buffer may still occupy a significant portion of the total storage capacity, e.g., one-eighth to one-third of the total storage capacity.

[0026] In order to reduce or eliminate the emergency buffer capacity and thereby increase the storage capacity, the storage system may include a second space reclamation as a fallback for the space reclamation running in the background. The second space reclamation may run in a process and share a fate with or be bound to a component that performs writes to the storage system. By running in a process, the second space reclamation is run by one or more threads of a computer program, and the one or more threads share the same memory address space with the thread that performs writes to the storage system. By sharing a fate or being bound, the second space reclamation may run on the same hardware and / or have the same software package as the processing component that performs writes. Therefore, if the second space reclamation fails due to hardware or software problems, the processing component that performs writes also fails, and thus the risk of exhausting the capacity in the storage system is mitigated. In the case where the second space reclamation is associated with the processing component that performs writes, the emergency buffer size may be reduced, thereby bringing savings in total storage costs.

[0027] For example, the second space reclamation can be implemented in a block-level storage system that exposes a block device network interface for a virtual machine. In a block storage system, a logical block of data can be written and overwritten multiple times throughout the life of the logical block. The hypervisor of the block-level storage system can intercept read / write requests from the virtual machine and append each new write to a backend log file. Background space reclamation can periodically traverse and merge log files. The hypervisor can monitor the log files to monitor the timestamp of the last merge. The hypervisor can determine that the timestamp of the last merge is equal to or earlier than a threshold, such as by comparing the timestamp of the last merge with a configurable threshold greater than the frequency of the background merge run. In response to the determination, the hypervisor can trigger the second space reclamation to perform the merge in process using the same software that performs the write.

[0028] Additionally or alternatively, if the hypervisor determines that the file density (e.g., the percentage of blocks that are not overwritten) is equal to or below a threshold, such as by comparing the file density to a configurable threshold that is below the threshold for background space reclamation merges, the hypervisor can trigger a second space reclamation to perform a merge in the process. As another example, while the second space reclamation is performing a merge in the process, the hypervisor can transcode the data into a more efficient encoding, for example, from a replicated encoding to a Reed-Solomon encoding or switch to a Reed-Solomon encoding with a wider pattern to further reduce storage costs. As yet another example, the hypervisor can receive an external trigger to initiate a merge in the process. For example, the hypervisor can receive a trigger from other software monitoring log files, file density, and / or storage capacity, or the hypervisor can receive a manual trigger provided by an operator. As yet another example, the hypervisor can determine that the available resources of the storage system are insufficient, and in response, can trigger a second space reclamation to perform a merge in the process. As yet another example, a storage system may include a third space reclamation (whether in-process or background) that runs on previous versions of software to offset potential software failures when newer software is deployed.

[0029] To further reduce or eliminate emergency buffer capacity and thereby increase storage capacity, the storage system may include storage usage monitoring. The monitoring may determine that storage usage is equal to or above a threshold, such as by comparing the amount of storage to a configurable percentage threshold. In response to the determination, the monitoring may trigger one or more mitigation mechanisms to slow down storage usage in the storage system. The one or more mitigation mechanisms may slow down storage usage by adjusting the basis of write requests (such as storage placement, encoding, and / or compression) rather than adjusting the manner in which write requests are serviced (such as by throttling, delaying, prioritizing, and / or canceling write requests). By dynamically implementing mitigation mechanisms, the emergency buffer size may be reduced, resulting in savings in overall storage costs.

[0030] One or more mitigation mechanisms may include selectively routing traffic in whole or in part to an additional storage device, such as a second standard buffer on a second storage device of the storage system. For example, if the storage system is a solid-state drive (SSD), traffic may be routed to a hard disk drive (HDD), if the storage system is a HDD, traffic may be routed to an SSD, and / or if the storage system is an SSD or HDD, traffic may be routed to a combination of HDD and SSD. As another example, traffic may be routed to a wider range of redundant arrays of independent disks (RAID) storage devices, remote storage devices (such as public storage providers), and / or storage devices with spare storage capacity at different geographical locations. One or more mitigation mechanisms may also include transcoding data to a more efficient storage encoding, such as Reed-Sol omon encoding or a wider range of Reed-Sol omon modes (if the storage system implements replicated writes). For example, the second storage device may be a storage device encoded by Reed-Sol omon, a more byte-efficient storage device, or a storage device encoded by a wider range of Reed-Sol omon. One or more mitigation mechanisms may also include compressing incoming traffic or implementing more efficient compression with a higher compression ratio, such as compression based on different compression levels (e.g., Zstandard compression level). One or more mitigation mechanisms may also include triggering in-process space reclamation, as described above.

[0031] For example, referring back to the block level storage system described above, the hypervisor can monitor storage usage. For example, the hypervisor can monitor the storage usage percentage and / or the storage usage amount. The hypervisor can determine that the storage usage is equal to or above a threshold, such as by comparing the storage usage to a configurable threshold percentage or storage usage amount. In response to the determination, the hypervisor can trigger one or more of the mitigation mechanisms described above. The hypervisor can continue to monitor storage usage while the mitigation mechanisms are occurring. The hypervisor can determine that the storage usage is below a threshold and, in response, resume normal operation by deactivating one or more mitigation mechanisms.

[0032] The storage system may include multiple thresholds associated with different severity levels, where different mitigation mechanisms are implemented depending on the thresholds reached. For example, multiple thresholds may include thresholds at 80% storage capacity, 90% storage capacity, and 95% storage capacity. At 80% storage capacity, the hypervisor may trigger partial writing of data to the HDD. At 90% storage capacity, the hypervisor may trigger the use of compression. At 95% storage capacity, the hypervisor may trigger a 50% bandwidth limit. Multiple thresholds may also be associated with different classes of storage devices. For example, a higher performance storage device (e.g., SSD) may have a higher threshold than a lower performance storage device (e.g., HDD) for triggering different mitigation mechanisms.

[0033] Figure 1 A block diagram of an example storage system 10 is depicted. The storage system 10 may include a plurality of storage devices 12 and a processing component 14. The storage devices 12 may include one or more logical volumes 16, a standard buffer 18, and an emergency buffer 20. The processing component 14 may include one or more write processes or threads 22. The processing component 14 may be configured to receive one or more write requests 24, map capacity from the standard buffer 18 to the logical volume 16, and route the write request 24 to the standard buffer 18 based on the mapping. If the capacity of the standard buffer 18 is insufficient, the processing component 14 may map capacity from the emergency buffer 20 to the logical volume 16 and route the write request 24 to the emergency buffer 20 based on the mapping.

[0034] The storage system 10 may implement background space reclamation 26 and / or thin provisioning 28 to optimize storage space in the storage device 12. Space reclamation may correspond to reclaiming storage space occupied by data that is no longer useful, such as data occupied by deleted files or objects, or data occupied by blocks that were subsequently overwritten. Background space reclamation 26 may be separate from the processing component 14 and run by a separate computing resource pool. Thin provisioning 28 may correspond to overcommitting storage resources of the storage device 12. For example, the capacity of the logical volume 16 may exceed the capacity of the storage device 12. Overcommitment may be based on a statistical model that predicts actual storage usage based on usage history data.

[0035] To reduce or eliminate the emergency buffer 20, the storage system 10 may include one or more in-process space reclamation 30 as a fallback to the background space reclamation 26. The in-process space reclamation 30 may share a fate with the write process 22 or be bound to the write process, such as by running on the same hardware and / or having the same software package as the write process 22. To further reduce or eliminate the emergency buffer 20, the storage system 10 may include a storage capacity management system 32. The storage capacity management system 32 may determine that one or more metrics for monitoring storage usage of the storage device 12 meet one or more corresponding thresholds. In response, the storage capacity management system 32 may trigger one or more mitigation mechanisms to slow down storage usage in the storage system 10. The one or more mitigation mechanisms may slow down storage usage by adjusting the basis of the write request 24 (such as storage placement, encoding and / or compression) rather than adjusting the way the write request 24 is serviced (such as by limiting, delaying, prioritizing and / or canceling the write request 24).

[0036] Figure 2 A block diagram of an example storage capacity management system 100 for one or more storage systems, such as a cloud-based platform, is depicted. The example cloud-based platform may provide a set of cloud computing services. The storage capacity management system 100 may be used with, for example, Figure 1 The depicted storage capacity management system 32 corresponds. The cloud-based platform can provide services that allow the configuration or maintenance of computing resources and / or applications, such as data centers, cloud environments, and / or container frameworks. For example, the cloud-based platform can be used as a service to provide software applications, such as accounting, word processing, inventory tracking, etc. As another example, the infrastructure of the platform can be divided in the form of virtual machines or containers on which software applications run.

[0037] The storage capacity management system 100 may be configured to receive input data 102 for controlling the storage capacity of the storage system. For example, the storage capacity management system 100 may receive the input data 102 as part of a call to an application programming interface (API), thereby exposing the storage capacity management system 100 to one or more computing devices. The input data 102 may also be provided to the storage capacity management system 100 via a storage medium, such as a remote storage device connected to one or more computing devices via a network. The input data 102 may also be provided as input via a user interface on a user computing device coupled to the storage capacity management system 100.

[0038] The input data 102 may include data associated with whether to initiate one or more in-process space reclamation and / or one or more mitigation mechanisms. For example, the input data 102 may include timestamps for merging log files associated with read / write requests to software applications of a cloud-based platform. As another example, the input data 102 may include file density of files of a storage system, such as a percentage of unoverwritten blocks of the storage system. As yet another example, the input data 102 may include storage usage levels, such as a percentage of a storage system or storage usage. As yet another example, the input data 102 may include instructions for triggering in-process space reclamation and / or one or more mitigation mechanisms, such as from a server computing device or a user computing device.

[0039] The storage capacity management system 100 can be configured to output one or more results related to controlling the storage capacity of the storage system from the input data 102, and the one or more results are generated as output data 104. The output data 104 may include instructions associated with one or more in-process space reclamation and / or one or more mitigation mechanisms. For example, the output data 104 may include instructions for triggering in-process space reclamation. In-process space reclamation can be executed by one or more threads of a computer program, and the one or more threads share the same memory address space as a processing component that performs writes to the storage system. In-process space reclamation can also share fate with the processing component that performs writes to the storage system or be bound to the processing component, such as running on the same hardware and / or having the same software package. In-process space reclamation can also be run on a previous version of the software package.

[0040] As another example, the output data 104 may include instructions for triggering one or more mitigation mechanisms to slow down storage usage in the storage system by adjusting the basis of write requests. The one or more mitigation mechanisms may include selectively routing traffic to other storage devices, transcoding data to a more efficient storage encoding, and / or performing compression on incoming traffic.

[0041] The storage capacity management system 100 can be configured to provide output data 104 as a set of computer-readable instructions, such as one or more computer programs. The computer program can be written in any type of programming language and according to any programming paradigm (e.g., declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative). The computer program can be written to perform one or more different functions and operate within a computing environment, for example, on a physical device, on a virtual machine, or across multiple devices. The computer program can also implement the functionality described herein, for example, as performed by a system, engine, module, or model.

[0042] The storage capacity management system 100 may also be configured to forward the output data 104 to one or more other devices configured to translate the output data 104 into an executable program written in a computer programming language or machine readable code and optionally as part of a framework for controlling storage capacity. The storage capacity management system 100 may also be configured to send the output data 104 to a storage device for storage and later retrieval. The storage capacity management system 100 may also be configured to send the output data 104 for display, such as on a user display.

[0043] The storage capacity management system 100 may include a monitoring engine 106, a space reclamation engine 108, and a storage usage mitigation engine 110. The monitoring engine 106, the space reclamation engine 108, and / or the storage usage mitigation engine 110 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination thereof.

[0044] The monitoring engine 106 may be configured to monitor one or more metrics to determine whether to initiate one or more in-process space reclamation and / or one or more mitigation mechanisms. One or metrics may include the timestamp of the last merge for a read / write request, file density, storage usage level, and / or whether a trigger is received. For example, the monitoring engine 106 may monitor a log file attached to a write to monitor the timestamp of the last merge included in the log file. The monitoring engine 106 may determine that the timestamp of the last merge meets a threshold, such as by being equal to or earlier than a threshold. The monitoring engine 106 may determine that the timestamp of the last merge meets a threshold by comparing the timestamp of the last merge with a configurable threshold greater than the frequency at which background merges can run. As another example, the monitoring engine 106 may monitor file density in a storage system, such as a percentage of overwritten blocks. The monitoring engine 106 may determine that the file density meets a threshold, such as by being equal to or lower than a threshold. The monitoring engine 106 may determine that the file density meets a threshold by comparing the file density with a configurable threshold lower than a threshold for background space reclamation merges. As yet another example, the monitoring engine 106 may monitor storage usage, such as a percentage of a storage system or storage usage, or available storage in a storage system. The monitoring engine 106 may determine that the storage usage meets the threshold, such as by being equal to or above a threshold. The monitoring engine 106 may determine that the storage usage meets the threshold by comparing the amount of storage, percentage available, and / or available capacity to a configurable threshold. As another example, the monitoring engine 106 may monitor and determine whether a trigger for a space reclamation and / or mitigation mechanism in an execution process is received.

[0045] In response to one or more of the determinations from the monitoring engine 106, the space reclamation engine 108 may be configured to initiate one or more space reclamations, such as in addition to background space reclamations already being run. The one or more space reclamations may include in-process space reclamations and / or unupdated space reclamations. In-process space reclamations may be run by one or more threads of a computer program that share the same memory address space as a thread that performs writes to a storage system. In-process space reclamations may also share a fate with or be bound to a thread that performs writes to a storage system, such as by running on the same hardware and / or having the same software package as the thread that performs the writes. Unupdated space reclamations may be run on previous versions of software to offset potential software failures when newer software is deployed. Unupdated space reclamations may also be in-process space reclamations.

[0046] In response to one or more determinations from the monitoring engine 106, the storage use mitigation engine 110 can be configured to initiate one or more mitigation mechanisms to slow down storage use by adjusting the basis of write requests rather than adjusting the way write requests are served. Adjusting the basis of write requests can include adjusting storage placement, encoding and / or compression, while adjusting the way write requests are served can include limiting, delaying, prioritizing and / or canceling write requests. For example, adjusting storage placement can include the storage use mitigation engine 110 selectively routing traffic to one or more additional storage devices in whole or in part. The storage use mitigation engine 110 can route traffic to additional buffers, different types of storage devices (such as SSDs and / or HDDs), more extensive RAID storage devices, remote storage devices (such as public storage devices) and / or storage devices at different geographical locations. As another example, adjusting encoding can include the storage use mitigation engine 110 transcoding data to a more efficient storage encoding. The storage use mitigation engine 110 can transcode data to Reed-Sol omon encoding, or if Reed-Sol omon encoding is already occurring, transcode to a more extensive Reed-Sol omon encoding mode. As yet another example, adjusting compression may include implementing more efficient compression by the storage mitigation engine 110. The storage mitigation engine 110 may implement compression of incoming traffic or implement more efficient compression with a higher compression ratio, such as compression based on different compression levels (e.g., Zstandard compression levels).

[0047] The space reclamation engine 108 and / or the storage usage mitigation engine 110 may initiate one or more space reclamations and / or one or more mitigation mechanisms based on determined thresholds from the monitoring engine 106 and / or different levels of thresholds determined by the monitoring engine 106, respectively. For example, the space reclamation engine 108 may initiate in-process space reclamation in response to the monitoring engine 106 determining that the last merged timestamp satisfies a threshold, and the storage usage mitigation engine 110 may initiate adjusting storage placement in response to the monitoring engine 106 determining that storage usage satisfies a threshold. As another example, the storage usage mitigation engine 110 may initiate adjusting storage placement in response to the monitoring engine 106 determining that storage usage satisfies a first threshold, and may initiate adjusting compression in response to the monitoring engine 106 determining that storage usage satisfies a second threshold that is higher than the first threshold. Multiple thresholds may be associated with different classes or storage devices, such as SSDs and HDDs.

[0048] Figure 3 A block diagram of an example environment 200 for implementing a storage capacity management system 218 is depicted. The storage capacity management system 218 may be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 202. The client computing devices 204 and the server computing devices 202 may be communicatively coupled to one or more storage devices 206 via a network 208. The storage devices 206 may be a combination of volatile and non-volatile memory and may be located in the same or different physical locations as the computing devices 202, 204. For example, the storage devices 206 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, an optical storage device, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a writable memory, and a read-only memory.

[0049] The server computing device 202 may include one or more processors 210 and memory 212. The memory 212 may store information accessible by the processor 210, including instructions 214 that may be executed by the processor 210. The memory 212 may also include data 216 that may be retrieved, manipulated, or stored by the processor 210. The memory 212 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processor 210, such as volatile memory and non-volatile memory. The processor 210 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0050] Instructions 214 may include one or more instructions that, when executed by processor 210, cause the processor or processors to perform actions defined by instructions 214. Instructions 214 may be stored in an object code format for direct processing by processor 210, or in other formats including interpretable scripts or independent source code modules that are interpreted on demand or compiled in advance. Instructions 214 may include instructions for implementing functions that may be used with Figure 2 The storage capacity management system 218 may be executed using the processor 210 and / or using other processors located remotely from the server computing device 202.

[0051] Data 216 may be retrieved, stored, or modified by processor 210 in accordance with instructions 214. Data 216 may be stored in a computer register, in a relational or non-relational database as a table with a plurality of different fields and records, or as a JSON, YAML, proto, or XML document. Data 216 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, data 216 may include information sufficient to identify the relevant information, such as a number, descriptive text, proprietary code, pointer, reference to data stored in other storage (including other network locations), or information used by a function to calculate the relevant data.

[0052] The client computing device 204 may also be configured similarly to the server computing device 202, with one or more processors 220, memory 222, instructions 224, and data 226. The client computing device 204 may also include user input 228 and user output 230. The user input 228 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, mouse, mechanical actuator, soft actuator, touch screen, microphone, and sensor.

[0053] The server computing device 202 may be configured to transmit data to the client computing device 204, and the client computing device 204 may be configured to display at least a portion of the received data on a display implemented as part of the user output 230. The user output 230 may also be used to display an interface between the client computing device 204 and the server computing device 202. The user output 230 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces, or other tactile feedback that provides non-visual and non-auditory information to a platform user of the client computing device 204.

[0054] Although Figure 3Processors 210, 220 and memories 212, 222 are shown as being within computing devices 202, 204, but the components described herein may include multiple processors and memories that can operate at different physical locations rather than within the same computing device. For example, some of the instructions 214, 224 and data 216, 226 may be stored on a removable SD card, while other instructions and data may be stored within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically away from the processors 210, 220 but still accessible to the processors. Similarly, processors 210, 220 may include a collection of processors that can perform concurrent and / or sequential operations. Computing devices 202, 204 may each include one or more internal clocks that provide timing information, which may be used for time measurement of operations and programs run by computing devices 202, 204.

[0055] The server computing device 202 may be configured to receive a request for processing data from the client computing device 204. For example, the environment 200 may be part of a computing platform configured to provide a variety of services to users through various user interfaces and / or application programming interfaces (APIs) that expose platform services. The various services may include techniques for storage capacity management. The client computing device 204 or another server computing device may transmit input data associated with whether to initiate one or more space reclamation and / or one or more mitigation mechanisms to reduce storage usage. The server computing device 202 may receive the input data and, in response, generate output data, the output data including instructions associated with one or more space reclamation and / or one or more mitigation mechanisms.

[0056] Devices 202, 204 may be capable of communicating directly and indirectly over network 208. For example, using network sockets, a client computing device 204 may connect to a service operating in a data center over the Internet Protocol. Devices 202, 204 may set up listening sockets that may accept initiated connections for sending and receiving information. Network 208 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, and a private network using a proprietary communication protocol of one or more companies. Network 208 may support a variety of short-range and long-range connections. Short-range and long-range connections may be made over different bandwidths, such as those typically associated with a wireless network. 2.402 GHz to 2.480 GHz, usually associated with the 2.4 GHz and 5 GHz associated with communication protocols; or using multiple communication standards, such as for wireless broadband communication Additionally or alternatively, the network 208 may also support wired connections between the devices 202, 204, including through various types of Ethernet connections.

[0057] Although Figure 3 A single server computing device 202, client computing device 204, and storage medium 206 are shown, but it should be understood that various aspects of the present disclosure may be implemented according to a variety of different configurations and numbers of computing devices (including in examples of sequential or parallel processing), or through a distributed network of multiple devices.

[0058] Figure 4 Describes a method for managing one or more storage systems such as Figure 1 10). The example process 300 may be performed in a system of one or more processors at one or more locations, such as a storage system 10. Figure 2 The depicted storage capacity management system 100 is executed on the storage capacity management system 100).

[0059] As shown in block 310, the storage capacity management system 100 may monitor one or more metrics associated with storage capacity. The one or more metrics may include the amount of time since background space reclamation has been performed, the file density of one or more storage systems, the storage usage of one or more storage systems, and / or whether an external trigger has been received. The amount of time since background space reclamation has been performed may be based on the last merged timestamp included in the file log attached to the read / write request. The file density of one or more storage systems may be based on the percentage of unoverwritten blocks in one or more storage systems. The storage usage of one or more storage systems may be based on the percentage or amount of available storage capacity in one or more storage systems. The external trigger may be based on receiving an instruction from a client device or other computing device.

[0060] As shown in block 320, the storage capacity management system 100 may determine that one or more metrics meet or exceed one or more thresholds. The storage capacity management system 100 may determine that the amount of time since background space reclamation has been performed meets or exceeds an idle threshold, the file density of one or more storage systems meets or falls below a density threshold, the storage usage of one or more storage systems meets or exceeds a storage usage threshold, and / or an external trigger is received. The storage capacity management system 100 may compare the amount of time since background space reclamation has been performed to an idle threshold, compare the file density percentage to a density threshold, and / or compare the storage usage percentage or capacity to a storage usage threshold.

[0061] As shown in box 330, the storage capacity management system 100 can trigger one or more space reclamation and / or one or more storage usage mitigation mechanisms in response to determining that one or more of the metrics meet or exceed corresponding thresholds. One or more space reclamation can include in-process space reclamation, which is bound to one or more write processes of one or more storage systems, such as by running on the same hardware and / or having the same software package as the one or more write processes. One or more space reclamation can also include space reclamation that runs on older software than in-process space reclamation. Space reclamation that runs on older software can also be in-process space reclamation. One or more storage usage mitigation mechanisms can include adjusting the basis of the write process, such as by adjusting traffic routing, data transcoding and / or data compression.

[0062] For example, the one or more storage usage mitigation mechanisms may include routing incoming traffic or a portion of incoming traffic to a HDD if the one or more storage systems include an SSD, routing to an SSD if the one or more storage systems include an HDD, and / or routing to a combination of HDDs and SSDs if the one or more storage systems include both HDDs and SSDs. As another example, the one or more storage usage mitigation mechanisms may include routing incoming traffic to a wider RAID storage device, a storage device that is remote from the one or more storage systems, and / or a storage device that is in a different geographic location than the one or more storage systems.

[0063] As another example, one or more storage usage mitigation mechanisms may include transcoding data to a more efficient storage encoding, such as Reed-Sol omon encoding or a more extensive Reed-Sol omon encoding mode. As another example, one or more storage usage mitigation mechanisms may include compressing data or implementing a more efficient compression with a higher compression ratio, such as compression based on different compression levels (e.g., Zstandard compression levels).

[0064] Figure 5 A flow chart of an example process 400 for managing storage capacity in one or more storage systems using different thresholds is depicted. Depending on the threshold levels met or exceeded, additional and / or alternative space reclamation or mitigation mechanisms may be executed. The example process 400 may be performed in a system of one or more processors at one or more locations, such as a Figure 2 The depicted storage capacity management system 100 is executed on the storage capacity management system 100).

[0065] As shown in block 410, the storage capacity management system 100 may monitor one or more metrics associated with storage capacity. The one or more metrics may include the amount of time since background space reclamation has been performed, the file density of one or more storage systems, the storage usage of one or more storage systems, and / or whether an external trigger has been received. The amount of time since background space reclamation has been performed may be based on the last merged timestamp included in the file log attached to the read / write request. The file density of one or more storage systems may be based on the percentage of unoverwritten blocks in one or more storage systems. The storage usage of one or more storage systems may be based on the percentage or amount of available storage capacity in one or more storage systems. The external trigger may be based on receiving an instruction from a client device or other computing device.

[0066] As shown in block 420, the storage capacity management system 100 may determine that the one or more metrics meet or exceed a first threshold. The first threshold may be an idle threshold, a density threshold, or a storage usage threshold. The storage capacity management system 100 may compare the amount of time since the background space was executed to the idle threshold, compare the file density percentage to the density threshold, and / or compare the storage usage percentage or capacity to the storage usage threshold.

[0067] As shown in box 430, the storage capacity management system 100 can trigger one or more space reclamation and / or one or more storage usage mitigation mechanisms in response to determining that one or more of the metrics meet or exceed a first threshold. The one or more space reclamation may include in-process space reclamation, which is bound to one or more write processes of one or more storage systems. The one or more space reclamation may also include space reclamation that runs on older software than the in-process space reclamation. The one or more storage usage mitigation mechanisms may include adjusting the basis of the write process, such as by adjusting traffic routing, data transcoding and / or data compression.

[0068] As shown in block 440, the storage capacity management system 100 may determine that one or more metrics exceed a second threshold. The second threshold may be a threshold different from the first threshold. For example, if the first threshold is an idle threshold, the second threshold may be a storage usage threshold. Additionally or alternatively, the second threshold may be a higher level of the first threshold. For example, if the first threshold is a storage usage threshold at 80% storage capacity, the second threshold may also be the storage usage threshold but at 90% storage capacity.

[0069] As shown in block 450, the storage capacity management system 100 may trigger one or more additional space reclamation and / or one or more additional mitigation mechanisms in response to determining that one or more of the metrics meet or exceed a second threshold. For example, if meeting the first threshold triggers in-process space reclamation, triggering the second threshold may trigger space reclamation running on older software. As another example, if meeting the first threshold triggers adjusting traffic routing, triggering the second threshold may trigger adjusting data compression. As yet another example, if meeting the first threshold triggers in-process space reclamation, triggering the second threshold may trigger adjusting traffic routing.

[0070] Figure 6 Describes a method for implementing a storage capacity management system such as Figure 2 1 is a block diagram of an example cloud-based platform 500 of a storage capacity management system 100 of FIG. 1. The platform 500 may include a collection 502 of host machines or physical machines 504 (e.g., hardware resources) that support or execute a virtual computing environment 550. The virtual computing environment 550 may include a virtual machine manager or hypervisor 552 and a virtual machine layer 554 that runs one or more virtual machines 556 that are configured to execute instances 558 of one or more applications 560.

[0071] Each physical machine 504 may include one or more physical processors 506 (e.g., data processing hardware) and associated physical memory 508 (e.g., memory hardware). Although each physical machine 504 is shown as having a single physical processor 506, the physical machine 504 may include multiple physical processors 506. The physical machine 504 may also include physical memory 508, which may be divided into virtual memory by a host operating system (OS) 510 and assigned for use by a virtual machine 556, a hypervisor 552, or a host OS 510. The physical memory 508 may include random access memory (RAM) and / or a disk storage device, such as an SSD storage device. The set 502 of host or physical machines 504 may also include background space reclamation 570, which is configured to reclaim storage space occupied by data that is no longer useful (such as data occupied by deleted files or objects, or data occupied by blocks that are subsequently overwritten).

[0072] The host OS 510 may be executed on a given one of the physical machines 504 or may be configured to operate across multiple host machines 504. In addition, although the host OS 510 is shown as part of the virtual computing environment 550, each physical machine 504 may be equipped with its own OS 512. The OS 512 on each physical machine 504 may be managed as a common OS 510 for the hypervisor 552 and the virtual machine layer 554.

[0073] The hypervisor 552 may correspond to a computing engine that includes at least one of software, firmware, or hardware configured to create, instantiate / deploy, and execute virtual machines 556. The hypervisor 552 may include a storage capacity management system 566. The storage capacity management system 566 may be associated with Figure 2 The storage capacity management system 100 corresponds to the storage capacity management system 100. Figure 6 552, but in other implementations, the storage capacity management system 566 may be part of the VM layer 554. The hypervisor may also include one or more in-process space reclamation 568 as a fallback to background space reclamation 570.

[0074] Each virtual machine 556 may be referred to as a guest machine. The hypervisor 552 may be configured to provide each virtual machine 556 with a corresponding guest OS 562 having a virtual operating platform and manage the execution of the corresponding guest OS 562 on the virtual machine 556. In some examples, multiple virtual machines 556 having multiple guest OS 562 may share virtualized resources. For example, virtual machines with different operating systems may all run on a single physical host machine.

[0075] The host OS 510 can virtualize the underlying host hardware and manage the concurrent execution of the guest OS 562 on one or more virtual machines 556. For example, the host OS 510 can manage the virtual machines 556 to include emulated versions of the underlying host hardware or different computer architectures. The emulated version of the hardware associated with each virtual machine 556 can be referred to as virtual hardware 564.

[0076] The virtual hardware 564 may include one or more virtual processors, such as virtual central processing units (vCPUs), that emulate one or more physical processors 506 of the physical machine 504. The virtual processors may be interchangeably referred to as computing resources associated with the virtual machine 556. The computing resources may include a target computing resource level required to execute a corresponding individual service instance 558 of the application 560.

[0077] The virtual hardware 564 may also include a virtual memory that communicates with the virtual processor and stores guest instructions that can be executed by the virtual processor to perform operations. The virtual memory may be interchangeably referred to as a memory resource associated with the virtual machine 556. The memory resource may include a target memory resource level required to execute the corresponding individual service instance 558.

[0078] The virtual hardware 564 may also include at least one virtual storage device that provides runtime capacity for services on the host machine 504. The at least one virtual storage device may be referred to as a storage resource associated with the virtual machine 556. The storage resource may include a target storage resource level required to execute the corresponding individual service instance 558.

[0079] The virtual processor may execute instructions from the virtual memory that cause the virtual processor to execute a corresponding separate service instance 558 of the software application 560. The separate service instance 558 may be referred to as a guest instance that cannot determine whether it is executed by the virtual hardware 564 or the physical machine 504. The processor 506 of the host machine 504 may enable the virtual hardware 564 to efficiently execute the software instance 558 of the application 560 by allowing the guest software instructions to be executed directly on the processor 506 of the host machine 504 without code rewriting, recompiling, or instruction emulation.

[0080] The guest OS 562 executing on each virtual machine 556 can include software that controls the execution of the corresponding individual service instance 558 of the application 560 by the virtual machine 556. The guest OS executing on the virtual machine can be the same as or different from other guest OSs executing on other virtual machines. The guest OS 562 executing on each virtual machine 556 can also assign network boundaries, for example, assign network addresses through which the corresponding guest software can communicate with other processes reachable through the infrastructure (such as an internal network). The network boundary can be referred to as a network resource associated with the virtual machine 556.

[0081] Aspects of the present disclosure may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, and / or in computer hardware (such as structures disclosed herein, structural equivalents of said structures, or combinations thereof). Aspects of the present disclosure may also be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible, non-transitory computer storage medium for execution by one or more data processing devices or for controlling the operation of one or more data processing devices. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. The computer program instructions may be encoded on an artificially generated propagated signal (such as a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0082] The term "configured" is used herein in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by one or more data processing devices, cause the devices to perform the operation or action.

[0083] The term "engine" refers to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. An engine can be implemented as one or more software modules or components or can be installed on one or more computers in one or more locations. A particular engine can have its own dedicated computer or computers, or multiple engines can be installed and run on the same computer or computers.

[0084] Unless otherwise stated, the aforementioned alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the examples described herein, and clauses expressed as "such as", "including", etc., should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method for reducing the emergency storage capacity of a storage system, characterized in that: The method comprises: monitoring, by one or more processors, an amount of time since background space reclamation has been performed; determining, by the one or more processors, that the amount of time meets or exceeds an idle threshold; and In response to determining that the amount of time meets or exceeds the idle threshold, in-process space reclamation is triggered by the one or more processors, the in-process space reclamation being bound to one or more write processes of the storage system.

2. The method according to claim 1, characterized in that The in-process space reclamation is bound to the one or more writer processes by running on the same hardware and having the same software package as the one or more writer processes.

3. The method according to claim 1, characterized in that The method further comprises: monitoring, by the one or more processors, a file density of the storage system; determining, by the one or more processors, that the density of the file meets or falls below a density threshold; and In response to determining that the file density meets or falls below the density threshold, the in-process space reclamation is triggered by the one or more processors.

4. The method according to claim 1, characterized in that The method further comprises: monitoring, by the one or more processors, storage usage of the storage system; determining, by the one or more processors, that the storage usage meets or exceeds a storage usage threshold; and In response to determining that the storage usage meets or exceeds the storage usage threshold, the in-process space reclamation is triggered by the one or more processors.

5. The method according to claim 4, characterized in that Also included is adjusting, by the one or more processors, a basis for the write process in response to determining that the storage usage meets or exceeds the storage usage threshold.

6. The method according to claim 5, characterized in that The basis for adjusting the write process also includes routing incoming traffic or a portion of the incoming traffic to a hard disk drive (HDD) if the storage system is a solid state drive (SSD), or to an SSD if the storage system is a HDD.

7. The method according to claim 5, characterized in that Adjusting the basis for the write process also includes transcoding data into a more efficient storage encoding.

8. The method according to claim 5, characterized in that Adjusting the basis for the write process also includes moving data to at least one of a more extensive redundant array of independent disks (RAID), a remote storage device, or a storage device at a different geographical location.

9. The method according to claim 5, characterized in that Adjusting the basis for the write process also includes initiating compression of incoming traffic or increasing the compression rate of the incoming traffic.

10. The method according to claim 4, characterized in that The storage usage threshold includes a plurality of threshold levels, and a different mitigation mechanism is performed at each of the plurality of threshold levels.

11. The method according to claim 1, characterized in that Also included is triggering, by the one or more processors, space reclamation running on older software than the in-process space reclamation in response to determining that the amount of time meets or exceeds the idle threshold.

12. A system, characterized in that: include: one or more processors; as well as one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for reducing emergency storage capacity, the operations comprising: Monitors the amount of time since background space reclamation has been performed; determining that the amount of time meets or exceeds an idle threshold; and In response to determining that the amount of time meets or exceeds the idle threshold, in-process space reclamation is triggered, the in-process space reclamation being bound to one or more write processes of the storage system.

13. The system of claim 12, wherein: The in-process space reclamation is bound to the one or more writer processes by running on the same hardware and having the same software package as the one or more writer processes.

14. The system of claim 12, wherein: The operations also include: monitoring file density of the storage system; Determining that the file density meets or falls below a density threshold; and In response to determining that the file density meets or falls below the density threshold, the in-process space reclamation is triggered.

15. The system of claim 12, wherein: The operations also include: monitoring storage usage of the storage system; determining that the storage usage meets or exceeds a storage usage threshold; and In response to determining that the storage usage meets or exceeds the storage usage threshold, the in-process space reclamation is triggered.

16. The system of claim 15, wherein: The operations further include, in response to determining that the storage usage meets or exceeds the storage usage threshold, adjusting a basis for the write process.

17. A non-transitory computer-readable medium for storing instructions, characterized in that: The instructions, when executed by one or more processors, cause the one or more processors to perform operations for reducing emergency storage capacity, the operations comprising: Monitors the amount of time since background space reclamation has been performed; determining that the amount of time meets or exceeds an idle threshold; and In response to determining that the amount of time meets or exceeds the idle threshold, in-process space reclamation is triggered, the in-process space reclamation being bound to one or more write processes of the storage system.

18. The non-transitory computer readable medium of claim 17, wherein: The operations also include: monitoring file density of the storage system; Determining that the file density meets or falls below a density threshold; and In response to determining that the file density meets or falls below the density threshold, the in-process space reclamation is triggered.

19. The non-transitory computer readable medium of claim 17, wherein: The operations also include: monitoring storage usage of the storage system; determining that the storage usage meets or exceeds a storage usage threshold; and In response to determining that the storage usage meets or exceeds the storage usage threshold, the in-process space reclamation is triggered.

20. The non-transitory computer readable medium of claim 19, wherein: The operations further include, in response to determining that the storage usage meets or exceeds the storage usage threshold, adjusting a basis for the write process.

Images