Determination Based on Snapshot Deletion Pattern of Ransomware Attack on Data Maintained by Storage System

Abstract

An exemplary method includes a data protection system detecting one or more deletion requests for deleting one or more recovery data sets of a storage system, determining that the one or more deletion requests are inconsistent with a recovery data set deletion pattern associated with the storage system, and determining, based on determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern, that data stored by the storage system may be targeted by a security threat.

Description

Technical Field

[0001] Related Applications

[0001] This application claims priority to U.S. Patent Application No. 17 / 846,301, filed on June 22, 2022. The entire disclosure of that application is incorporated herein by reference in its entirety.

[0002] Brief Description of the Drawings

[0002] The accompanying drawings illustrate various embodiments and form a part of this specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals indicate the same or similar elements.

Brief Description of the Drawings

[0003]

Figure 1A

[0003] FIG. 1A shows a first exemplary system for data storage according to some implementations.

Figure 1B

[0004] FIG. 1B shows a second exemplary system for data storage according to some implementations.

Figure 1C

[0005] FIG. 1C shows a third exemplary system for data storage according to some implementations.

Figure 1D

[0006] FIG. 1D shows a fourth exemplary system for data storage according to some implementations.

Figure 2A

[0007] FIG. 2A is a perspective view of a storage cluster having a plurality of storage nodes and internal storage coupled to each storage node for providing network-connected storage according to some embodiments.

Figure 2B

[0008] FIG. 2B is a block diagram showing an interconnect switch that couples a plurality of storage nodes according to some embodiments.

Figure 2C

[0009] Figure 2C is a multi-level block diagram showing the content of a storage node and the content of one of the non-volatile solid state storage units, according to some embodiments.

Figure 2D

[0010] Figure 2D shows a storage server environment using embodiments of the storage node and storage unit of some of the previous figures, according to some embodiments.

[0011] Figure 2E is a hardware block diagram of a blade showing the control plane, the compute and storage planes, and the authority to interact with the underlying physical resources, according to some embodiments.

Figure 2E

[0012] Figure 2F shows an elastic software layer within a blade of a storage cluster, according to some embodiments.

Figure 2F

[0013] Figure 2G shows the authorities and storage resources within a blade of a storage cluster, according to some embodiments.

Figure 2G

[0014] Figure 3A illustrates a diagram of a storage system coupled for data communication with a cloud service provider, according to some embodiments of the present disclosure.

Figure 3A

[0015] Figure 3B illustrates a diagram of a storage system, according to some embodiments of the present disclosure.

Figure 3B

[0016] Figure 3C illustrates an example of a cloud-based storage system, according to some embodiments of the present disclosure.

Figure 3C

[0017] Figure 3D shows an exemplary computing device that can be specifically configured to perform one or more of the processes described herein.

Figure 3D

[0018] Figure 4 shows an exemplary monitoring system, according to some embodiments of the present disclosure.

Figure 4

[0019] FIG. 5 shows an exemplary configuration in which a storage system can receive requests according to some embodiments of the present disclosure.

Figure 5

[0020] FIG. 6 shows an exemplary configuration in which a cloud-based monitoring system is communicably coupled to a storage system via a network according to some embodiments of the present disclosure.

Figure 6

[0021] FIG. 7 shows an exemplary method according to some embodiments of the present disclosure.

Figure 7

[0021] FIG. 8 shows an exemplary method according to some embodiments of the present disclosure.

Figure 8

[0021] FIG. 9 shows an exemplary method according to some embodiments of the present disclosure.

Figure 9

[0021] FIG. 10 shows an exemplary method according to some embodiments of the present disclosure.

Figure 10

[0021] FIG. 11 shows an exemplary method according to some embodiments of the present disclosure.

Figure 11

[0021] FIG. 12 shows an exemplary method according to some embodiments of the present disclosure.

Figure 12

[0021] FIG. 13 shows an exemplary method according to some embodiments of the present disclosure.

Figure 13

[0021] FIG. 14 shows an exemplary method according to some embodiments of the present disclosure.

Figure 14

[0021] FIG. 15 shows an exemplary method according to some embodiments of the present disclosure.

Figure 15

[0021] FIG. 16 shows an exemplary method according to some embodiments of the present disclosure.

Figure 16

[0021] FIG. 17 shows an exemplary method according to some embodiments of the present disclosure.

Figure 17

[0021] FIG. 18 shows an exemplary method according to some embodiments of the present disclosure.

Figure 18

[0022] FIG. 19A shows an exemplary diagram of a series of time windows according to some embodiments of the present disclosure.

Figure 19A

Figure 19B

[0023] FIG. 19B shows an exemplary diagram of a time window according to some embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0004] Description of Embodiments

[0024] Referring to the accompanying drawings beginning with FIG. 1A, an exemplary method, apparatus, and product for reducing the operation related to the storage structure according to embodiments of the present disclosure are described. FIG. 1A shows an exemplary system for data storage according to some implementation forms. System 100 (also referred to herein as a "storage system") includes a number of elements for illustrative purposes only, without limitation. It may be noted that in other implementation forms, System 100 may include the same, more, or fewer elements configured in the same or different ways.

[0005]

[0025] System 100 includes a number of computing devices 164A - B. A computing device (also referred to herein as a "client device") may be embodied, for example, as a server, a workstation, a personal computer, a notebook, or the like within a data center. Computing devices 164A - B may be coupled to one or more storage arrays 102A - B through a storage area network ("SAN") 158 or a local area network ("LAN") 160 for data communication.

[0006]

[0026] The SAN 158 can be implemented using various data communication fabrics, devices, and protocols. For example, the fabric for the SAN 158 can include Fibre Channel, Ethernet, InfiniBand, Serial Attached Small Computer System Interface (「SAS」), or the like. Data communication protocols for use with the SAN 158 can include Advanced Technology Attachment (「ATA」), Fibre Channel Protocol, Small Computer System Interface (「SCSI」), Internet Small Computer System Interface (「iSCSI」), HyperSCSI, Non-Volatile Memory Express over Fabric (「NVMe」), or the like. It may be noted that the SAN 158 is provided by way of example and not limitation. Other data communication couplings can also be implemented between the computing devices 164A - B and the storage arrays 102A - B.

[0007]

[0027] LAN 160 can also be implemented using various fabrics, devices, and protocols. For example, the fabric for LAN 160 can include Ethernet (802.3), wireless (802.11), or the like. Data communication protocols for use in LAN 160 can include Transmission Control Protocol ("TCP"), User Datagram Protocol ("UDP"), Internet Protocol ("IP"), HyperText Transfer Protocol ("HTTP"), Wireless Access Protocol ("WAP"), Handheld Device Transport Protocol ("HDTP"), Session Initiation Protocol ("SIP"), Real Time Protocol ("RTP"), or the like.

[0008]

[0028] Storage arrays 102A - B may provide persistent data storage for computing devices 164A - B. Storage array 102A may, in some implementations, be housed within a chassis (not shown), and storage array 102B may be housed within a separate chassis (not shown). Storage arrays 102A and 102B may include one or more storage array controllers 110A - D (also referred to herein as "controllers"). Storage array controllers 110A - D may be embodied as modules of an automated computing machine that includes computer hardware, computer software, or a combination of computer hardware and software. In some implementations, storage array controllers 110A - D may be configured to perform various storage tasks. Storage tasks may include writing data received from computing devices 164A - B to storage arrays 102A - B, erasing data from storage arrays 102A - B, retrieving data from storage arrays 102A - B and providing the data to computing devices 164A - B, monitoring and reporting disk utilization and performance, performing redundancy operations such as redundant array of independent drives ("RAID") or RAID - like data redundancy operations, compressing data, encrypting data, and the like.

[0009]

[0029] Storage array controllers 110A - D can be implemented in various ways, including being implemented as a field programmable gate array ("FPGA" (Field Programmable Gate Array)), a programmable logic chip ("PLC" (Programmable Logic Chip)), an application specific integrated circuit ("ASIC" (Application Specific Integrated Circuit)), a system on chip ("SOC" (System-on-Chip)), or any computing device including individual components such as a processing device, a central processing unit, computer memory, or various adapters. Storage array controllers 110A - D can include, for example, a data communication adapter configured to support communication via SAN 158 or LAN 160. Depending on the implementation, storage array controllers 110A - D can be coupled independently to LAN 160. In various implementations, storage array controllers 110A - D can include an I / O controller or the like that couples the storage array controllers 110A - D for data communication, through a midplane (not shown), to persistent storage resources 170A - B (also referred to herein as "storage resources"). Persistent storage resources 170A - B mainly include any number of storage drives 171A - F (also referred to herein as "storage devices") and any number of non-volatile random access memory ("NVRAM" (non-volatile Random Access Memory)) devices (not shown).

[0010]

[0030] Depending on the implementation form, the NVRAM devices of the persistent storage resources 170A - B can be configured to receive data to be stored in the storage drives 171A - F from the storage array controllers 110A - D. In some examples, the data can be transmitted from the computing devices 164A - B. In some examples, writing data to the NVRAM device can be performed more quickly than writing the data directly to the storage drives 171A - F. In various implementations, the storage array controllers 110A - D can be configured to use the NVRAM device as a quickly accessible buffer for the data to be written to the storage drives 171A - F. The latency for write requests using the NVRAM device as a buffer can be improved compared to a system where the storage array controllers 110A - D write the data directly to the storage drives 171A - F. Depending on the implementation form, the NVRAM device can be implemented using computer memory in the form of high - bandwidth, low - latency RAM. Since the NVRAM device can receive or include a dedicated power source to maintain the state of the RAM after a main power loss to the NVRAM device, the NVRAM device is referred to as "non - volatile". Such a power source can be a battery, one or more capacitors, or the like. In response to a power loss, the NVRAM device can be configured to write the contents of the RAM to persistent storage such as the storage drives 171A - F.

[0011]

[0031] In various implementations, storage drives 171A - F can refer to any device configured to permanently record data. Here, "permanently" or "persistent" refers to the ability of a device to maintain the recorded data after a power loss. Depending on the implementation, storage drives 171A - F can support non - disk storage media. For example, storage drives 171A - F can be one or more solid - state drives ("SSD" (solid - state drive)), flash - memory - based storage, any type of solid - state non - volatile memory, or any other type of non - mechanical storage device. In other implementations, storage drives 171A - F can include mechanical or rotating hard disks such as hard - disk drives ("HDD" (hard - disk drive)).

[0012]

[0032] Depending on the implementation form, the storage array controllers 110A to 110D may be configured to offload the device management responsibility from the storage drives 171A to 171F within the storage arrays 102A to 102B. For example, the storage array controllers 110A to 110D may manage control information that can describe the state of one or more memory blocks within the storage drives 171A to 171F. The control information can indicate, for example, that a particular memory block is faulty and should no longer be written to, that a particular memory block contains boot code for the storage array controllers 110A to 110D, the number of program-erase ("P / E") cycles performed on a particular memory block, the elapsed time of the data stored in a particular memory block, the type of data stored in a particular memory block, and so on. Depending on the implementation form, the control information may be stored together with the associated memory blocks as metadata. In other implementation forms, the control information for the storage drives 171A to 171F may be stored within one or more particular memory blocks of the storage drives 171A to 171F selected by the storage array controllers 110A to 110D. The selected memory blocks may be tagged with an identifier that indicates that the selected memory blocks contain control information. The identifier may be used in association with the storage drives 171A to 171F by the storage array controllers 110A to 110D to quickly identify the memory blocks that contain the control information. For example, the storage controllers 110A to 110D may issue a command to identify the location of the memory blocks that contain the control information. It may be noted that since the control information can become very large, portions of the control information may be stored in multiple locations, or the control information may be stored in multiple locations for redundancy purposes, or the control information may otherwise be spread across multiple memory blocks within the storage drives 171A to 171F.

[0013]

[0033] In various implementations, storage array controllers 110A - D can offload device management responsibilities from storage drives 171A - F of storage arrays 102A - B by retrieving control information that describes the state of one or more memory blocks within storage drives 171A - F. Retrieving control information from storage drives 171A - F can be implemented, for example, by storage array controllers 110A - D querying storage drives 171A - F about the location of control information for a particular storage drive 171A - F. Storage drives 171A - F can be configured to execute instructions that enable the storage drives 171A - F to identify the location of the control information. The instructions can be executed by a controller (not shown) associated with or disposed on storage drives 171A - F, and can cause the storage drives 171A - F to scan portions of each memory block to identify a memory block that stores control information for the storage drive 171A - F. Storage drives 171A - F can respond by sending a response message to storage array controllers 110A - D that includes the location of control information for the storage drive 171A - F. In response to receiving the response message, storage array controllers 110A - D can issue a request to read data stored at an address associated with the location of control information for storage drives 171A - F.

[0014]

[0034] In other implementation forms, in response to receiving control information, the storage array controllers 110A to D can further offload device management responsibilities from the storage drives 171A to F by performing storage drive management operations. The storage drive management operations can include, for example, operations typically performed by the storage drives 171A to F (e.g., a controller (not shown) associated with a specific storage drive among the storage drives 171A to F). The storage drive management operations can include, for example, ensuring that data is not written into a failed memory block within the storage drives 171A to F, ensuring that data is written into the memory blocks within the storage drives 171A to F in such a way that appropriate wear leveling is achieved, and the like.

[0015]

[0035] In various implementations, the storage arrays 102A - B may implement two or more storage array controllers 110A - D. For example, storage array 102A may include storage array controller 110A and storage array controller 110B. In a given case, a single storage array controller 110A - D of the storage system 100 (e.g., storage array controller 110A) may be designated with a primary status (also referred to herein as the "primary controller"), and the other storage array controllers 110A - D (e.g., storage array controller 110A) may be designated with a secondary status (also referred to herein as the "secondary controller"). The primary controller may have special rights, such as permission to change data within the persistent storage resources 170A - B (e.g., write data to the persistent storage resources 170A - B). At least some of the rights of the primary controller may take precedence over the rights of the secondary controller. For example, when the primary controller has the right, the secondary controller may not have permission to change data within the persistent storage resources 170A - B. The status of the storage array controllers 110A - D may change. For example, storage array controller 110A may be designated with a secondary status, and storage array controller 110B may be designated with a primary status.

[0016]

[0036] Depending on the implementation form, a primary controller such as the storage array controller 110A can serve as a primary controller for one or more storage arrays 102A - B, and a second controller such as the storage array controller 110B can serve as a secondary controller for one or more storage arrays 102A - B. For example, the storage array controller 110A can be a primary controller for the storage array 102A and the storage array 102B, and the storage array controller 110B can be a secondary controller for the storage arrays 102A and 102B. Depending on the implementation form, the storage array controllers 110C and 110D (also referred to as "memory processing modules") may not have either primary or secondary status. The storage array controllers 110C and 110D implemented as memory processing modules can serve as a communication interface between the primary and secondary controllers (e.g., the storage array controllers 110A and 110B respectively) and the storage array 102B. For example, the storage array controller 110A of the storage array 102A can send a write request to the storage array 102B via the SAN 158. The write request can be received by both storage array controllers 110C and 110D of the storage array 102B. The storage array controllers 110C and 110D can facilitate communication, for example, send the write request to the appropriate storage drives 171A - F. It can be noted that depending on the implementation form, the memory processing module can be used to increase the number of storage drives controlled by the primary and secondary controllers.

[0017]

[0037] In various implementations, storage array controllers 110A - D are communicatively coupled via a midplane (not shown) to one or more storage drives 171A - F and to one or more non - volatile random - access memory (NVRAM) devices (not shown) included as part of storage arrays 102A - B. Storage array controllers 110A - D may be coupled to the midplane via one or more data communication links, and the midplane may be coupled to storage drives 171A - F and NVRAM devices via one or more data communication links. The data communication links described herein are collectively represented by data communication links 108A - D and may include, for example, a Peripheral Component Interconnect Express (「PCIe」) bus.

[0018]

[0038] FIG. 1B shows an exemplary system for data storage according to some implementations. The storage array controller 101 shown in FIG. 1B may be similar to the storage array controllers 110A - D described with reference to FIG. 1A. In one example, the storage array controller 101 may be similar to storage array controller 110A or storage array controller 110B. The storage array controller 101 includes a number of elements for purposes of illustration and not limitation. It may be noted that in other implementations, the storage array controller 101 may include the same, more, or fewer elements configured in the same or different ways. It may be noted that elements of FIG. 1A may be included hereinafter to help illustrate the features of the storage array controller 101.

[0019]

[0039] Storage array controller 101 may include one or more processing devices 104 and random access memory ("RAM") 111. Processing device 104 (or controller 101) represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More specifically, processing device 104 (or controller 101) may be a complex instruction set computing ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 104 (or controller 101) may also be one or more dedicated processing devices such as an ASIC, an FPGA, a digital signal processor ("DSP"), a network processor, or the like.

[0020]

[0040] Processing device 104 may be connected to RAM 111 via a data communication link 106, which may be embodied as a high-speed memory bus such as a double data rate 4 ("DDR4") bus. An operating system 112 is stored in RAM 111. In some implementations, instructions 113 are stored in RAM 111. Instructions 113 may include computer program instructions for performing operations within a direct mapped flash storage system. In one embodiment, a direct mapped flash storage system directly addresses data blocks in a flash drive without address translation being performed by a flash drive storage controller.

[0021]

[0041] In various embodiments, the storage array controller 101 includes one or more host bus adapters 103A - C coupled to the processing device 104 via data communication links 105A - C. In various embodiments, the host bus adapters 103A - C can be computer hardware that connects a host system (e.g., a storage array controller) to other networks and storage arrays. By way of example, the host bus adapters 103A - C can be a fiber channel adapter that enables the storage array controller 101 to connect to a SAN, an Ethernet adapter that enables the storage array controller 101 to connect to a LAN, or the like. The host bus adapters 103A - C can be coupled to the processing device 104 via data communication links 105A - C, such as a PCIe bus, for example.

[0022]

[0042] In various embodiments, the storage array controller 101 can include a host bus adapter 114 coupled to an expander 115. The expander 115 can be used to attach a host system to a greater number of storage drives. The expander 115 can be, for example, a SAS expander that is utilized to enable the host bus adapter 114 to attach to storage drives in an implementation where the host bus adapter 114 is embodied as a SAS controller.

[0023]

[0043] In various embodiments, the storage array controller 101 can include a switch 116 coupled to the processing device 104 via a data communication link 109. The switch 116 can be a computer hardware device that can create multiple endpoints from a single endpoint, thereby enabling multiple devices to share a single endpoint. The switch 116 can be, for example, a PCIe switch coupled to a PCIe bus (e.g., the data communication link 109) that provides multiple PCIe connection points to a midplane.

[0024]

[0044] In various implementations, the storage array controller 101 includes a data communication link 107 for coupling the storage array controller 101 to other storage array controllers. In some examples, the data communication link 107 can be a QuickPath Interconnect (QPI) interconnect.

[0025]

[0045] Traditional storage systems using traditional flash drives can perform processes that span across flash drives that are part of the traditional storage system. For example, higher-level processes of the storage system can initiate and control processes that span across flash drives. However, the flash drives of traditional storage systems can include their own storage controllers that perform processes in a similar manner. Therefore, for traditional storage systems, both higher-level processes (e.g., initiated by the storage system) and lower-level processes (e.g., initiated by the storage controllers of the storage system) can be performed.

[0026]

[0046] To address various deficiencies of traditional storage systems, operations can be performed by higher-level processes and not by lower-level processes. For example, a flash storage system can include flash drives that do not include storage controllers that provide processes. Therefore, the operating system of the flash storage system itself can initiate and control processes. This can be achieved by a direct map flash storage system that directly addresses data blocks in the flash drive without address translation being performed by a flash drive storage controller.

[0027]

[0047] The operating system of a flash storage system may identify and maintain a list of allocation units that span multiple flash drives of the flash storage system. The allocation unit may be an entire erase block or multiple erase blocks. The operating system may maintain a map or address range that directly maps addresses to erase blocks of the flash drives of the flash storage system.

[0028]

[0048] The direct mapping to the erase blocks of the flash drive may be used to rewrite data and erase data. For example, the operation may be performed on one or more allocation units that include first data that is to be retained and second data that is no longer used by the flash storage system. The operating system may start a process of writing the first data to a new location within another allocation unit, erasing the second data, and marking the allocation unit as available for use for subsequent data. Therefore, the process may be performed only by the higher-level operating system of the flash storage system without additional lower-level processes being performed by the controller of the flash drive.

[0029]

[0049] The advantage of the process being performed only by the operating system of the flash storage system includes that the reliability of the flash drives of the flash storage system is increased because unnecessary or duplicate write operations are not performed during the process. One possible novel aspect here is the concept of starting and controlling the process in the operating system of the flash storage system. Additionally, the process may be controlled by the operating system across multiple flash drives. This is in contrast to the process being performed by the storage controller of the flash drive.

[0030]

[0050] A storage system can consist of two storage array controllers that share a set of drives for failover purposes, or it can consist of a single storage array controller that provides a storage service using multiple drives, or it can consist of a distributed network of storage array controllers each having some number of drives or some amount of flash storage, and the storage array controllers within the network cooperate to provide a complete storage service and cooperate in various aspects of the storage service including storage allocation and garbage collection.

[0031]

[0051] FIG. 1C shows a third exemplary system 117 for data storage according to some implementations. System 117 (also referred to herein as a “storage system”) includes a number of elements for purposes of illustration and not limitation. It may be noted that in other implementations, system 117 may include the same, more, or fewer elements configured in the same or different ways.

[0032]

[0052] In one embodiment, system 117 includes a dual peripheral component interconnect ("PCI") flash storage device 118 having separately addressable high-speed write storage. System 117 may include a storage controller 119. In one embodiment, storage controllers 119A - D may be a CPU, an ASIC, an FPGA, or any other circuitry that may implement the necessary control structures according to the present disclosure. In one embodiment, system 117 includes flash memory devices (e.g., including flash memory devices 120a - n) operably coupled to various channels of storage device controller 119. Flash memory devices 120a - n may be presented to controllers 119A - D as an addressable set of flash pages, erase blocks, and / or control elements sufficient to allow storage device controllers 119A - D to program and retrieve various aspects of the flash. In one embodiment, storage device controllers 119A - D may perform operations on flash memory devices 120a - n including storing and retrieving page data content, placing and erasing any block, tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells, tracking and predicting error codes and failures within the flash memory, and controlling voltage levels associated with programming and retrieving the content of flash cells.

[0033]

[0053] In one embodiment, system 117 may include RAM 121 for storing separately addressable high-speed write data. In one embodiment, RAM 121 may be one or more separate individual devices. In another embodiment, RAM 121 may be integrated within storage device controllers 119A - D or multiple storage device controllers. RAM 121 may also be utilized for other purposes, such as a temporary program memory for a processing device (e.g., a CPU) within storage device controller 119.

[0034]

[0054] In one embodiment, system 117 may include an energy storage device 122, such as a rechargeable battery or capacitor. The energy storage device 122 may store sufficient energy to power the storage device controller 119, some amount of RAM (e.g., RAM 121), and some amount of flash memory (e.g., flash memories 120a - 120n) for a time sufficient to write the contents of the RAM to the flash memory. In one embodiment, when the storage device controller detects a loss of external power, the storage device controllers 119A - D may write the contents of the RAM to the flash memory.

[0035]

[0055] In one embodiment, system 117 includes two data communication links 123a, 123b. In one embodiment, the data communication links 123a, 123b may be PCI interfaces. In another embodiment, the data communication links 123a, 123b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). The data communication links 123a, 123b may be based on the Non - Volatile Memory Express ( "NVMe") or NVMe over Fabrics ( "NVMf") specification that enables external connections from other components within the storage system 117 to the storage device controllers 119A - D. Note that for convenience, the data communication links may be referred to interchangeably as a PCI bus in this specification.

[0036]

[0056] System 117 may also include an external power source (not shown) that may be provided through one or both of the data communication links 123a, 123b or may be provided separately. An alternative embodiment includes a dedicated separate flash memory (not shown) for storing the contents of the RAM 121. The storage device controllers 119A - D may present individual portions of the logical address space of the storage device 118 that may be presented as a logical device through a PCI bus including addressable high - speed write logic devices, or as PCI memory, or as persistent storage. In one embodiment, operations for storage within the device are directed into the RAM 121. In the event of a power failure, the storage device controllers 119A - D may write the stored contents associated with the addressable high - speed write logic storage to a flash memory (e.g., flash memories 120a - n) for long - term persistent storage.

[0037]

[0057] In one embodiment, the logical device may include some or all of the presentation of the contents of the flash memory devices 120a - n, and that presentation enables the storage system (e.g., storage system 117) including the storage device 118 to directly address flash memory pages and directly reprogram erase blocks from storage system components external to the storage device through the PCI bus. The presentation may also include some or all of tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells across all flash memory devices, tracking and predicting error codes and failures within or across flash memory devices, and controlling the voltage levels associated with programming and retrieving the contents of flash cells, etc., such that one or more of the external components can also control and retrieve other aspects of the flash memory.

[0038]

[0058] In one embodiment, the energy storage device 122 may be sufficient to ensure the completion of ongoing operations for the flash memory devices 120a - 120n. The energy storage device 122 may power the storage device controllers 119A - D and associated flash memory devices (e.g., 120a - n) for those operations and for the storage of high - speed write RAM to the flash memory. The energy storage device 122 may be used to store accumulated statistics and other parameters held and tracked by the flash memory devices 120a - n and / or the storage device controller 119. A separate capacitor or energy storage device (such as a smaller capacitor near or incorporated within the flash memory device itself) may be used for some or all of the operations described herein.

[0039]

[0059] Various schemes may be used to track and optimize the life of the energy storage components, such as adjusting the voltage level over time, partially discharging the energy storage device 122, and measuring the corresponding discharge characteristics. If the available energy decreases over time, the effective available capacity of the addressable high - speed write storage may be reduced to ensure that the addressable high - speed write storage can be safely written based on the currently available stored energy.

[0040]

[0060] FIG. 1D shows a third exemplary system 124 for data storage according to some implementations. In one embodiment, the system 124 includes storage controllers 125a, 125b. In one embodiment, the storage controllers 125a, 125b are operatively coupled to dual - PCI storage devices 119a, 119b and 119c, 119d, respectively. The storage controllers 125a, 125b may be operatively coupled to a number of host computers 127a - n (e.g., via a storage network 130).

[0041]

[0061] In one embodiment, two storage controllers (e.g., 125a and 125b) provide storage services such as a SCSI block storage array, a file server, an object server, a database, or a data analysis service. The storage controllers 125a, 125b can provide services to external host computers 127a - n of the storage system 124 through a number of network interfaces (e.g., 126a - d). The storage controllers 125a, 125b can provide integrated services or applications that are entirely within the storage system 124, forming a centralized storage and computing system. The storage controllers 125a, 125b utilize high - speed write memory within or spanning the storage devices 119a - d to store ongoing operations to ensure that operations are not lost in the event of a power failure, removal of a storage controller, shutdown of a storage controller or storage system, or some failure of one or more software or hardware components within the storage system 124.

[0042]

[0062] In one embodiment, the controllers 125a, 125b operate as a PCI master to one or the other of the PCI buses 128a, 128b. In another embodiment, 128a and 128b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). Embodiments of other storage systems may operate the storage controllers 125a, 125b as multi-masters for both of the PCI buses 128a, 128b. Alternatively, a PCI / NVMe / NVMf switching infrastructure or fabric may connect multiple storage controllers. Embodiments of some storage systems may enable storage devices to communicate directly with each other rather than only with the storage controller. In one embodiment, the storage device controller 119a may be operable to synthesize and transfer data to be stored in the flash memory device from data stored in RAM (e.g., RAM 121 of FIG. 1C) under the instruction of the storage controller 125a. For example, to ensure improving data security or to free up addressable high-write capacity for reuse, after the storage controller determines that the operation has been completed across the storage system, or when the high-write memory on the device reaches a certain used capacity, or after a certain amount of time, a recomputed version of the contents of the RAM may be transferred. This mechanism may be used, for example, to avoid a second transfer through the buses (e.g., 128a, 128b) from the storage controllers 125a, 125b. In one embodiment, recomputation may include compressing data, adding indexing or other metadata, combining multiple data segments together, performing erasure code calculations, etc.

[0043]

[0063] In one embodiment, under the instructions from storage controllers 125a and 125b, storage device controllers 119a and 119b can be operable to calculate data from data stored in a RAM (e.g., RAM 121 in FIG. 1C) without the involvement of storage controllers 125a and 125b and transfer the data to other storage devices. This operation can be used to mirror data stored in one controller 125a to another controller 125b, or alternatively, it can be used to offload the calculations of compression, data aggregation, and / or erasure coding and transfer to storage devices in order to reduce the load on the storage controllers or the storage controller interfaces 129a and 129b to the PCI buses 128a and 128b.

[0044]

[0064] Storage device controllers 119A - D may include mechanisms for implementing high - availability primitives for use by other parts of the storage system external to the dual - PCI storage device 118. For example, in a storage system having two storage controllers that provide a high - availability storage service, reservation or exclusive primitives may be provided such that one storage controller can prevent the other storage controller from accessing or continuing to access the storage device. This may be used, for example, when one controller detects that the other controller is not functioning properly, or when there is a possibility that the interconnect between the two storage controllers itself is not functioning properly.

[0045]

[0065] In one embodiment, a storage system for use with a dual PCI direct map mode storage device having separately addressable high-speed write storage manages erase blocks, or groups of erase blocks, as allocation units for storing data in place of a storage service, or for storing metadata (e.g., indexes, logs, etc.) associated with the storage service, or for proper management of the storage system itself. When data arrives, or when the storage system is to hold data for a long time interval (e.g., exceeding a specified time threshold), flash pages that can be several kilobytes in size can be written. To commit data more quickly or to reduce the number of writes to the flash memory device, the storage controller can first write the data into separately addressable high-speed write storage on one or more storage devices.

[0046]

[0066] In one embodiment, storage controllers 125a, 125b can start using erase blocks within and across storage devices (e.g., 118) according to the elapsed time and predicted remaining life of the storage devices, or based on other statistics. Storage controllers 125a, 125b can start garbage collection and data transfer between storage devices according to pages that are no longer needed, and for managing flash page and erase block lifetimes, and for managing overall system performance.

[0047]

[0067] In one embodiment, the storage system 124 may utilize mirroring and / or an erasure coding scheme as part of storing data in addressable high-speed write storage and / or as part of writing data within an allocation unit associated with an erase block. The erasure code may be used across storage devices, within an erase block or allocation unit, or within or across flash memory devices on a single storage device to provide redundancy against failures of single or multiple storage devices, or to prevent internal corruption of flash memory pages resulting from the operation of flash memory or degradation of flash memory cells. Separately or in combination, mirroring and erasure coding at various levels may be used to recover from multiple types of failures that occur separately or in combination.

[0048]

[0068] The embodiments shown with reference to FIGS. 2A - G illustrate a storage cluster that stores user data, such as user data originating from one or more users or client systems, or other sources external to the storage cluster. The storage cluster distributes user data across storage nodes housed within a chassis or across multiple chassis, using metadata erasure coding and redundant copies. Erasure coding refers to a method of data protection or reconstruction where data is stored across a set of different locations, such as disks, storage nodes, or geographical locations. Flash memory is a type of solid - state memory that can be integrated with the embodiments. However, the embodiments can be extended to other storage media, including other types of solid - state memory or non - solid - state memory. Control of storage locations and workloads is distributed across storage locations within a clustered peer - to - peer system. Tasks such as mediating communication between various storage nodes, detecting when a storage node becomes unavailable, and balancing I / O (input and output) across various storage nodes are all handled in a distributed manner. Data is, in some embodiments, spread or distributed across multiple storage nodes in the form of data fragments or stripes that support data recovery. Ownership of data can be re - assigned within the cluster independent of input and output patterns. This architecture, described in more detail below, allows a system to remain operational while storage nodes within the cluster fail because data can be reconstructed from other storage nodes and thus remain available for input and output operations. In various embodiments, the storage nodes can be referred to as cluster nodes, blades, or servers.

[0049]

[0069] A storage cluster can be included within a chassis, i.e., an enclosure that houses one or more storage nodes. A mechanism for providing power to each storage node, such as a power distribution bus, and a communication mechanism, such as a communication bus that enables communication between the storage nodes, are included within the chassis. According to some embodiments, the storage cluster can operate as an independent system within one location. In one embodiment, the chassis includes at least two instances of both a power distribution bus and a communication bus that can be independently enabled or disabled. The internal communication bus can be an Ethernet bus, although other technologies such as PCIe, InfiniBand, and others are equally suitable. The chassis provides ports for an external communication bus to enable communication between multiple chassis, either directly or through a switch, as well as with a client system. External communication can use technologies such as Ethernet, InfiniBand, Fibre Channel, etc. Depending on the embodiment, the external communication bus uses different communication bus technologies for inter-chassis and client communication. When switches are placed within or between the chassis, the switches can serve the role of conversion between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, the storage cluster can be accessed by a client using either a proprietary interface or a standard interface such as a Network File System (“NFS”), Common Internet File System (“CIFS”), Small Computer System Interface (“SCSI”), or Hypertext Transfer Protocol (“HTTP”). Conversion from the client protocol can be performed at the switch, on the chassis external communication bus, or within each storage node. Depending on the embodiment, multiple chassis can be coupled or connected to each other through an aggregator switch. Some and / or all of the coupled or connected chassis can be designated as a storage cluster.As described above, each chassis can have a plurality of blades, and each blade has a media access control ("MAC") address, but the storage cluster, depending on the embodiment, is presented to the external network as having a single cluster IP address and a single MAC address.

[0050]

[0070] Each storage node can be one or more storage servers, and each storage server is connected to one or more non-volatile solid state memory units, which can be referred to as storage units or storage devices. One embodiment includes a single storage server within each storage node and one to eight non-volatile solid state memory units, but this example is not intended to be limiting. The storage server can include a processor, DRAM, and interfaces for internal communication buses and power distribution for each power bus. Inside the storage node, the interface and the storage unit may share a communication bus, such as PCI Express, depending on the embodiment. The non-volatile solid state memory unit can directly access the internal communication bus interface through the storage node communication bus or can request the storage node to access the bus interface. The non-volatile solid state memory unit includes, depending on the embodiment, an embedded CPU, a solid state storage controller, and a large amount of solid state large scale storage, such as, for example, 2 to 32 terabytes ("TB"). Embedded volatile memory media, such as DRAM, and energy storage devices are included within the non-volatile solid state memory unit. Depending on the embodiment, the energy storage device is a capacitor, a supercapacitor, or a battery that enables the transfer of a partial subset of the content of the DRAM to a stable storage medium in the event of a power loss. Depending on the embodiment, the non-volatile solid state memory unit is constructed using storage class memory, such as phase change or magnetoresistive random access memory ("MRAM"), which replaces the DRAM and enables a reduced power retention device.

[0051]

[0071] One of the many features of storage nodes and non-volatile solid state storage is the ability to proactively reconstruct data within a storage cluster. The storage node and non-volatile solid state storage can determine when a storage node or non-volatile solid state storage within the storage cluster is unreachable, regardless of whether there is an attempt to read data that involves that storage node or non-volatile solid state storage. Next, the storage node and non-volatile solid state storage cooperate to recover and reconstruct the data, at least partially, into a new location. The system provides proactive reconstruction because it reconstructs the data without waiting for the data to be needed for a read access initiated from a client system using the storage cluster. These and further details of the memory and its operation are described below.

[0052]

[0072] FIG. 2A is a perspective view of a storage cluster 161 having a plurality of storage nodes 150 and internal solid state memories coupled to each storage node for providing network connected storage or a storage area network in accordance with some embodiments. Network connected storage, a storage area network, or a storage cluster, or other storage memory can include one or more storage clusters 161 each having one or more storage nodes 150 in a flexible and reconfigurable arrangement of both physical components and the amount of storage memory provided thereby. The storage cluster 161 is designed to fit snugly within a rack, and one or more racks can be set up and implemented as desired for storage memory. The storage cluster 161 has a chassis 138 having a plurality of slots 142. It should be understood that the chassis 138 can be referred to as a housing, enclosure, or rack unit. In one embodiment, the chassis 138 has 14 slots 142, although other numbers of slots can be readily devised. For example, some embodiments have 4 slots, 8 slots, 16 slots, 32 slots, or other suitable numbers of slots. Each slot 142 can accommodate one storage node 150 depending on the embodiment. The chassis 138 includes flaps 148 that can be used to mount the chassis 138 onto a rack. Fans 144 provide air circulation for cooling the storage nodes 150 and their components. However, other cooling components can be used, or embodiments without cooling components can be devised. A switch fabric 146 couples the storage nodes 150 within the chassis 138 to each other and to the network for communication to the memory. In one embodiment shown herein, the slots 142 on the left side of the switch fabric 146 and the fans 144 are shown as occupied by the storage nodes 150, while the slots 142 on the right side of the switch fabric 146 are empty and available for insertion of storage nodes 150 for illustrative purposes.This configuration is an example, and one or more storage nodes 150 could occupy the slots 142 in various additional arrangements. The arrangement of the storage nodes need not be continuous or adjacent in some embodiments. The storage nodes 150 are hot-pluggable. That is, the storage nodes 150 can be inserted into or removed from the slots 142 within the chassis 138 without stopping the system or turning off its power. When a storage node 150 is inserted into or removed from a slot 142, the system automatically reconfigures to recognize and adapt to the change. The reconfiguration may include, in some embodiments, restoring redundancy and / or rebalancing the data or load again.

[0053]

[0073] Each storage node 150 can have a plurality of components. In the embodiment shown here, the storage node 150 includes a printed circuit board 159 on which a CPU 156, i.e., a processor, a memory 154 coupled to the CPU 156, and a non-volatile solid-state storage 152 coupled to the CPU 156 are implemented. However, in further embodiments, other mounts and / or components may also be used. The memory 154 has instructions executed by the CPU 156 and / or data operated on by the CPU 156. As further described below, the non-volatile solid-state storage 152 includes flash or, in further embodiments, other types of solid-state memory.

[0054]

[0074] Referring to FIG. 2A, the storage cluster 161 is scalable. That is, as described above, storage capacities with non-uniform memory sizes can be easily added. One or more storage nodes 150 can be plugged into or removed from each chassis, and in some embodiments, the storage cluster self-configures. The plug-in storage nodes 150 can have different sizes regardless of whether they are installed in the chassis at the time of shipment or added later. For example, in one embodiment, the storage node 150 can have any multiple of 4TB, such as 8TB, 12TB, 16TB, 32TB, etc. In further embodiments, the storage node 150 could have any multiple of other storage amounts or capacities. The storage capacity of each storage node 150 is communicated in parallel and affects the decision of how to stripe the data. For maximum storage efficiency, one embodiment can self-configure as extensively as possible within a stripe, subject to the requirement of continued operation in the event of the loss of up to one or up to two non-volatile solid state storage units 152 or storage nodes 150 within the chassis.

[0055]

[0075] Figure 2B is a block diagram showing a communication interconnect 173 and a power distribution bus 172 that couple a plurality of storage nodes 150. Referring again to Figure 2A, the communication interconnect 173 may, in some embodiments, be included within or implemented using a switch fabric 146. If a plurality of storage clusters 161 occupy a rack, the communication interconnect 173 may, in some embodiments, be included within or implemented using an upper portion of a rack switch. As shown in Figure 2B, the storage cluster 161 is housed within a single chassis 138. External ports 176 are coupled to the storage nodes 150 through the communication interconnect 173, while external ports 174 are directly coupled to the storage nodes. An external power port 178 is coupled to the power distribution bus 172. The storage nodes 150 may include non-volatile solid state storage 152 in various amounts and different capacities, as described with reference to Figure 2A. Additionally, one or more of the storage nodes 150 may be compute-only storage nodes, as shown in Figure 2B. Permissions 168 are implemented as, for example, a list or data structure stored in memory on the non-volatile solid state storage 152. In some embodiments, the permissions are stored within the non-volatile solid state storage 152 and supported by software running on a controller or other processor of the non-volatile solid state storage 152. In further embodiments, the permissions 168 are implemented as, for example, a list or other data structure stored in memory 154 on the storage node 150 and supported by software running on the CPU 156 of the storage node 150. The permissions 168 control, in some embodiments, how and where data is stored within the non-volatile solid state storage 152. This control helps determine which type of erasure coding scheme is applied to the data and which storage nodes 150 have which portions of the data. Each permission 168 may be allocated to a non-volatile solid state storage 152.In various embodiments, each permission may control a range of inode numbers, segment numbers, or other data identifiers assigned to data by a file system, by storage node 150, or by non-volatile solid state storage 152.

[0056]

[0076] In some embodiments, all data pieces and all metadata pieces have redundancy within the system. Additionally, all data pieces and all metadata pieces have an owner, which may be referred to as a permission. There is a succession plan for how to discover the data or its metadata if the permission becomes inaccessible, e.g., through the failure of a storage node. In various embodiments, there are redundant copies of the permission 168. The permission 168 has a relationship to the storage node 150 and the non-volatile solid state storage 152 in some embodiments. Each permission 168 is responsible for a range of data segment numbers, or other identifiers of data, and may be assigned to a particular non-volatile solid state storage 152. In some embodiments, the permissions 168 for all such ranges are distributed across the non-volatile solid state storage 152 of the storage cluster. Each storage node 150 has a network port that provides access to the non-volatile solid state storage 152 of that storage node 150. Data can be stored within segments associated with segment numbers, which are, in some embodiments, indirect references for the configuration of a RAID (redundant array of independent disks) stripe. Therefore, the assignment and use of the permission 168 establish an indirect reference to the data. The indirect reference can, according to some embodiments, in this case, be referred to as the ability to indirectly reference the data through the permission 168. A segment identifies a set of non-volatile solid state storage 152 and a local identifier within the set of non-volatile solid state storage 152 that can contain data. In some embodiments, the local identifier is an offset into the device and can be continuously reused by multiple segments. In other embodiments, the local identifier is unique for a particular segment and is never reused. The offset within the non-volatile solid state storage 152 is applied to identify the location of the data for writing to or reading from the non-volatile solid state storage 152 (in the form of a RAID stripe).The data either includes non-volatile solid-state storage 152 having the authority 168 for a specific data segment or is striped across a plurality of units of non-volatile solid-state storage 152, which may be different therefrom.

[0057]

[0077] For example, during data movement or data reorganization, if there is a change in where a particular segment of data is located, the authority 168 for that data segment must be referenced in the non-volatile solid state storage 152 or storage node 150 that has that authority 168. To identify the location of a particular piece of data, embodiments calculate a hash value for the data segment or apply an i-node number or data segment number. The output of this operation indicates the non-volatile solid state storage 152 that has the authority 168 for that particular piece of data. Depending on the embodiment, there are two stages to this operation. The first stage maps an entity identifier (ID (identifier)), such as a segment number, i-node number, or directory number, to an authority identifier. This mapping may include calculations such as a hash or bitmask. The second stage is to map the authority identifier to a particular non-volatile solid state storage 152, which can be done through an explicit mapping. The operation is reproducible, such that when the calculation is performed, the result of the calculation reproducibly and surely indicates the particular non-volatile solid state storage 152 that has that authority 168. The operation may include as input a set of reachable storage nodes. If the set of reachable non-volatile solid state storage units changes, the optimal set changes. Depending on the embodiment, the persistent value is the current allocation (which is always true), and the calculated value is the target allocation that the cluster will attempt to reconfigure to. This calculation can be used to determine the optimal non-volatile solid state storage 152 for an authority when there is a set of non-volatile solid state storage 152 that make up the same cluster that are reachable. The calculation also determines an ordered set of equivalent non-volatile solid state storage 152s that will record the authority in the non-volatile solid state storage mapping such that the authority can be determined even if the allocated non-volatile solid state storage is unreachable. Depending on the embodiment, if a particular authority 168 is unavailable, a replicated or alternative authority 168 may be referenced.

[0058]

[0078] Referring to FIGS. 2A and 2B, two of the many tasks of the CPU 156 on the storage node 150 are to split the write data and reconstruct the read data. When the system determines that data is to be written, the location of the authority 168 for that data is identified as described above. When the segment ID for the data has already been determined, the write request is transferred to the non-volatile solid state storage 152 that is currently determined to be the host of the authority 168 determined from the segment. Next, the host CPU 156 of the storage node 150 where the non-volatile solid state storage 152 and the corresponding authority 168 exist splits or shards the data and sends the data to various non-volatile solid state storages 152. The transmitted data is written as a data stripe according to an erasure coding scheme. Depending on the embodiment, the data is requested to be pulled, and in other embodiments, the data is pushed. Conversely, when data is read, the location of the authority 168 for the segment ID containing the data is identified as described above. Next, the host CPU 156 of the storage node 150 where the non-volatile solid state storage 152 and the corresponding authority 168 exist requests the data from the non-volatile solid state storage indicated by the authority and the corresponding storage node. Depending on the embodiment, the data is read from the flash storage as a data stripe. Next, the host CPU 156 of the storage node 150 reconstructs the read data while correcting any errors (if any) according to an appropriate erasure coding scheme and transfers the reconstructed data to the network. In further embodiments, some or all of these tasks may be handled within the non-volatile solid state storage 152. Depending on the embodiment, the segment host requests pages from the storage and then requests that the data be sent to the storage node 150 by sending the data to the storage node that made the original request.

[0059]

[0079] In some systems, such as a UNIX-style file system for example, data is handled together with an index node or i-node that specifies a data structure representing an object in the file system. The object can be, for example, a file or a directory. Among the attributes, metadata such as permission data and creation timestamps can be associated with the object. Segment numbers may be assigned to all or part of such objects in the file system. In other systems, data segments are handled with segment numbers assigned elsewhere. For purposes of explanation, a unit of distribution is an entity, and the entity can be a file, a directory, or a segment. That is, an entity is a unit of data or metadata stored by a storage system. Entities are grouped into a set called permissions. Each permission has a permission owner, which is a storage node having an exclusive right to update the entities within the permission. In other words, a storage node encompasses a permission, and the permission, in turn, encompasses an entity.

[0060]

[0080] A segment, according to some embodiments, is a logical container of data. A segment is an address space between media address spaces, and the physical flash location, i.e., the data segment number, is within this address space. A segment may also include metadata that enables the restoration of data redundancy (rewriting to different flash locations or devices) without the involvement of higher-level software. In one embodiment, the internal form of a segment includes client data and a media mapping for determining the location of that data. Each data segment is protected, where applicable, from memory and other failures by dividing the segment into a number of data and parity shards, e.g., by striping the data and parity shards across non-volatile solid state storage 152 coupled to host CPU 156 (see FIGS. 2E and 2G), i.e., according to an erasure coding scheme. The term "segment" is used, in some embodiments, to refer to a container and its location within the address space of the segment. The term "stripe" is used, according to some embodiments, to refer to the same set of shards as a segment and includes how the shards are distributed with redundancy or parity information.

[0061]

[0081] A series of address translations occur across the storage system. At the top level, there is a directory entry (file name) that links to an i-node. The i-node points to within a media address space where the data is logically stored. The media address can be mapped through a series of indirect media to spread the load of large files or to perform data services such as deduplication or snapshots. The media address can be mapped through a series of indirect media to spread the load of large files or to perform data services such as deduplication or snapshots. Next, the segment address is translated to a physical flash location. The physical flash location has an address range bounded by the amount of flash in the system in some embodiments. The media address and segment address are logical containers and in some embodiments use identifiers of 128 bits or more to be effectively infinite, and the potential for reuse is calculated to be longer than the expected lifetime of the system. Addresses from the logical containers are assigned in a hierarchical manner in some embodiments. First, each non-volatile solid state storage unit 152 can be allocated a range of address space. Within this allocated range, the non-volatile solid state storage 152 can assign addresses without synchronizing with other non-volatile solid state storage 152.

[0062]

[0082] Data and metadata are stored by a set of underlying storage layouts optimized for various workload patterns and storage devices. These layouts incorporate multiple redundancy schemes, compression formats, and indexing algorithms. Some of these layouts store information regarding permissions and permission masters, while others store file metadata and file data. Redundancy schemes include error correction codes that tolerate broken bits within a single storage device (such as a NAND flash chip), erasure codes that tolerate failures of multiple storage nodes, and replication schemes that tolerate data center or regional failures. In some embodiments, low density parity check (LDPC) codes are used within a single storage unit. In some embodiments, Reed-Solomon encoding is used within a storage cluster, and mirroring is used within a storage grid. Metadata may be stored using an ordered log-structured index (such as a log-structured merge tree), and large data may not need to be stored within a log-structured layout.

[0063]

[0083] To maintain consistency across multiple copies of an entity, storage nodes implicitly agree through computation on two things: (1) the authority that encompasses the entity, and (2) the storage nodes that encompass the authority. The assignment of an entity to an authority can be done by pseudo-randomly assigning the entity to the authority, by dividing the entity into ranges based on an externally generated key, or by placing a single entity within each authority. Examples of pseudo-random schemes are the replication under scalable hashing (RUSH) group, which includes linear hashing and controlled replication under scalable hashing (CRUSH). In some embodiments, since the set of nodes can change, pseudo-random assignment is used only to assign authorities to nodes. The set of authorities cannot change, and thus, in these embodiments, any subjective function can be applied. Some placement schemes automatically place authorities on storage nodes, while others rely on an explicit mapping of authorities to storage nodes. In some embodiments, a pseudo-random scheme is used to map from each authority to a set of candidate authority owners. The pseudo-random data distribution function associated with CRUSH can assign authorities to storage nodes and create a list of where the authorities are assigned. Each storage node has a copy of the pseudo-random data distribution function and can reach the same computation to disperse and later discover or identify the location of the authority. Each of the pseudo-random schemes requires, in some embodiments, a set of reachable storage nodes as input to conclude on the same target node. Once an entity is placed within an authority, the entity can be stored on a physical device such that an assumed failure does not result in unexpected data loss.In some embodiments, the rebalancing algorithm attempts to store copies of all entities within the authority on the same set of machines and with the same layout.

[0064]

[0084] Examples of assumed failures include device failures, theft of machines, fires in data centers, and regional disasters such as nuclear or geological events. Different failures result in different levels of acceptable data loss. In some embodiments, the theft of a storage node does not affect the security or reliability of the system, while depending on the system configuration, a regional event could result in no data loss, loss of updates for seconds or minutes, or even complete data loss.

[0065]

[0085] In embodiments, the placement of data for storage redundancy is independent of the placement of authorities for data consistency. In some embodiments, storage nodes that contain authorities do not contain any persistent storage at all. Instead, the storage nodes are connected to non-volatile solid-state storage units that do not contain authorities. The communication interconnect between the storage nodes and the non-volatile solid-state storage units consists of multiple communication technologies and has heterogeneous performance and fault tolerance characteristics. In some embodiments, as described above, the non-volatile solid-state storage units are connected to the storage nodes via PCI Express, the storage nodes are interconnected within a single chassis using an Ethernet backplane, the chassis are interconnected to form a storage cluster. The storage cluster is connected to clients using Ethernet or Fibre Channel in some embodiments. When multiple storage clusters are configured into a storage grid, the multiple storage clusters are connected using other long-distance networking links such as the Internet, or "metro scale" links or private links that do not cross the Internet.

[0066]

[0086] The authority owner has the exclusive right to move an entity from one non-volatile solid-state storage unit to another non-volatile solid-state storage unit, to change the entity, and to add and remove copies of the entity. This makes it possible to maintain the redundancy of the underlying data. When the authority owner fails, is scheduled to be shut down, or is in an overloaded state, the authority is transferred to a new storage node. In the case of a temporary failure, it becomes important to ensure that all healthy machines agree on the new location of the authority. Ambiguities arising from temporary failures can be resolved automatically by a consensus protocol such as Paxos, a hot-warm failover scheme, through manual intervention by a remote system administrator, or by a local hardware administrator (e.g., physically removing the failed machine from the cluster or pressing a button on the failed machine). In some embodiments, a consensus protocol is used and the failover is automatic. If too many failures or replication events occur within too short a period, the system, according to some embodiments, enters a self-preservation mode and stops replication and data movement activities until an administrator intervenes.

[0067]

[0087] Permissions are transferred between storage nodes, and as the owners of the permissions update entities in those permissions, the system transfers messages between the storage nodes and the non-volatile solid state storage units. For persistent messages, messages with different purposes are of different types. Depending on the type of message, the system maintains different orderings and persistence guarantees. As persistent messages are processed, the messages are temporarily stored within multiple durable and non-durable storage hardware technologies. In some embodiments, the messages are stored in RAM, within NVRAM, and on NAND flash devices, and various protocols are used to efficiently utilize each storage medium. Latency-sensitive client requests can be held in replicated NVRAM and then in NAND, while background rebalancing operations are held directly in NAND.

[0068]

[0088] Persistent messages are permanently stored before being transmitted. This enables the system to continue to respond to client requests despite failures and component replacements. Many hardware components include unique identifiers visible to the system administrator, manufacturer, hardware supply chain, and continuous monitoring quality management infrastructure, but applications running on infrastructure addresses virtualize the addresses. These virtualized addresses do not change over the lifetime of the storage system regardless of component failures and replacements. This enables each component of the storage system to be replaced over time without involving reconfiguration or interruption of client request processing. That is, the system supports non-stop upgrades.

[0069]

[0089] Depending on the embodiment, the virtual address is stored with sufficient redundancy. The always-on monitoring system correlates the hardware and software status with the hardware identifier. This enables the detection and prediction of failures due to faulty components and manufacturing details. The monitoring system also, depending on the embodiment, enables proactive transfer of rights and entities from affected devices before a failure occurs by removing components from the critical path.

[0070]

[0090] Figure 2C is a multi-level block diagram showing the contents of storage node 150 and the contents of non-volatile solid state storage 152 of storage node 150. Data is communicated to and from storage node 150 by a network interface controller (“NIC” (network interface controller)) 202 in some embodiments. Each storage node 150 has a CPU 156 and one or more non-volatile solid state storages 152 as described above. Moving down one level in FIG. 2C, each non-volatile solid state storage 152 has relatively fast non-volatile solid state memory such as non-volatile random access memory (“NVRAM” (nonvolatile random access memory)) 204 and flash memory 206. In some embodiments, NVRAM 204 can be a component (DRAM, MRAM, PCM) that does not require program / erase cycles and can be a memory that supports being written much more frequently than the memory is read. Moving down another level in FIG. 2C, NVRAM 204 is implemented as high-speed volatile memory such as dynamic random access memory (DRAM (dynamic random access memory)) 216 that is backed up by energy storage 218 in one embodiment. Energy storage 218 provides enough power to keep DRAM 216 powered for long enough for the contents to be transferred to flash memory 206 in the event of a power failure. In some embodiments, energy storage 218 is a capacitor, supercapacitor, battery, or other device that supplies a suitable energy supply sufficient to enable transfer of the contents of DRAM 216 to a stable storage medium in the event of a power loss. Flash memory 206 is implemented as a plurality of flash dies 222, which can also be referred to as a package of flash dies 222 or an array of flash dies 222.It should be understood that the flash die 222 can be packaged in various ways within the hybrid package, such as a bare die on a printed circuit board or other substrate, as an encapsulated die, etc., so as to have a single die per package or multiple dies per package (i.e., a multi-chip package). In the illustrated embodiment, the non-volatile solid-state storage 152 has a controller 212 or other processor, and an input / output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or the network interface controller 202 of the flash storage node 150. A flash input / output (I / O) port 220 is coupled to the flash die 222, and a direct memory access unit (DMA) 214 is coupled to the controller 212, the DRAM 216, and the flash die 222. In the illustrated embodiment, the I / O port 210, the controller 212, the DMA unit 214, and the flash I / O port 220 are implemented on a programmable logic device (「PLD」(programmable logic device)) 208, for example, an FPGA. In this embodiment, each flash die 222 has pages organized as 16 kB (kilobyte) pages 224, and registers 226 through which data can be written to or read from the flash die 222. In a further embodiment, instead of or in addition to the flash memory shown within the flash die 222, other types of solid-state memory are also used.

[0071]

[0091] In various embodiments as disclosed herein, the storage cluster 161 can generally be contrasted with a storage array. The storage nodes 150 are part of the set that creates the storage cluster 161. Each storage node 150 owns a piece of data and the computing required to provide the data. Multiple storage nodes 150 cooperate to store and retrieve data. Memory or storage devices are generally not much involved in the processing and manipulation of data when used within a storage array. The memory or storage devices within the storage array receive commands to read, write, or erase data. The memory or storage devices within the storage array do not recognize the larger system in which they are incorporated or what the data means. The memory or storage devices within the storage array can include various types of memory such as RAM, solid state drives, hard disk drives, etc. The storage unit 152 described herein has multiple interfaces that operate simultaneously and serve multiple purposes. In some embodiments, some of the functionality of the storage node 150 is moved into the storage unit 152, transforming the storage unit 152 into a combination of the storage unit 152 and the storage node 150. Placing the computing (with respect to the storage data) within the storage unit 152 places this computing closer to the data itself. Embodiments of various systems have a hierarchy of storage node layers with different capabilities. In contrast, within a storage array, a controller owns and is aware of everything regarding all the data that the controller manages within a shelf or storage device. Within the storage cluster 161, as described herein, multiple controllers within multiple storage units 152 and / or storage nodes 150 cooperate in various ways (e.g., for erasure coding, data sharding, metadata communication and redundancy, storage capacity expansion or contraction, data recovery, etc.).

[0072]

[0092] FIG. 2D shows a storage server environment using the embodiments of the storage nodes 150 and the storage units 152 of FIGS. 2A - C. In this version, each storage unit 152 includes a processor such as a controller 212 (see FIG. 2C), an FPGA, a flash memory 206, and an NVRAM 204 (see FIGS. 2B and 2C, which is DRAM 216 backed up by supercapacitors on a PCIe (Peripheral Component Interconnect Express) board within a chassis 138). The storage unit 152 can be implemented as a single board containing storage and can be the largest acceptable failure domain within the chassis. In some embodiments, up to two storage units 152 can fail and the device will continue without data loss.

[0073]

[0093] Physical storage, in some embodiments, is divided into named regions based on application usage. The NVRAM 204 is a contiguous block of reserved memory within the DRAM 216 of the storage unit 152 and is backed up by NAND flash. The NVRAM 204 is logically divided into multiple memory regions (e.g., spool regions) that are written as spools for two. The space within the spools of the NVRAM 204 is independently managed by each authority 168. Each device provides a certain amount of storage space for each authority 168. That authority 168 further manages the lifespan and allocation within that space. Examples of spools include distributed transactions or notions. When the primary power to the storage unit 152 fails, the on - board supercapacitor provides short - term power retention. During this retention interval, the contents of the NVRAM 204 are flushed to the flash memory 206. Upon the next power - on, the contents of the NVRAM 204 are recovered from the flash memory 206.

[0074]

[0094] Regarding the storage unit controller, the responsibility of the logical "controller" is distributed across each of the blades encompassing authority 168. This distribution of logical control is shown in FIG. 2D as host controller 242, intermediate layer controller 244, and storage unit controller 246. The management of the control plane and the storage plane is handled independently, although the components may be physically co-located on the same blade. Each authority 168 effectively performs the role of an independent controller. Each authority 168 provides its own data and metadata structure, its own background worker, and maintains its own life cycle.

[0075]

[0095] FIG. 2E is a hardware block diagram of blade 252 showing authority 168 interacting with control plane 254, compute and storage planes 256, 258, and underlying physical resources, using the embodiments of storage nodes 150 and storage unit 152 of FIGS. 2A - C within the storage server environment of FIG. 2D. The control plane 254 is divided into a number of authorities 168 that can use compute resources within compute plane 256 to execute on any of blades 252. The storage plane 258 is divided into a set of devices each providing access to the resources of flash 206 and NVRAM 204. In one embodiment, the compute plane 256 may perform the operation of a storage array controller as described herein on one or more devices of the storage plane 258 (e.g., a storage array).

[0076]

[0096] In the calculations and storage planes 256, 258 of FIG. 2E, the authority 168 interacts with the underlying physical resources (i.e., devices). From the perspective of the authority 168, the resources are striped across all of the physical devices. From the perspective of the device, it provides the resources to all authorities 168 regardless of where the authority happens to be executing. Each authority 168 has assigned or has been assigned one or more segments 260 of the storage memory within the storage unit 152, e.g., segments 260 within the flash memory 206 and the NVRAM 204. Each authority 168 uses those assigned segments 260 belonging to it to write to or read user data. Authorities can be associated with different amounts of physical storage of the system. For example, one authority 168 could have a greater number of segments 260 or segments 260 of a larger size within one or more storage units 152 than one or more other authorities 168.

[0077]

[0097] Figure 2F shows an elastic software layer within blade 252 of a storage cluster, according to some embodiments. In the elastic structure, the elastic software is symmetric. That is, the compute module 270 of each blade executes three identical layers of the process shown in Figure 2F. The storage manager 274 executes read and write requests from other blades 252 for data and metadata stored in the NVRAM 204 and flash 206 of the local storage unit 152. The privilege 168 fulfills client requests by issuing the necessary reads and writes to the blade 252 on which the corresponding data or metadata resides on the storage unit 152. The endpoint 272 parses client connection requests received from the monitoring software of the switch fabric 146, relays the client connection requests to the privilege 168 responsible for fulfillment, and relays the response of the privilege 168 to the client. The symmetric three-layer structure enables a high degree of parallelism in the storage system. In these embodiments, elasticity scales out efficiently and reliably. Additionally, elasticity evenly balances work across all resources regardless of the client access pattern, and typically maximizes parallelism by eliminating many of the requirements for blade-to-blade coordination that accompany traditional distributed locking by implementing a unique scale-out technique.

[0078]

[0098] Referring still to FIG. 2F, the authority 168 executed within the calculation module 270 of the blade 252 performs the internal operations required to fulfill the client request. One characteristic of elasticity is that the authorities 168 are stateless, that is, they cache their active data and metadata in the DRAM of their own blade 252 for fast access, but the authorities store all updates within their partitions of the NVRAM 204 on three separate blades 252 until the updates are written to the flash 206. All writes of the storage system to the NVRAM 204 are, in some embodiments, tripled to partitions on three separate blades 252. By having triple-mirrored NVRAM 204 and persistent storage protected by parity and Reed-Solomon RAID checksums, the storage system can survive the simultaneous failure of two blades 252 without loss of access to data, metadata, or either.

[0079]

[0099] Since the authorities 168 are stateless, they can move between blades 252. Each authority 168 has a unique identifier. The partitions of the NVRAM 204 and the flash 206 are associated with the identifier of the authority 168 and are not, in part, associated with the blade 252 on which they are executing. Therefore, when an authority 168 moves, the authority 168 continues to manage the same storage partition from its new location. When a new blade 252 is implemented within an embodiment of the storage cluster, the system partitions the storage of the new blade 252 for use by the authorities 168 of the system, moves the selected authorities 168 to the new blade 252, starts the endpoints 272 on the new blade 252, and automatically rebalances the load by including them in the client connection distribution algorithm of the switch fabric 146.

[0080]

[0100] From those new locations, the relocated authority 168 holds the contents of its partition of NVRAM 204 on flash 206, processes read and write requests from other authorities 168, and the endpoint 272 fulfills client requests directed to it. Similarly, if blade 252 fails or is removed, the system redistributes its authority 168 among the remaining blades 252 of the system. The redistributed authority 168 continues to perform its original functions from its new location.

[0081]

[0101] FIG. 2G shows the authority 168 and storage resources within a blade 252 of a storage cluster, according to some embodiments. Each authority 168 is solely responsible for a partition of flash 206 and NVRAM 204 on each blade 252. The authority 168 manages the contents and integrity of its partition independently of other authorities 168. The authority 168 compresses incoming data, temporarily stores it within its partition of NVRAM 204, then integrates, RAID protects, and stores it within a segment of storage within its partition of flash 206. When the authority 168 writes data to flash 206, the storage manager 274 performs the necessary flash translations to optimize write performance and maximize media life. In the background, the authority 168 "frees" or reclaims space occupied by data made obsolete by a client overwriting the data. It should be understood that since the partitions of authority 168 have no common elements, distributed locks are not necessary for clients and writes to execute or for background functions to perform.

[0082]

[0102] The embodiments described herein may utilize various software, communication and / or networking protocols. Additionally, the hardware and / or software configuration may be adjusted to accommodate various protocols. For example, embodiments may be WINDOWS TMAn Active Directory can be utilized in a system based on databases that provide authentication, directory, policy, and other services within an environment. In these embodiments, LDAP (Lightweight Directory Access Protocol) is one exemplary application protocol for querying and modifying entries within a directory service provider such as an Active Directory. In some embodiments, a Network Lock Manager (“NLM”) is utilized as a mechanism that works in cooperation with a Network File System (“NFS”) to provide System V style advisory files and record locks across a network. The Server Message Block (“SMB”) protocol, also known as the Common Internet File System (“CIFS”) in one version, can be integrated with the storage systems described herein. SMB operates as an application layer network protocol commonly used to provide shared access to files, printers, and serial ports, as well as various communications between nodes on a network. SMB also provides an authenticated inter-process communication mechanism. AMAZON TMS3 (Simple Storage Service) is a web service provided by Amazon Web Services, and the systems described herein can interface with Amazon S3 through a web service interface (REST (representational state transfer), SOAP (simple object access protocol), and BitTorrent). The RESTful API (application programming interface) decomposes a transaction and creates a series of small modules. Each module addresses a particular underlying part of the transaction. The control or permission provided by these embodiments, particularly for object data, may include the use of an access control list (“ACL” (access control list)). An ACL is a list of permissions attached to an object, and the ACL specifies which users or system processes are authorized access to the object and what actions are permitted for a given object. The system provides an identification and location system for computers on a network and may utilize Internet Protocol version 6 (“IPv6”) and IPv4 for communication protocols that select a path for traffic over the Internet. Packet routing between networked systems can include equal-cost multi-path routing (“ECMP”), which is a routing strategy where the next-hop packet forwarding to a single destination can be done through multiple “best paths” that tie for first place in the calculation of the routing metric. Multi-path routing can generally be used in conjunction with most routing protocols as it is a hop-by-hop decision limited to a single router.Software can support multi-tenancy, which is an architecture in which a single instance of a software application serves multiple customers. Each customer can be referred to as a tenant. A tenant can be given the ability to customize some parts of the application, although in some embodiments, it may not be necessary to customize the application's code. Embodiments can maintain an audit log. An audit log is a document that records events within a computing system. In addition to documenting which resources were accessed, audit log entries typically include a destination and source address, a timestamp, and user login information for compliance with various regulations. Embodiments can support various key management policies, such as encryption key rotation. Additionally, the system can support a dynamic root password, or some variation that dynamically changes the password.

[0083]

[0103] Figure 3A illustrates a diagram of a storage system 306 coupled for data communication with a cloud service provider 302 according to some embodiments of the present disclosure. Although shown more generally, the storage system 306 shown in Figure 3A may be similar to the storage systems described above with reference to Figures 1A - 1D and Figures 2A - 2G. Depending on the embodiment, the storage system 306 shown in Figure 3A may be embodied as a storage system including a non - balanced active / active controller, as a storage system including a balanced active / active controller, as a storage system including active / active controllers where a minority of all of the resources of each controller are utilized such that each controller has reserved resources that can be used to support failover, as a storage system including fully active / active controllers, as a storage system including controllers by data set, as a storage system including a two - layer architecture having a front - end controller and a back - end integrated storage controller, as a storage system including a scale - out cluster of dual - controller arrays, and as combinations of such embodiments.

[0084]

[0104] In the example shown in FIG. 3A, the storage system 306 is coupled to the cloud service provider 302 via a data communication link 304. The data communication link 304 can be embodied as a dedicated data communication link, as a data communication path provided through the use of one or more data communication networks such as a wide area network ("WAN") or a LAN, or as some other mechanism having the ability to transport digital information between the storage system 306 and the cloud service provider 302. Such a data communication link 304 can be entirely wired, entirely wireless, or some combination of wired and wireless data communication paths. In such an example, digital information can be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using one or more data communication protocols. For example, digital information can be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using a handheld device transfer protocol ("HDTP"), a hypertext transfer protocol ("HTTP"), an Internet protocol ("IP"), a real-time transfer protocol ("RTP"), a transmission control protocol ("TCP"), a user datagram protocol ("UDP"), a wireless application protocol ("WAP"), or other protocols.

[0085]

[0105] The cloud service provider 302 shown in FIG. 3A can be embodied as a system and computing environment that provides a vast number of services to users of the cloud service provider 302 through sharing of computing resources via, for example, a data communication link 304. The cloud service provider 302 can provide on-demand access to a shared pool of configurable computing resources such as computer networks, servers, storage, applications, and services. The shared pool of configurable resources can be rapidly provisioned with minimal administrative effort and made available to users of the cloud service provider 302. Generally, users of the cloud service provider 302 do not recognize the exact computing resources being utilized by the cloud service provider 302 to provide the services. In many cases, such a cloud service provider 302 can be accessible via the Internet, but those skilled in the art will recognize that any system that abstracts the use of shared resources to provide services to users through any data communication link can be considered a cloud service provider 302.

[0086]

[0106] In the example shown in FIG. 3A, cloud service provider 302 may be configured to provide various services to storage system 306 and the users of storage system 306 through the implementation of various service models. For example, cloud service provider 302 may provide access to its storage infrastructure for use by storage system 306 and the users of storage system 306 through the implementation of an infrastructure-as-a-service (“IaaS”) service model, through the implementation of a platform-as-a-service (“PaaS”) service model, through the implementation of a software-as-a-service (“SaaS”) service model, through the implementation of an authentication-as-a-service (“AaaS”) service model, and through the implementation of a storage-as-a-service model, and through others. The reader will understand that the service models described above are included for illustrative purposes only and in no way limit the services that may be provided by cloud service provider 302 or the service models that may be implemented by cloud service provider 302, so cloud service provider 302 may be configured to provide additional services to storage system 306 and the users of storage system 306 through the implementation of additional service models.

[0087]

[0107] In the example shown in FIG. 3A, the cloud service provider 302 can be embodied, for example, as a private cloud, as a public cloud, or as a combination of a private cloud and a public cloud. In one embodiment where the cloud service provider 302 is embodied as a private cloud, the cloud service provider 302 may be specialized to provide services to a single organization rather than to multiple organizations. In one embodiment where the cloud service provider 302 is embodied as a public cloud, the cloud service provider 302 may provide services to multiple organizations. Further alternative embodiments, the cloud service provider 302 may be embodied as a hybrid of private and public cloud services using a hybrid cloud deployment.

[0088]

[0108] Although not explicitly shown in FIG. 3A, the reader will understand that a vast amount of additional hardware components and additional software components may be required to facilitate the storage system 306 and the provision of cloud services to users of the storage system 306. For example, the storage system 306 may be coupled to (or even include) a cloud storage gateway. Such a cloud storage gateway may be embodied as, for example, hardware-based or software-based equipment located on-premises with the storage system 306. Such a cloud storage gateway may operate as a bridge between a local application running on the storage array 306 and remote cloud-based storage utilized by the storage array 306. Through the use of a cloud storage gateway, an organization may move primary iSCSI or NAS to a cloud service provider 302, thereby enabling the organization to save space on their on-premises storage systems. Such a cloud storage gateway may be configured to emulate a disk array, a block-based device, a file server, or other storage systems that can translate SCSI commands, file server commands, or other appropriate commands into a REST space protocol that facilitates communication with the cloud service provider 302.

[0089]

[0109] To enable the storage system 306 and users of the storage system 306 to utilize the services provided by the cloud service provider 302, a cloud transfer process can be performed in which data, applications, or other elements from the organization's local system (or even from another cloud environment) are moved to the cloud service provider 302. To successfully transfer data, applications, or other elements to the environment of the cloud service provider 302, middleware such as a cloud transfer tool can be utilized to bridge the gap between the environment of the cloud service provider 302 and the organization's environment. Such cloud transfer tools can also address the potentially high network costs and long transfer times associated with transferring large volumes of data to the cloud service provider 302, and can be configured to address security concerns associated with confidential data to the cloud service provider 302 over a data communication network. To further enable the storage system 306 and users of the storage system 306 to utilize the services provided by the cloud service provider 302, a cloud orchestrator can also be used to prepare and coordinate automated tasks to create an integrated process or workflow. Such a cloud orchestrator can perform tasks such as configuring various components, regardless of whether those components are cloud components or on-premises components, and managing the interconnections between such components. The cloud orchestrator can simplify communication and connections between components to ensure that links are properly configured and maintained.

[0090]

[0110] In the example shown in FIG. 3A, and as briefly described above, the cloud service provider 302 may be configured to provide services to the storage system 306 and the users of the storage system 306 through the use of a SaaS service model, eliminating the need to install and run applications on local computers. This can simplify the maintenance and support of the applications. Such applications can take many forms according to various embodiments of the present disclosure. For example, the cloud service provider 302 may be configured to provide access to a data analysis application to the storage system 306 and the users of the storage system 306. Such a data analysis application may be configured to receive, for example, a vast amount of telemetry data reported (phoned home) by the storage system 306. Such telemetry data may describe various operating characteristics of the storage system 306 and can be analyzed for a vast number of purposes, including, for example, determining the health of the storage system 306, identifying the workload being run on the storage system 306, predicting when the storage system 306 will run out of various resources, recommending configuration changes, hardware or software upgrades, workflow migrations, or other actions that may improve the operation of the storage system 306.

[0091]

[0111] Cloud service provider 302 may also be configured to provide access to the virtualized computing environment to storage system 306 and users of storage system 306. Such a virtualized computing environment may be embodied, for example, as virtual machines or other virtualized computer hardware platforms, virtual storage devices, virtualized computer network resources, and the like. Examples of such virtualized environments include virtual machines created to emulate actual computers, virtual desktop environments that separate logical desktops from physical machines, virtualized file systems that enable uniform access to different types of specific file systems, and many others.

[0092]

[0112] For further illustration, FIG. 3B depicts a diagram of storage system 306 according to some embodiments of the present disclosure. Although shown more generally, the storage system may include many of the components described above, so storage system 306 shown in FIG. 3B may be similar to the storage systems described above with reference to FIGS. 1A-1D and FIGS. 2A-2G.

[0093]

[0113] The storage system 306 shown in FIG. 3B can include a vast amount of storage resources 308 that can be embodied in many forms. For example, the storage resources 308 can include nano-RAM, or another form of non-volatile random access memory that utilizes carbon nanotubes deposited on a substrate, 3D cross-point non-volatile memory, single-level cell ( "SLC" (single-level cell)) NAND flash, multi-level cell ( "MLC" (multi-level cell)) NAND flash, triple-level cell ( "TLC" (triple-level cell)) NAND flash, quad-level cell ( "QLC" (quad-level cell)) NAND flash, or flash memory including others. Similarly, the storage resources 308 can include magnetoresistive random-access memory ( "MRAM" (magnetoresistive random-access memory)) including spin transfer torque ( "STT" (spin transfer torque)) MRAM. Exemplary storage resources 308 can alternatively include non-volatile phase change memory ( "PCM" (phase-change memory)), quantum memory that enables storage and retrieval of optical quantum information, resistive random-access memory ( "ReRAM" (resistive random-access memory)), storage class memory ( "SCM" (storage class memory)), or other forms of storage resources including any combination of the resources described herein. The reader will understand that other forms of computer memory and storage devices, including DRAM, SRAM, EEPROM, universal memory, and many others, can also be utilized by the storage system described above.The storage resource 308 shown in FIG. 3A can be embodied in various form factors including, but not limited to, dual in-line memory modules (“DIMMs”), non-volatile dual in-line memory modules (“NVDIMMs”), M.2, U.2, and others.

[0094]

[0114] The storage resource 308 shown in FIG. 3A can include various forms of SCM. SCM can effectively treat high-speed non-volatile memory (e.g., NAND flash) as an extension of DRAM, whereby a data set can be treated as an in-memory data set that is entirely present within DRAM. SCM can include, for example, non-volatile media such as NAND flash. Such NAND flash can be accessed using NVMe, which uses a PCIe bus as its transport mechanism and provides a relatively low access latency compared to older protocols. In fact, network protocols used for SSDs in all-flash arrays can include Ethernet (ROCE, NVME TCP), Fibre Channel (NVMe FC), InfiniBand (iWARP), and NVMe that uses other things that enable treating high-speed non-volatile memory as an extension of DRAM. Considering the fact that DRAM is often byte-addressable and high-speed non-volatile memory such as NAND flash is block-addressable, a controller software / hardware stack may be required to convert block data into bytes stored in the medium. Examples of media and software that can be used as SCM can include, for example, 3D XPoint, Intel Memory Drive Technology, Samsung's Z-SSD, and others.

[0095]

[0115] The exemplary storage system 306 shown in FIG. 3B may implement various storage architectures. For example, a storage system according to some embodiments of the present disclosure may utilize block storage, where data is stored within blocks, and each block essentially serves the role of an individual hard drive. A storage system according to some embodiments of the present disclosure may utilize object storage, where data is managed as objects. Each object may include the data itself, a variable amount of metadata, and a globally unique identifier, and object storage may be implemented at multiple levels (e.g., device level, system level, interface level). A storage system according to some embodiments of the present disclosure utilizes file storage, where data is stored within a hierarchical structure. Such data is stored within files and folders and may be presented in the same format to both the system storing it and the system retrieving it.

[0096]

[0116] The exemplary storage system 306 shown in FIG. 3B may be embodied as a storage system in which additional storage resources can be added through the use of a scale-up model, or additional storage resources can be added through the use of a scale-out model, or through some combination thereof. In a scale-up model, additional storage can be added by adding additional storage devices. However, in a scale-out model, additional storage nodes can be added to a cluster of storage nodes, and such storage nodes can include additional processing resources, additional networking resources, and the like.

[0097]

[0117] The storage system 306 shown in FIG. 3B also includes communication resources 310 that can be useful in facilitating data communication between components within the storage system 306 and between the storage system 306 and computing devices external to the storage system 306, including embodiments where those resources are separated by a relatively large extent. The communication resources 310 can be configured to utilize a variety of different protocols and data communication fabrics to facilitate data communication between components within the storage system and computing devices external to the storage system. For example, the communication resources 310 can include fibre channel (「FC」) technology such as an FC fabric and an FC protocol that can transport SCSI commands over an FC network, FC over ethernet (「FCoE」) technology where FC frames are encapsulated and transmitted over an ethernet network, InfiniBand (「IB」) technology where a switched fabric topology is utilized to facilitate transmission between channel adapters, NVM express (「NVMe」) technology and NVMe over fabric (「NVMeoF」) technology where non-volatile memory media attached via a PCI express (「PCIe」) bus can be accessed, and others. In fact, the storage systems described above can utilize neutrino communication technologies and devices in which information (including binary information) is transmitted directly or indirectly using a neutrino beam.

[0098]

[0118] The communication resource 310 can also include a serial attached SCSI (SAS) for connecting the storage resource 308 in the storage system 306 to a host bus adapter in the storage system 306, a serial ATA (SATA) bus interface, an internet small computer systems interface (iSCSI) technology for providing block-level access to the storage resource 308 in the storage system 306, a mechanism for accessing the storage resource 308 in the storage system 306, and other communication resources that can be useful for facilitating data communication between components within the storage system 306 and between the storage system 306 and a computing device external to the storage system 306.

[0099]

[0119] The storage system 306 shown in FIG. 3B also includes processing resources 312 that can be useful for executing computer program instructions and performing other computational tasks within the storage system 306. The processing resources 312 can include one or more ASICs customized for some specific purpose and one or more CPUs. The processing resources 312 can also include one or more DSPs, one or more FPGAs, one or more system on a chip (SoC), or other forms of processing resources 312. The storage system 306 can utilize the storage resources 312 to perform various tasks, including supporting the execution of software resources 314, which will be described in more detail below.

[0100]

[0120] The storage system 306 shown in FIG. 3B also includes software resources 314 that can perform a vast number of tasks when executed by processing resources 312 within the storage system 306. The software resources 314 can include, for example, one or more modules of computer program instructions useful for implementing various data protection techniques for preserving the integrity of data stored within the storage system when executed by processing resources 312 within the storage system 306. The reader will understand that such data protection techniques can be implemented, for example, by system software executing on computer hardware within the storage system, by a cloud service provider, or in other ways. Such data protection techniques can include, for example, data archiving techniques that move data that is no longer being actively used to a separate storage device or separate storage system for long-term retention, data backup techniques where data stored within the storage system is copied and stored in a separate location to avoid data loss in the event of equipment failure or some other catastrophic situation occurring in the storage system, data replication techniques where data stored within the storage system is replicated to another storage system so that the data can be made accessible via multiple storage systems, data snapshot techniques where the state of data within the storage system is captured at various points in time, data and database cloning techniques where duplicate copies of data and databases can be created, and other data protection techniques.

[0101]

[0121] Software resource 314 may also include software that is useful for implementing software-defined storage (“SDS”). In such an example, software resource 314 may include one or more modules of computer program instructions that, when executed, are useful for policy-based provisioning and management of data storage independent of the underlying hardware. Such software resource 314 may be useful for implementing storage virtualization to separate storage hardware from the software that manages the storage hardware.

[0102]

[0122] Software resource 314 may also include software that is useful for facilitating and optimizing I / O operations directed to storage resource 308 within storage system 306. For example, software resource 314 may include software modules that perform various data conditioning techniques such as, for example, data compression, data deduplication, and the like. Software resource 314 may include software modules that intelligently group I / O operations together to facilitate better utilization of the underlying storage resource 308, software modules that perform data transfer operations for transfer from within the storage system, and software modules that perform other functions. Such software resource 314 may be embodied as one or more software containers or in many other ways.

[0103]

[0123] For further explanation, FIG. 3C illustrates an example of a cloud-based storage system 318 according to some embodiments of the present disclosure. In the example shown in FIG. 3C, the cloud-based storage system 318 is fully created within a cloud computing environment 316 such as, for example, Amazon Web Services (“AWS”), Microsoft Azure, Google Cloud Platform, IBM Cloud, Oracle Cloud, and others. The cloud-based storage system 318 can be used to provide services similar to those provided by the storage systems described above. For example, the cloud-based storage system 318 can be used to provide a block storage service to users of the cloud-based storage system 318, the cloud-based storage system 318 can be used to provide a storage service to users of the cloud-based storage system 318 through the use of solid-state storage, and so on.

[0104]

[0124] The cloud-based storage system 318 shown in FIG. 3C includes two cloud computing instances 320, 322 each used to support the execution of storage controller applications 324, 326. The cloud computing instances 320, 322 can be embodied as instances of cloud computing resources (e.g., virtual machines) that can be provided by the cloud computing environment 316 to support the execution of software applications such as the storage controller applications 324, 326. In one embodiment, the cloud computing instances 320, 322 can be embodied as Amazon Elastic Compute Cloud (「EC2」) instances. In such an example, an Amazon Machine Image (「AMI」) including the storage controller applications 324, 326 can be booted to create and configure virtual machines on which the storage controller applications 324, 326 can execute.

[0105]

[0125] In the exemplary method shown in FIG. 3C, storage controller applications 324, 326, when executed, may be embodied as modules of computer program instructions that perform various storage tasks. For example, storage controller applications 324, 326, when executed, write data received from a user of cloud-based storage system 318 to cloud-based storage system 318, erase data from cloud-based storage system 318, retrieve data from cloud-based storage system 318 and provide such data to a user of cloud-based storage system 318, monitor and report disk utilization and performance, perform redundancy operations such as RAID or RAID-like data redundancy operations, compress data, encrypt data, deduplicate data, etc., and may be embodied as modules of computer program instructions that perform the same tasks as controllers 110A, 110B in FIG. 1A described above. The reader will understand that, since there are two cloud computing instances 320, 322 each including a storage controller application 324, 326, in some embodiments, one cloud computing instance 320 may operate as the primary controller as described above, while the other cloud computing instance 322 may operate as the secondary controller as described above. The reader will also understand that the storage controller applications 324, 326 shown in FIG. 3C may include the same source code executed within different cloud computing instances 320, 322.

[0106]

[0126] Consider an example where the cloud computing environment 316 is embodied as AWS and the cloud computing instances are embodied as EC2 instances. In such an example, a cloud computing instance 320 operating as a primary controller can be deployed on one of the instance types having a relatively large amount of memory and processing power, while a cloud computing instance 322 operating as a secondary controller can be deployed on one of the instance types having a relatively small amount of memory and processing power. In such an example, when a failover event occurs in which the primary and secondary roles are switched, a double failover can be actually implemented, whereby 1) a first failover event in which the cloud computing instance 322 that previously operated as a secondary controller begins to operate as a primary controller, and 2) a third cloud computing instance (not shown) that is of the instance type having a relatively large amount of memory and processing power is spun up along with a copy of the storage controller application, and the third cloud computing instance begins to operate as a primary controller, while the cloud computing instance 322 that originally operated as a secondary controller begins to operate as a secondary controller again. In such an example, the cloud computing instance 320 that previously operated as a primary controller can be terminated. The reader will understand that in an alternative embodiment, the cloud computing instance 320 operating as a secondary controller after a failover event can continue to operate as a secondary controller, and the cloud computing instance 322 that operated as a primary controller after the occurrence of the failover event can be terminated when the primary role is taken over by a third cloud computing instance (not shown).

[0107]

[0127] The reader will understand that while the above-described embodiments relate to embodiments in which one cloud computing instance 320 operates as a primary controller and a second cloud computing instance 322 operates as a secondary controller, other embodiments are also within the scope of the present disclosure. For example, each cloud computing instance 320, 322 may operate as a primary controller for some portion of the address space supported by the cloud-based storage system 318, each cloud computing instance 320, 322 may operate as a primary controller, and servicing of I / O operations directed to the cloud-based storage system 318 may be split in some other way, and so on. In fact, in other embodiments where cost savings may be prioritized over performance requirements, there may be only a single cloud computing instance that includes the storage controller application.

[0108]

[0128] The cloud-based storage system 318 shown in FIG. 3C includes cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338. The cloud computing instances 340a, 340b, 340n shown in FIG. 3C can be embodied as instances of cloud computing resources that can be provided by the cloud computing environment 316, for example, to support the execution of software applications. The cloud computing instances 340a, 340b, 340n in FIG. 3C have resources of local storage 330, 334, 338, whereas the cloud computing instances 320, 322 that support the execution of the storage controller applications 324, 326 need not have local storage resources, so the cloud computing instances 340a, 340b, 340n in FIG. 3C can be different from the cloud computing instances 320, 322 described above. The cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 can be embodied, for example, as EC2 M5 instances including one or more SSDs, as EC2 R5 instances including one or more SSDs, as EC2 I3 instances including one or more SSDs, and others. According to embodiments, the local storage 330, 334, 338 must be embodied as solid state storage (e.g., SSD), rather than storage using hard disk drives.

[0109]

[0129] In the example shown in FIG. 3C, each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 can include software daemons 328, 332, 336 that can appear to storage controller applications 324, 326 as if the cloud computing instances 340a, 340b, 340n were physical storage devices (e.g., one or more SSDs) when executed by the cloud computing instances 340a, 340b, 340n. In such an example, the software daemons 328, 332, 336 can include computer program instructions similar to those that would typically be included on a storage device, whereby the storage controller applications 324, 326 can send and receive the same commands that a storage controller would send to a storage device. In this way, the storage controller applications 324, 326 can include code that is the same (or substantially the same) as the code that would be executed by a controller in the storage system described above. In these and similar embodiments, the communication between the storage controller applications 324, 326 and the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 can utilize iSCSI, NVMe over TCP, messaging, a custom protocol, or be by some other mechanism.

[0110]

[0130] In the example shown in FIG. 3C, each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 can also be coupled to block storage 342, 344, 346 provided by the cloud computing environment 316. The block storage 342, 344, 346 provided by the cloud computing environment 316 can be embodied, for example, as Amazon Elastic Block Store (EBS) volumes. For example, a first EBS volume can be coupled to the first cloud computing instance 340a, a second EBS volume can be coupled to the second cloud computing instance 340b, and a third EBS volume can be coupled to the third cloud computing instance 340n. In such an example, software daemons 328, 332, 336 (or some other module) running within a particular cloud computing instance 340a, 340b, 340n, upon receiving a request to write data, can initiate writing of the data to its attached EBS volume and to the resources of its local storage 330, 334, 338. Thus, the block storage 342, 344, 346 provided by the cloud computing environment 316 can be utilized in a manner similar to how the above-described NVRAM devices are utilized. According to alternative embodiments, the data can be written only to the resources of the local storage 330, 334, 338 within a particular cloud computing instance 340a, 340b, 340n. In an alternative embodiment, instead of using the block storage 342, 344, 346 provided by the cloud computing environment 316 as NVRAM, the actual RAM on each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 can be used as NVRAM, thereby reducing the network usage cost that would be associated with using the EBS volumes as NVRAM.

[0111]

[0131] In the example shown in FIG. 3C, cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338 can be utilized to service service I / O operations directed to cloud-based storage system 318 by cloud computing instances 320, 322 that support execution of storage controller applications 324, 326. Consider an example where a first cloud computing instance 320 executing storage controller application 324 is operating as a primary controller. In such an example, the first cloud computing instance 320 executing storage controller application 324 can receive (either directly or indirectly via a secondary controller) a request from a user of cloud-based storage system 318 to write data to cloud-based storage system 318. In such an example, the first cloud computing instance 320 executing storage controller application 324 can perform various tasks such as deduplicating data included in the request, compressing data included in the request, determining where to write the data included in the request, etc. before finally sending a request to write a deduplicated, encrypted, or otherwise, possibly updated version of the data to one or more of cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338. Either cloud computing instance 320, 322 can, depending on the embodiment, receive a request to read data from cloud-based storage system 318 and can finally send the request to read the data to one or more of cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338.

[0112]

[0132] When a reader's request to write data is received by a particular cloud computing instance 340a, 340b, 340n having local storage 330, 334, 338, software daemons 328, 332, 336, or any other module of computer program instructions running on the particular cloud computing instance 340a, 340b, 340n can be configured not only to write data to the resources of its own local storage 330, 334, 338 and to any suitable block storage 342, 344, 346 provided by the cloud computing environment 316, but also to write data to cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n. Cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n can be embodied, for example, as Amazon Simple Storage Service ("S3") storage accessible by the particular cloud computing instance 340a, 340b, 340n. In other embodiments, cloud computing instances 320, 322 each including storage controller applications 324, 326 can initiate storage of data into the local storage 330, 334, 338 of cloud computing instances 340a, 340b, 340n and into cloud-based object storage 348.

[0113]

[0133] As described above, the reader will understand that the cloud-based storage system 318 can be used to provide a block storage service to users of the cloud-based storage system 318. The resources of the local storage 330, 334, 338 and the resources of the block storage 342, 344, 346 utilized by the cloud computing instances 340a, 340b, 340n can support block-level access, while the cloud-based object storage 348 attached to a particular cloud computing instance 340a, 340b, 340n supports only object-based access. To address this, the software daemons 328, 332, 336, or some other module of the computer program instructions running on a particular cloud computing instance 340a, 340b, 340n, can be configured to take blocks of data, package those blocks into objects, and write the objects to the cloud-based object storage 348 attached to a particular cloud computing instance 340a, 340b, 340n.

[0114]

[0134] Consider an example where data is written to the resources of local storage 330, 334, 338 and the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n in 1MB blocks. In such an example, assume that a user of cloud - based storage system 318 issues a request to write data that, after being compressed and deduplicated by storage controller applications 324, 326, requires writing 5MB of data. In such an example, since five 1MB - sized blocks are written to the resources of local storage 330, 334, 338 and the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n, it is relatively easy to write data to the resources of local storage 330, 334, 338 and the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n. In such an example, software daemons 328, 332, 336, or some other module of computer program instructions running on a particular cloud computing instance 340a, 340b, 340n can be configured to: 1) create a first object containing the first 1MB of data, write the first object to cloud - based object storage 348; 2) create a second object containing the second 1MB of data, write the second object to cloud - based object storage 348; 3) create a third object containing the third 1MB of data, write the third object to cloud - based object storage 348, and so on. Therefore, in some embodiments, each object written to cloud - based object storage 348 can be of the same size (or approximately the same size). The reader will understand that in such an example, metadata associated with the data itself can be included within each object (e.g., the first 1MB of the object is data, and the remaining portion is metadata associated with the data).

[0115]

[0135] The reader will understand that a cloud-based object storage 348 can be incorporated within a cloud-based storage system 318 to increase the durability of the cloud-based storage system 318. Continuing with the above example where the cloud computing instances 340a, 340b, 340n are EC2 instances, the reader will understand that the EC2 instances are only guaranteed to have 99.9% monthly uptime and that data stored in the local instance store will only persist for the lifetime of the EC2 instance. Therefore, relying on the cloud-based storage system 318 as the sole source of persistent data storage within the cloud computing instances 340a, 340b, 340n with local storage 330, 334, 338 can result in a relatively unreliable storage system. Similarly, EBS volumes are designed for a 99.999% availability rate. Therefore, even relying on EBS as the persistent data store within the cloud-based storage system 318 can result in a storage system that is not sufficiently durable. However, Amazon S3 is designed to provide 99.999999999% durability. That is, a cloud-based storage system 318 that can incorporate S3 within its storage pool is significantly more durable than various other options.

[0116]

[0136] Readers will understand that the cloud-based storage system 318 into which S3 can be incorporated within its storage pool is considerably more durable than various other options, while using S3 as the primary storage pool can result in a storage system with a relatively slow response time and a relatively long I / O latency. Therefore, the cloud-based storage system 318 shown in FIG. 3C not only stores data in S3, but the cloud-based storage system 318 also stores data within the resources of the local storage 330, 334, 338 and the resources of the block storage 342, 344, 346 utilized by the cloud computing instances 340a, 340b, 340n, such that read operations can be serviced from the resources of the local storage 330, 334, 338 and the resources of the block storage 342, 344, 346 utilized by the cloud computing instances 340a, 340b, 340n, thereby reducing the read latency when a user of the cloud-based storage system 318 attempts to read data from the cloud-based storage system 318.

[0117]

[0137] Depending on the embodiment, all data stored by the cloud-based storage system 318 can be stored in both 1) cloud-based object storage 348 and 2) at least one of the resources of local storage 330, 334, 338 or the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n. In such an embodiment, the resources of local storage 330, 334, 338 and the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n can generally operate effectively as a cache containing all the data stored in S3 as well, such that all reads of data can be serviced by cloud computing instances 340a, 340b, 340n without the need for the cloud computing instances 340a, 340b, 340n to access the cloud-based object storage 348. The reader will understand, however, that in other embodiments, all data stored by the cloud-based storage system 318 can be stored in the cloud-based object storage 348, but at least one of the resources of local storage 330, 334, �38 or the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n can store less than all of the data stored by the cloud-based storage system 318. In such an example, various policies can be utilized to determine which subset of the data stored by the cloud-based storage system 318 should be present in both 1) cloud-based object storage 348 and 2) at least one of the resources of local storage 330, 334, 338 or the resources of block storage 342, 344, 346 utilized by cloud computing instances 340a, 340b, 340n.

[0118]

[0138] As described above, when cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338 are embodied as EC2 instances, the cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338 are only guaranteed to have 99.9% monthly uptime, and the data stored in the local instance store only persists for the lifetime of each cloud computing instance 340a, 340b, 340n having local storages 330, 334, 338. Therefore, one or more modules of the computer program instructions executing within the cloud-based storage system 318 (e.g., a monitoring module executing on its own EC2 instance) can be designed to handle failures of one or more of the cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338. In such an example, the monitoring module can create one or more new cloud computing instances having local storage, retrieve the data stored on the failed cloud computing instances 340a, 340b, 340n from the cloud-based object storage 348, and store the data retrieved from the cloud-based object storage 348 in the local storage on the newly created cloud computing instances, thereby handling failures of one or more of the cloud computing instances 340a, 340b, 340n having local storages 330, 334, 338. The reader will understand that many variations of this process can be implemented.

[0119]

[0139] Consider an example where all cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 fail. In such an example, the monitoring module may create a new cloud computing instance having local storage, and a high-bandwidth instance type that enables a maximum data transfer rate between the newly created high-bandwidth cloud computing instance having local storage and cloud-based object storage 348 is selected. The reader will understand that an instance type that enables a maximum data transfer rate between the new cloud computing instance and cloud-based object storage 348 is selected, whereby the new high-bandwidth cloud computing instance can rehydrate data from cloud-based object storage 348 as quickly as possible. When the new high-bandwidth cloud computing instance has rehydrated data from cloud-based object storage 348, a less expensive, lower-bandwidth cloud computing instance can be created, the data can be transferred to the less expensive, lower-bandwidth cloud computing instance, and the high-bandwidth cloud computing instance can be terminated.

[0120]

[0140] The reader will understand that, in some embodiments, the number of new cloud computing instances created can substantially exceed the number of cloud computing instances required to locally store all of the data stored by the cloud-based storage system 318. Since each new cloud computing instance can (in parallel) retrieve some portion of the data stored by the cloud-based storage system 318, the number of new cloud computing instances created can substantially exceed the number of cloud computing instances required to locally store all of the data stored by the cloud-based storage system 318 in order to more quickly pull the data from the cloud-based object storage 348 into the new cloud computing instances. In such embodiments, when the data stored by the cloud-based storage system 318 is pulled into the newly created cloud computing instances, the data can be integrated within a subset of the newly created cloud computing instances and those newly created cloud computing instances that have become redundant can be terminated.

[0121]

[0141] Consider an example where 1,000 cloud computing instances are required to locally store all the valid data written by a user of the cloud-based storage system 318 to the cloud-based storage system 318. In such an example, assume that all 1,000 cloud computing instances fail. In such an example, the monitoring module may cause 100,000 cloud computing instances to be created, and each cloud computing instance is responsible for retrieving an individual 1 / 100,000th chunk of the valid data written by the user of the cloud-based storage system 318 to the cloud-based storage system 318 from the cloud-based object storage 348 and locally storing the individual chunk of the retrieved data set. In such an example, since each of the 100,000 cloud computing instances can retrieve data from the cloud-based object storage 348 in parallel, the caching layer can be restored 100 times faster compared to an embodiment where the monitoring module only creates 1,000 replacement cloud computing instances. In such an example, over time, the data locally stored within the 100,000 can be integrated into the 1,000 cloud computing instances, and the remaining 99,000 cloud computing instances can be terminated.

[0122]

[0142] The reader will understand that various performance aspects of the cloud-based storage system 318 can be monitored (e.g., by a monitoring module running within an EC2 instance), whereby the cloud-based storage system 318 can be scaled up or out as needed. Consider an example where the monitoring module monitors the communication between cloud computing instances 320, 322, 340a, 340b, 340n and the cloud-based object storage 348, either via communication with one or more of the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 respectively, or via monitoring the communication between cloud computing instances 320, 322, 340a, 340b, 340n, or by some other means, to monitor the performance of the cloud-based storage system 318. In such an example, assume that the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 are of insufficient size and are determined not to be serving the I / O requests issued by the users of the cloud-based storage system 318 adequately. In such an example, the monitoring module can create a new, more powerful cloud computing instance (e.g., a type of cloud computing instance that includes greater processing power, more memory, etc.) that includes the storage controller application, whereby the new, more powerful cloud computing instance can begin to operate as the primary controller.Similarly, if the monitoring module determines that the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 are too large and could achieve cost savings by switching to smaller, lower-capability cloud computing instances, the monitoring module may create a new, lower-capability (and less expensive) cloud computing instance that includes the storage controller application, whereby the new, lower-capability cloud computing instance can begin operating as the primary controller.

[0123]

[0143] As an additional example of dynamically setting the size of the cloud-based storage system 318, consider an example where the monitoring module determines that the utilization of the local storage collectively provided by the cloud computing instances 340a, 340b, 340n has reached a predetermined utilization threshold (e.g., 95%). In such an example, the monitoring module may create additional cloud computing instances with local storage to expand the pool of local storage provided by the cloud computing instances. Alternatively, the monitoring module may create one or more new cloud computing instances having a larger amount of local storage than the existing cloud computing instances 340a, 340b, 340n, thereby enabling the transfer of data stored within the existing cloud computing instances 340a, 340b, 340n to the one or more new cloud computing instances and terminating the existing cloud computing instances 340a, 340b, 340n, as a result expanding the pool of local storage provided by the cloud computing instances. Similarly, if the pool of local storage provided by the cloud computing instances is unnecessarily large, data can be consolidated and some of the cloud computing instances can be terminated.

[0124]

[0144] The reader will understand that the cloud-based storage system 318 can be automatically sized up and down by the monitoring module applying a given set of rules that can be relatively complex or relatively simple. In fact, the monitoring module can not only consider the current state of the cloud-based storage system 318, but the monitoring module can also apply predictive policies based on, for example, observed behavior (e.g., usage from 10 PM to 6 AM every night of the storage system is relatively light), a given fingerprint (e.g., every time a virtual desktop infrastructure adds 100 virtual desktops, the number of IOPS directed to the storage system increases by X), etc. In such examples, the dynamic scaling of the cloud-based storage system 318 can be based on current performance metrics, predicted workloads, and many other factors, including these combinations.

[0125]

[0145] The reader will further understand that since the cloud-based storage system 318 can be dynamically scaled, the cloud-based storage system 318 can even operate in a more dynamic manner. Consider an example of garbage collection. In a traditional storage system, the amount of storage is fixed. Therefore, at some point, the amount of available storage becomes highly constrained and the storage system may have to perform garbage collection because it is about to run out of storage. In contrast, the cloud-based storage system 318 described herein can "add" additional storage at any time (e.g., by adding more cloud computing instances with local storage). Since the cloud-based storage system 318 described herein can "add" additional storage at any time, the cloud-based storage system 318 can make more intelligent decisions regarding when to perform garbage collection. For example, the cloud-based storage system 318 may implement a policy where garbage collection is only performed when the number of IOPS served by the cloud-based storage system 318 falls below a certain level. Depending on the embodiment, given that the size of the cloud-based storage system 318 is not constrained in the same way as a traditional storage system, other system-level features (e.g., deduplication, compression) can also be turned off and on depending on the system load.

[0126]

[0146] The reader will understand that embodiments of the present disclosure solve problems related to block storage services provided by some cloud computing environments, since some cloud computing environments only allow one cloud computing instance to be connected to a block storage volume at a single point in time. For example, in Amazon AWS, only a single EC2 instance can be connected to an EBS volume. Through the use of EC2 instances with local storage, embodiments of the present disclosure can provide a multiple connection capability that allows multiple EC2 instances to be connected to another EC2 instance with local storage (a "drive instance"). In such embodiments, the drive instance may include software that runs within the drive instance that enables the drive instance to support I / O directed to a specific volume from each of the connected EC2 instances. Therefore, some embodiments of the present disclosure may be embodied as a multiple connection block storage service that does not have to include all of the components shown in FIG. 3C.

[0127]

[0147] In some embodiments, especially in embodiments where the resources of the cloud-based object storage 348 are implemented as Amazon S3, the cloud-based storage system 318 may include one or more modules (e.g., modules of computer program instructions that execute on an EC2 instance) configured to ensure that appropriate data is actually in S3 when the local storage of a particular cloud computing instance rehydrates data from S3. The reason this problem occurs is largely because S3 implements a result consistency model in which when overwriting an existing object, the read of the object will eventually become consistent (if not immediately) and will eventually return the overwritten version of the object (if not immediately). To address this problem, in some embodiments of the present disclosure, objects in S3 are never overwritten. Instead, a traditional "overwrite" will result in the creation of a new object (including the updated version of the data) and the eventual deletion of the old object (including the previous version of the data).

[0128]

[0148] In some embodiments of the present disclosure, as part of an attempt to never (or rarely) overwrite an object, when data is written to S3, the resulting object can be tagged with a sequential number. In some embodiments, these sequential numbers can be maintained elsewhere (e.g., within a database) so that at any point in time, the sequential numbers associated with the most recent version of some data pieces can be known. In this way, simply by reading the sequential numbers associated with an object - and without actually reading the data from S3 - a determination can be made as to whether S3 has the most recent version of some data pieces. Since it is not desirable to rehydrate the local storage of an exchange cloud computing instance with old data, this ability to make this determination can be particularly important when a cloud computing instance with local storage crashes. In fact, since the cloud-based storage system 318 does not need to access the data to verify its effectiveness, the data can remain encrypted and access charges can be avoided.

[0129]

[0149] The storage system described above can implement an intelligent data backup technique in which data stored within the storage system can be copied and stored in a separate location in order to avoid data loss when a device failure or some other form of catastrophic situation occurs. For example, the storage system described above can be configured to examine each backup in order to avoid restoring the storage system to an undesirable state. Consider an example in which malware infects the storage system. In such an example, the storage system can include software resource 314 that can scan each backup to identify backups that were taken before the malware infected the storage system and those backups that were taken after the malware infected the storage system. In such an example, the storage system can restore itself from a backup that does not contain malware - or at least, not restore the portion of the backup that contains the malware. In such an example, the storage system can identify the presence of malware (or a virus, or some other undesirable thing) by, for example, identifying write operations originating from a network subnet serviced by the storage system and suspected of sending out the malware, by identifying write operations originating from a user serviced by the storage system and suspected of sending out the malware, by identifying write operations serviced by the storage system and comparing the content of the write operations against the fingerprint of the malware, and in many other ways, and can include software resource 314 that can scan each backup to identify the presence of malware (or a virus, or some other undesirable thing).

[0130]

[0150] The reader will further understand that backups (often in the form of one or more snapshots) may also be utilized to effect a rapid recovery of the storage system. Consider an example where a storage system has been infected with ransomware that locks the user out of the storage system. In such an example, the software resources 314 within the storage system may be configured to detect the presence of the ransomware and further configured to use the stored backup to restore the storage system to a point in time prior to when the ransomware infected the storage system. In such an example, the presence of the ransomware may be explicitly detected through the use of software tools utilized by the system, through the use of a key (e.g., a USB drive) inserted into the storage system, or in a similar manner. Similarly, the presence of the ransomware may be inferred in response to the system activity meeting a predetermined fingerprint, such as reads or writes not being performed on the system for a predetermined period of time.

[0131]

[0151] The reader will understand that the various components described above may be grouped into one or more optimized computing packages as converged infrastructure. Such converged infrastructure may include a pool of computer, storage, and networking resources that are shared by multiple applications and can be managed in an aggregated fashion using policy-driven processes. Such converged infrastructure may be implemented using a converged infrastructure reference architecture, using stand-alone devices, using a software-driven hyper-converged approach (e.g., hyper-converged infrastructure), or in other ways.

[0132]

[0152] The reader will understand that the storage system described above can be useful for supporting various types of software applications. For example, the storage system 306 can be useful for supporting artificial intelligence (「AI」) applications, database applications, DevOps projects, electronic design automation tools, event-driven software applications, high-performance computing applications, simulation applications, high-speed data capture and analysis applications, machine learning applications, media production applications, media serving applications, picture archiving and communication system (「PACS」) applications, software development applications, virtual reality applications, augmented reality applications, and many other types of applications, by providing storage resources to such applications.

[0133]

[0153] The storage system described above can operate to support a wide variety of applications. Considering the fact that the storage system includes computing resources, storage resources, and a wide variety of other resources, the storage system can be well-suited for supporting resource-intensive applications such as, for example, AI applications. AI applications can be deployed in various fields, including predictive maintenance in manufacturing and related fields, healthcare applications such as patient data and risk analysis, retail and marketing deployments (e.g., search advertising, social media advertising), supply chain solutions, fintech solutions such as business analytics and reporting tools, real-time analytics tools, application performance management tools, operational deployments such as IT infrastructure management tools, and many others.

[0134]

[0154] Such AI applications can enable a device to perceive its environment and take actions that maximize the likelihood of success of some goal. Examples of such AI applications can include IBM Watson, Microsoft Oxford, Google DeepMind, Baidu Minwa, and others. The storage system described above may also be well-suited to support other types of resource-intensive applications, such as, for example, machine learning applications. Machine learning applications can perform various types of data analysis to automate analytical model building. By using algorithms that learn iteratively from data, machine learning applications can enable a computer to learn without being explicitly programmed. One particular area of machine learning is called reinforcement learning and involves taking actions suitable to maximize a reward in a particular situation. Reinforcement learning can be employed to discover the best possible behavior or path that a particular software application or machine should take in a particular situation. Reinforcement learning differs from other areas of machine learning (e.g., supervised learning, unsupervised learning) in that correct input / output pairs need not be provided for reinforcement learning and sub-optimal actions need not be explicitly corrected.

[0135]

[0155] In addition to the resources already described, the storage system described above may also sometimes include a graphics processing unit (referred to as a "GPU" (graphics processing unit)), which is sometimes referred to as a visual processing unit (referred to as a "VPU" (visual processing unit)). Such a GPU can be embodied as a special electronic circuit that rapidly manipulates and modifies memory to accelerate the creation of images within a frame buffer intended for output to a display device. Such a GPU can be included, for example, within any of the computing devices that are part of the storage system described above, as one of many individually scalable components of the storage system. Here, other examples of such individually scalable components of the storage system can include storage components, memory components, computing components (e.g., CPUs, FPGAs, ASICs), networking components, software components, and the like. In addition to the GPU, the storage system described above may also include a neural network processor (referred to as an "NNP" (neural network processor)) for use in various aspects of neural network processing. Such an NNP can be used instead of (or in addition to) the GPUs and can also be independently scalable.

[0136]

[0156] As described above, the storage system described herein can be configured to support artificial intelligence applications, machine learning applications, big data analytics applications, and many other types of applications. The rapid growth of these types of applications is driven by three technologies: deep learning (DL), GPU processors, and big data. Deep learning is a computational model that utilizes large-scale parallel neural networks inspired by the human brain. Instead of an expert manually creating software, a deep learning model writes its own software by learning from many examples. Such GPUs can contain thousands of cores well-suited for executing algorithms that roughly represent the parallelism of the human brain.

[0137]

[0157] The progress of deep neural networks has fueled a new wave of algorithms and tools for data scientists to leverage their data using artificial intelligence (AI). With improved algorithms, larger datasets, and various frameworks (including open-source software libraries for machine learning across a wide range of tasks), data scientists are tackling new use cases such as self-driving vehicles, natural language processing and understanding, computer vision, machine inference, strong AI, and many others. Applications of such techniques include machine and vehicle object detection, identification, and avoidance; visual recognition, classification, and tagging; algorithmic financial trading strategy performance management; simultaneous localization and mapping; predictive maintenance of high-value machinery; prevention of cybersecurity threats, automation of expertise; image recognition and classification; question answering; robotics; text analysis (extraction, classification) and text generation and transformation; and many others. The application of AI techniques has been realized in a variety of products, such as the voice recognition technology of Amazon Echo that enables users to speak to their machines, Google Translate (trademark) that enables machine-based language translation, Spotify's Discover Weekly that provides recommendations on new songs and artists likely to be liked by users based on their usage and traffic analysis, Quill's text generation offering that acquires structured data and turns it into a narrative story, chatbots that provide real-time contextually specific responses to interactive questions, and many others.

[0138]

[0158] Data is at the heart of modern AI and deep learning algorithms. Before training can begin, one problem that must be addressed is the collection of labeled data, which is extremely important for training accurate AI models. Full-scale AI deployments may require continuously collecting, cleaning, transforming, labeling, and storing large amounts of data. Adding additional high-quality data points directly leads to more accurate models and better insights. Data samples include, but are not limited to: 1) importing data from external sources into the training system and storing the data in its raw form; 2) cleaning the data and converting it into a convenient format for training, including linking data samples to appropriate labels; 3) exploring parameters and models, quickly testing with smaller datasets, and iterating to converge on the most promising models for pushing into the production cluster; 4) performing a training phase to select a random batch of input data, including both new and older samples, and inputting them into a production GPU server for calculations to update model parameters; and 5) evaluating, including using a held-back portion of data not used in training to evaluate model accuracy based on holdout data. This lifecycle can apply to any type of parallelized machine learning, not just neural networks or deep learning. For example, standard machine learning frameworks may rely on CPUs instead of GPUs, but the data ingestion and training workflows can be the same. The reader will understand that a single shared storage data hub creates a point of coordination throughout the lifecycle that does not require extra data copies between the ingestion, preprocessing, and training phases. It is rare for the ingested data to be used for only one purpose, and shared storage provides flexibility for training multiple different models or applying traditional analytics to the data.

[0139]

[0159] The reader will understand that each stage within the AI data pipeline can have various requirements from a data hub (e.g., a storage system, or a collection of storage systems). A scale-out storage system must exhibit uncompromising performance for every access type and pattern - from small, metadata-rich files to large files, from random access patterns to sequential access patterns, and from low parallelism to high parallelism. Since the system can serve unstructured workloads, the storage system described above can play the role of an ideal AI data hub. In the first stage, to avoid excessive data copying, data is preferably ingested and stored on the same data hub that subsequent stages will use. The next two steps can be performed on a standard computing server optionally including GPUs, and then, in the fourth and final stage, a complete training production job is executed on a powerful GPU-accelerated server. Often, there is a production pipeline running in parallel with an experimental pipeline operating on the same dataset. Additionally, GPU-accelerated servers can be connected to each other, either for independent use for different models, or for training using one larger model, or even across multiple systems for distributed training. If the shared storage layer is slow, then data has to be copied to local storage at each stage, creating wasted time staging the data on different servers. An ideal data hub for an AI training pipeline exhibits similar performance to data stored locally on server nodes, while also having the simplicity and performance to enable all pipeline stages to operate simultaneously.

[0140]

[0160] While the above paragraphs describe deep learning applications, the reader will understand that the storage system described herein may also be part of a distributed deep learning ("DDL") platform to support the execution of DDL algorithms. The storage system described above may also be paired with other technologies, such as TensorFlow, an open source software library for dataflow programming across a range of tasks that can be used for machine learning applications such as neural networks to facilitate such machine learning models, applications, and other developments.

[0141]

[0161] The storage system described above may also be used in a neuromorphic computing environment. Neuromorphic computing is a form of computing that mimics brain cells. To support neuromorphic computing, an architecture of interconnected "neurons" replaces traditional computing models with low-power signals that directly traverse between neurons for more efficient computing. Neuromorphic computing can utilize very-large-scale integration (VLSI) systems that include electronic analog circuits mimicking the neurobiological architectures present in the nervous system, as well as analog, digital, hybrid-mode analog / digital VLSI, and software systems that implement models of neural systems for perception, motor control, or multisensory integration.

[0142]

[0162] The reader will understand that the storage system described above may be configured to support (among other data types) the storage or use of blockchain. In addition to supporting the storage and use of blockchain technology, the storage system described above may also, for example, be used by IBM TMOpen source blockchain and related tools that are part of the Hyperledger project, permissioned blockchains that enable a specific number of trusted parties to access the blockchain, blockchain products that enable developers to build their own distributed ledger projects, and others can also support the storage and use of derivative items. The blockchain and the storage systems described herein can be utilized to support on-chain storage of data and off-chain storage of data.

[0143]

[0163] Off-chain storage of data can be implemented in various ways and can be done when the data itself is not stored within the blockchain. For example, in one embodiment, a hash function can be utilized and the data itself can be put into the hash function to generate a hash value. In such an example, instead of the data itself, the hash of a large data piece can be incorporated within a transaction. The reader will understand that in other embodiments, alternatives to the blockchain can be used to facilitate the decentralized storage of information. For example, one alternative to the blockchain that can be used is a blockweave. Conventional blockchains store every transaction to achieve validity confirmation, while a blockweave enables secure decentralization without using the entire chain, thereby enabling low-cost on-chain storage of data. Such a blockweave can utilize a consensus mechanism based on proof of access (PoA) and proof of work (PoW).

[0144]

[0164] The storage system described above can be used, either alone or in combination with other computing devices, to support in-memory computing applications. In-memory computing requires the storage of information in RAM distributed across a cluster of computers. The reader will understand that the storage system described above, particularly those that can be configured using a customizable amount of processing resources, storage resources, and memory resources (e.g., those systems in which blades include a configurable amount of each type of resource), can be configured in such a way as to provide an infrastructure that can support in-memory computing. Similarly, the storage system described above can include component parts (e.g., NVDIMMs that provide persistent high-speed random access memory, 3D cross-point storage) that can actually provide an improved in-memory computing environment compared to in-memory computing environments that rely on RAM distributed across dedicated servers.

[0145]

[0165] In some embodiments, the storage system described above may be configured to operate as a hybrid in-memory computing environment that includes a universal interface to all storage media (e.g., RAM, flash storage, 3D cross-point storage). In such embodiments, users may not have knowledge of the details of where their data is stored, but they can still use the same complete unified API to address the data. In such embodiments, the storage system can (in the background) move data to the fastest available layer - including intelligently placing the data based on various characteristics of the data or some other rule of thumb. In such an example, the storage system may even utilize existing products such as Apache Ignite and GridGain to move data between the various storage layers, or the storage system may utilize custom software to move data between the various storage layers. The storage systems described herein may implement various optimizations to improve the performance of in-memory computing, such as, for example, causing computations to be performed as close as possible to the data.

[0146]

[0166] The reader will further understand that, depending on the embodiment, the storage system described above may be paired with other resources to support the application described above. For example, one infrastructure may include primary computing in the form of servers and workstations specialized for using general-purpose computing on graphics processing units ( "GPGPU") by a graphics processing unit to accelerate deep learning applications interconnected within a computing engine to train parameters for a deep neural network. Each system may have Ethernet external connectivity, InfiniBand external connectivity, some other form of external connectivity, or some combination of these. In such an example, GPUs may be grouped for a single large training or used independently to train multiple models. The infrastructure may also include a storage system such as those described above to provide a scale-out all-flash file or object store through which data can be accessed via a high-performance protocol such as NFS, S3, etc. The infrastructure may also include, for example, redundant top-of-rack Ethernet switches connected to storage and computing via ports within an MLAG port channel for redundancy. The infrastructure may also include additional computing in the form of a white-box server, optionally with GPUs, for data ingestion, preprocessing, and model debugging. The reader will understand that additional infrastructure is also possible.

[0147]

[0167] The reader will understand that the storage system described above can be configured to support other AI-related tools, either alone or in cooperation with other computers. For example, the storage system can utilize tools such as ONNX or other open neural network exchange formats that make it easier to transfer models written in different AI frameworks. Similarly, the storage system can be configured to support tools such as Amazon's Gluon that enable developers to create, build, and train prototypes of deep learning models. In fact, the storage system described above can be part of a larger platform, including integrated data science, data engineering, and application building services, such as IBM TM Cloud Private for Data.

[0148]

[0168] The reader will further understand that the storage system described above can also be deployed as an edge solution. Such edge solutions can be implemented to optimize cloud computing systems by performing data processing at the edge of the network, near the source of the data. Edge computing can push applications, data, and computing power (i.e., services) from a central point to the logical limits of the network. Through the use of edge solutions such as the storage system described above, computing tasks can be performed using the computing resources provided by such a storage system, data can be stored using the storage resources of the storage system, and cloud-based services can be accessed through the use of various resources of the storage system (including networking resources). By performing computing tasks on the edge solution, storing data on the edge solution, and generally utilizing the edge solution, consumption of expensive cloud-based resources can be avoided, and in fact, performance improvements can be experienced compared to relying more strongly on cloud-based resources.

[0149]

[0169] Many tasks can benefit from the use of edge solutions, but some specific applications may be particularly suitable for deployment in such an environment. For example, devices such as drones, autonomous vehicles, robots, and others may require extremely high-speed processing - so high that, in fact, sending data to and receiving data processing support from a cloud environment can simply become too slow. As an additional example, some IoT devices, such as connected video cameras, may simply be unsuitable for the use of cloud-based resources because (not only from a privacy, security, or economic perspective) it may be impossible to send the data to the cloud due to the sheer volume of data involved. Therefore, many tasks that rely on data processing, storage, or communication may be better suited by a platform that includes an edge solution such as the storage system described above.

[0150]

[0170] The storage system described above can serve as a network edge platform that combines computing resources, storage resources, networking resources, cloud technology, network virtualization technology, etc., either alone or in combination with other computing resources. As part of the network, the edge can exhibit characteristics similar to other network institutions from the customer premise and the backhaul aggregation mechanism to the Point of Presence (PoP) and the regional data center. Readers will understand that network workloads, such as virtualized network functions (VNFs) and others, will exist on the network edge platform. Enabled by the combination of containers and virtual machines, the network edge platform can rely on controllers and schedulers that are no longer geographically co-located with data processing resources. The functions can be divided as microservices into a control plane, user and data planes, or even a state machine, enabling the application of independent optimization and scaling techniques. Such user and data planes can be enabled through increased accelerators, both of which exist within the server platform, such as FPGAs and smart NICs, and through SDN-enabled merchant silicon and programmable ASICs.

[0151]

[0171] The storage system described above can also be optimized for use in big data analytics. Big data analytics can generally be described as a process for investigating large and diverse datasets to reveal hidden patterns, unknown correlations, market trends, customer preferences, and other useful information that can help organizations make more informed business decisions. As part of that process, semi-structured and unstructured data, such as, for example, Internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone call detail records, IoT sensor data, and other data, can be converted into a structured form.

[0152]

[0172] The storage system described above may also support (including implementing as a system interface) applications that perform tasks in response to human speech. For example, the storage system may support the execution of intelligent personal assistant applications such as, for example, Amazon's Alexa, Apple Siri, Google Voice, Samsung Bixby, Microsoft Cortana, and others. The examples described in the previous sentence utilize speech as an input, but the storage system described above may also support chatbots, talkbots, chatterbots, or artificial conversation entities, or other applications configured to conduct conversations via auditory or textual means. Similarly, the storage system may actually execute such applications to enable a user, such as a system administrator, to interact with the storage system via speech. Such applications generally have the ability to provide voice interaction, music playback, to-do list creation, alarm setting, podcast streaming, audio book playback, and other real-time information such as weather, traffic, and news. However, in embodiments according to the present disclosure, such applications may be utilized as an interface to various system management operations.

[0153]

[0173] The above-described storage system may also implement an AI platform to achieve the vision of self-driven storage. Such an AI platform may be configured to provide holistic predictive intelligence by collecting and analyzing a large number of storage system telemetry data points to enable easy management, analysis, and support. In fact, such a storage system has the ability to predict both capacity and performance and generate intelligent advice regarding workload deployment, interaction, and optimization. Such an AI platform may be configured to scan all incoming storage system telemetry data against a library of problem fingerprints in real time to predict incidents before they affect the customer environment and resolve them, and incorporate hundreds of variables related to performance used to predict performance loads.

[0154]

[0174] The above-described storage system may support the serialized or simultaneous execution of artificial intelligence applications, machine learning applications, data analysis applications, data transformation, and other tasks that may collectively form an AI ladder. Such an AI ladder may be effectively formed by combining such elements to form a complete data science pipeline, and there are dependencies between the elements of the AI ladder. For example, AI may require that some form of machine learning has been performed, machine learning may require that some form of analysis has been performed, analysis may require that some form of data and information construction has been performed, and so on. Therefore, each element may be considered a rung in an AI ladder that can collectively form a complete and sophisticated AI solution.

[0155]

[0175] The storage system described above can also be used, either alone or in combination with other computing environments, to provide an AI experience wherever AI penetrates into a wide range of aspects of business and life. For example, AI can play an important role in the provision of deep learning solutions, deep reinforcement learning solutions, general artificial intelligence solutions, self-driving vehicles, cognitive computing solutions, commercial UAVs or drones, conversational user interfaces, enterprise taxonomies, ontology management solutions, machine learning solutions, smart dust, smart robots, smart workplaces, and many others.

[0156]

[0176] The storage system described above can also be used, either alone or in combination with other computing environments, to provide a wide range of pervasive immersion experiences (including those using digital twins of various "things" such as humans, places, processes, systems, etc.) that can bring about permeability between humans, businesses, and things. Such pervasive immersion experiences can be provided as augmented reality technology, connected homes, virtual reality technology, brain-computer interfaces, human augmentation technology, nanotube electronics, volumetric displays, 4D printing technology, or others.

[0157]

[0177] The storage system described above can also be used, either alone or in combination with other computing environments, to support a wide variety of digital platforms. Such digital platforms can include, for example, 5G wireless systems and platforms, digital twin platforms, edge computing platforms, IoT platforms, quantum computing platforms, serverless PaaS, software-defined security, neuromorphic computing platforms, and others.

[0158]

[0178] The storage system described above can also be part of a multi-cloud environment where multiple cloud computing and storage services are deployed within a single heterogeneous architecture. To facilitate the operation of such a multi-cloud environment, DevOps tools can be deployed to enable orchestration across the clouds. Similarly, continuous development and continuous integration tools can be deployed to standardize the processes around continuous integration and delivery, new feature rollouts, and the provisioning of cloud workloads. By standardizing these processes, a multi-cloud strategy can be implemented that enables the best provider utilization for each workload.

[0159]

[0179] The storage system described above can be used as part of a platform that enables the use of crypto anchors to authenticate the origin and content of a product to ensure that it matches the blockchain record associated with the product. Similarly, as part of a suite of tools for protecting the data stored on the storage system, the storage system described above can implement various encryption techniques and methods, including lattice cryptography. Lattice cryptography can involve the construction of cryptographic primitives that include lattices, either in the construction itself or in the proof of security. Unlike public-key schemes such as RSA, Diffie-Hellman, or elliptic curve cryptosystems, which are vulnerable to attack by quantum computers, some lattice-based constructions appear to be resistant to attack by both classical and quantum computers.

[0160]

[0180] A quantum computer is a device that performs quantum computing. Quantum computing is computing that uses quantum mechanical phenomena such as superposition and entanglement. A quantum computer is different from a traditional transistor-based computer because such a traditional computer requires data to be encoded in binary digits (bits), each of which is always in one of two distinct states (0 or 1). In contrast to traditional computers, quantum computers use qubits, which can be in a superposition of states. A quantum computer maintains a series of qubits. Here, a single qubit can represent 1, 0, or any quantum superposition of those two qubit states. A pair of qubits can be in any quantum superposition of four states, and three qubits can be in any superposition of eight states. A quantum computer using n qubits can generally be in any superposition of up to 2^n different states simultaneously, whereas a traditional computer can only be in one of these states at any given time. A quantum Turing machine is a theoretical model of such a computer.

[0161]

[0181] The storage system described above can also be paired with an FPGA-accelerated server as part of a larger AI or ML infrastructure. Such an FPGA-accelerated server can be located near the storage system described above (e.g., within the same data center), or alternatively, can be incorporated within a device that includes one or more storage systems, one or more FPGA-accelerated servers, a networking infrastructure that supports communication between one or more storage systems and one or more FPGA-accelerated servers, and other hardware and software components. Alternatively, the FPGA-accelerated server can exist within a cloud computing environment that is used to perform compute-related tasks for AI and ML jobs. Any of the embodiments described above can be used to collectively serve the role of an FPGA-based AI or ML platform. The reader will understand that depending on the embodiment of the FPGA-based AI or ML platform, the FPGA incorporated within the FPGA-accelerated server can be reconfigured for different types of ML models (e.g., LSTM, CNN, GRU). The ability to reconfigure the FPGA incorporated within the FPGA-accelerated server can enable acceleration of ML or AI applications based on optimal numerical precision and the memory model being used. The reader will understand that by treating a collection of FPGA-accelerated servers as a pool of FPGAs, any CPU within the data center can utilize the pool of FPGAs as a shared hardware micro-service, rather than limiting the server to a dedicated accelerator plugged into it.

[0162]

[0182] The above-described FPGA-accelerated server and GPU-accelerated server can implement a computing model in which machine learning models and parameters are fixed in high-bandwidth on-chip memory and a large amount of data flows through the high-bandwidth on-chip memory, instead of holding a small amount of data in the CPU and flowing a long instruction stream thereon as is done in more traditional computing models. Since an FPGA can be programmed using only the instructions required to execute this type of computational model, the FPGA can be more efficient than a GPU even for this computational model.

[0163]

[0183] The above-described storage system can be configured to provide parallel storage, for example, through the use of a parallel file system such as BeeGFS. Such a parallel file system can include a distributed metadata architecture. For example, a parallel file system can include a plurality of metadata servers where metadata is distributed, as well as components including services for clients and storage servers.

[0164]

[0184] The above-described system can support the execution of various software applications. Such software applications can be deployed in various ways, including a container-based deployment model. Containerized applications can be managed using various tools. For example, containerized applications can be managed using Docker Swarm, Kubernetes, and others. Containerized applications can be used to facilitate serverless, cloud-native computing deployment and management models for software applications. In support of serverless, cloud-native computing deployment and management models for software applications, containers can be used as part of an event processing mechanism (e.g., AWS Lambda), whereby various events spin up the containerized application to act as an event handler.

[0165]

[0185] The above-described system can be deployed in various ways, including being deployed in a manner that supports a fifth-generation ("5G") network. A 5G network can support data communication at significantly higher speeds than previous-generation mobile communication networks, and as a result, modern large-scale data centers may become less prominent and can, for example, be replaced by more local micro data centers located near mobile network towers, resulting in the decentralization of data and computing resources. The above-described system can be included within such local micro data centers and can be part of or paired with a multi-access edge computing ("MEC") system. Such an MEC system can enable cloud computing capabilities and an IT service environment at the edge of a cellular network. By executing applications and performing related processing tasks closer to cellular customers, network congestion can be reduced and applications can operate better.

[0166]

[0186] For further illustration, FIG. 3D shows an exemplary computing device 350 that can be specifically configured to perform one or more of the processes described herein. As shown in FIG. 3D, the computing device 350 can include a communication interface 352, a processor 354, a storage device 356, and an input / output ("I / O") module 358 that are communicatively connected to each other via a communication infrastructure 360. Although an exemplary computing device 350 is shown in FIG. 3D, the components shown in FIG. 3D are not intended to be limiting. In other embodiments, additional or alternative components may be used. Next, the components of the computing device 350 shown in FIG. 3D are described in more detail.

[0167]

[0187] The communication interface 352 may be configured to communicate with one or more computing devices. Examples of the communication interface 352 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0168]

[0188] The processor 354 generally represents any type or form of processing unit having the ability to process data and / or interpret, execute, and / or direct the execution of one or more of the instructions, processes, and / or operations described herein. The processor 354 may perform operations by executing computer-executable instructions 362 (such as applications, software, code, and / or other executable data instances) stored in the storage device 356.

[0169]

[0189] The storage device 356 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, the storage device 356 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data including the data described herein may be stored temporarily and / or permanently within the storage device 356. For example, data representing computer-executable instructions 362 configured to direct the processor 354 to perform any of the operations described herein may be stored within the storage device 356. In some examples, the data may be arranged within one or more databases present within the storage device 356.

[0170]

[0190] The I / O module 358 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 358 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 358 may include, without limitation, hardware and / or software for capturing user input including a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0171]

[0191] The I / O module 358 may include one or more devices for presenting output to the user, including, without limitation, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O module 358 is configured to provide graphical data to a display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content, as may be provided for a particular implementation. By way of example, any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 350.

[0172]

[0192] The advantages and features of the present disclosure may be further illustrated by the following statements.

[0173]

[0193] 1. A method, comprising: detecting, by a data protection system, one or more deletion requests for deleting one or more recovery data sets of a storage system; determining, by the data protection system, that the one or more deletion requests are inconsistent with a recovery data set deletion pattern associated with the storage system; and determining, by the data protection system and based on determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern, that data stored by the storage system may be targeted by a security threat.

[0174]

[0194] 2. The method according to statement 1, wherein the recovery data set deletion pattern is associated with one or more time windows and one or more of a threshold number of retained recovery data sets per time window or a recovery point distance between two consecutive retained recovery data sets per time window.

[0175]

[0195] 3. The method according to statement 1 or 2, wherein the recovery data set deletion pattern is associated with a first time window and a second time window prior to the first time window, the recovery data set deletion pattern is associated with a first threshold number of retained recovery data sets for the first time window and a second threshold number of retained recovery data sets for the second time window, and the first threshold number of retained recovery data sets is greater than the second threshold number of retained recovery data sets.

[0176]

[0196] 4. The method according to any one of statements 1 to 3, wherein the recovery data set deletion pattern is associated with a first time window and a second time window prior to the first time window, the recovery data set deletion pattern is associated with a first recovery point distance between two consecutive retained recovery data sets for the first time window and a second recovery point distance between two consecutive retained recovery data sets for the second time window, and the first recovery point distance is shorter than the second recovery point distance.

[0177]

[0197] 5. Determining that one or more deletion requests are inconsistent with a recovery dataset deletion pattern, determining that one or more deletion requests are attempting to delete a first number of recovery datasets generated within a first time window, determining that one or more deletion requests are attempting to delete a second number of recovery datasets generated within a second time window, where the second time window is before the first time window, and determining that the first number of recovery datasets is greater than the second number of recovery datasets, the method according to any one of statements 1 to 4.

[0178]

[0198] 6. Determining that one or more deletion requests are inconsistent with a recovery dataset deletion pattern, determining that one or more deletion requests are attempting to delete a first number of recovery datasets generated within a time window, determining a second number of recovery datasets that would be retained for the time window when the first number of recovery datasets are deleted, and determining that the second number of recovery datasets that would be retained for the time window is less than a threshold number of retained recovery datasets corresponding to the time window as specified by the recovery dataset deletion pattern, the method according to any one of statements 1 to 5.

[0179]

[0199] 7. Determining that one or more deletion requests are inconsistent with a recovery dataset deletion pattern, identifying one or more recovery datasets retained based on the recovery dataset deletion pattern, and determining that one or more deletion requests are attempting to delete at least one of the one or more retained recovery datasets, the method according to any one of statements 1 to 6.

[0180]

[0200] 8. Further comprising, by a data protection system, performing a corrective action on the storage system in response to determining that data stored by the storage system may be targeted by a security threat, the method according to any one of statements 1 to 7.

[0181]

[0201] 9. Performing a corrective action on a storage system includes identifying one or more recovery data sets retained based on a recovery data set deletion pattern, and converting at least one of the retained one or more recovery data sets into a temporarily protected recovery data set having a higher protection level compared to the retained one or more recovery data sets. The method according to any one of statements 1 to 8.

[0182]

[0202] 10. Determining that one or more additional deletion requests are inconsistent with the recovery data set deletion pattern, and in response to determining that one or more additional deletion requests are inconsistent with the recovery data set deletion pattern, converting the temporarily protected recovery data set into a fully protected recovery data set having an even higher protection level compared to the temporarily protected recovery data set. The method according to any one of statements 1 to 9.

[0183]

[0203] 11. The data protection system determines that one or more additional deletion requests are inconsistent with the recovery data set deletion pattern, and in response to the data protection system determining that one or more additional deletion requests are inconsistent with the recovery data set deletion pattern, the data protection system instructs the storage system to block one or more subsequent deletion requests associated with the data stored by the storage system. The method according to any one of statements 1 to 10.

[0184]

[0204] 12. A system comprising: a memory storing instructions; and a processor communicatively coupled to the memory, the processor configured to execute the instructions and detect one or more deletion requests for deleting one or more recovery data sets of a storage system, determine that the one or more deletion requests are inconsistent with a recovery data set deletion pattern associated with the storage system, and based on determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern, determine that data stored by the storage system may be targeted by a security threat.

[0185]

[0205] 13. The system of statement 12, wherein the recovery data set deletion pattern is associated with one or more time windows and a threshold number of retained recovery data sets per time window, or a recovery point distance between two consecutive retained recovery data sets per time window.

[0186]

[0206] 14. The system of statement 12 or 13, wherein the recovery data set deletion pattern is associated with a first time window and a second time window prior to the first time window, the recovery data set deletion pattern is associated with a first threshold number of retained recovery data sets for the first time window and a second threshold number of retained recovery data sets for the second time window, and the first threshold number of retained recovery data sets is greater than the second threshold number of retained recovery data sets.

[0187]

[0207] 15. The system of any one of statements 12 - 14, wherein the recovery data set deletion pattern is associated with a first time window and a second time window prior to the first time window, the recovery data set deletion pattern is associated with a first recovery point distance between two consecutive retained recovery data sets for the first time window and a second recovery point distance between two consecutive retained recovery data sets for the second time window, and the first recovery point distance is shorter than the second recovery point distance.

[0188]

[0208] 16. Determining that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, determining that one or more deletion requests are attempting to delete a first number of recovery datasets generated within a first time window, determining that one or more deletion requests are attempting to delete a second number of recovery datasets generated within a second time window, where the second time window is before the first time window, and determining that the first number of recovery datasets is greater than the second number of recovery datasets, the system according to any one of statements 12 to 15.

[0189]

[0209] 17. Determining that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, determining that one or more deletion requests are attempting to delete a first number of recovery datasets generated within a time window, determining a second number of recovery datasets that would be retained for the time window when the first number of recovery datasets are deleted, and determining that the second number of recovery datasets that would be retained for the time window is less than the threshold number of retained recovery datasets corresponding to the time window as specified by the recovery dataset deletion pattern, the system according to any one of statements 12 to 16.

[0190]

[0210] 18. Determining that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, identifying one or more recovery datasets retained based on the recovery dataset deletion pattern, and determining that one or more deletion requests are attempting to delete at least one of the one or more retained recovery datasets, the system according to any one of statements 12 to 17.

[0191]

[0211] 19. The system according to any one of statements 12 to 18, further configured such that, in response to the processor executing instructions and determining that data stored by the storage system may be targeted by a security threat, the processor performs corrective actions on the storage system.

[0192]

[0212] 20. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor of a computing device to detect one or more deletion requests to delete one or more recovery data sets of a storage system, determine that the one or more deletion requests are inconsistent with a recovery data set deletion pattern associated with the storage system, and based on determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern, determine that data stored by the storage system may be targeted by a security threat.

[0193]

[0213] In one or more embodiments, in this specification, method steps may be used to describe specific functions and their relationships. The boundaries and orders of these functional building blocks and method steps are arbitrarily defined in this specification for convenience of explanation. Alternative boundaries and orders may be defined as long as the specific functions and relationships are properly performed. Therefore, any such alternative boundaries or orders are included in the scope and spirit of the claims. Further, the boundaries of these functional building blocks are arbitrarily defined for convenience of explanation. Alternative boundaries may be defined as long as the specific important functions are properly performed. Similarly, flow diagram blocks may also be arbitrarily defined in this specification to show specific important functionality.

[0194]

[0214] Within the scope of use, the boundaries and order of the blocks in the flowchart may be defined differently, and yet it would still be possible to perform certain important functionality. Therefore, such alternative definitions of both the functional building blocks and the blocks and order of the flowchart are included within the scope and spirit of the claims. The average person skilled in the art will also recognize that the functional building blocks, as well as the other exemplary blocks, modules, and components herein, can be implemented as illustrated, or by individual components, application-specific integrated circuits, processors executing appropriate software, and the like, or any combination thereof.

[0195]

[0215] Specific combinations of the various functions and features of one or more embodiments are explicitly described herein, but other combinations of these features and functions are equally possible. The present disclosure is not limited by the specific examples disclosed herein and explicitly incorporates these other combinations.

[0196]

[0216] Malicious entities (e.g., hackers, malware, and / or other entities) may gain unauthorized access to storage systems, such as any of the storage systems described herein. Using such access, a malicious entity may attempt to perform operations that affect the capacity of storage structures (e.g., snapshots, volumes, and / or other logical structures within a storage system that contain and / or reference data stored by the storage system).

[0197]

[0217] The methods and systems described herein can mitigate (e.g., prevent, suppress, and / or otherwise address) malicious actions against storage structures. For example, as described herein, a monitoring system can detect that a storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. The monitoring system can identify at least one attribute of the request and the storage system. Based on the attribute, the monitoring system can determine that the request indicates a malicious action. In response to this determination, the monitoring system can perform a corrective action against the requested operation, thereby ensuring that the request does not render the data irrecoverable before the administrator has time to notice and take action. As used herein, a target storage structure is one that is the target of a request that may potentially be malicious in nature.

[0198]

[0218] According to the methods and systems described herein, various advantages and benefits can be realized. For example, by preventing or otherwise mitigating malicious actions against storage structures within a storage system, the methods and systems described herein can result in a storage system that is robust, secure, and efficient in its operation. Additionally, by implementing the monitoring system at the storage level, the methods and systems described herein can provide a last line of defense against malicious attacks in the event that other data security measures taken at levels higher than the storage level (e.g., at the client level) fail to identify and / or prevent malicious ones. This can improve the operation of computing devices at both the storage level and other levels higher than the storage level.

[0199]

[0219] Figure 4 shows an exemplary monitoring system 400 (the "system 400"). As shown, the system 400 may include, without limitation, a memory mechanism 402 and a processing mechanism 404 that are selectively and communicatively coupled to each other. Each of the mechanisms 402 and 404 may include or be implemented by hardware and / or software components (e.g., a processor, memory, communication interface, instructions stored in memory for execution by the processor, etc.). In some examples, the mechanisms 402 and 404 may be distributed among multiple devices and / or multiple locations so as to be provided in a particular implementation form.

[0200]

[0220] The memory mechanism 402 may maintain (e.g., store) executable data used by the processing mechanism 404 to perform any of the operations described herein. For example, the memory mechanism 402 may store instructions 406 executable by the processing mechanism 404 to perform any of the operations described herein. The instructions 406 may be implemented by any suitable application, software, code, and / or other executable data instance.

[0201]

[0221] The memory mechanism 402 may also maintain any data received, generated, managed, used, and / or transmitted by the processing mechanism 404. For example, as shown, the memory mechanism 402 may maintain attribute data 408. The attribute data 408 represents one or more attributes of a request and / or a storage system, as described herein. Additionally, the memory mechanism 402 may maintain any other suitable type of data such as may be provided in a particular implementation form.

[0202]

[0222] The processing mechanism 404 can be configured to perform various processing operations associated with reducing malicious operations on the storage structures within the storage system (e.g., execute instructions 406 stored within the storage mechanism 402 for performing them). For example, the processing mechanism 404 can detect that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. The processing mechanism 404 can identify an attribute of at least one of the request and the storage system. Based on the attribute, the processing mechanism 404 can determine that the request indicates a malicious action. In response to this determination, the processing mechanism 404 can perform a corrective action for the requested operation. These and other operations that can be performed by the system 400 (e.g., the processing mechanism 404) are described herein.

[0203]

[0223] FIG. 5 shows an exemplary configuration 500 in which a storage system 502 can receive a request 504 to perform an operation that affects the capacity of a storage structure within the storage system 502. The storage system 502 can be implemented by any of the storage systems, devices, and / or components described herein. For example, the storage system 502 can be implemented by a local storage system (e.g., a storage system deployed on-site at a customer's facility) and / or by a remote storage system (e.g., a storage system deployed within the cloud).

[0204]

[0224] As shown in the figure, the storage system 502 includes a plurality of storage structures 506 (e.g., storage structures 506-1 to 506-N) and a controller 508. Each storage structure 506 can include any logical structure in which data can be stored internally and / or organized. For example, the storage structure 506 can include one or more snapshots, volumes, file systems, object stores, key values or relational or other databases, backup data sets, objects that manage groups of volumes, container objects, blocks, etc. In some examples, the storage structure 506 is maintained within one or more storage elements (e.g., storage arrays, memories, etc.).

[0205]

[0225] The controller 508 can be configured to control the operation of the elements included within the storage system 502 and can be implemented by any suitable combination of a processor, an operating system, and / or other components as described herein. Specifically, the controller 508 can be configured to generate control data 510 configured to control the storage structure 506. For example, the control data 510 can represent one or more instructions for creating, modifying, writing to, reading from, deleting, eradicating, and / or otherwise interacting with the storage structure 506.

[0206]

[0226] Requirement 504 can be issued from a source (e.g., a host communicating with storage system 502) and can be provided by an entity such as a person or a software application. Requirement 504 can include instructions for the controller 508 to perform one or more operations that affect the capacity of one or more storage structures 506. Such operations can include complete deletion of the storage structure 506, deletion of the storage structure 506, replacement of data within the storage structure 506 with less compressible or non-compressible data (a ransomware attack on the storage structure 506 that can issue a request to the storage system 502, and / or any other operation that destroys, modifies, renders unusable, or otherwise affects the storage structure 506 and / or the original data within the storage structure 506).

[0207]

[0227] In some cases, requirement 504 can be issued from a legitimate source. For example, an administrator of the storage system 506 can log in to the storage system as part of routine maintenance procedures for the storage system 502 and provide requirement 504. However, in other cases, requirement 504 can be issued from a malicious entity such as a hacker who has illegally accessed the storage system 502 and / or malware that has invaded the storage system 502. In these cases, requirement 504 can instruct malicious actions configured to damage or otherwise harm one or more of the storage structure 506 and / or the data within the storage structure 506.

[0208]

[0228] Therefore, as described herein, system 400 is configured to monitor for requirements indicating malicious actions. When a requirement is determined by system 400 to indicate a malicious action, system 400 can perform a corrective action for the requested operation.

[0209]

[0229] In some examples, system 400 is implemented by storage system 502. For example, system 400 can be implemented by controller 508. In this implementation, system 400 can detect that storage system 502 has received request 504 by receiving request 504 using an application executed by controller 508.

[0210]

[0230] Additionally, or alternatively, system 400 can be at least partially implemented by one or more computing devices or systems that are separate from and communicate with storage system 502.

[0211]

[0231] For example, FIG. 6 shows an exemplary configuration 600 in which a cloud-based monitoring system 602 is communicatively coupled to storage system 502 via network 604. The cloud-based monitoring system 602 can at least partially implement system 400.

[0212]

[0232] Network 604 can include the Internet, a wide area network, a local area network, a provider-specific wired or wireless network (e.g., a cable or satellite carrier network or a cellular phone network), a content delivery network, and / or any other suitable network. Data can flow between storage system 502 and the cloud-based monitoring system 604 using any communication technology, device, medium, and protocol so as to be suitable for a particular implementation.

[0213]

[0233] The cloud-based monitoring system 602 can be implemented by one or more server-side computing devices configured to communicate with storage system 502 via network 604. For example, the cloud-based monitoring system 602 can be implemented by one or more servers or other physical computing devices.

[0214]

[0234] The cloud-based monitoring system 602 can be configured to perform one or more remote monitoring operations on the storage system 502. For example, the cloud-based monitoring system 602 can be configured to remotely monitor requests received by the storage system 502 that indicate malicious actions. To achieve this purpose, as shown in the figure, the cloud-based monitoring system 602 can receive a phone-home log 606 from the controller 508 of the storage system 502 via the network 604. The phone-home log 606 can include various types of data that can be used by the cloud-based monitoring system 602 to monitor various types of operations performed by the storage system 502. Specifically, the phone-home log 606 can include data representing the request 504.

[0215]

[0235] As shown in the figure, the cloud-based monitoring system 602 includes an extractor 608 configured to receive the phone-home log 606 and extract from the phone-home log 606 data representing the request 504. The extractor 608 can be implemented by any suitable combination of hardware and software so as to be provided for a particular implementation form.

[0216]

[0236] As shown in the figure, the extractor 608 can generate request data 610 representing the request 504. The request data 610 is processed by the processor 612 of the cloud-based monitoring system 602, which can be configured to determine whether the request 504 is legitimate or indicates a malicious action, by any of the methods described herein.

[0217]

[0237] Processor 612 may return instruction 614 to controller 508 via network 604. Instruction 614 is configured to instruct controller 508 on how to process request 504. For example, if processor 612 determines that request 504 indicates a malicious action, instruction 614 may be configured to instruct controller 508 to refrain from performing the requested operation and / or otherwise prevent target storage structure 506 from being adversely affected.

[0218]

[0238] When system 400 detects that storage system 502 has received request 504, system 400 may identify request 504 and / or one or more attributes of storage system 502. Based on the identified one or more attributes, system 400 may determine whether request 504 indicates a malicious action. Next, various examples are described in which system 400 identifies the attributes of request 504 and / or storage system 502 and, based on the attributes, determines that request 504 indicates a malicious action. Each of the identification and determination operations described herein may be performed alone and / or in combination with one or more of the other identification and determination operations described herein.

[0219]

[0239] In some examples, system 400 may identify the attributes by determining that request 504 is included within a plurality of requests of the same type (e.g., all full deletion requests, all write requests, etc.) received by storage system 502 during a period. In these examples, system 400 may determine that request 504 indicates a malicious action by determining that the plurality of requests received by storage system 502 during the period exceed a threshold.

[0220]

[0240] The threshold value can be determined, set, and / or changed in any suitable manner. For example, the threshold value can be set by an administrator of the storage system 502 by providing user input representing the threshold value to the system 400. In other examples, the threshold value can be automatically determined by the system 400 based on one or more characteristics of the storage system 502. For example, the threshold value can be set to be a percentage of the total number of storage structures 506 within the storage system 502. Similarly, the period during which the total number of requests is counted and compared to the threshold value can be determined (e.g., set) in any suitable manner. For example, an exemplary period can be a few minutes.

[0221]

[0241] As an example, the threshold number of requests can be set to N. Here, N is an integer greater than 0. During a predetermined period, the system 400 can detect that the storage system 502 has received N + 1 requests, all of which are full deletion requests. In response, the system 400 can determine that at least the N + 1th request indicates a malicious action (and, in some cases, all of the N + 1 requests received during the predetermined period indicate malicious actions). In response, the system 400 can perform any of the corrective actions described herein. For example, the system 400 can provide a notification to the user associated with the storage system 502 (e.g., by transmitting a message to the administrator and / or provider of the storage system 502), delay the full deletion of the target storage structure 506 for a predetermined amount of time to allow the administrator to determine whether the request is valid, and / or perform any other suitable action such as subjecting it to a particular implementation.

[0222]

[0242] Additionally, or alternatively, the system 400 may identify an attribute by determining that the request 504 is among a plurality of requests received by the storage system 502 during a period, where the requests are for a plurality of storage structures 506 within the storage system 502. In these examples, the system 400 may determine that the request 504 indicates a malicious action by determining that the number of storage structures 506 exceeds a predetermined ratio compared to the total number of storage structures 506 within the storage system. The predetermined ratio and period in this example may be determined (e.g., set) in any suitable manner.

[0223]

[0243] By way of example, the system 400 may determine that the storage system 502 received requests for M storage structures 506 during a predetermined period. Here, M is an integer greater than 0. The system 400 may also determine that a total of N storage structures 506 are present within the storage system 502 at some point during the predetermined period. If the ratio of M to N is greater than the predetermined ratio, the system 400 may determine that at least some of the requests indicate a malicious action. Accordingly, the system 400 may perform any of the corrective actions described herein.

[0224]

[0244] Additionally, or alternatively, the system 400 may identify an attribute by determining the source of the request 504. In these examples, the system 400 may determine that the request 504 indicates a malicious action by determining that the source is a malicious source.

[0225]

[0245] The system 400 may determine the source of the request 504 in any suitable manner. For example, the system 400 may identify an IP address, login credentials, and / or any other characteristic associated with the request 504.

[0226]

[0246] System 400 can determine that the source is a malicious source in any suitable manner. For example, System 400 can determine that the IP address used to provide Request 504 is abnormal and / or is within a geographical location (e.g., in a different country) that is otherwise on a blacklist. As another example, System 400 can determine that Request 504 was sent from a user using login credentials that are not typically used to fulfill a particular type of request and / or are not used during a particular time period when Request 504 was made. In response to any of these types of determinations, System 400 can perform any of the corrective actions described herein.

[0227]

[0247] In addition or alternatively, System 400 can identify attributes by determining that the request includes a write request (e.g., a request to write data to a particular storage structure 506). In these examples, System 400 can determine that Request 504 indicates a malicious action by determining that the write request includes an attempt to overwrite compressible data within storage structure 506 with incompressible data.

[0228]

[0248] By way of example, Request 504 can be configured to modify storage structure 506 by replacing good data that is typically compressible within storage structure 506 with bad data that is typically incompressible. Therefore, by determining that Request 504 includes an instruction to replace some or all of the compressible data within storage structure 506 with incompressible data, System 400 can determine that Request 504 indicates a malicious action. In response, System 400 can perform any of the corrective actions described herein.

[0229]

[0249] Additionally, or alternatively, the system 400, or the storage system 502, may identify an attribute by determining that it has received a request to change an operation time delay associated with a storage structure 506 within the storage system 502. In these examples, the system 400 may determine that request 504 indicates a malicious action by determining that a request to change the operation time delay was received by the storage system 502 within a predetermined amount of time of request 504. The predetermined amount of time may be determined (e.g., set) in any suitable manner. As used herein, the operation time delay refers to the amount of time that the storage system is configured to wait until it actually performs the requested operation.

[0230]

[0250] By way of example, the storage system 502 may maintain data representing an operation time delay that determines when the storage structure 506 will be operated after receiving a corresponding request. For example, the operation time delay may be set to 24 hours or any other suitable amount of time. This means that in response to receiving a request for a particular storage structure 506, the storage system 502 is configured to wait 24 hours before actually performing the requested operation. In some instances, the operation time delay may vary for different days. For example, the operation time delay may be set to 24 hours for days following a workday (e.g., Monday - Friday), such as days from Sunday - Thursday. However, the operation time delay may be set to 48 or 72 hours for days following a non - workday (e.g., Saturday and Sunday), such as days from Friday - Saturday.

[0231]

[0251] In some cases, the storage system 502, in response to receiving a request, and to the source of the request, may immediately indicate that the operation has been performed, but may not actually perform the operation until the completion of the operation time delay.

[0232]

[0252] The operation time delay can be configured to allow an administrator or other user to determine whether a particular requested operation should actually be performed. Therefore, a malicious entity may attempt to change the operation time delay by setting it to zero or any other relatively short amount of time. In this way, a malicious entity may attempt to prevent the administrator or other user from reviewing the request and / or having time to prevent it from being implemented.

[0233]

[0253] Thus, by determining that a request to change the operation time delay was received by the storage system 502 within a predetermined amount of time of request 504 (e.g., within a few minutes before request 504 is received by the storage system 502), the system 400 may determine that request 504 indicates a malicious action. In response, the system 400 may perform any of the corrective actions described herein.

[0234]

[0254] In some examples, different storage structures 506 within the storage system 502 may have different associated operation time delays. For example, storage structure 506-1 may have an associated operation time delay of 24 hours, while storage structure 506-2 may have an associated operation time delay of 48 hours. In some examples, the operation time delay may be associated with (i.e., fixed to) the storage structure 506 at the time the storage structure 506 is created. For example, if the operation time delay setting for the entire system specifies a 24-hour operation time delay when the storage structure 506 is created, this 24-hour operation time delay may be associated with the storage structure 506 regardless of future changes to the operation time delay setting. In these cases, the operation time delay for the storage structure 506 cannot be changed, even if a malicious entity changes the operation time delay setting to a shorter time (e.g., zero minutes). The particular operation time delay may be associated with the storage structure 506 in any other suitable manner.

[0235]

[0255] Additionally, or alternatively, system 400 may identify an attribute by detecting an abnormal pattern of interaction with storage system 502 during a period. In these examples, system 400 may determine that request 504 indicates a malicious action by determining that the requests were received by the storage system during the period. The period may be determined (e.g., set) in any suitable manner.

[0236]

[0256] System 400 may detect an abnormal pattern of interaction with storage system 502 in any suitable manner. For example, system 400 may detect an abnormal pattern of interaction with storage system 502 by determining that the operations performed on storage system 502 during a period are different from the historical operations performed on storage system 502 by a threshold amount. Such a determination may be made in any suitable manner, such as by comparing a metric associated with the operation to a threshold, using a machine learning heuristic, etc.

[0237]

[0257] By way of example, system 400 may detect a sudden increase in the writing of non-compressible and / or non-deduplicable data to storage structure 506 within storage system 502 within a relatively short amount of time. This sudden increase may be different from the historical average for writing of such types of data by a threshold amount. Accordingly, system 400 may perform any of the corrective actions described herein.

[0238]

[0258] Additionally, or alternatively, system 400 may identify an attribute by determining the elapsed time of other storage structures 506 within storage system 502 (i.e., storage structures 506 other than the particular storage structure 506 that is the target of request 504). In these examples, system 400 may determine that request 504 indicates a malicious action by determining that the elapsed time is older than a predetermined elapsed time.

[0239]

[0259] For illustration purposes, the system 400 may determine that the request 504 relates to a relatively new (e.g., recently created) target snapshot, while all other snapshots of the same data are relatively old (e.g., created relatively long before the target snapshot). In this case, the performance of operations that affect the capacity of the target snapshot may result in at least some data becoming irrecoverable. Thus, the system 400 may determine that the request 504 exhibits a malicious action and may perform any of the corrective actions described herein.

[0240]

[0260] In addition, or alternatively, the system 400 may identify the attribute by determining the amount of the non-disruptive capacity of the storage system 502, where the non-disruptive capacity is not affected by a plurality of requests including the request 504. In these examples, the system 400 may determine that the request 504 exhibits a malicious action by determining that the non-disruptive capacity is less than a threshold. The threshold may be determined (e.g., set) in any suitable manner.

[0241]

[0261] For illustration purposes, the system 400 may determine that a plurality of requests 504 affect a relatively large number of storage structures 506 compared to the remaining capacity of the storage system 502. This may indicate an attempt by a malicious entity to overwhelm the storage system 502 and / or force the storage system 506 to actually change (e.g., completely delete) the storage structure, for example, without waiting for the operation time delay until the storage structure is changed. Thus, the system 400 may determine that the request 504 exhibits a malicious action and may perform any of the corrective actions described herein.

[0242]

[0262] Additionally, or alternatively, the system 400 may identify attributes by determining that the target storage structure 506 is flagged as being a ransomware recovery structure. In these examples, the system 400 may determine that the request 504 indicates a malicious action by determining that the request 504 is for a particular storage structure 506 that is flagged as being a ransomware recovery structure.

[0243]

[0263] For example, the storage system 502 may be configured to periodically (e.g., once a day, once a week, or at any other suitable time interval) create a ransomware recovery structure (e.g., a ransomware recovery snapshot). These ransomware recovery structures may have one or more attributes that instruct legitimate users that such storage structures should not be deleted, fully deleted, or modified. Thus, if the system 400 detects an attempt to delete, fully delete, or modify such a storage structure, the system 400 may use one or more attributes to determine that the attempt is malicious.

[0244]

[0264] In some examples, the storage system 502 may require that a particular sequence of operations be performed to delete, fully delete, or modify a storage structure that is flagged as being a ransomware recovery structure. For example, the storage system 502 may require data from multiple sources to delete, fully delete, or modify the storage structure. Such data may include, but is not limited to, one or more decryption keys from multiple sources. Thus, if the system 400 detects a request for such a storage structure that does not include the required data from multiple sources, the system 400 may determine that the request indicates a malicious action. Accordingly, the system 400 may perform any of the corrective actions described herein.

[0245]

[0265] In some examples, the storage structure designated as a ransomware recovery structure may be read-only, which can make attempts to delete, fully erase, or otherwise modify the storage structure impossible. One or more read-only ransomware recovery structures are created along with one or more associated metadata catalogs and can then be used to directly recover data.

[0246]

[0266] Next, various corrective actions that can be performed by system 400 in response to determining that the request indicates a malicious action are described. System 400 can perform one or more of the corrective actions described herein in any suitable combination.

[0247]

[0267] In some examples, system 400 can perform a corrective action by providing a notification indicating that the request indicates a malicious action. The notification can be in any suitable format. For example, the notification can include a message (e.g., a text message and / or an email), a notification within a user interface used by a user (e.g., an administrator) to manage storage system 502, a phone call, and / or any other suitable type of notification that can be provided in a particular implementation.

[0248]

[0268] In addition or alternatively, system 400 can perform a corrective action by instructing storage system 502 to refrain from actually performing the requested operation for a predetermined time after storage system 502 receives the request. As described herein, this predetermined time can be specified by an operation time delay setting and / or any other suitable type of setting that can be provided in a particular implementation.

[0249]

[0269] In some cases, for security reasons, the system 400 may encrypt data included within a target storage structure 506 that is targeted for deletion, complete deletion, or modification but is waiting until the completion of an operational delay before being deleted, completely deleted, or modified. Such encryption may ensure that confidential data within the target storage structure 506 cannot be easily accessed during the operational delay. In some cases, the encrypted data may require multiple keys from different sources in order to be decrypted. Such sources may include, but are not limited to, a cloud-based monitoring system 602 and / or one or more other independent servers and / or systems that may be provided for a particular implementation.

[0250]

[0270] In addition, or alternatively, the system 400 may perform a corrective action by instructing the storage system 502 to refrain from actually performing a requested operation (e.g., completely deleting the target storage structure 506) until the garbage collection process is ready to be performed on the storage structure 506. In this way, the target storage structure 506 may be made recoverable by a system administrator between the time of the request and the time when the garbage collection process is scheduled to be performed.

[0251]

[0271] In addition, or alternatively, the system 400 may perform a corrective action by blocking the request, suppressing the performance of the requested operation in response to the request (e.g., slowing down), and / or disabling the storage system 502 (e.g., by taking the storage system 502 offline).

[0252]

[0272] FIG. 7 shows an exemplary method 700. FIG. 7 shows exemplary operations according to one embodiment, but other embodiments may omit, add to, rearrange, and / or modify any of the operations shown in FIG. 7. One or more of the operations shown in FIG. 7 may be performed by the system 400, any component included therein, and / or any implementation thereof.

[0253]

[0273] In operation 702, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 702 can be performed by any of the methods described herein.

[0254]

[0274] In operation 704, the monitoring system identifies at least one attribute of the request and the storage system. Operation 704 can be performed by any of the methods described herein.

[0255]

[0275] In operation 706, the monitoring system determines, based on the attribute, that the request indicates a malicious action. Operation 706 can be performed by any of the methods described herein.

[0256]

[0276] In operation 708, the monitoring system performs a corrective action on the requested operation in response to the determination that the request indicates a malicious action. Operation 708 can be performed by any of the methods described herein.

[0257]

[0277] FIG. 8 shows a particular exemplary method 800 that includes operations that can be performed in a monitoring system 400, any component included therein, and / or any implementation thereof.

[0258]

[0278] In operation 802, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 802 can be performed by any of the methods described herein.

[0259]

[0279] In operation 804, the monitoring system determines that the request is included within a plurality of requests of the same type received by the storage system during the period. Operation 804 can be performed by any of the methods described herein.

[0260]

[0280] In operation 806, the monitoring system determines that the plurality of requests exceeds a threshold. Operation 806 can be performed by any of the methods described herein.

[0261]

[0281] In operation 808, in response to determining that the plurality of requests exceeds a threshold, the monitoring system performs a corrective action for the requested operation. Operation 808 can be performed by any of the methods described herein.

[0262]

[0282] FIG. 9 shows another specific method 900 that includes operations that can be performed in the monitoring system 400, any component included therein, and / or any implementation thereof.

[0263]

[0283] In operation 902, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 902 can be performed by any of the methods described herein.

[0264]

[0284] In operation 904, the monitoring system determines that the request is included within a plurality of requests received by the storage system during the period, where the request is for a large number of storage structures within the storage system. Operation 904 can be performed by any of the methods described herein.

[0265]

[0285] In operation 906, the monitoring system determines that the number of storage structures exceeds a predetermined ratio compared to the total number of storage structures in the storage system. Operation 906 can be performed by any of the methods described herein.

[0266]

[0286] In operation 908, the monitoring system performs a corrective action on the requested operation in response to determining that the number of storage structures exceeds a predetermined ratio compared to the total number of storage structures in the storage system. Operation 908 can be performed by any of the methods described herein.

[0267]

[0287] FIG. 10 shows another specific method 1000 that includes operations that can be performed in the monitoring system 400, any components included therein, and / or any implementation thereof.

[0268]

[0288] In operation 1002, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of the storage structures in the storage system. Operation 1002 can be performed by any of the methods described herein.

[0269]

[0289] In operation 1004, the monitoring system determines the source of the request. Operation 1004 can be performed by any of the methods described herein.

[0270]

[0290] In operation 1006, the monitoring system determines that the source is a malicious source. Operation 1006 can be performed by any of the methods described herein.

[0271]

[0291] In operation 1008, the monitoring system performs a corrective action on the requested operation in response to determining that the source is a malicious source. Operation 1008 can be performed by any of the methods described herein.

[0272]

[0292] Figure 11 shows another specific method 1100 that includes the monitoring system 400, any components included therein, and / or operations that may be performed in any implementation thereof.

[0273]

[0293] In operation 1102, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of the storage structure within the storage system. Operation 1102 may be performed by any of the methods described herein.

[0274]

[0294] In operation 1104, the monitoring system determines that the request includes a write request. Operation 1104 may be performed by any of the methods described herein.

[0275]

[0295] In operation 1106, the monitoring system determines that the write request includes an attempt to overwrite compressible data within the storage structure with non-compressible data. Operation 1106 may be performed by any of the methods described herein.

[0276]

[0296] In operation 1108, in response to the determination that the write request includes an attempt to overwrite compressible data within the storage structure with non-compressible data, the monitoring system performs a corrective action for the requested operation. Operation 1108 may be performed by any of the methods described herein.

[0277]

[0297] Figure 12 shows another specific method 1200 that includes the monitoring system 400, any components included therein, and / or operations that may be performed in any implementation thereof.

[0278]

[0298] In operation 1202, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 1202 can be performed in any of the ways described herein.

[0279]

[0299] In operation 1204, the monitoring system determines that the storage system has received a request to change an operation time delay associated with a storage structure within the storage system. Operation 1204 can be performed in any of the ways described herein.

[0280]

[0300] In operation 1206, the monitoring system determines that the request to change the operation time delay was received by the storage system within a predetermined amount of time of the request. Operation 1206 can be performed in any of the ways described herein.

[0281]

[0301] In operation 1208, in response to determining that the request to change the operation time delay was received by the storage system within a predetermined amount of time of the request, the monitoring system performs a corrective action for the requested operation. Operation 1208 can be performed in any of the ways described herein.

[0282]

[0302] FIG. 13 shows another specific method 1300 that includes operations that can be performed in a monitoring system 400, any components included therein, and / or any implementation thereof.

[0283]

[0303] In operation 1302, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 1302 can be performed in any of the ways described herein.

[0284]

[0304] In operation 1304, the monitoring system detects an abnormal pattern of interaction with the storage system during a period. Operation 1304 can be performed by any of the methods described herein.

[0285]

[0305] In operation 1306, the monitoring system determines that a request was received by the storage system during a period. Operation 1306 can be performed by any of the methods described herein.

[0286]

[0306] In operation 1308, in response to the determination that a request was received by the storage system during a period, the monitoring system performs a corrective action for the requested operation. Operation 1308 can be performed by any of the methods described herein.

[0287]

[0307] FIG. 14 shows another specific method 1400 that includes operations that can be performed in a monitoring system 400, any component included therein, and / or any implementation thereof.

[0288]

[0308] In operation 1402, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 1402 can be performed by any of the methods described herein.

[0289]

[0309] In operation 1404, the monitoring system determines the elapsed time of another storage structure within the storage system. Operation 1404 can be performed by any of the methods described herein.

[0290]

[0310] In operation 1406, the monitoring system determines that the elapsed time is older than a predetermined elapsed time. Operation 1406 can be performed by any of the methods described herein.

[0291]

[0311] In operation 1408, the monitoring system performs a corrective action on the requested operation in response to a determination that the elapsed time is older than a predetermined elapsed time. Operation 1408 can be performed by any of the methods described herein.

[0292]

[0312] FIG. 15 shows another specific method 1500 that includes operations that can be performed in the monitoring system 400, any components included therein, and / or any implementation thereof.

[0293]

[0313] In operation 1502, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 1502 can be performed by any of the methods described herein.

[0294]

[0314] In operation 1504, the monitoring system determines an amount of non-disruptive capacity of the storage system, where the non-disruptive capacity is not affected by a plurality of requests including the request. Operation 1504 can be performed by any of the methods described herein.

[0295]

[0315] In operation 1506, the monitoring system determines that the non-disruptive capacity is less than a threshold. Operation 1506 can be performed by any of the methods described herein.

[0296]

[0316] In operation 1508, the monitoring system performs a corrective action on the requested operation in response to a determination that the non-disruptive capacity is less than a threshold. Operation 1508 can be performed by any of the methods described herein.

[0297]

[0317] FIG. 16 shows another specific method 1600 that includes operations that can be performed in the monitoring system 400, any components included therein, and / or any implementation thereof.

[0298]

[0318] In operation 1602, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 1602 can be performed by any of the methods described herein.

[0299]

[0319] In operation 1604, the monitoring system determines that the storage structure is flagged as being a ransomware recovery structure. Operation 1604 can be performed by any of the methods described herein.

[0300]

[0320] In operation 1606, the monitoring system determines that the request is for a specific storage structure that is flagged as being a ransomware recovery structure. Operation 1606 can be performed by any of the methods described herein.

[0301]

[0321] In operation 1608, in response to determining that the request is for a specific storage structure that is flagged as being a ransomware recovery structure, the monitoring system performs a corrective action for the requested operation. Operation 1608 can be performed by any of the methods described herein.

[0302]

[0322] FIG. 17 shows another specific method 1700 that includes operations that can be performed in the monitoring system 400, any component included therein, and / or any implementation thereof.

[0303]

[0323] In operation 1702, the monitoring system detects that the storage system has received a request to perform an operation that affects the capacity of a storage structure within the storage system. Operation 1702 can be performed by any of the methods described herein.

[0304]

[0324] In operation 1704, the monitoring system identifies at least one attribute of the request and the storage system. Operation 1704 can be performed by any of the methods described herein.

[0305]

[0325] In decision 1706, the monitoring system determines whether the attribute indicates a malicious action. Decision 1706 can be performed by any of the methods described herein.

[0306]

[0326] If decision 1706 is "yes", the monitoring system performs a corrective action on the requested operation (operation 1708). Operation 1708 can be performed by any of the methods described herein.

[0307]

[0327] If decision 1706 is "no", the monitoring system instructs the storage system to perform the requested operation (operation 1710).

[0308]

[0328] FIG. 18 shows an exemplary recovery dataset deletion-based security threat detection method 1800 that can be performed by system 400 and / or any implementation thereof. In some embodiments, system 400 performs one or more operations of method 1800 and / or instructs storage system 502 to perform one or more operations of method 1800. Method 1800 can be used alone or in combination with any of the other security threat detection methods described herein.

[0309]

[0329] In operation 1802, system 400 can detect one or more deletion requests for deleting one or more recovery datasets of storage system 502. For example, storage system 502 can receive one or more deletion requests from the same source or multiple sources, and the one or more deletion requests can request that one or more recovery datasets for data stored within storage system 502 be deleted.

[0310]

[0330] In some embodiments, the recovery data set may include a snapshot, a backup copy, a backup image, an ordered log of metadata describing an ordered application of updates to data maintained by the storage system 502, and / or other data structures that can be used to restore the data of the storage system 502 to an intact state at a specific point in time, such as the time of recovery when the recovery data set was generated. In some embodiments, the storage system 502 and / or the system 400 may generate a recovery data set for the data stored within the storage system 502 at a predefined backup interval. For example, the storage system 502 and / or the system 400 may generate a recovery data set for the data stored within the storage system 502 every hour. Therefore, a recovery data set representing the version of the data for that time may be generated every hour. The recovery data set may be generated for all the data stored by the storage system 502, the data stored on a specific storage structure (e.g., volume, directory) of the storage system 502, the data stored within the storage system 502 associated with a specific host, and / or any other partial set of the data stored within the storage system 502.

[0311]

[0331] In operation 1804, the system 400 may determine that one or more deletion requests to delete one or more recovery data sets of the storage system 502 are inconsistent with the recovery data set deletion pattern associated with the storage system 502.

[0312]

[0332] According to some embodiments, the recovery dataset deletion pattern may indicate a pattern in which one or more of the multiple recovery datasets in the storage system 502 are deleted in order to reduce a large number of recovery datasets maintained for the storage system 502 without causing security threats, data corruption, or other adverse effects on the storage system 502. Thanks to the deletion of one or more recovery datasets, the storage system 502 can avoid unnecessarily storing a large number of recovery datasets, thereby reducing the storage space occupied by the recovery datasets of the storage system 502. According to some embodiments, the deletion of one or more recovery datasets of the storage system 502 to reduce the number of a large number of recovery datasets maintained for the storage system 502 without causing security threats, data corruption, or other adverse effects on the storage system 502 may be referred to as a pruning operation. According to some embodiments, the pruning operation may be performed for the storage system 502 at a predetermined pruning interval (e.g., every 24 hours).

[0313]

[0333] According to some embodiments, the recovery dataset deletion pattern may be established and implemented on the storage system 502, and the system 400 may obtain the recovery dataset deletion pattern from the storage system 502. Additionally, or alternatively, the system 400 may monitor various operations (e.g., creation operations, deletion operations, copy operations, etc.) associated with one or more recovery datasets of the storage system 502 and determine the recovery dataset deletion pattern based on the operations. According to some embodiments, the recovery dataset deletion pattern may be associated with one or more past time windows (also referred to herein as time windows) prior to the current time. According to the recovery dataset deletion pattern, each time window may be associated with a threshold number of retained recovery datasets for the time window and / or a recovery point distance between two consecutive retained recovery datasets for the time window. According to some embodiments, there may be various recovery dataset deletion patterns in which recovery datasets are deleted to perform the pruning operation for the storage system 502.

[0314]

[0334] In some embodiments, the system 400 may determine that one or more deletion requests to delete one or more recovery data sets of the storage system 502 are inconsistent with the recovery data set deletion pattern as described herein. For example, the recovery data set deletion pattern may specify a first threshold number of retained recovery data sets for a first time window and a second threshold number of retained recovery data sets for a second time window prior to the first time window. The first threshold number of retained recovery data sets corresponding to the first time window may be greater than the second threshold number of retained recovery data sets corresponding to the second time window. Therefore, according to the recovery data set deletion pattern, since the first time window is closer to the current time than the second time window, the storage system 502 may retain a greater number of recovery data sets for the first time window than for the second time window.

[0315]

[0335] However, in contrast to the recovery data set deletion pattern, the system 400 may determine that one or more deletion requests are attempting to delete a first number of recovery data sets generated within the first time window and a second number of recovery data sets generated within the second time window, and that the first number of recovery data sets is greater than the second number of recovery data sets. Therefore, instead of retaining a greater number of recovery data sets for the first time window, which is closer to the current time than the second time window, one or more deletion requests may attempt to delete a greater number of recovery data sets corresponding to the first time window compared to the second time window. Thus, the system 400 may determine that one or more deletion requests are inconsistent with the recovery data set deletion pattern associated with the storage system 502. Other scenarios where one or more deletion requests are inconsistent with the recovery data set deletion pattern associated with the storage system 502 are possible and contemplated.

[0316]

[0336] In operation 1806, based on determining that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, system 400 may determine that the data stored by storage system 502 may be targeted by a security threat. This may be accomplished by any of the methods described herein.

[0317]

[0337] In response to determining that the data stored by storage system 502 may be targeted by a security threat, system 400 may perform a corrective action on storage system 502. Additionally, or alternatively, system 400 may instruct storage system 502 to perform a corrective action. The corrective action may include any of the corrective actions described herein.

[0318]

[0338] As an example of performing a corrective action, system 400 may identify one or more recovery datasets maintained based on a recovery dataset deletion pattern and instruct storage system 502 to convert at least one of these recovery datasets to a temporarily protected recovery dataset having a higher level of protection compared to the recovery datasets maintained for storage system 502. In some embodiments, the temporarily protected recovery dataset may require additional authorization from one or more authenticated entities (e.g., an authenticated administrator, a third - party AI engine, etc.) for deletion or modification compared to a regular recovery dataset (e.g., one or more recovery datasets maintained for storage system 502 according to a recovery dataset deletion pattern). Other types of corrective actions are possible and contemplated.

[0319]

[0339] Therefore, system 400 can detect a potential security threat to the data stored by storage system 502 based on an inconsistency between a deletion request attempting to delete one or more recovery data sets of storage system 502 and a recovery data set deletion pattern that is expected to conform to the deletion of recovery data sets to reduce the number of recovery data sets stored for storage system 502 in an appropriate manner. In some embodiments, the recovery data set deletion pattern can be associated with one or more past time windows prior to the current time. Each past time window (also referred to herein as a time window) can be in the past, can extend from a start time to an end time, and the end time of the time window can coincide with or precede the current time.

[0320]

[0340] FIG. 19A shows FIG. 1900 illustrating an exemplary time window associated with a recovery data set deletion pattern. As shown in FIG. 19A, the recovery data set deletion pattern can specify a series of time windows 1910-1... 1910-n (collectively referred to as time window 1910) prior to a particular time t0. The particular time t0 can be, for example, the current time, the pruning time at which a pruning operation to delete one or more recovery data sets from the recovery data sets generated for storage system 502 is performed, etc. Other types of particular times t0 are possible and contemplated.

[0321]

[0341] According to some embodiments, a series of time windows may include a plurality of time windows 1910 arranged in chronological order. Each pair of consecutive time windows 1910 within the series of time windows (e.g., time windows 1910-2 and 1910-1) may include a previous time window 1910 and a subsequent time window 1910 that continuously follows the previous time window 1910 without having other time windows in between. According to some embodiments, the end time of the previous time window 1910 may coincide with the start time of the subsequent time window 1910. Alternatively, there may be a small time gap (e.g., 1 s or less) between the end time of the previous time window 1910 and the start time of the subsequent time window 1910. For example, the previous time window 1910 may extend from a start time of 12:00:00 to an end time of 14:00:00 on a specific date, and the subsequent time window 1910 may extend from a start time of 14:00:01 to an end time of 15:30:00 on the same specific date. Therefore, the subsequent time window 1910 may start when the previous time window 1910 ends, and the subsequent time window 1910 and the previous time window 1910 may not overlap. According to some embodiments, the previous time window 1910 may be considered to continuously precede or be earlier than the subsequent time window 1910 within the series of time windows, and the subsequent time window 1910 may be considered to continuously follow or be later than the previous time window 1910 within the series of time windows.

[0322]

[0342] According to some embodiments, the start time of the first time window 1910 within a series of time windows may be after the end time of the second time window 1910 within the series of time windows (the first time window 1910 and the second time window 1910 may or may not be consecutive time windows). Therefore, the second time window 1910 may be before the first time window 1910, and the first time window 1910 may be after the second time window 1910. Thus, the first time window 1910 may be closer to a specific time t0 (e.g., the current time, pruning time) than the second time window 1910. For example, as shown in FIG. 19A, the time window 1910-2 may be closer to a specific time t0 than the time windows 1910-n and 1910-3. Similarly, the time window 1910-1 may be closer to a specific time t0 than the time windows 1910-n, 1910-3, and 1910-2. The closeness of the time window 1910 to a specific time t0 (e.g., the current time, pruning time) may be referred to as the recency of the time window 1910.

[0323]

[0343] According to some embodiments, the recency of the time window 1910 relative to the recency of other time windows 1910 may be indicated by a recency value of the time window 1910. For example, the time window 1910 closest to a specific time t0 (e.g., the time window 1910-1) may have the highest recency value, the time window 1910 furthest in the past relative to a specific time t0 (e.g., the time window 1910-n) may have the lowest recency value, and the first time window 1910 that is closer to a specific time t0 than the second time window 1910 may have a higher recency value than the second time window 1910.

[0324]

[0344] Depending on the embodiment, each time window 1910 within a series of time windows may have a duration that indicates the amount of time between the start time and the end time of the time window 1910. Depending on the embodiment, various time windows 1910 within a series of time windows may have the same duration or different durations. Depending on the embodiment, the duration of the time window 1910 may depend on the recency of the time window 1910. For example, the most recent time window 1910 (e.g., time window 1910-1) for a particular time t0 may have the shortest duration, the most past time window 1910 (time window 1910-n) for a particular time t0 may have the longest duration, and the first time window 1910 that is more recent than a second time window 1910 for a particular time t0 may have a shorter duration than the second time window 1910.

[0325]

[0345] For example, as shown in FIG. 19A, the time window 1910-1 may extend from a start time at timestamp t1 to an end time at a particular time t0. The time window 1910-1 may have a duration of 12 hours between timestamp t1 and the particular time t0. Thus, timestamp t1 may be equal to (particular time t0 - 12 hours).

[0326]

[0346] As shown in FIG. 19A, the time window 1910-2 that continuously precedes the time window 1910-1 may extend from a start time at timestamp t2 to an end time at timestamp t1. The time window 1910-2 may have a duration of 12 hours between timestamp t2 and timestamp t1. Thus, timestamp t2 may be equal to (timestamp t1 - 12 hours), and thus may be equal to (particular time t0 - 24 hours).

[0327]

[0347] As shown in FIG. 19A, the time window 1910-3 that continuously precedes the time window 1910-2 can extend from the start time at the time stamp t3 to the end time at the time stamp t2. The time window 1910-3 can have a duration of two days (e.g., 48 hours) between the time stamp t3 and the time stamp t2. Thus, the time stamp t3 is equal to (time stamp t2 - 48 hours), and thus can be equal to (specific time t0 - 72 hours) or (specific time t0 - 3 days).

[0328]

[0348] As shown in FIG. 19A, the time window 1910-4 that continuously precedes the time window 1910-3 can extend from the start time at the time stamp t4 to the end time at the time stamp t3. The time window 1910-4 can have a duration of four days (e.g., 96 hours) between the time stamp t4 and the time stamp t3. Thus, the time stamp t4 is equal to (time stamp t3 - 96 hours), and thus can be equal to (specific time t0 - 168 hours) or (specific time t0 - 7 days).

[0329]

[0349] In this example, a series of time windows can include five time windows 1910, and the most past time window 1910 (e.g., time window 1910-n) can extend from the start time at the starting time stamp (e.g., the starting time stamp can be the initial point of the timeline associated with the storage system 502) to the end time at the time stamp t4. Thus, the time window 1910-n can cover the entire period before the time stamp t4, while the seven-day period between the time stamp t4 and the specific time t0 can be collectively covered by the time windows 1910-1, 1910-2, 1910-3, and 1910-4.

[0330]

[0350] Therefore, in this example, time windows 1910-1 and 1910-2 may have the same 12-hour duration, while time window 1910-3 may have a 2-day duration, time window 1910-4 may have a 4-day duration, and time window 1910-n may have a significantly long duration. Thus, as the recency value of time window 1910 decreases, the duration of time window 1910 may increase.

[0331]

[0351] In some embodiments, each time window 1910 within a series of time windows may correspond to one or more recovery data sets of the storage system 502. As shown in FIG. 19A, the storage system 502 and / or system 400 may generate one or more recovery data sets 1920-1... 1920-z (collectively referred to as recovery data sets 1920) for the data stored within the storage system 502. As described herein, the recovery data sets 1920 may be generated at a predefined backup interval (e.g., every hour). Each recovery data set 1920 may correspond to a recovery timestamp (also referred to herein as "recovery time" or "recovery point in time") at which the recovery data set 1920 was generated and may be used to restore the data of the storage system 502 to the state of the data at the recovery timestamp. In some embodiments, the one or more recovery data sets 1920 may be organized in chronological order of their recovery timestamps. For example, the first recovery data set 1920 may correspond to a first recovery timestamp, and the second recovery data set 1920 may correspond to a second recovery timestamp that is later than the first recovery timestamp. In this case, the second recovery data set 1920 may be considered to be more recent or newer than the first recovery data set 1920, and the first recovery data set 1920 may be considered to be older or past relative to the second recovery data set 1920. Therefore, the recovery timestamps of the recovery data sets 1920 may represent the relative elapsed time of the recovery data sets 1920.

[0332]

[0352] In some embodiments, one or more recovery data sets 1920 may be generated at one or more recovery timestamps within a particular time window 1910, and thus, the one or more recovery data sets 1920 may be considered to correspond to the particular time window 1910. Therefore, the recovery data set 1920 corresponding to the particular time window 1910 may include the recovery data set 1920 generated at the recovery time within the particular time window 1910.

[0333]

[0353] In some embodiments, each time window 1910 may be associated with a threshold number of retained recovery data sets and a recovery point distance according to a recovery data set deletion pattern. For example, the recovery data set deletion pattern may specify the threshold number of retained recovery data sets and the recovery point distance for each time window 1910.

[0334]

[0354] In some embodiments, the threshold number of retained recovery data sets associated with the time window 1910 may indicate the number of recovery data sets 1920 retained for the time window 1910 according to the recovery data set deletion pattern. To conform to the recovery data set deletion pattern, the deletion of one or more recovery data sets 1920 corresponding to the time window 1910 may not result in the number of recovery data sets 1920 remaining for the time window 1910 after deletion being less than the threshold number of retained recovery data sets associated with the time window 1910.

[0335]

[0355] Depending on the embodiment, the threshold number of retention recovery data sets associated with the time window 1910 may depend on the recency of the time window 1910. For example, the threshold number of retention recovery data sets associated with the time window 1910 may be proportional (e.g., directly proportional) to the recency value of the time window 1910. Thus, for a first time window 1910 and a second time window 1910 that is before the first time window 1910 and thus has a lower recency value than the first time window 1910, the first threshold number of retention recovery data sets specified for the first time window 1910 by the recovery data set deletion pattern may be greater than the second threshold number of retention recovery data sets specified for the second time window 1910 by the recovery data set deletion pattern. Therefore, to conform to the recovery data set deletion pattern, more recovery data sets 1920 may be retained for the first time window 1910 than for the second time window 1910, and more recovery data sets 1920 corresponding to the second time window 1910 may be deleted compared to the first time window 1910.

[0336]

[0356] Depending on the embodiment, since the duration of the time window 1910 can vary, instead of or in addition to the threshold number of the retention recovery data set associated with the time window 1910, the recovery data set deletion pattern can specify the ratio of the threshold number of the retention recovery data set associated with the time window 1910 to the duration of the time window 1910. As described herein, the time window 1910 that is more recent with respect to a particular time t0 can be associated with a larger threshold number of the retention recovery data set and can have a shorter duration. For example, for the first time window 1910 and the second time window 1910 that is prior to the first time window, the first threshold number of the retention recovery data set associated with the first time window 1910 can be larger than the second threshold number of the retention recovery data set associated with the second time window 1910, as described herein. In addition, the first duration of the first time window 1910 can be shorter than the second duration of the second time window 1910, as described herein. Therefore, the first ratio of the first threshold number of the retention recovery data set to the first duration, specified for the first time window 1910 by the recovery data set deletion pattern, can be larger than the second ratio of the second threshold number of the retention recovery data set to the second duration, specified for the second time window 1910 by the recovery data set deletion pattern.

[0337]

[0357] According to some embodiments, the recovery point distance associated with the time window 1910 may indicate the time distance between two recovery times corresponding to two consecutive recovery data sets 1920 retained for the time window 1910 according to the recovery data set deletion pattern. Therefore, the recovery point distance may represent the difference in elapsed time between two consecutive recovery data sets 1920 retained for the time window 1910 according to the recovery data set deletion pattern. According to some embodiments, the two consecutive recovery data sets 1920 retained for the time window 1910 may include a first recovery data set 1920 and a second recovery data set 1920 retained for the time window 1910 without having other recovery data sets retained for the time window 1910 therebetween. To conform to the recovery data set deletion pattern, the time distance between the recovery timestamp of the first recovery data set 1920 and the recovery timestamp of the second recovery data set 1920 may be equal to the recovery point distance specified by the recovery data set deletion pattern for the time window 1910. According to some embodiments, the recovery point distance associated with the time window 1910 may be calculated by dividing the duration of the time window 1910 by the threshold number of retained recovery data sets associated with the time window 1910 specified by the recovery data set deletion pattern.

[0338]

[0358] As an example, the first recovery dataset 1920 and the second recovery dataset 1920 can be two consecutive recovery datasets 1920 held for a specific time window 1910 without having other recovery datasets held for the specific time window 1910 in between. The first recovery dataset 1920 can be generated at a first recovery timestamp, the second recovery dataset 1920 can be generated at a second recovery timestamp, and the difference between the first recovery timestamp and the second recovery timestamp is two days and seven hours. In this example, the recovery point distance specified by the recovery dataset deletion pattern for the specific time window 1910 can be two days. Therefore, the time distance between two consecutive recovery datasets 1920 held for the specific time window 1910 can be longer than the recovery point distance specified by the recovery dataset deletion pattern. Thus, the system 400 can determine that the recovery dataset deletion pattern is not complied with, and therefore can determine that the data stored by the storage system 502 may be targeted by a security threat.

[0339]

[0359] In some embodiments, the recovery point distance associated with the time window 1910 may depend on the recency of the time window 1910. For example, the recovery point distance associated with the time window 1910 may be proportional (e.g., inversely proportional) to the recency value of the time window 1910. Thus, for a first time window 1910 and a second time window 1910 that is before the first time window 1910 and thus has a lower recency value than the first time window 1910, the first recovery point distance specified for the first time window 1910 by the recovery dataset deletion pattern can be shorter than the second recovery point distance specified for the second time window 1910 by the recovery dataset deletion pattern. In this case, depending on the difference in the recovery point distances with respect to the difference in the durations between the first time window 1910 and the second time window 1910, more recovery datasets 1920 than the second time window 1910 can be held for the first time window 1910, or vice versa.

[0340]

[0360] For example, in the example described above with reference to FIG. 19A, the recovery dataset 1920 can be generated at a default backup interval of one hour. Accordingly, the time windows 1910-1 and 1910-2 having a duration of 12 hours can each correspond to 12 recovery datasets 1920 of the storage system 502. The time window 1910-3 having a duration of two days (e.g., 48 hours) can correspond to 48 recovery datasets 1920 of the storage system 502. The time window 1910-4 having a duration of four days (e.g., 96 hours) can correspond to 96 recovery datasets 1920 of the storage system 502. The time window 1910-n can correspond to a large number of recovery datasets 1920 of the storage system 502 due to its significantly long duration.

[0341]

[0361] In this example, the recovery dataset deletion pattern can specify a recovery point distance of one hour for the time window 1910-1 having a duration of 12 hours. Accordingly, in order to conform to the recovery dataset deletion pattern, one recovery dataset 1920 can be retained every hour in the time window 1910-1. Therefore, out of the 12 recovery datasets 1920 corresponding to the time window 1910-1, 12 recovery datasets 1920 can be retained and the recovery datasets 1920 cannot be deleted.

[0342]

[0362] The recovery dataset deletion pattern can also specify a recovery point distance of four hours for the time window 1910-2 having a duration of 12 hours. Accordingly, in order to conform to the recovery dataset deletion pattern, one recovery dataset 1920 can be retained every four hours in the time window 1910-2. Therefore, out of the 12 recovery datasets 1920 corresponding to the time window 1910-2, 3 recovery datasets 1920 can be retained and 9 recovery datasets 1920 can be deleted. In this example, the recovery point distance (e.g., four hours) specified for the time window 1910-2 can be four times longer than the recovery point distance (e.g., one hour) specified for the time window 1910-1 that is continuously after the time window 1910-2.

[0343]

[0363] The recovery dataset deletion pattern may also specify a recovery point distance of 8 hours for the time window 1910-3 having a duration of 48 hours. Thus, in order to conform to the recovery dataset deletion pattern, one recovery dataset 1920 may be retained every 8 hours for the time window 1910-3. Therefore, out of the 48 recovery datasets 1920 corresponding to the time window 1910-3, 6 recovery datasets 1920 may be retained and 42 recovery datasets 1920 may be deleted. In this example, the recovery point distance (e.g., 8 hours) specified for the time window 1910-3 may be twice as long as the recovery point distance (e.g., 4 hours) specified for the time window 1910-2 that is continuously after the time window 1910-3.

[0344]

[0364] The recovery dataset deletion pattern may also specify a recovery point distance of 1 day (e.g., 24 hours) for the time window 1910-4 having a duration of 4 days (e.g., 96 hours). Thus, in order to conform to the recovery dataset deletion pattern, one recovery dataset 1920 may be retained every 1 day for the time window 1910-4. Therefore, out of the 96 recovery datasets 1920 corresponding to the time window 1910-4, 4 recovery datasets 1920 may be retained and 92 recovery datasets 1920 may be deleted. In this example, the recovery point distance (e.g., 1 day or 24 hours) specified for the time window 1910-4 may be three times as long as the recovery point distance (e.g., 8 hours) specified for the time window 1910-3 that is continuously after the time window 1910-4.

[0345]

[0365] The recovery dataset deletion pattern may also specify a recovery point distance of one week (e.g., 7 days or 168 hours) for the time window 1910-n covering the entire period before the timestamp t4. Thus, in order to conform to the recovery dataset deletion pattern, one recovery dataset 1920 may be retained for each one-week period of the time window 1910-n. Therefore, out of the multiple recovery datasets 1920 corresponding to the time window 1910-n, only a few recovery datasets 1920 may be retained and most of the recovery datasets 1920 may be deleted. In this example, the recovery point distance (e.g., one week) specified for the time window 1910-n may be seven times longer than the recovery point distance (e.g., one day) specified for the time window 1910-4 that is continuously after the time window 1910-n.

[0346]

[0366] Thus, in order to conform to the recovery dataset deletion pattern, most or all of the recovery datasets 1920 corresponding to the most recent time window 1910 (e.g., time window 1910-1) may be retained, as shown using this example. For time windows 1910 with lower recency values such as time windows 1910-2 and 1910-3, the number of recovery datasets 1920 retained for the time window 1910 is considerably less than the number of recovery datasets 1920 deleted from the one or more recovery datasets 1920 corresponding to the time window 1910. As the recency value of the time window 1910 decreases, the number of recovery datasets 1920 corresponding to the time window 1910 that are deleted may increase significantly. Therefore, for time windows 1910 with low recency values such as time windows 1910-4 and 1910-n, most of the recovery datasets 1920 corresponding to the time window 1910 may be deleted, and thus, potentially stale recovery datasets 1920 may be deleted before they become very old.

[0347]

[0367] As shown in FIG. 19A, each time window 1910 can be associated with one or more recovery data sets 1930. Each recovery data set 1930 can become the recovery data set 1920 held for the time window 1910 if the recovery data set deletion pattern is matched. In some embodiments, the system 400 and / or the storage system 502 can identify one or more recovery data sets 1930 for the time window 1910 among the one or more recovery data sets 1920 corresponding to the time window 1910. The identification of the recovery data set 1930 for the time window 1910 can be performed using the recovery point distance specified for the time window 1910 by the recovery data set deletion pattern.

[0348]

[0368] For example, FIG. 19B shows a diagram 1950 showing the time window 1910-x. As shown in FIG. 19B, the time window 1910-x can extend from the start time at time stamp t x-1 to the end time at time stamp t x and can be associated with the recovery point distance d x specified for the time window 1910-x by the recovery data set deletion pattern. In some embodiments, to identify one or more recovery data sets 1930 for the time window 1910-x, the system 400 can select a reference time stamp (e.g., t reference ) within the time window 1910-x. The reference time stamp can be the start time t x-1 of the time window 1910-x, the end time t x of the time window 1910-x, the recovery time stamp of the recovery data set 1920 generated within the time window 1910-x, and the like. Other types of reference time stamps are also possible and contemplated.

[0349]

[0369] In some embodiments, the system 400 selects, from the recovery data set 1920 corresponding to the time window 1910-x, the recovery point distance d xOne or more recovery data sets 1920 having a time distance between those recovery timestamps and the reference timestamp equal to one or more multiples of can be identified. Next, the system 400 can identify the one or more recovery data sets 1920 to become recovery data sets 1930 held for the time window 1910 - x according to a recovery data set deletion pattern. Therefore, to conform to the recovery data set deletion pattern, the recovery data set 1930 can be held for the time window 1910 - x, and other recovery data sets 1920 corresponding to the time window 1910 - x can be deleted. Other implementations for identifying the recovery data set 1930 held for the time window 1910 - x according to the recovery data set deletion pattern are also possible and contemplated.

[0350]

[0370] Depending on the embodiment, the storage system 502 may store temporary data that is used for a limited period of time. For example, the storage system 502 may store data (e.g., temporary data) used to generate a stage report for a particular stage of an investigation project that includes multiple stages. As described herein, the system 400 and / or the storage system 502 may generate one or more recovery data sets 1920 for the data at a predetermined backup interval (e.g., every hour). Depending on the embodiment, in order to perform a pruning operation for the recovery data set 1920 of the data when a particular stage has not yet been completed, the system 400 temporarily holds one or more recovery data sets 1920 of the data according to a recovery data set deletion pattern, as described herein, and deletes other recovery data sets 1920 of the data. The retained recovery data set 1920 can be used to restore data for regenerating the stage report of a particular stage as needed. When a particular stage is completed and the investigation project proceeds to a subsequent stage, the system 400 may delete all the recovery data sets 1920 of the previously retained data. Alternatively, the system 400 may continue to hold one or more recovery data sets 1920 of the data generated at one or more specific timestamps (e.g., the timestamp when a particular stage started, the timestamp when a particular stage ended, the end-of-quarter timestamp, etc.), and other recovery data sets 1920 of the previously retained data may be deleted.

[0351]

[0371] Thus, the recovery data set 1920 can be deleted according to a recovery data set deletion pattern to appropriately reduce the number of recovery data sets 1920 held for the storage system 502. Depending on the embodiment, if one or more deletion requests attempting to delete one or more recovery data sets 1920 of the storage system 502 are inconsistent with the recovery data set deletion pattern, the system 400 may determine that the data stored by the storage system 502 may be targeted by a security threat.

[0352]

[0372] In some embodiments, to determine that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, system 400 may determine that the one or more deletion requests are attempting to delete a first number of recovery datasets 1920 generated within a first time window 1910. System 400 may also determine that the one or more deletion requests are attempting to delete a second number of recovery datasets 1920 generated within a second time window 1910, where the second time window 1910 is prior to the first time window 1910. In some embodiments, system 400 may determine that the first number of recovery datasets 1920 is greater than the second number of recovery datasets 1920, and thus may determine that the one or more deletion requests are attempting to delete a greater number of recovery datasets 1920 corresponding to the more recent time window 1910.

[0353]

[0373] In contrast, deletion of datasets 1920 that conform to the recovery dataset deletion pattern may result in more recovery datasets 1920 corresponding to the more recent time window 1910 being retained and more recovery datasets 1920 corresponding to older time windows 1910 being deleted. As a result, system 400 may determine that the one or more deletion requests are inconsistent with the recovery dataset deletion pattern, and thus may determine that the one or more deletion requests are attempting to inappropriately delete recovery datasets 1920 of storage system 502. Accordingly, system 400 may determine that the data stored by storage system 502 may be targeted by a security threat.

[0354]

[0374] In some embodiments, to determine that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, system 400 may determine that the one or more deletion requests are attempting to delete a first number of recovery datasets 1920 generated within time window 1910. Next, system 400 may determine a second number of recovery datasets 1920 that would be retained for time window 1910 when the first number of recovery datasets 1920 are deleted upon execution of the one or more deletion requests. To calculate the second number of recovery datasets 1920, system 400 may subtract the first number of recovery datasets 1920 that the one or more deletion requests are attempting to delete from the number of recovery datasets 1920 generated within time window 1910.

[0355]

[0375] In some embodiments, system 400 may determine that the second number of recovery datasets 1920 that would be retained for time window 1910 when the one or more deletion requests are executed is less than the threshold number of retained recovery datasets specified for time window 1910 by the recovery dataset deletion pattern. Thus, system 400 may determine that the one or more deletion requests are attempting to delete an unduly large number of recovery datasets 1920 corresponding to time window 1910, such that the number of recovery datasets 1920 retained for time window 1910 may become insufficient to conform to the recovery dataset deletion pattern. As a result, system 400 may determine that the one or more deletion requests are inconsistent with the recovery dataset deletion pattern, and thus that the one or more deletion requests are attempting to inappropriately delete recovery datasets 1920 of storage system 502. Accordingly, system 400 may determine that the data stored by storage system 502 may be targeted by a security threat.

[0356]

[0376] In some embodiments, to determine that one or more deletion requests are inconsistent with the recovery dataset deletion pattern, system 400 may identify one or more recovery datasets 1920 that are retained based on the recovery dataset deletion pattern. These recovery datasets 1920 may be retained for one or more time windows 1910 if the recovery dataset deletion pattern is complied with. In some embodiments, the recovery datasets 1920 retained for a particular time window 1910 according to the recovery dataset deletion pattern may be identified using the recovery point distance specified for the particular time window 1910 by the recovery dataset deletion pattern as described herein.

[0357]

[0377] In some embodiments, system 400 may determine that one or more deletion requests are attempting to delete at least one of the one or more recovery datasets 1920 retained according to the recovery dataset deletion pattern. Thus, system 400 may determine that one or more deletion requests are attempting to delete at least one recovery dataset 1920 that should be retained if the recovery dataset deletion pattern is complied with. As a result, system 400 may determine that one or more deletion requests are inconsistent with the recovery dataset deletion pattern and, thus, that one or more deletion requests are attempting to inappropriately delete the recovery datasets 1920 of storage system 502. Accordingly, system 400 may determine that the data stored by storage system 502 may be targeted by a security threat.

[0358]

[0378] In some embodiments, system 400 may determine that one or more deletion requests are attempting to delete most or all of the recovery datasets 1920 corresponding to one or more of the most recent time windows 1910.

[0359]

[0379] Additionally, or alternatively, the system 400 may determine that one or more deletion requests are attempting to indiscriminately delete (e.g., regardless of their relative elapsed times) multiple recovery data sets 1920.

[0360]

[0380] Additionally, or alternatively, the system 400 may determine that one or more deletion requests are attempting to delete multiple recovery data sets 1920 in the order of their recovery timestamps (e.g., in a forward or reverse temporal direction), and that due to the deletion of the recovery data sets, the recovery data sets 1920 are not retained for a particular time window 1910 before proceeding to the next time window 1910. The next time window 1910 may be either continuously before or continuously after a particular time window 1910.

[0361]

[0381] Additionally, or alternatively, the system 400 may determine that one or more deletion requests are attempting to delete multiple recovery data sets 1920 where the time distance between the recovery times corresponding to the recovery data sets 1920 retained for the most recent time window 1910 is longer than the time distance between the recovery times corresponding to the recovery data sets 1920 retained for a more past time window 1910. In other words, the more recent, or newer, retained recovery data sets 1920 may have a time distance between their recovery timestamps that is longer than the time distance between the recovery timestamps of the more past, or older, retained recovery data sets 1920.

[0362]

[0382] In these situations, the system 400 may determine that one or more deletion requests are inconsistent with the recovery data set deletion pattern, and thus, may determine that one or more deletion requests are attempting to inappropriately delete the recovery data sets 1920 of the storage system 502. Accordingly, the system 400 may determine that the data stored by the storage system 502 may be targeted by a security threat.

[0363]

[0383] Depending on the embodiment, in response to determining that the storage system 502 is targeted by a security threat, the system 400 may perform a corrective action on the storage system 502. Additionally, or alternatively, the system 400 may instruct the storage system 502 to perform a corrective action.

[0364]

[0384] Depending on the embodiment, to perform a corrective action for the storage system 502, the system 400 may identify one or more recovery data sets 1920 that are retained based on a recovery data set deletion pattern. As described herein, these recovery data sets 1920 may be retained if the recovery data set deletion pattern is met and may be identified using a recovery point distance specified for the corresponding time window 1910 by the recovery data set deletion pattern. Depending on the embodiment, the system 400 may convert at least one of the recovery data sets 1920 retained according to the recovery data set deletion pattern into a temporarily protected recovery data set of the storage system 502. The temporarily protected recovery data set may have a higher level of protection than the recovery data set 1920 retained according to the recovery data set deletion pattern. For example, the temporarily protected recovery data set may require additional authorization from an additional authorization entity (e.g., a third-party authorization application such as a system administrator, an AI engine, etc.) to delete and / or modify compared to the recovery data set 1920 retained according to the recovery data set deletion pattern.

[0365]

[0385] In some embodiments, system 400 continues to monitor deletion requests that attempt to delete one or more recovery data sets 1920 of storage system 502, and may determine that one or more additional deletion requests are also inconsistent with the recovery data set deletion pattern. For example, system 400 may identify additional deletion requests received later by storage system 502 that are inconsistent with the recovery data set deletion pattern in the manner described herein, and may determine that the number of additional deletion requests exceeds a predetermined threshold number. In this case, system 400 may verify that the recovery data set deletion pattern has been violated and that the data stored by storage system 502 may be targeted by a security threat. Additionally or alternatively, system 400 may verify that the data stored by storage system 502 may be targeted by a security threat using any of the security threat detection methods described herein.

[0366]

[0386] In some embodiments, it is determined that one or more additional deletion requests are inconsistent with the recovery data set deletion pattern, and thus, in response to verifying that the recovery data set deletion pattern has been violated, system 400 may convert the temporarily protected recovery data set of storage system 502 to a fully protected recovery data set of storage system 502. The fully protected recovery data set may have an even higher level of protection compared to the temporarily protected recovery data set. For example, the fully protected recovery data set may require additional authorization from additional authorizing entities (e.g., a default set of system managers, one or more CxO-level authenticated users, etc.) to delete and / or modify compared to the temporarily protected recovery data set. Additionally or alternatively, the fully protected recovery data set may be prevented from being deleted and / or modified for a predetermined period of time from its creation.

[0367]

[0387] In some embodiments, one or more additional deletion requests are determined to be inconsistent with the recovery dataset deletion pattern, and thus, in response to verifying that the recovery dataset deletion pattern has been violated, system 400 may instruct storage system 502 to block one or more subsequent deletion requests associated with the data stored by storage system 502. For example, system 400 may instruct storage system 502 to block any future deletion requests received by storage system 502 after the determination that one or more additional deletion requests are also inconsistent with the recovery dataset deletion pattern. In some embodiments, system 400 may instruct storage system 502 to block deletion requests attempting to delete one or more data items from storage system 502 for a predetermined period (e.g., 24 hours), or until a security threat to storage system 502 is investigated and / or addressed by one or more authorizing entities.

[0368]

[0388] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various changes and modifications can be made to them without departing from the scope of the invention as set forth in the appended claims, and additional embodiments can be implemented. For example, the specific features of one embodiment described herein can be combined with or substituted for the features of another embodiment described herein. Accordingly, the description and drawings are to be regarded in an illustrative rather than a limiting sense.

Claims

1. A method comprising: detecting, by a data protection system, one or more deletion requests for deleting one or more recovery data sets of a storage system; determining, by the data protection system, that the one or more deletion requests are inconsistent with a recovery data set deletion pattern associated with the storage system; determining, by the data protection system and based on the determination that the one or more deletion requests are inconsistent with the recovery data set deletion pattern, that data stored by the storage system may be targeted by a security threat; A method comprising the above.

2. The method of claim 1, wherein the recovery data set deletion pattern is associated with one or more time windows and one or more of a threshold number of retained recovery data sets per time window or a recovery point distance between two consecutive retained recovery data sets per time window.

3. The recovery data set deletion pattern is associated with a first time window and a second time window prior to the first time window, The recovery data set deletion pattern is associated with a first threshold number of retained recovery data sets for the first time window and a second threshold number of retained recovery data sets for the second time window, and the first threshold number of retained recovery data sets is greater than the second threshold number of retained recovery data sets. The method of claim 1.

4. The recovery data set deletion pattern is associated with a first time window and a second time window prior to the first time window, The recovery data set deletion pattern is associated with a first recovery point distance between two consecutive retained recovery data sets for the first time window and a second recovery point distance between two consecutive retained recovery data sets for the second time window, and the first recovery point distance is shorter than the second recovery point distance. The method of claim 1.

5. The determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern comprises: determining that the one or more deletion requests are attempting to delete a first number of recovery data sets generated within a first time window; determining that the one or more deletion requests are attempting to delete a second number of recovery data sets generated within a second time window, wherein the second time window is prior to the first time window, and determining that the first number of recovery data sets is greater than the second number of recovery data sets, The method according to claim 1, comprising: **Claim 6** wherein determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern comprises determining that the one or more deletion requests are attempting to delete a first number of recovery data sets generated within a time window, determining a second number of recovery data sets to be retained for the time window when the first number of recovery data sets are deleted, and determining that the second number of recovery data sets to be retained for the time window is less than a threshold number of retained recovery data sets corresponding to the time window as specified by the recovery data set deletion pattern, The method according to claim 1, comprising: **Claim 7** wherein determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern comprises identifying one or more recovery data sets retained based on the recovery data set deletion pattern, and determining that the one or more deletion requests are attempting to delete at least one of the one or more retained recovery data sets, The method according to claim 1, comprising: **Claim 8** The method according to claim 1, further comprising, in response to determining that the data stored by the storage system may be targeted by the security threat, performing a corrective action on the storage system by the data protection system. **Claim 9** Performing the corrective action on the storage system comprises identifying one or more recovery data sets retained based on the recovery data set deletion pattern, and converting at least one of the one or more retained recovery data sets to a temporary protected recovery data set having a higher protection level compared to the one or more retained recovery data sets, The method according to claim 8, comprising: **Claim 10** Determine that one or more additional deletion requests are inconsistent with the recovery dataset deletion pattern, Convert the temporarily protected recovery dataset to a fully protected recovery dataset having an even higher protection level compared to the temporarily protected recovery dataset, in response to determining that the one or more additional deletion requests are inconsistent with the recovery dataset deletion pattern, The method according to claim 9, further comprising.

11. Determine by the data protection system that one or more additional deletion requests are inconsistent with the recovery dataset deletion pattern, Instruct the storage system to block one or more subsequent deletion requests associated with the data stored by the storage system, by the data protection system and in response to determining that the one or more additional deletion requests are inconsistent with the recovery dataset deletion pattern, The method according to claim 1, further comprising.

12. A system comprising: A memory storing instructions; A processor communicatively coupled to the memory, the processor executing the instructions, Detect one or more deletion requests for deleting one or more recovery datasets of a storage system, Determine that the one or more deletion requests are inconsistent with a recovery dataset deletion pattern associated with the storage system, Based on determining that the one or more deletion requests are inconsistent with the recovery dataset deletion pattern, determine that the data stored by the storage system may be targeted by a security threat, A processor configured to perform; A system comprising.

13. The system according to claim 12, wherein the recovery dataset deletion pattern is associated with one or more of one or more time windows, and a threshold number of retained recovery datasets per time window, or a recovery point distance between two consecutive retained recovery datasets per time window.

14. The recovery dataset deletion pattern is associated with a first time window and a second time window prior to the first time window, The recovery dataset deletion pattern is associated with a first threshold number of retained recovery datasets for the first time window and a second threshold number of retained recovery datasets for the second time window, and the first threshold number of retained recovery datasets is greater than the second threshold number of retained recovery datasets, the system according to claim 12.

15. The recovery dataset deletion pattern is associated with a first time window and a second time window prior to the first time window, The recovery dataset deletion pattern is associated with a first recovery point distance between two consecutive retained recovery datasets for the first time window and a second recovery point distance between two consecutive retained recovery datasets for the second time window, and the first recovery point distance is shorter than the second recovery point distance, the system according to claim 12.

16. Determining that the one or more deletion requests are inconsistent with the recovery dataset deletion pattern is Determining that the one or more deletion requests are attempting to delete a first number of recovery datasets generated within a first time window, Determining that the one or more deletion requests are attempting to delete a second number of recovery datasets generated within a second time window, the second time window being prior to the first time window, and Determining that the first number of recovery datasets is greater than the second number of recovery datasets, The system according to claim 12, comprising.

17. Determining that the one or more deletion requests are inconsistent with the recovery dataset deletion pattern is Determining that the one or more deletion requests are attempting to delete a first number of recovery datasets generated within a time window, Determining a second number of recovery datasets that would be retained for the time window when the first number of recovery datasets are deleted, and Determining that the second number of recovery datasets that would be retained for the time window is less than the threshold number of retained recovery datasets corresponding to the time window as specified by the recovery dataset deletion pattern, The system according to claim 12, comprising.

18. Determining that the one or more deletion requests are inconsistent with the recovery dataset deletion pattern is identifying one or more recovery data sets retained based on the recovery data set deletion pattern, and determining that the one or more deletion requests are attempting to delete at least one of the one or more retained recovery data sets, The system according to claim 12, comprising:

19. the processor executes the instructions, further configured to perform a corrective action on the storage system in response to determining that the data stored by the storage system may be targeted by the security threat, the system according to claim 12.

20. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor of a computing device to detect one or more deletion requests for deleting one or more recovery data sets of a storage system; determine that the one or more deletion requests are inconsistent with a recovery data set deletion pattern associated with the storage system; determine that the data stored by the storage system may be targeted by a security threat based on determining that the one or more deletion requests are inconsistent with the recovery data set deletion pattern; A non-transitory computer-readable medium instructing to perform.

Images