Coordinated snapshots among storage systems implementing promotion / degradation models

Abstract

Implementing coordinated snapshots among storage systems of a promotion / degradation model, including: monitoring a set of two or more target storage systems, where the two or more target storage systems are replication targets for respective portions of a source data set; associating a first coordinated tracking dataset with two or more local tracking datasets maintained on the two or more target storage systems, wherein each local tracking dataset is a tracking copy of a portion of the source dataset; and advancing the first coordination tracking dataset depending on a first coordination target checkpoint.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0001] Figure 1A A first example system for data storage according to some embodiments is described.

[0002] Figure 1B A second example system for data storage is described in accordance with some embodiments.

[0003] Figure 1C A third example system for data storage is described in accordance with some embodiments.

[0004] Figure 1D A fourth example system for data storage is described in accordance with some embodiments.

[0005] Figure 2A is a perspective diagram of a storage cluster having multiple storage nodes and internal storage devices coupled to each storage node to provide network attached storage according to some embodiments.

[0006] Figure 2B is a block diagram illustrating an interconnect switch coupling a plurality of storage nodes according to some embodiments.

[0007] Figure 2C is a multi-level block diagram showing the contents of a storage node and the contents of one of the non-volatile solid-state storage units according to some embodiments.

[0008] Figure 2D A storage server environment is shown according to some embodiments using the storage nodes and storage units of some of the previous figures.

[0009] Figure 2E is a hardware block diagram showing a control plane, compute and storage planes, and authorized blades interacting with underlying physical resources according to some embodiments.

[0010] Figure 2F Depicted is a resiliency software layer in a blade of a storage cluster according to some embodiments.

[0011] Figure 2G Depicted are authorization and storage resources in blades of a storage cluster according to some embodiments.

[0012] Figure 3A A diagram of a storage system coupled for data communication with a cloud service provider is presented in accordance with some embodiments of the present disclosure.

[0013] Figure 3B A diagram illustrating a storage system according to some embodiments of the present disclosure is presented.

[0014] Figure 3CAn example of a cloud-based storage system according to some embodiments of the present disclosure is presented.

[0015] Figure 3D An exemplary computing device is illustrated that may be specifically configured to perform one or more of the processes described herein.

[0016] Figure 3E A team of exemplary storage systems providing storage services according to some embodiments of the present disclosure is described.

[0017] Figure 4 A flow chart illustrating an example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0018] Figure 5 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0019] Figure 6 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0020] Figure 7 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0021] Figure 8 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0022] Fig. 9 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0023] Fig.10 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0024] Fig.11 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0025] Fig.12 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0026] Fig.13A flow chart illustrating an example of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is presented.

[0027] Fig.14 A flow chart illustrating another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth.

[0028] Fig.15 A flow chart illustrating an example method for coordinating snapshots across multiple storage systems according to some embodiments of the present disclosure is presented.

[0029] Fig.16 Flowcharts illustrating additional example methods for coordinating snapshots across multiple storage systems according to some embodiments of the present disclosure are set forth.

[0030] Fig.17 Flowcharts illustrating additional example methods for coordinating snapshots across multiple storage systems according to some embodiments of the present disclosure are set forth.

[0031] Fig.18 Flowcharts illustrating additional example methods for coordinating snapshots across multiple storage systems according to some embodiments of the present disclosure are set forth.

[0032] Fig.19 Flowcharts illustrating additional example methods for coordinating snapshots across multiple storage systems according to some embodiments of the present disclosure are set forth.

[0033] Fig. 20 Flowcharts illustrating additional example methods for coordinating snapshots across multiple storage systems according to some embodiments of the present disclosure are set forth.

[0034] Fig.21 A flow chart illustrating an example method of coordinating snapshots among a storage system implementing a promotion / demotion model according to some embodiments of the present disclosure is set forth.

[0035] Fig. 22 Flowcharts illustrating additional example methods of coordinating snapshots among storage systems implementing a promotion / demotion model according to some embodiments of the present disclosure are set forth.

[0036] Fig.23 Flowcharts illustrating additional example methods of coordinating snapshots among storage systems implementing a promotion / demotion model according to some embodiments of the present disclosure are set forth.

[0037] Fig.24 Flowcharts illustrating additional example methods of coordinating snapshots among storage systems implementing a promotion / demotion model according to some embodiments of the present disclosure are set forth. DETAILED DESCRIPTION

[0038] by Figure 1A Initially, example methods, devices, and products for coordinating snapshots in a storage system implementing a promotion / demotion model according to embodiments of the present disclosure are described with reference to the accompanying drawings. Figure 1A An example system for data storage according to some embodiments is described. For purposes of illustration and not limitation, system 100 (also referred to herein as a "storage system") includes numerous elements. It may be noted that in other embodiments, system 100 may include the same, more, or fewer elements configured in the same or different manners.

[0039] The system 100 includes a plurality of computing devices 164A-B. The computing devices (also referred to herein as "client devices") may be embodied as servers, workstations, personal computers, notebooks, or the like in a data center, for example. The computing devices 164A-B may be coupled for data communication with one or more storage arrays 102A-B via a storage area network ('SAN') 158 or a local area network ('LAN') 160.

[0040] SAN 158 may be implemented with a variety of data communication architectures, devices, and protocols. For example, the architecture for SAN 158 may include Fibre Channel, Ethernet, InfiniBand, Serial Attached Small Computer System Interface ('SAS'), or the like. Data communication protocols used with SAN 158 may include Advanced Technology Attachment ('ATA'), Fibre Channel Protocol, Small Computer System Interface ('SCSI'), Internet Small Computer System Interface ('iSCSI'), Ultra SCSI, Non-Volatile Memory Express Architecture ('NVMe'), or the like. It may be noted that SAN 158 is provided for illustration and not limitation. Other data communication couplings may be implemented between computer devices 164A-B and storage arrays 102A-B.

[0041] LAN 160 may also be implemented with various architectures, devices, and protocols. For example, the architecture for LAN 160 may include Ethernet (802.3), wireless (802.11), or the like. Data communication protocols used in LAN 160 may include Transmission Control Protocol ('TCP'), User Datagram Protocol ('UDP'), Internet Protocol ('IP'), Hypertext Transfer Protocol ('HTTP'), Wireless Access Protocol ('WAP'), Handheld Device Transport Protocol ('HDTP'), Session Initiation Protocol ('SIP'), Real Time Protocol ('RTP'), or the like. LAN 160 may also be connected to Internet 162.

[0042] Storage arrays 102A-B may provide persistent data storage for computing devices 164A-B. In an embodiment, storage array 102A may be housed in a chassis (not shown) and storage array 102B may be housed in another chassis (not shown). Storage arrays 102A and 102B may include one or more storage array controllers 110A-D (also referred to herein as "controllers"). Storage array controllers 110A-D may be embodied as modules of automated computing machinery including computer hardware, computer software, or a combination of computer hardware and software. In some embodiments, storage array controllers 110A-D may be configured to perform various storage tasks. Storage tasks may include writing data received from computing devices 164A-B to storage arrays 102A-B, erasing data from storage arrays 102A-B, retrieving data from storage arrays 102A-B and providing data to computing devices 164A-B, monitoring and reporting disk utilization and performance, performing redundancy operations (e.g., redundant array of independent drives ('RAID') or RAID-like data redundancy operations), compressing data, encrypting data, and the like.

[0043] The storage array controllers 110A-D may be implemented in various ways, including as a field programmable gate array ('FPGA'), a programmable logic chip ('PLC'), an application specific integrated circuit ('ASIC'), a system on a chip ('SOC'), or any computing device including discrete components such as a processing device, a central processing unit, a computer memory, or various adapters. The storage array controllers 110A-D may include, for example, a data communications adapter configured to support communications via a SAN 158 or a LAN 160. In some implementations, the storage array controllers 110A-D may be independently coupled to a LAN 160. In an implementation, the storage array controllers 110A-D may include an I / O controller or the like that couples the storage array controllers 110A-D for data communications to persistent storage resources 170A-B (also referred to herein as "storage resources") through a midplane (not shown). The persistent storage resources 170A-B primarily include any number of storage drives 171A-F (also referred to herein as "storage devices") and any number of non-volatile random access memory ('NVRAM') devices (not shown).

[0044] In some embodiments, the NVRAM device of the persistent storage resource 170A to B may be configured to receive data to be stored in the storage drive 171A to F from the storage array controller 110A to D. In some examples, the data may originate from the computing device 164A to B. In some examples, writing data to the NVRAM device may be faster than writing data directly to the storage drive 171A to F. In an embodiment, the storage array controller 110A to D may be configured to utilize the NVRAM device as a fast accessible buffer for data intended to be written to the storage drive 171A to F. Relative to a system in which the storage array controller 110A to D writes data directly to the storage drive 171A to F, the latency of a write request using the NVRAM device as a buffer may be improved. In some embodiments, the NVRAM device may be implemented with a computer memory in the form of a high-bandwidth, low-latency RAM. The NVRAM device is called "non-volatile" because the NVRAM device may receive or contain a unique power source that maintains the state of the RAM after the NVRAM device loses the main power supply. This power source may be a battery, one or more capacitors, or the like. In response to a loss of power, the NVRAM device may be configured to write the contents of the RAM to a persistent storage device, such as storage drives 171A-F.

[0045] In an implementation, storage drives 171A-F may refer to any device configured to persistently record data, where "persistently" or "persistent" refers to the ability of the device to maintain the recorded data after power is lost. In some implementations, storage drives 171A-F may correspond to non-disk storage media. For example, storage drives 171A-F may be one or more solid-state drives ('SSD'), flash memory-based storage devices, any type of solid-state non-volatile memory, or any other type of non-mechanical storage device. In other implementations, storage drives 171A-F may include a mechanical or rotating hard disk, such as a hard disk drive ('HDD').

[0046] In some embodiments, the storage array controllers 110A-D may be configured to offload device management responsibilities from the storage drives 171A-F in the storage arrays 102A-B. For example, the storage array controllers 110A-D may manage control information that may describe the status of one or more memory blocks in the storage drives 171A-F. The control information may indicate, for example, that a particular memory block has failed and should no longer be written to, that a particular memory block contains boot code for the storage array controllers 110A-D, the number of program-erase ('P / E') cycles that have been performed on a particular memory block, the age of the data stored in the particular memory block, the type of data stored in the particular memory block, and the like. In some embodiments, the control information may be stored as metadata with the associated memory blocks. In other embodiments, the control information of the storage drives 171A-F may be stored in one or more specific memory blocks of the storage drives 171A-F selected by the storage array controllers 110A-D. The selected memory blocks may be tagged with an identifier indicating that the selected memory blocks contain the control information. The identifier may be used by the storage array controllers 110A-D in conjunction with the storage drives 171A-F to quickly identify a memory block containing control information. For example, the storage controllers 110A-D may issue a command to locate a memory block containing control information. It may be noted that the control information may be so large that portions of the control information may be stored in multiple locations, that the control information may be stored in multiple locations, for example, for redundancy purposes, or that the control information may be otherwise distributed across multiple memory blocks in the storage drives 171A-F.

[0047] In an embodiment, the storage array controller 110A-D may offload device management responsibilities from the storage drives 171A-F of the storage array 102A-B by retrieving control information describing the state of one or more memory blocks in the storage drives 171A-F from the storage drives 171A-F. Retrieving the control information from the storage drives 171A-F may be performed, for example, by the storage array controller 110A-D querying the storage drives 171A-F for the location of the control information for a particular storage drive 171A-F. The storage drives 171A-F may be configured to execute instructions that enable the storage drives 171A-F to identify the location of the control information. The instructions may be executed by a controller (not shown) associated with or otherwise located on the storage drives 171A-F and may cause the storage drives 171A-F to scan a portion of each memory block to identify a memory block storing the control information for the storage drives 171A-F. The storage drives 171A-F may respond by sending a response message including the location of the control information for the storage drives 171A-F to the storage array controllers 110A-D. In response to receiving the response message, the storage array controllers 110A-D may issue a request to read data stored at the address associated with the location of the control information for the storage drives 171A-F.

[0048] In other embodiments, the storage array controllers 110A-D may further offload device management responsibilities from the storage drives 171A-F by performing storage drive management operations in response to receiving the control information. The storage drive management operations may include, for example, operations that are typically performed by the storage drives 171A-F, e.g., a controller (not shown) associated with the particular storage drives 171A-F. The storage drive management operations may include, for example, ensuring that data is not written to failed memory blocks within the storage drives 171A-F, ensuring that data is written to memory blocks within the storage drives 171A-F in a manner such that adequate wear leveling is achieved, and the like.

[0049] In an embodiment, the storage array 102A to B may implement two or more storage array controllers 110A to D. For example, the storage array 102A may include a storage array controller 110A and a storage array controller 110B. In a given example, a single storage array controller 110A to D (e.g., storage array controller 110A) of the storage system 100 may be designated as having a primary status (also referred to herein as a "primary controller"), and other storage array controllers 110A to D (e.g., storage array controller 110A) may be designated as having a secondary status (also referred to herein as a "secondary controller"). The primary controller may have specific rights, such as permission to change data in the persistent storage resources 170A to B (e.g., write data to the persistent storage resources 170A to B). At least some rights of the primary controller may replace the rights of the secondary controller. For example, when the primary controller has permission to change data in the persistent storage resources 170A to B, the secondary controller may not have the rights. The status of the storage array controllers 110A to D may change. For example, storage array controller 110A may be designated as having a secondary status and storage array controller 110B may be designated as having a primary status.

[0050] In some embodiments, a primary controller (e.g., storage array controller 110A) may be used as a primary controller for one or more storage arrays 102A to B, and a secondary controller (e.g., storage array controller 110B) may be used as a secondary controller for one or more storage arrays 102A to B. For example, storage array controller 110A may be the primary controller for storage array 102A and storage array 102B, and storage array controller 110B may be the secondary controller for storage arrays 102A and 102B. In some embodiments, storage array controllers 110C and 110D (also referred to as "storage processing modules") may not have primary or secondary status. Storage array controllers 110C and 110D implemented as storage processing modules may be used as a communication interface between primary and secondary controllers (e.g., storage array controllers 110A and 110B, respectively) and storage array 102B. For example, storage array controller 110A of storage array 102A may send a write request to storage array 102B via SAN 158. The write request may be received by both storage array controllers 110C and 110D of the storage array 102B. The storage array controllers 110C and 110D facilitate communication, for example, sending the write request to the appropriate storage drives 171A through F. It may be noted that in some embodiments, the storage processing module may be used to increase the number of storage drives controlled by the primary and secondary controllers.

[0051] In an embodiment, storage array controllers 110A-D are communicatively coupled to one or more storage drives 171A-F and to one or more NVRAM devices (not shown) included as part of storage arrays 102A-B via a mid-plane (not shown). Storage array controllers 110A-D may be coupled to the mid-plane via one or more data communications links, and the mid-plane may be coupled to storage drives 171A-F and NVRAM devices via one or more data communications links. For example, the data communications links described herein are collectively illustrated by data communications links 108A-D and may include a Peripheral Component Interconnect Express ('PCIe') bus.

[0052] Figure 1B An example system for data storage is described according to some embodiments. Figure 1B The storage array controller 101 described in Figure 1A 1. Storage array controller 101 may be similar to storage array controller 110A or storage array controller 110B. For purposes of illustration and not limitation, storage array controller 101 includes numerous elements. It is noted that in other embodiments, storage array controller 101 may include the same, more, or fewer elements configured in the same or different manners. It is noted that the following may include Figure 1A 1 to help illustrate the features of storage array controller 101.

[0053] The storage array controller 101 may include one or more processing devices 104 and a random access memory ('RAM') 111. The processing device 104 (controller 101) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device 104 (or controller 101) may be a complex instruction set computing ('CISC') microprocessor, a reduced instruction set computing ('RISC') microprocessor, a very long instruction word ('VLIW') microprocessor, or a processor that implements other instruction sets or a number of processors that implement a combination of instruction sets. The processing device 104 (controller 101) may also be one or more special-purpose processing devices, such as an ASIC, an FPGA, a digital signal processor ('DSP'), a network processor, or the like.

[0054] Processing device 104 may be connected to RAM 111 via data communication link 106, which may be embodied as a high-speed memory bus, such as a double data rate 4 ('DDR4') bus. Stored in RAM 111 is an operating system 112. In some embodiments, instructions 113 are stored in RAM 111. Instructions 113 may include computer program instructions for performing operations in a direct-mapped flash storage system. In one embodiment, a direct-mapped flash storage system is a system that directly addresses data blocks within a flash drive and does not require address translation performed by a storage controller of the flash drive.

[0055] In an embodiment, storage array controller 101 includes one or more host bus adapters 103A-C coupled to processing device 104 via data communication links 105A-C. In an embodiment, host bus adapters 103A-C may be computer hardware that connects a host system (e.g., storage array controller) to other networks and storage arrays. In some examples, host bus adapters 103A-C may be Fibre Channel adapters that enable storage array controller 101 to connect to a SAN, Ethernet adapters that enable storage array controller 101 to connect to a LAN, or the like. Host bus adapters 103A-C may be coupled to processing device 104 via data communication links 105A-C, such as a PCIe bus, for example.

[0056] In an embodiment, storage array controller 101 may include a host bus adapter 114 coupled to an expander 115. Expander 115 may be used to attach a host system to a greater number of storage drives. In an embodiment where host bus adapter 114 is embodied as a SAS controller, expander 115 may be, for example, a SAS expander for enabling host bus adapter 114 to attach to storage drives.

[0057] In an embodiment, storage array controller 101 may include a switch 116 coupled to processing device 104 via data communication link 109. Switch 116 may be a computer hardware device that can create multiple endpoints from a single endpoint, thereby enabling multiple devices to share a single endpoint. Switch 116 may, for example, be a PCIe switch that couples to a PCIe bus (e.g., data communication link 109) and presents multiple PCIe connection points to a midplane.

[0058] In an embodiment, storage array controller 101 includes a data communication link 107 for coupling storage array controller 101 to other storage array controllers. In some examples, data communication link 107 may be a Quick Path Interconnect (QPI) interconnect.

[0059] A conventional storage system using conventional flash drives may implement processes across the flash drives that are part of the conventional storage system. For example, a higher level process of the storage system may initiate and control processes across the flash drives. However, the flash drives of the conventional storage system may include their own storage controllers that also perform the processes. Thus, for the conventional storage system, both higher level processes (e.g., initiated by the storage system) and lower level processes (e.g., initiated by the storage controller of the storage system) may be performed.

[0060] To address various shortcomings of conventional storage systems, operations may be performed by higher level processes rather than by lower level processes. For example, a flash storage system may include a flash drive that does not include a storage controller that provides the process. Thus, the operating system of the flash storage system itself may initiate and control the process. This may be accomplished by a direct mapped flash storage system that directly addresses data blocks within the flash drive and does not require address translation performed by the flash drive's storage controller.

[0061] In an embodiment, storage drive 171A to F may be one or more partitioned storage devices. In some embodiments, one or more partitioned storage devices may be shingled HDDs. In an embodiment, one or more storage devices may be flash-based SSDs. In a partitioned storage device, the partition namespace on the partitioned storage device may be addressed by block groups, which are grouped and aligned in natural sizes to form several addressable areas. In an embodiment utilizing an SSD, the natural size may be an erase block size based on the SSD. In some embodiments, the zones of the partitioned storage device may be defined during the initialization of the partitioned storage device. In an embodiment, the zones may be dynamically defined as data is written to the partitioned storage device.

[0062] In some embodiments, the zones may be heterogeneous, with some zones each being a page group and other zones being multiple page groups. In an embodiment, some zones may correspond to an erase block and other zones may correspond to multiple erase blocks. In an embodiment, for a heterogeneous mix of programming modes, manufacturers, product types, and / or product generations of storage devices such as those suitable for heterogeneous assembly, updating, distributed storage, etc., zones may be any combination of different numbers of pages in units of page groups and / or erase blocks. In some embodiments, zones may be defined as having usage characteristics, such as properties of supporting data with a particular kind of lifespan (e.g., very short lifespan or very long lifespan). These properties may be used by the partitioned storage device to determine how the zone will be managed over the expected lifespan of the zone.

[0063] It should be understood that a zone is a virtual construction. Any particular zone may not have a fixed position at a storage device. Before allocation, a zone may not have any position at a storage device. A zone may correspond to a number, which represents a block of virtual allocatable space, which is the size of an erase block or other block sizes in various embodiments. When the system allocates or opens a zone, the zone is allocated to a flash memory or other solid-state storage memory, and as the system writes to the zone, the page is written to that mapped flash memory or other solid-state storage memory of the partition storage device. When the system closes the zone, the associated erase block or other size block is completed. At some point in the future, the system may delete the zone, which will release the allocated space of the zone. During its life cycle, for example, when the partition storage device performs internal maintenance, the zone may be moved to a different location of the partition storage device.

[0064] In an embodiment, the zone of a partitioned storage device may be in different states. A zone may be in an empty state, where data has not yet been stored at the zone. An empty zone may be explicitly opened or implicitly opened by writing data to the zone. This is the initial state of a zone on a newly partitioned storage device, but it may also be the result of a zone reset. In some embodiments, an empty zone may have a specified location within the flash memory of a partitioned storage device. In an embodiment, the location of the zone may be selected when the empty zone is first opened or first written to (or if the write is buffered in a memory, then later). A zone may be explicitly or implicitly in an open state, where a zone in an open state may be written to store data with a write or append command. In an embodiment, a zone in an open state may also be written using a copy command that copies data from different zones. In some embodiments, a partitioned storage device may have a limit on the number of open zones at a particular time.

[0065] A zone in a closed state is a zone that has been partially written to but has entered a closed state after an explicit close operation has been issued. A zone in a closed state may be left available for future writes, but may reduce some of the runtime overhead consumed by keeping the zone in an open state. In an embodiment, a partitioned storage device may have a limit on the number of closed zones at a particular time. A zone in a full state is a zone that is storing data and can no longer be written to. A zone may be in a full state after a write has written data to the entire zone or due to a zone completion operation. Prior to a completion operation, a zone may or may not have been fully written. However, after a completion operation, it may not be possible to open a zone for further writing without first performing a zone reset operation.

[0066] The mapping from zones to erase blocks (or to shingled tracks in an HDD) can be arbitrary, dynamic, and hidden from view. The process of opening a zone can be an operation that allows a new zone to be dynamically mapped to the underlying storage of a partitioned storage device and then allows data to be written by additional writes to the zone until the zone reaches capacity. A zone can be completed at any time, after which additional data may not be written to the zone. When the data stored at a zone is no longer needed, the zone can be reset, which effectively deletes the contents of the zone from the partitioned storage device, making the physical storage held by that zone available for subsequent data storage. Once a zone has been written and completed, the partitioned storage device ensures that the data stored at the zone will not be lost until the zone is reset. In the time between writing data to a zone and resetting the zone, a zone can be moved between shingled tracks or erase blocks as part of a maintenance operation within the partitioned storage device, such as by copying data to keep the data refreshed or handling memory cell aging in an SSD.

[0067] In embodiments utilizing HDDs, resetting a zone may allow shingled tracks to be assigned to a new open zone that may be opened at some point in the future. In embodiments utilizing SSDs, resetting a zone may cause the zone's associated physical erase blocks to be erased and subsequently reused for storage of data. In some embodiments, a zoned storage device may have a limit on the number of open zones at a point in time to reduce the amount of overhead dedicated to keeping zones open.

[0068] The operating system of the flash storage system can identify and maintain a list of allocation units across multiple flash drives of the flash storage system. An allocation unit can be a whole erase block or multiple erase blocks. The operating system can maintain a mapping or address range that maps addresses directly to erase blocks of the flash drives of the flash storage system.

[0069] The erase blocks mapped directly to the flash drive can be used to rewrite data and erase data. For example, an operation can be performed on one or more allocation units containing first data and second data, where the first data will be retained and the second data is no longer used by the flash storage system. The operating system can initiate a process to write the first data to a new location within other allocation units and erase the second data and mark the allocation unit as available for subsequent data. Therefore, the process can be performed only by the higher-level operating system of the flash storage system, and additional lower-level processes do not need to be performed by the controller of the flash drive.

[0070] Advantages of the process being performed only by the operating system of the flash storage system include improving the reliability of the flash drives of the flash storage system because no unnecessary or redundant write operations are performed during the process. Here, a possible novelty is the concept of initiating and controlling the process at the operating system of the flash storage system. In addition, the process can be controlled by the operating system across multiple flash drives. This is in contrast to the process being performed by the storage controller of the flash drive.

[0071] A storage system may consist of two storage array controllers that share a set of drives for failover purposes, or it may consist of a single storage array controller that provides storage services that utilize multiple drives, or it may consist of a distributed network of storage array controllers, each having a certain number of drives or a certain amount of flash storage, where the storage array controllers in the network cooperate to provide a complete storage service and cooperate in various aspects of the storage service including storage allocation and obsolete item collection.

[0072] Figure 1C A third example system 117 for data storage according to some embodiments is described. For purposes of illustration and not limitation, the system 117 (also referred to herein as a "storage system") includes numerous elements. It may be noted that in other embodiments, the system 117 may include the same, more, or fewer elements configured in the same or different manner.

[0073] In one embodiment, system 117 includes dual peripheral component interconnect ('PCI') flash memory devices 118 with individually addressable fast write storage. System 117 may include a storage device controller 119. In one embodiment, storage device controllers 119A-D may be a CPU, ASIC, FPGA, or any other circuitry that can implement the necessary control structures according to the present disclosure. In one embodiment, system 117 includes flash memory devices (e.g., including flash memory devices 120a-n) operably coupled to various channels of storage device controller 119. Flash memory devices 120a-n may be presented to controllers 119A-D as an addressable collection of flash pages, erase blocks, and / or control elements sufficient to allow storage device controllers 119A-D to program and retrieve various aspects of the flash memory. In one embodiment, the storage device controllers 119A to D can perform operations on the flash memory devices 120a to n, including storing and retrieving the data contents of a page, arranging and erasing any blocks, tracking statistics related to the use and reuse of flash memory pages, erase blocks and cells, tracking and predicting error codes and failures within the flash memory, controlling voltage levels associated with programming and retrieving the contents of flash memory cells, etc.

[0074] In one embodiment, the system 117 may include a RAM 121 to store individually addressable fast write data. In one embodiment, the RAM 121 may be one or more separate discrete devices. In another embodiment, the RAM 121 may be integrated into the storage device controller 119A to D or multiple storage device controllers. The RAM 121 may also be used for other purposes, such as temporary program storage for a processing device (e.g., a CPU) in the storage device controller 119.

[0075] In one embodiment, system 117 may include energy storage device 122, such as a rechargeable battery or capacitor. Energy storage device 122 may store enough energy to power storage controller 119, some amount of RAM (e.g., RAM 121), and some amount of flash memory (e.g., flash memory 120a-120n) for a sufficient time to write the contents of the RAM to the flash memory. In one embodiment, if the storage controller detects a loss of external power, storage controller 119A-D may write the contents of the RAM to the flash memory.

[0076] In one embodiment, the system 117 includes two data communication links 123a, 123b. In one embodiment, the data communication links 123a, 123b may be PCI interfaces. In another embodiment, the data communication links 123a, 123b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). The data communication links 123a, 123b may be based on the Non-Volatile Memory Express ('NVMe') or NVMe Architecture ('NVMf') specification, which allows external connection to the storage device controllers 119A to D from other components in the storage system 117. It should be noted that for convenience, the data communication links may be interchangeably referred to as PCI buses herein.

[0077] The system 117 may also include an external power source (not shown) which may be provided via one or both of the data communication links 123a, 123b, or may be provided separately. An alternative embodiment includes a separate flash memory (not shown) dedicated to storing the contents of the RAM 121. The storage device controllers 119A-D may present a logical device (which may include an addressable fast write logical device) or a distinct portion of the logical address space of the storage device 118 (which may be presented as PCI memory or as a persistent storage device) via the PCI bus. In one embodiment, operations to store into the device are directed into the RAM 121. In the event of a power failure, the storage device controllers 119A-D may write the stored contents associated with the addressable fast write logical storage to a flash memory (e.g., flash memory 120a-n) for long-term persistent storage.

[0078] In one embodiment, the logic device may include some representation of some or all of the contents of flash memory devices 120a-n, wherein the representation allows a storage system (e.g., storage system 117) including storage device 118 to directly address flash memory pages and directly reprogram erase blocks from storage system components external to the storage device over a PCI bus. The representation may also allow one or more of the external components to control and retrieve other aspects of the flash memory, including some or all of the following: tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells across all flash memory devices; tracking and predicting error codes and failures within and across flash memory devices; controlling voltage levels associated with programming and retrieving the contents of flash cells; etc.

[0079] In one embodiment, the energy storage device 122 may be sufficient to ensure completion of ongoing operations on the flash memory devices 120a to 120n, and the energy storage device 122 may power the storage device controllers 119A to D and the associated flash memory devices (e.g., 120a to n) for those operations, as well as for storing fast write RAM to flash memory. The energy storage device 122 may be used to store accumulated statistics and other parameters maintained and tracked by the flash memory devices 120a to n and / or the storage device controller 119. A separate capacitor or energy storage device (e.g., a smaller capacitor near or embedded within the flash memory device itself) may be used for some or all of the operations described herein.

[0080] Various schemes may be used to track and optimize the life of the energy storage component, such as adjusting voltage levels over time, partially discharging the energy storage device 122 to measure corresponding discharge characteristics, etc. If the available energy decreases over time, the effective available capacity of the addressable fast write storage device may be reduced to ensure it can be safely written based on the currently available stored energy.

[0081] Figure 1D A third example storage system 124 for data storage is illustrated in accordance with some embodiments. In one embodiment, the storage system 124 includes storage controllers 125a, 125b. In one embodiment, the storage controllers 125a, 125b are operably coupled to dual PCI storage devices. The storage controllers 125a, 125b are operably coupled (e.g., via a storage network 130) to a number of host computers 127a to n.

[0082] In one embodiment, two storage controllers (e.g., 125a and 125b) provide storage services, such as an SCS block storage array, a file server, an object server, a database or data analysis service, etc. The storage controllers 125a, 125b may provide services to host computers 127a to n external to the storage system 124 through a number of network interfaces (e.g., 126a to d). The storage controllers 125a, 125b may provide integrated services or applications entirely within the storage system 124, thereby forming a converged storage and computing system. The storage controllers 125a, 125b may utilize fast write memory within or across the storage devices 119a to d to journal ongoing operations to ensure that operations are not lost in the event of a power failure, storage controller removal, storage controller or storage system shutdown, or some failure of one or more software or hardware components within the storage system 124.

[0083] In one embodiment, the storage controllers 125a, 125b operate as PCI masters for one or the other PCI bus 128a, 128b. In another embodiment, 128a, 128b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). Other storage system embodiments may operate the storage controllers 125a, 125b as multi-masters for both PCI buses 128a, 128b. Alternatively, a PCI / NVMe / NVMf switching infrastructure or architecture may connect multiple storage controllers. Some storage system embodiments may allow storage devices to communicate directly with each other, rather than just with the storage controllers. In one embodiment, the storage device controller 119a may operate under guidance from the storage controller 125a to retrieve data that has been stored in RAM (e.g., Figure 1C The recalculated version of the RAM contents may be transferred after the storage controller has determined that the operation has been fully committed across the storage system or when the fast write memory on the device has reached a certain usage capacity or after a certain amount of time to ensure improved data security or free up addressable fast write capacity for reuse. This mechanism can be used, for example, to avoid secondary transfers from the storage controller 125a, 125b via a bus (e.g., 128a, 128b). In one embodiment, the recalculation may include compressing the data, appending indexes or other metadata, combining multiple data segments together, performing erasure code calculations, etc.

[0084] In one embodiment, under direction from the storage controllers 125a, 125b, the storage device controllers 119a, 119b are operable to retrieve data from a RAM (eg, Figure 1CThe memory device 120 can be used to calculate data from the data in the RAM 121 of the memory device 125a and transfer the data to other memory devices without involving the memory controllers 125a, 125b. This operation can be used to mirror the data stored in one memory controller 125a to another memory controller 125b, or it can be used to offload the compression, data aggregation and / or erasure coding calculations and transfers to the memory devices to reduce the load on the memory controllers or the memory controller interfaces 129a, 129b to the PCI buses 128a, 128b.

[0085] The storage device controllers 119A-D may include mechanisms for implementing high availability primitives for use by other components of the storage system external to the dual PCI storage device 118. For example, a reservation or exclusion primitive may be provided so that in a storage system having two storage controllers providing highly available storage services, one storage controller may prevent the other storage controller from accessing or continuing to access the storage device. This may be used, for example, in situations where one of the controllers detects that the other controller is not functioning properly or where the interconnect between the two storage controllers itself may not be functioning properly.

[0086] In one embodiment, a storage system for use with dual PCI direct mapped storage devices with individually addressable fast write storage includes a system that manages erase blocks or groups of erase blocks as allocation units for storing data on behalf of a storage service or for storing metadata associated with the storage service (e.g., indexes, logs, etc.) or for proper management of the storage system itself. Flash pages, which may be several kilobytes in size, may be written as data arrives or when the storage system will hold data for a long time interval (e.g., exceeding a defined time threshold). To commit data more quickly, or to reduce the number of writes to the flash memory device, the storage controller may first write the data to an individually addressable fast write storage device on one or more storage devices.

[0087] In one embodiment, the storage controllers 125a, 125b may initiate the use of erase blocks within and across storage devices such as 118 based on the age and expected remaining useful life of the storage devices or based on other statistics. The storage controllers 125a, 125b may initiate obsolete item collection and data migration data between storage devices based on pages that are no longer needed and in order to manage flash page and erase block useful life and in order to manage overall system performance.

[0088] In one embodiment, the storage system 124 may utilize mirroring and / or erasure coding schemes as part of storing data in addressable fast write storage devices and / or as part of writing data to allocation units associated with erase blocks. Erasure codes may be used across storage devices and within erase blocks or allocation units, or within and across flash memory devices on a single storage device to provide redundancy against single or multiple storage device failures or to protect against corruption within flash memory pages caused by flash memory operations or by flash memory cell degradation. Mirroring and erasure coding at various levels may be used to recover from multiple types of failures occurring alone or in combination.

[0089] refer to Figure 2A The embodiments depicted in FIG. 1 and FIG. 2 illustrate a storage cluster that stores user data, such as user data originating from one or more users or client systems or other sources external to the storage cluster. The storage cluster uses erasure coding and redundant copies of metadata to distribute user data across storage nodes housed in a chassis or across multiple chassis. Erasure coding refers to a method of data protection or reconstruction in which data is stored across a set of different locations (such as disks, storage nodes, or geographic locations). Flash memory is a type of solid-state memory that can be integrated with embodiments, but embodiments can be extended to other types of solid-state memory or other storage media, including non-solid-state memory. Control of storage locations and workloads is distributed across storage locations in a cluster peer system. Tasks such as mediating communications between storage nodes, detecting when a storage node becomes unavailable, and balancing I / O (input and output) across storage nodes are all handled on a distributed basis. In some embodiments, data is laid out or distributed across multiple storage nodes in data segments or stripes that support data recovery. Ownership of data can be reassigned within the cluster, independent of the input and output type. This architecture, described in more detail below, allows a storage node in a cluster to fail while the system remains operational because data can be reconstructed from other storage nodes and thus remain available for input and output operations.In various embodiments, a storage node may be referred to as a cluster node, a blade, or a server.

[0090] The storage cluster may be housed in a chassis (i.e., a housing that houses one or more storage nodes). A mechanism for providing power to each storage node (e.g., a power distribution bus) and a communication mechanism (e.g., a communication bus that enables communication between storage nodes) are included in the chassis. According to some embodiments, the storage cluster may operate as a standalone system in one location. In one embodiment, the chassis houses at least two instances of both power distribution and communication buses that can be enabled or disabled individually. The internal communication bus may be an Ethernet bus, however, other technologies such as PCIe, Infiniband, and others are equally suitable. The chassis provides ports for an external communication bus for communication between multiple chassis and with client systems, either directly or through a switch. External communications may use technologies such as Ethernet, Infiniband, Fibre Channel, etc. In some embodiments, the external communication bus uses different communication bus technologies for inter-chassis and client communications. If a switch is deployed within a chassis or between chassis, the switch may be used as a translator between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, the storage cluster may be accessed by clients using a proprietary interface or a standard interface, such as a Network File System ('NFS'), Common Internet File System ('CIFS'), Small Computer System Interface ('SCSI'), or Hypertext Transfer Protocol ('HTTP'). Protocol translation from the client may occur at a switch, a chassis external communication bus, or within each storage node. In some embodiments, multiple chassis may be coupled or connected to each other via an aggregator switch. A portion and / or all of the coupled or connected chassis may be designated as a storage cluster. As discussed above, each chassis may have multiple blades, each blade having a media access control ('MAC') address, but in some embodiments, the storage cluster appears to the external network as having a single cluster IP address and a single MAC address.

[0091] Each storage node may be one or more storage servers, and each storage server is connected to one or more non-volatile solid-state memory units, which may be referred to as storage units or storage devices. One embodiment includes a single storage server in each storage node and between 1 and 8 non-volatile solid-state memory units, however, this one example is not meant to be limiting. The storage server may include a processor, DRAM, and interface for an internal communication bus and power distribution for each of the power buses. In some embodiments, inside the storage node, the interface and the storage unit share a communication bus, such as PCI Express. The non-volatile solid-state memory unit may directly access the internal communication bus interface through the storage node communication bus, or request the storage node to access the bus interface. The non-volatile solid-state memory unit contains an embedded CPU, a solid-state storage controller, and a certain number of solid-state mass storage devices, for example, in some embodiments, between 2 and 32 terabytes ('TB'). Embedded volatile storage media (such as DRAM) and energy storage devices are included in the non-volatile solid-state memory unit. In some embodiments, the energy storage device is a capacitor, supercapacitor, or battery that enables a subset of DRAM content to be transferred to a stable storage medium in the event of a power loss. In some embodiments, the non-volatile solid-state memory cell is constructed with a storage class memory such as phase change or magnetoresistive random access memory ('MRAM') that replaces DRAM and enables reduced power retention devices.

[0092] One of the many features of storage nodes and non-volatile solid-state storage devices is the ability to actively rebuild data in a storage cluster. Storage nodes and non-volatile solid-state storage devices can determine when a storage node or non-volatile solid-state storage device in a storage cluster is unreachable, independent of whether an attempt is made to read data involving that storage node or non-volatile solid-state storage device. The storage nodes and non-volatile solid-state storage devices then cooperate to recover and rebuild the data in at least a portion of the new location. This constitutes an active reconstruction because the system does not need to wait until the data is needed for a read access initiated from a client system employing the storage cluster to rebuild the data. These and other details of the storage memory and its operation are discussed below.

[0093] Figure 2A1 is a perspective view of a storage cluster 161 according to some embodiments, which has multiple storage nodes 150 and internal solid-state storage coupled to each storage node to provide a network attached storage device or storage area network. A network attached storage device, storage area network or storage cluster or other storage storage may include one or more storage clusters 161, each storage cluster having one or more storage nodes 150, arranging both the physical components and the amount of storage storage provided thereby in a flexible and reconfigurable manner. The storage cluster 161 is designed to be assembled in a rack, and one or more racks can be configured and populated for storage storage as needed. The storage cluster 161 has a chassis 138 with multiple slots 142. It should be understood that the chassis 138 can be referred to as a housing, a shell, or a rack unit. In one embodiment, the chassis 138 has fourteen slots 142, but other numbers of slots can be easily designed. For example, some embodiments have 4 slots, 8 slots, 16 slots, 32 slots, or other suitable numbers of slots. In some embodiments, each slot 142 can accommodate one storage node 150. The chassis 138 includes fins 148 that can be used to mount the chassis 138 on a rack. The fans 144 provide air circulation for cooling the storage nodes 150 and their components, but other cooling components can be used, or embodiments without cooling components can be designed. The switch architecture 146 couples the storage nodes 150 within the chassis 138 together and couples to a network for communication with the storage. In the embodiments depicted herein, for illustrative purposes, the slots 142 on the left side of the switch architecture 146 and the fans 144 are shown as being occupied by the storage nodes 150, while the slots 142 on the right side of the switch architecture 146 and the fans 144 are empty and can be used to insert the storage nodes 150. This configuration is an example, and in various other arrangements, one or more storage nodes 150 can occupy the slots 142. In some embodiments, the storage node arrangement need not be sequential or adjacent. The storage nodes 150 are hot-swappable, meaning that the storage nodes 150 can be inserted into or removed from the slots 142 in the chassis 138 without stopping or powering off the system. After the storage nodes 150 are inserted or removed from the slots 142, the system automatically reconfigures to recognize and adapt to the changes. In some embodiments, the reconfiguration includes restoring storage redundancy and / or rebalancing data or load.

[0094] Each storage node 150 may have multiple components. In the embodiment shown here, the storage node 150 includes a printed circuit board 159, which is populated by a CPU 156 (i.e., a processor), a memory 154 coupled to the CPU 156, and a non-volatile solid-state storage device 152 coupled to the CPU 156, but in other embodiments, other mountings and / or components may be used. The memory 154 has instructions executed by the CPU 156 and / or data operated on by the CPU 156. As further explained below, the non-volatile solid-state storage device 152 includes flash memory, or in other embodiments, includes other types of solid-state memory.

[0095] refer to Figure 2A , the storage cluster 161 is scalable, which means that storage capacity with non-uniform storage size can be easily added, as described above. One or more storage nodes 150 can be inserted into or removed from each chassis, and in some embodiments, the storage cluster self-configures. The plug-in storage node 150 can have different sizes whether it is installed in the chassis at the time of delivery or added later. For example, in one embodiment, the storage node 150 can have any multiple of 4TB, such as 8TB, 12TB, 16TB, 32TB, etc. In another embodiment, the storage node 150 can have any multiple of other storage amounts or capacities. The storage capacity of each storage node 150 is broadcast and affects the decision of how to stripe data. For maximum storage efficiency, the embodiment can be self-configured in the stripe to be as wide as possible, subject to predetermined continuous operation requirements, wherein up to one or up to two non-volatile solid-state storage devices 152 units or storage nodes 150 are lost in the chassis.

[0096] Figure 2B 1 is a block diagram showing a communication interconnect 173 and a power distribution bus 172 coupling a plurality of storage nodes 150. Return to Reference Figure 2A In some embodiments, the communication interconnect 173 may be included in or implemented with the switch architecture 146. In the case where multiple storage clusters 161 occupy a rack, in some embodiments, the communication interconnect 173 may be included in or implemented with the top of the rack switch. Figure 2B 1, storage cluster 161 is enclosed within a single chassis 138. External port 176 is coupled to storage node 150 via communication interconnect 173, while external port 174 is directly coupled to storage node. External power port 178 is coupled to power distribution bus 172. Storage node 150 may include non-volatile solid-state storage devices 152 of varying amounts and capacities, as described in reference to FIG. Figure 2A In addition, one or more storage nodes 150 may be computing-only storage nodes, such as Figure 2B. The authorization 168 is implemented on the non-volatile solid-state storage device 152, for example as a list or other data structure stored in a memory. In some embodiments, the authorization is stored in the non-volatile solid-state storage device 152 and supported by software executed on a controller or other processor of the non-volatile solid-state storage device 152. In another embodiment, the authorization 168 is implemented on the storage node 150, for example as a list or other data structure stored in a memory 154 and supported by software executed on a CPU 156 of the storage node 150. In some embodiments, the authorization 168 controls how data is stored in the non-volatile solid-state storage device 152 and where the data is stored in the non-volatile solid-state storage device 152. This control helps determine which type of erasure coding scheme is applied to the data and which storage nodes 150 have which parts of the data. Each authorization 168 can be assigned to a non-volatile solid-state storage device 152. In various embodiments, each authorization may control a range of inode numbers, segment numbers, or other data identifiers assigned to data by the file system, by the storage node 150, or by the non-volatile solid-state storage device 152.

[0097] In some embodiments, each piece of data and each piece of metadata has redundancy in the system. In addition, each piece of data and each piece of metadata has an owner, which can be called an authorization. If that authorization is unreachable, such as by the failure of a storage node, there is a succession plan for how to find that data or that metadata. In various embodiments, there are redundant copies of authorization 168. In some embodiments, authorization 168 has a relationship with a storage node 150 and a non-volatile solid-state storage device 152. Each authorization 168 covering the range of data segment numbers or other identifiers of data can be assigned to a specific non-volatile solid-state storage device 152. In some embodiments, authorization 168 for all such ranges is distributed on the non-volatile solid-state storage device 152 of the storage cluster. Each storage node 150 has a network port that provides access to the non-volatile solid-state storage device 152 of that storage node 150. Data can be stored in segments, and in some embodiments, the segment is associated with a segment number and that segment number is the indirectness of the configuration of a RAID (Redundant Array of Independent Disks) stripe. The assignment and use of authorization 168 therefore establishes indirectness to data. According to some embodiments, indirectness may be referred to as the ability to reference data indirectly (in this case, via authorization 168). A segment identifies a set of non-volatile solid-state storage devices 152 and a local identifier to the set of non-volatile solid-state storage devices 152 that may contain data. In some embodiments, the local identifier is an offset into the device and may be reused sequentially by multiple segments. In other embodiments, the local identifier is unique to a particular segment and is never reused. The offset in the non-volatile solid-state storage device 152 is applied to locate data for writing to or reading from the non-volatile solid-state storage device 152 (in the form of a RAID stripe). Data is striped across multiple units of the non-volatile solid-state storage device 152, which may include a non-volatile solid-state storage device 152 having an authorization 168 for a particular data segment or a non-volatile solid-state storage device 152 different from the non-volatile solid-state storage device 152.

[0098] If the location where a particular data segment is located changes, for example, during data movement or data reconstruction, the authorization 168 for that data segment should be consulted at the non-volatile solid-state storage device 152 or storage node 150 with that authorization 168. In order to locate a particular piece of data, an embodiment calculates a hash value of the data segment or applies an index node number or a data segment number. The output of this operation points to the non-volatile solid-state storage device 152 with the authorization 168 for that particular piece of data. In some embodiments, there are two stages for this operation. The first stage maps an entity identifier (ID) (such as a segment number, an index node number, or a directory number) to an authorization identifier. This mapping may include, for example, the calculation of a hash or a bit mask. The second stage is to map the authorization identifier to a specific non-volatile solid-state storage device 152, which can be accomplished by explicit mapping. The operation is repeatable so that when the calculation is performed, the result of the calculation can be repeatedly and reliably pointed to a specific non-volatile solid-state storage device 152 with that authorization 168. The operation may include a reachable storage node set as input. If the set of reachable non-volatile solid-state storage units changes, the best set changes. In some embodiments, the value maintained is the current assignment (which is always true) and the calculated value is the target assignment that the cluster will attempt to reconfigure. This calculation can be used to determine the best non-volatile solid-state storage device 152 for authorization when there is a group of non-volatile solid-state storage devices 152 that are reachable and constitute the same cluster. The calculation also determines an ordered set of peer non-volatile solid-state storage devices 152, which will also record the authorization mapped to the non-volatile solid-state storage device so that the authorization can be determined even when the assigned non-volatile solid-state storage device is unreachable. In some embodiments, if a specific authorization 168 is not available, a duplicate or alternative authorization 168 can be consulted.

[0099] refer to Figure 2A and 2B, two of the many tasks of the CPU 156 on the storage node 150 are to decompose write data and reassemble read data. When the system has determined that data is to be written, the authorization 168 for that data is located as described above. When the segment ID of the data has been determined, the write request is forwarded to the non-volatile solid-state storage device 152 of the host that is currently determined to be the authorization 168 determined from the segment. Then, the host CPU 156 of the storage node 150 on which the non-volatile solid-state storage device 152 and the corresponding authorization 168 reside decomposes or splits the data and transmits the data out to various non-volatile solid-state storage devices 152. The transmitted data is written as a data stripe according to an erasure coding scheme. In some embodiments, the request pulls the data, and in other embodiments, the data is pushed. In contrast, when data is read, the authorization 168 for the segment ID containing the data is located as described above. The host CPU 156 of the storage node 150 on which the non-volatile solid-state storage device 152 and the corresponding grant 168 reside requests data from the non-volatile solid-state storage device and the corresponding storage node pointed to by the grant. In some embodiments, the data is read from the flash storage device as a data stripe. Then, the host CPU 156 of the storage node 150 reorganizes the read data, thereby correcting any errors (if any) according to the appropriate erasure coding scheme and forwarding the reorganized data to the network. In other embodiments, some or all of these tasks can be handled in the non-volatile solid-state storage device 152. In some embodiments, the segment host requests data to be sent to the storage node 150 by requesting a page from the storage device and then sending the data to the storage node that issued the original request.

[0100] In an embodiment, authorization 168 operates to determine how operations will be performed for specific logical elements. Each of the logical elements can be operated by specific authorization across multiple storage controllers of the storage system. Authorization 168 can communicate with multiple storage controllers so that multiple storage controllers perform operations together for those specific logical elements.

[0101] In embodiments, a logical element may be, for example, a file, a directory, an object bucket, an individual object, a descriptive portion of a file or object, other forms of key-value databases or tables. In embodiments, performing an operation may involve, for example, ensuring consistency, structural integrity, and / or recoverability of other operations on the same logical element, reading metadata and data associated with that logical element, determining what data should be persistently written to the storage system to preserve any changes to the operation, or metadata and data may be determined where modular storage devices attached to multiple storage controllers in the storage system are to be stored.

[0102] In some embodiments, operations are token-based transactions that are efficiently passed within a distributed system. Each transaction may be accompanied or associated with a token that gives permission to perform the transaction. In some embodiments, authorization 168 can maintain the pre-transaction state of the system until the operation is completed. Token-based communications can be completed without global locks across the system and can also restart operations in the event of corruption or other failures.

[0103] In some systems, such as in UNIX-style file systems, data is handled with index nodes (inodes) that specify a data structure representing an object in the file system. For example, an object may be a file or a directory. Metadata may accompany the object as attributes, such as permission data and a creation timestamp, among other attributes. Segment numbers may be assigned to all or part of this object in the file system. In other systems, data segments are handled with segment numbers assigned elsewhere. For discussion purposes, a distribution unit is an entity, and an entity may be a file, a directory, or a segment. That is, an entity is a unit of data or metadata stored by a storage system. Entities are grouped into sets called grants. Each grant has a grant owner, which is a storage node that has the exclusive right to update the entities in the grant. In other words, a storage node contains a grant, and a grant contains an entity.

[0104] According to some embodiments, a segment is a logical container for data. A segment is an address space between the media address space and the physical flash location, i.e., the data segment number is in this address space. A segment may also contain metadata that enables data redundancy to be restored (rewritten to a different flash location or device) without involving higher-level software. In one embodiment, the internal format of a segment contains client data and a media map to determine the location of that data. Where applicable, each data segment is protected by breaking it into several data and parity fragments, e.g., to prevent the effects of memory and other failures. The data and parity fragments are distributed (i.e., striped) across the non-volatile solid-state storage device 152 coupled to the host CPU 156 according to an erasure coding scheme (see Figure 2E and 2G ). In some embodiments, the use of the term segment refers to a container and its location in the address space of a segment. According to some embodiments, the use of the term stripe refers to the same set of fragments as a segment and includes how the fragments and redundant or parity information are distributed.

[0105] A series of address space transformations occur across the entire storage system. At the top is a directory entry (file name) linked to an index node. The index node points to a media address space where data is logically stored. The media address can be mapped through a series of indirect media to expand the load of large files or implement data services such as deduplication or snapshots. The media address can be mapped through a series of indirect media to expand the load of large files or implement data services such as deduplication or snapshots. Then, the segment address is translated into a physical flash location. According to some embodiments, the physical flash location has an address range defined by the amount of flash memory in the system. The media address and segment address are logical containers, and in some embodiments, 128-bit or larger identifiers are used so that they are actually infinite, where the possibility of reuse is calculated to be longer than the expected life of the system. In some embodiments, addresses from logical containers are allocated in a hierarchical manner. Initially, each non-volatile solid-state storage device 152 unit can be assigned an address space range. Within this assigned range, the non-volatile solid-state storage device 152 is able to assign addresses without synchronizing with other non-volatile solid-state storage devices 152.

[0106] Data and metadata are stored by a set of underlying storage layouts optimized for different workload types and storage devices. These layouts incorporate a variety of redundancy schemes, compression formats, and indexing algorithms. Some of these layouts store information about authorization and authorization masters, while other layouts store file metadata and file data. Redundancy schemes include error correction codes that tolerate corrupted bits within a single storage device (e.g., a NAND flash chip), erasure codes that tolerate failures of multiple storage nodes, and replication schemes that tolerate data center or regional failures. In some embodiments, low-density parity check ('LDPC') codes are used within a single storage unit. In some embodiments, Reed-Solomon encoding is used within a storage cluster, and mirroring is used within a storage grid. Metadata can be stored using an ordered log structure index (e.g., a log structure merge tree), and large data may not be stored in a log structure layout.

[0107] In order to maintain consistency across multiple copies of an entity, storage nodes implicitly agree on two things through computation: (1) the authorization containing the entity, and (2) the storage node containing the authorization. The assignment of entities to authorizations can be accomplished by pseudo-randomly assigning entities to authorizations, by splitting entities into ranges based on an externally generated key, or by placing a single entity into each authorization. Examples of pseudo-random schemes are linear hashing and hashing of the replication under scalable hashing ('RUSH') family, including controlled replication under scalable hashing ('CRUSH'). In some embodiments, pseudo-random assignment is only used to assign authorizations to nodes, because the set of nodes can change. The set of authorizations cannot change, so any subjective function can be applied in these embodiments. Some placement schemes automatically place authorizations on storage nodes, while other placement schemes rely on an explicit mapping of authorizations to storage nodes. In some embodiments, a pseudo-random scheme is used to map from each authorization to a set of candidate authorization owners. A pseudo-random data distribution function related to CRUSH can assign authorizations to storage nodes and create a list of where to assign authorizations. Each storage node has a copy of the pseudo-random data distribution function and can derive the same calculations to distribute and later find or locate the grant. In some embodiments, each of the pseudo-random schemes requires a reachable set of storage nodes as input in order to derive the same target node. Once an entity has been placed in a grant, the entity can be stored on a physical device so that expected failures will not result in unexpected data loss. In some embodiments, the rebalancing algorithm attempts to store copies of all entities within a grant in the same layout and on the same set of machines.

[0108] Examples of expected failures include device failures, stolen machines, data center fires, and regional disasters, such as nuclear or geological events. Different failures result in different levels of acceptable data loss. In some embodiments, the theft of a storage node affects neither the security nor the reliability of the system, while regional events may result in no data loss, a few seconds or minutes of update loss, or even complete data loss, depending on the system configuration.

[0109] In an embodiment, the placement of data for storage redundancy is independent of the placement of authorization for data consistency. In some embodiments, the storage node containing authorization does not contain any persistent storage device. Instead, the storage node is connected to a non-volatile solid-state storage unit that does not contain authorization. The communication interconnection between the storage node and the non-volatile solid-state storage unit consists of a variety of communication technologies and has non-uniform performance and fault tolerance characteristics. In some embodiments, as mentioned above, the non-volatile solid-state storage unit is connected to the storage node via PCI Express, and the storage node is connected together in a single chassis using an Ethernet backplane, and the chassis are connected together to form a storage cluster. In some embodiments, the storage cluster is connected to the client using Ethernet or Fiber Channel. If multiple storage clusters are configured as a storage grid, then multiple storage clusters are connected using the Internet or other long-distance networking links (e.g., "metro scale" links or private links that do not cross the Internet).

[0110] The authorized owner has the exclusive right to modify the entity, migrate the entity from one non-volatile solid-state storage unit to another non-volatile solid-state storage unit, and add and remove copies of the entity. This allows the redundancy of the underlying data to be maintained. When the authorized owner fails, is to be stopped or overloaded, the authorization is transferred to the new storage node. Transient failures make it very important to ensure that all non-faulty machines agree on the new authorized location. Ambiguities caused by transient failures can be automatically implemented through consensus protocols (such as Paxos) and hot-warm failover schemes via manual intervention by a remote system administrator or by a local hardware administrator (for example, by physically removing the failed machine from the cluster, or pressing a button on the failed machine). In some embodiments, a consensus protocol is used and failover is automatic. According to some embodiments, if too many failures or replication events occur in too short a period of time, the system enters a self-protection mode and stops replication and data movement activities until the administrator intervenes.

[0111] When authorization is transferred between a storage node and an authorization owner to update an authorization of an entity, the system transmits messages between the storage node and the non-volatile solid-state storage unit. Regarding persistent messages, messages with different purposes have different types. Depending on the type of message, the system maintains different sorting and durability guarantees. When processing persistent messages, messages are temporarily stored with multiple persistent and non-persistent storage hardware technologies. In some embodiments, messages are stored in RAM, NVRAM, and on NAND flash memory devices, and various protocols are used to effectively use each storage medium. Latency-sensitive client requests can be kept in replicated NVRAM, and then, later kept in NAND, while background rebalancing operations are kept directly to NAND.

[0112] Persistent messages are persistently stored before being transmitted. This allows the system to continue to serve client requests despite failures and component replacements. Although many hardware components contain unique identifiers visible to system administrators, manufacturers, hardware supply chains, and ongoing monitoring quality control infrastructure, applications running on infrastructure addresses virtualize the addresses. These virtualized addresses do not change over the life of the storage system, regardless of component failures and replacements. This allows each component of the storage system to be replaced over time without reconfiguring or interrupting client request processing, i.e., the system supports non-disruptive upgrades.

[0113] In some embodiments, virtualized addresses are stored with sufficient redundancy. The continuous monitoring system correlates hardware and software states with hardware identifiers. This allows for detection and prediction of failures due to faulty components and manufacturing details. In some embodiments, the monitoring system can also proactively transfer authorizations and entities away from affected devices before a failure occurs by removing components from critical paths.

[0114] Figure 2C 1 is a multi-level block diagram showing the contents of a storage node 150 and the contents of a non-volatile solid-state storage device 152 of the storage node 150. In some embodiments, data is passed to and from the storage node 150 by a network interface controller ('NIC') 202. Each storage node 150 has a CPU 156 and one or more non-volatile solid-state storage devices 152, as discussed above. Figure 2C Moving down one level in FIG. 1 , each non-volatile solid-state storage device 152 has relatively fast non-volatile solid-state memory, such as non-volatile random access memory ('NVRAM') 204 and flash memory 206. In some embodiments, NVRAM 204 may be a component (DRAM, MRAM, PCM) that does not require program / erase cycles and may be a memory that can support writes much more frequently than reads from the memory. Figure 2CMoving down another level in FIG. 2 , in one embodiment, NVRAM 204 is implemented as a high-speed volatile memory, such as dynamic random access memory (DRAM) 216, backed up by an energy reserve 218. The energy reserve 218 provides sufficient power to keep the DRAM 216 powered long enough to transfer the contents to the flash memory 206 in the event of a power failure. In some embodiments, the energy reserve 218 is a capacitor, supercapacitor, battery, or other device that supplies a suitable energy supply sufficient to enable the contents of the DRAM 216 to be transferred to a stable storage medium in the event of a loss of power. The flash memory 206 is implemented as a plurality of flash die 222, which may be referred to as a package of flash die 222 or an array of flash die 222. It should be appreciated that the flash die 222 may be packaged in any number of ways, with a single die per package, multiple die per package (i.e., a multi-chip package), in a hybrid package, as an exposed die on a printed circuit board or other substrate, as an encapsulated die, and the like. In the embodiment shown, the non-volatile solid-state storage device 152 has a controller 212 or other processor and an input-output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or the network interface controller 202 of the flash storage node 150. The flash input-output (I / O) port 220 is coupled to the flash die 222, and the direct memory access unit (DMA) 214 is coupled to the controller 212, the DRAM 216, and the flash die 222. In the embodiment shown, the I / O port 210, the controller 212, the DMA unit 214, and the flash I / O port 220 are implemented on a programmable logic device ('PLD') 208, such as an FPGA. In this embodiment, each flash die 222 has pages organized as 16kB (kilobyte) pages 224 and registers 226 through which data can be written to or read from the flash die 222. In further embodiments, other types of solid-state memory are used in place of or in addition to the flash memory illustrated within flash die 222 .

[0115] In various embodiments as disclosed herein, storage cluster 161 can be generally contrasted with storage array. Storage node 150 is part of a collection that creates storage cluster 161. Each storage node 150 owns a slice of data and provides the calculations required for the data. Multiple storage nodes 150 cooperate to store and retrieve data. As commonly used in storage arrays, storage memory or storage devices are less involved in processing and manipulating data. Storage memory or storage devices in storage arrays receive commands to read, write or erase data. Storage memory or storage devices in storage arrays do not know the larger system in which they are embedded, or what the data means. Storage memory or storage devices in storage arrays may include various types of storage memory, such as RAM, solid-state drives, hard disk drives, etc. The non-volatile solid-state storage device 152 unit described herein has multiple interfaces that are simultaneously effective and serve multiple purposes. In some embodiments, a certain functionality of storage node 150 is shifted into storage unit 152, thereby transforming storage unit 152 into a combination of storage unit 152 and storage node 150. Placing computation (as opposed to storing data) into the storage unit 152 places this computation closer to the data itself. Various system embodiments have a hierarchy of storage node tiers with different capabilities. In contrast, in a storage array, the controller owns and knows everything about all the data that the controller manages in a shelf or storage device. In a storage cluster 161, as described herein, multiple non-volatile solid-state storage device 152 units and / or multiple controllers in a storage node 150 cooperate in various ways (e.g., for erasure coding, data partitioning, metadata communication and redundancy, storage capacity expansion or contraction, data recovery, etc.).

[0116] Figure 2D Demonstration of use Figure 2A 150 and storage device 152 units. In this version, each non-volatile solid-state storage device 152 unit has a storage server environment in the chassis 138 (see Figure 2A ) on a PCIe (Peripheral Component Interconnect Express) board, for example, a controller 212 (see Figure 2C ) of the processor, FPGA, flash memory 206 and NVRAM 204 (which is a supercapacitor-backed DRAM 216, see Figure 2B and 2C ). The non-volatile solid-state storage device 152 unit may be implemented as a single board containing the storage device and may be the maximum tolerable failure domain within the chassis. In some embodiments, up to two non-volatile solid-state storage device 152 units may fail and the device will continue without data loss.

[0117] In some embodiments, the physical storage device is divided into named areas based on application use. NVRAM 204 is a continuous block of memory retained in the non-volatile solid-state storage device 152 DRAM 216 and is supported by NAND flash memory. NVRAM 204 is logically divided into multiple memory areas, two of which are written as spools (e.g., spool_region). The space within the NVRAM 204 spool is managed independently by each authorization 168. Each device provides a certain amount of storage space to each authorization 168. That authorization 168 further manages the life cycle and allocation within that space. The instance of the spool includes a distributed transaction or concept. When the main power to the non-volatile solid-state storage device 152 unit fails, the on-board supercapacitor provides a shorter duration of power retention. During this retention interval, the content of NVRAM 204 is refreshed to the flash memory 206. At the next power-on, the content of NVRAM 204 is restored from the flash memory 206.

[0118] With respect to the storage unit controller, the responsibility of the logical "controller" is distributed across each of the blades containing the authorization 168. This logical control is distributed across Figure 2D 1 and 2 are shown as host controller 242, mid-tier controller 244, and storage unit controller 246. The management of the control plane and storage plane are treated independently, but the components can be physically co-located on the same blade. Each authorization 168 effectively acts as an independent controller. Each authorization 168 provides its own data and metadata structures, its own background workers, and maintains its own life cycle.

[0119] Figure 2E It is displayed in Figure 2D Used in a storage server environment Figure 2A 1. A hardware block diagram of a blade 252 of an embodiment of a storage node 150 and storage unit 152 of FIG. 10 interacts with underlying physical resources, a control plane 254, a compute and storage plane 256, 258, and an authorization 168. The control plane 254 is partitioned into a number of authorizations 168 that can run on any of the blades 252 using the compute resources in the compute plane 256. The storage plane 258 is partitioned into a set of devices, each of which provides access to the flash memory 206 and NVRAM 204 resources. In one embodiment, the compute plane 256 can perform the operations of a storage array controller on one or more devices of the storage plane 258 (e.g., a storage array), as described herein.

[0120] exist Figure 2EIn the compute and storage planes 256, 258 of the system, the authorizations 168 interact with the underlying physical resources (i.e., devices). From the perspective of the authorizations 168, their resources are striped across all physical devices. From the perspective of the devices, they provide resources to all authorizations 168, no matter where the authorizations happen to be running. Each authorization 168 has or has been assigned one or more partitions 260 of storage memory in the storage unit 152, such as the partitions 260 in the flash memory 206 and the NVRAM 204. Each authorization 168 uses those allocated partitions 260 belonging to it for writing or reading user data. Authorizations can be associated with different amounts of physical storage of the system. For example, one authorization 168 can have a greater number of partitions 260 or larger size partitions 260 in one or more storage units 152 than one or more other authorizations 168.

[0121] Figure 2F Depicts the elasticity software layer in the blade 252 of a storage cluster according to some embodiments. In the elastic architecture, the elasticity software is symmetrical, i.e., the computing module 270 of each blade runs Figure 2F The storage manager 274 executes read and write requests from other blades 252 for data and metadata stored in the local storage unit 152 NVRAM 204 and flash memory 206. The authorization 168 implements the client request by issuing the necessary reads and writes to the blade 252 on whose storage unit 152 the corresponding data or metadata resides. The endpoint 272 parses the client connection request received from the switch architecture 146 monitoring software, relays the client connection request to the authorization 168 responsible for implementation, and relays the response of the authorization 168 to the client. The symmetric three-tier structure realizes a high degree of concurrency in the storage system. In these embodiments, elasticity is efficiently and reliably scaled out. In addition, elasticity implements a unique scale-out technology that evenly balances work across all resources regardless of the client access pattern, and maximizes concurrency by eliminating most of the need for inter-blade coordination that usually occurs for conventional distributed locking.

[0122] Still refer to Figure 2F, the authority 168 running in the compute module 270 of the blade 252 performs the internal operations required to complete the client request. One feature of resiliency is that the authority 168 is stateless, that is, it caches valid data and metadata in the DRAM of its own blade 252 for fast access, but the authority stores each update in its partition of NVRAM 204 on three separate blades 252 until the update has been written to flash memory 206. In some embodiments, all storage system writes to NVRAM 204 are triplicated to the partitions on the three separate blades 252. With three mirrored NVRAM 204 and persistent storage protected by parity and Reed-Solomon RAID checksums, the storage system can withstand the concurrent failure of two blades 252 without losing data, metadata, or access to either.

[0123] Because the authorization 168 is stateless, it can be migrated between blades 252. Each authorization 168 has a unique identifier. In some, the partitions of NVRAM 204 and flash memory 206 are associated with the identifier of the authorization 168, rather than the blade 252 on which it is running. Therefore, when the authorization 168 migrates, the authorization 168 continues to manage the same storage partitions from its new location. When a new blade 252 is installed in an embodiment of the storage cluster, the system automatically rebalances the load by partitioning the storage device of the new blade 252 for use by the authorization 168 of the system, migrating the selected authorization 168 to the new blade 252, opening the endpoint 272 on the new blade 252, and including it in the client connection distribution algorithm of the switch architecture 146.

[0124] The migrated authority 168 maintains the contents of its NVRAM 204 partition on flash memory 206 from its new location, processes read and write requests from other authorities 168, and completes client requests directed to it by endpoint 272. Similarly, if a blade 252 fails or is removed, the system redistributes its authorities 168 among the remaining blades 252 in the system. The redistributed authorities 168 continue to perform their original functions from their new location.

[0125] Figure 2GDepicts the authorities 168 and storage resources in the blades 252 of a storage cluster according to some embodiments. Each authority 168 is specifically responsible for a partition of flash memory 206 and NVRAM 204 on each blade 252. An authority 168 manages the content and integrity of its partition independently of other authorities 168. An authority 168 compresses incoming data and temporarily saves it in a partition of its NVRAM 204, and then consolidates, RAID protects, and maintains the data in a segment of the storage device in its flash memory 206 partition. As the authority 168 writes data to the flash memory 206, the storage manager 274 performs the necessary flash translations to optimize write performance and maximize media life. In the background, the authority 168 "does garbage collection" or reclaims space occupied by data that has been made obsolete by clients overwriting the data. It should be appreciated that because the partitions of the authority 168 are disjoint, no distributed locking is required to execute clients and writes or to perform background functions.

[0126] The embodiments described herein may utilize various software, communication and / or networking protocols. In addition, the configuration of the hardware and / or software may be adjusted to accommodate various protocols. For example, embodiments may utilize Active Directory, which is a Windows TM A database-based system that provides authentication, directory, policy and other services in an environment. In these embodiments, LDAP (Lightweight Directory Access Protocol) is an example application protocol for querying and modifying items in a directory service provider (such as an active directory). In some embodiments, a network lock manager ('NLM') is used as a facility that works in cooperation with a network file system ('NFS') to provide System V-style advisory file and record locking via a network. The Server Message Block ('SMB') protocol (a version of which is also called the Common Internet File System ('CIFS')) can be integrated with the storage system discussed herein. SMP operations are application layer network protocols that are typically used to provide shared access to files, printers, and serial ports, as well as miscellaneous communications between nodes on a network. SMB also provides an authenticated inter-process communication mechanism. AMAZON TMS3 (Simple Storage Service) is a web service provided by Amazon Web Services, and the system described herein can interface with Amazon S3 through web service interfaces (REST (Representational State Transfer), SOAP (Simple Object Access Protocol), and BitTorrent). The RESTful API (Application Programming Interface) decomposes transactions to create a series of small modules. Each module addresses a specific underlying part of the transaction. The control or permissions provided by these embodiments, especially for object data, can include the use of access control lists ('ACLs'). An ACL is a list of permissions attached to an object, and the ACL specifies which users or system processes are authorized to access an object and what operations are allowed on a given object. The system can utilize Internet Protocol version 6 ('IPv6') as well as IPv4, a communication protocol for providing identification and location systems and routing services across the Internet for computers on a network. Packet routing between networked systems can include equal cost multipath routing ('ECMP'), which is a routing strategy in which next-hop packet forwarding to a single destination can occur via multiple "best paths" that are at the top of the routing metric calculation. Multipath routing can be used with most routing protocols because it is a per-hop decision limited to a single router. The software can support multi-tenancy, which is an architecture in which a single instance of a software application serves multiple customers. Each customer may be referred to as a tenant. In some embodiments, tenants may be given the ability to customize some parts of the application, but may not be able to customize the code of the application. An embodiment may maintain an audit log. An audit log is a document that records events in a computing system. In addition to recording what resources are accessed, audit log entries typically include destination and source addresses, timestamps, and user login information to comply with various regulations. An embodiment may support various key management strategies, such as encryption key rotation. In addition, the system may support a dynamic root password or a change in a dynamic password.

[0127] Figure 3A A diagram of a storage system 306 coupled for data communication with a cloud service provider 302 is presented in accordance with some embodiments of the present disclosure. Although depicted in less detail, Figure 3A The storage system 306 depicted in FIG. 3 may be similar to the storage system 306 described above with reference to FIG. Figures 1A to 1D and Figures 2A to 2G In some embodiments, Figure 3AThe storage system 306 depicted in the diagram may be embodied as a storage system including unbalanced active / active controllers, a storage system including balanced active / active controllers, a storage system including active / active controllers (in which less than all resources of each controller are utilized such that each controller has reserved resources available to support failover), a storage system including fully active / active controllers, a storage system including data set isolation controllers, a storage system including a two-tier architecture with a front-end controller and a back-end integrated storage controller, a storage system including a scale-out cluster of dual-controller arrays, and combinations of such embodiments.

[0128] exist Figure 3A 304, the storage system 306 is coupled to the cloud service provider 302 via a data communication link 304. The data communication link 304 may be embodied as a dedicated data communication link, a data communication path provided by using one or more data communication networks, such as a wide area network ('WAN') or a LAN, or some other mechanism capable of transmitting digital information between the storage system 306 and the cloud service provider 302. This data communication link 304 may be completely wired, completely wireless, or some aggregation of wired and wireless data communication paths. In this example, digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using one or more data communication protocols. For example, the digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using the Handheld Device Transfer Protocol ('HDTP'), the Hypertext Transfer Protocol ('HTTP'), the Internet Protocol ('IP'), the Real-time Transport Protocol ('RTP'), the Transmission Control Protocol ('TCP'), the User Datagram Protocol ('UDP'), the Wireless Application Protocol ('WAP'), or other protocols.

[0129] For example, Figure 3A 304. The cloud service provider 302 depicted in FIG. 304 may be embodied as a system and computing environment that provides a number of services to users of the cloud service provider 302 by sharing computing resources via a data communication link 304. The cloud service provider 302 may provide on-demand access to a pool of shared configurable computing resources (e.g., computer networks, servers, storage devices, applications and services, etc.). The shared pool of configurable resources may be rapidly provisioned and released to users of the cloud service provider 302 with minimal management effort. Generally, users of the cloud service provider 302 are unaware of the precise computing resources used by the cloud service provider 302 to provide services. Although in many cases such a cloud service provider 302 may be accessed via the Internet, readers in the art will recognize that any system that abstracts the use of shared resources to provide services to users via any data communication link may be considered a cloud service provider 302.

[0130] exist Figure 3A , cloud service provider 302 may be configured to provide various services to storage system 306 and users of storage system 306 by implementing various service models. For example, cloud service provider 302 may be configured to provide services by implementing an infrastructure as a service ('IaaS') service model, by implementing a platform as a service ('PaaS') service model, by implementing a software as a service ('SaaS') service model, by implementing an authentication as a service ('AaaS') service model, by implementing a storage as a service model (in which cloud service provider 302 provides access to its storage infrastructure for use by storage system 306 and users of storage system 306), etc. The reader will appreciate that cloud service provider 302 may be configured to provide additional services to storage system 306 and users of storage system 306 by implementing additional service models, as the service models described above are included for purposes of explanation only and in no way represent a limitation on services that may be provided by cloud service provider 302 or a limitation on service models that may be implemented by cloud service provider 302.

[0131] exist Figure 3A In the example depicted in , cloud service provider 302 may be embodied as, for example, a private cloud, a public cloud, or a combination of private and public clouds. In embodiments where cloud service provider 302 is embodied as a private cloud, cloud service provider 302 may be dedicated to providing services to a single organization, rather than providing services to multiple organizations. In embodiments where cloud service provider 302 is embodied as a public cloud, cloud service provider 302 may provide services to multiple organizations. In yet alternative embodiments, cloud service provider 302 may be embodied as a mix of private and public cloud services with a hybrid cloud deployment.

[0132] although Figure 3A 306 and users of the storage system 306. For example, the storage system 306 may be coupled to (or even include) a cloud storage gateway. For example, this cloud storage gateway may be embodied as a hardware-based or software-based device that is located on premise with the storage system 306. This cloud storage gateway may operate as a bridge between local applications executed on the storage system 306 and remote cloud-based storage devices utilized by the storage system 306. By using a cloud storage gateway, an organization can move the primary iSCSI or NAS to the cloud service provider 302, thereby enabling the organization to save space on its locally deployed storage system. This cloud storage gateway may be configured to emulate a disk array, a block-based device, a file server, or other storage system that can translate SCSI commands, file server commands, or other appropriate commands into a REST space protocol that facilitates communication with the cloud service provider 302.

[0133] In order to enable storage system 306 and users of storage system 306 to use services provided by cloud service provider 302, a cloud migration process may be performed during which data, applications, or other elements from an organization's local systems (or even from another cloud environment) are moved to cloud service provider 302. In order to successfully migrate data, applications, or other elements to the environment of cloud service provider 302, middleware such as cloud migration tools may be used to bridge the gap between the environment of cloud service provider 302 and the environment of the organization. Such cloud migration tools may also be configured to address the potential high network costs and long transfer times associated with migrating large amounts of data to cloud service provider 302, as well as to address security issues associated with sensitive data to cloud service provider 302 via data communications networks. In order to enable storage system 306 and users of storage system 306 to further use services provided by cloud service provider 302, a cloud orchestrator may also be used to arrange and coordinate automated tasks to create a merged process or workflow. This cloud orchestrator may perform tasks such as configuring various components, whether those components are cloud components or locally deployed components, and managing the interconnections between such components. Cloud orchestrators simplify communication and connectivity between components to ensure links are properly configured and maintained.

[0134] exist Figure 3A In the example depicted in , and as briefly described above, the cloud service provider 302 may be configured to provide services to the storage system 306 and users of the storage system 306 by using a SaaS service model, thereby eliminating the need to install and run applications on local computers, which can simplify the maintenance and support of the applications. According to various embodiments of the present disclosure, such applications may take many forms. For example, the cloud service provider 302 may be configured to provide access to data analysis applications to the storage system 306 and users of the storage system 306. Such data analysis applications may be configured, for example, to receive a large amount of telemetry data that is phoned home by the storage system 306. Such telemetry data may describe various operating characteristics of the storage system 306 and may be analyzed for a number of purposes, including, for example, determining the health of the storage system 306, identifying workloads executed on the storage system 306, predicting when the storage system 306 will run out of various resources, recommending configuration changes, hardware or software upgrades, workflow migrations, or other actions that may improve the operation of the storage system 306.

[0135] The cloud service provider 302 may also be configured to provide access to a virtualized computing environment to the storage system 306 and users of the storage system 306. For example, such a virtualized computing environment may be embodied as a virtual machine or other virtualized computer hardware platform, a virtual storage device, virtualized computer network resources, etc. Examples of such a virtualized environment may include a virtual machine created to emulate an actual computer, a virtualized desktop environment that separates a logical desktop from a physical machine, a virtualized file system that allows uniform access to different types of specific file systems, and many others.

[0136] although Figure 3A The example depicted in FIG. 3 illustrates storage system 306 coupled for data communication with cloud service provider 302, but in other embodiments, storage system 306 may be part of a hybrid cloud deployment in which private cloud elements (e.g., private cloud services, on-premises infrastructure, etc.) and public cloud elements (e.g., public cloud services, infrastructure, etc. that may be provided by one or more cloud service providers) are combined to form a single solution, orchestrated among various platforms. Such a hybrid cloud deployment may utilize hybrid cloud management software, such as, for example, Microsoft Azure. TM Azure TM Arc, which centralizes the management of hybrid cloud deployments to any infrastructure and enables deployment of services anywhere. In this example, the hybrid cloud management software can be configured to create, update, and delete resources (both physical and virtual) that form a hybrid cloud deployment, allocate compute and storage to specific workloads, monitor workloads and resources for performance, policy compliance, updates and patches, security status, or perform a variety of other tasks.

[0137] The reader will appreciate that by pairing the storage system described herein with one or more cloud service providers, a variety of products can be implemented. For example, where cloud resources are used to protect applications and data from damage caused by disasters, including in embodiments where the storage system can be used as primary data storage, Disaster Recovery as a Service ('DRaaS') can be provided. In such embodiments, full system backups can be performed, which allows business continuity to be maintained in the event of a system failure. In such embodiments, cloud data backup technology (either independently or as part of a larger DRaaS solution) can also be integrated into an overall solution including the storage system described herein and a cloud service provider.

[0138] The storage systems and cloud service providers described herein can be used to provide a number of security features. For example, the storage system can encrypt data at rest (and data can be sent to and from the encrypted storage system) and can use Key Management as a Service ('KMaaS') to manage encryption keys, keys for locking and unlocking storage devices, and the like. Likewise, a cloud data security gateway or similar mechanism can be used to ensure that data stored within the storage system does not end up being improperly stored in the cloud as part of a cloud data backup operation. In addition, micro-segmentation or identity-based segmentation can be used in the data center containing the storage system or within the cloud service provider to create security zones in data center and cloud deployments, which enables workloads to be isolated from each other.

[0139] To explain further, Figure 3B A diagram of a storage system 306 is presented in accordance with some embodiments of the present disclosure. Although not depicted in detail, Figure 3B The storage system 306 depicted in FIG. 3 may be similar to the storage system 306 described above with reference to FIG. Figures 1A to 1D and Figures 2A to 2G The storage system described above is because the storage system may include many of the components described above.

[0140] Figure 3B The storage system 306 depicted in can include a number of storage resources 308, which can be embodied in many forms. For example, the storage resources 308 can include nano-RAM or another form of non-volatile random access memory utilizing carbon nanotubes deposited on a substrate, 3D cross-point non-volatile memory, flash memory (including single-level cell ('SLC') NAND flash memory, multi-level cell ('MLC') NAND flash memory, three-level cell ('TLC') NAND flash memory, four-level cell ('QLC') NAND flash memory), or other. Likewise, the storage resources 308 can include non-volatile magnetoresistive random access memory ('MRAM'), including spin transfer torque ('STT') MRAM. The example storage resources 308 can alternatively include non-volatile phase change memory ('PCM'), quantum memory that allows storage and retrieval of photon quantum information, resistive random access memory ('ReRAM'), storage class memory ('SCM'), or other forms of storage resources, including any combination of the resources described herein. The reader will appreciate that other forms of computer memory and storage devices may be utilized by the memory system described above, including DRAM, SRAM, EEPROM, general purpose memory, and many others. Figure 3A The storage resources 308 depicted in may be embodied in various form factors, including, but not limited to, dual in-line memory modules ('DIMMs'), non-volatile dual in-line memory modules ('NVDIMMs'), M.2, U.2, and others.

[0141] Figure 3B The storage resources 308 depicted in FIG. 3 may include various forms of SCM. The SCM may effectively treat fast non-volatile memory (e.g., NAND flash) as an extension of DRAM, so that the entire data set may be treated as an in-memory data set that resides entirely in DRAM. The SCM may include non-volatile media such as, for example, NAND flash. This NAND flash may be accessed using NVMe, which may use the PCIe bus as its transport, providing relatively low access latency compared to older protocols. In practice, network protocols for SSDs in an all-flash array may include NVMe using Ethernet (ROCE, NVME TCP), Fibre Channel (NVMe FC), InfiniBand (iWARP), and others that make it possible to treat fast non-volatile memory as an extension of DRAM. Given the fact that DRAM is typically byte-addressable and fast non-volatile memory (e.g., NAND flash) is block-addressable, a controller software / hardware stack may be required to convert block data into bytes stored in the media. Examples of media and software that may be used as SCM may include, for example, 3D XPoint, Intel memory drive technology, Samsung's Z-SSD, and others.

[0142] Figure 3B The storage resources 308 depicted in FIG. 3 may also include racetrack memory (also referred to as domain-wall memory). Such racetrack memory may be embodied in a form of non-volatile solid-state memory that relies, in addition to the charge of the electrons, on the intrinsic strength and orientation of the magnetic field created by the electrons in the solid-state device as they spin. By using spin-coherent current to move magnetic domains along nanopermalloy wires, the magnetic domains may be transferred by a magnetic read / write head positioned near the wire as the current is passed through the wire, which changes the magnetic domains to record a bit pattern. To create a racetrack memory device, many such wires and read / write elements may be packaged together.

[0143] Figure 3B The example storage system 306 depicted in can implement various storage architectures. For example, a storage system according to some embodiments of the present disclosure may utilize block storage, where data is stored in blocks, and each block essentially acts as an individual hard drive. A storage system according to some embodiments of the present disclosure may utilize object storage, where data is managed as objects. Each object may include the data itself, a variable amount of metadata, and a globally unique identifier, where object storage can be implemented at multiple levels (e.g., device level, system level, interface level). A storage system according to some embodiments of the present disclosure utilizes file storage, where data is stored in a hierarchical structure. This data can be saved in files and folders and presented to both the system that stores it and the system that retrieves it in the same format.

[0144] Figure 3B The example storage system 306 depicted in FIG. 300 may be embodied as a storage system in which additional storage resources may be added using a scale-up model, additional storage resources may be added using a scale-out model, or some combination thereof. In a scale-up model, additional storage may be added by adding additional storage devices. However, in a scale-out model, additional storage nodes may be added to a cluster of storage nodes, wherein such storage nodes may include additional processing resources, additional networking resources, and the like.

[0145] Figure 3B The example storage system 306 depicted in FIG. 300 can utilize the storage resources described above in a variety of different ways. For example, some portion of the storage resources can be used as a write cache, storage resources within the storage system can be used as a read cache, or tiering can be implemented within the storage system by placing data within the storage system according to one or more tiering policies.

[0146] Figure 3B The storage system 306 depicted in also includes communication resources 310, which can be used to facilitate data communication between components within the storage system 306 and between the storage system 306 and computing devices external to the storage system 306, including embodiments in which those resources are separated by relatively wide areas. The communication resources 310 can be configured to utilize a variety of different protocols and data communication architectures to facilitate data communication between components within the storage system and computing devices external to the storage system. For example, the communication resources 310 can include: Fibre Channel ('FC') technology, such as an FC architecture and FC protocol that can transmit SCSI commands over an FC network; FC over Ethernet ('FCoE') technology, by which FC frames are encapsulated and transmitted over an Ethernet network; InfiniBand ('IB') technology, in which a switch fabric topology is used to facilitate transmissions between channel adapters; NVM Express ('NVMe') technology and Fabric-based NVMe ('NVMeoF') technology, by which non-volatile storage media attached via a PCI Express ('PCIe') bus can be accessed; and others. In fact, the storage system described above may directly or indirectly use neutrino communication techniques and devices by which information (including binary information) is transmitted using neutrino beams.

[0147] The communication resources 310 may also include mechanisms for accessing storage resources 308 within the storage system 306 utilizing Serial Attached SCSI ('SAS'), a Serial ATA ('SATA') bus interface for connecting storage resources 308 within the storage system 306 to a host bus adapter within the storage system 306, Internet Small Computer System Interface ('iSCSI') technology for providing block-level access to storage resources 308 within the storage system 306, and other communication resources that may be used to facilitate data communications between components within the storage system 306 and data communications between the storage system 306 and computing devices external to the storage system 306.

[0148] Figure 3B The storage system 306 depicted in FIG. 3 also includes processing resources 312 that can be used to execute computer program instructions and perform other computing tasks within the storage system 306. The processing resources 312 can include one or more ASICs and one or more CPUs customized for a particular purpose. The processing resources 312 can also include one or more DSPs, one or more FPGAs, one or more systems on a chip ('SoC'), or other forms of processing resources 312. The storage system 306 can utilize the storage resources 312 to perform a variety of tasks, including (but not limited to) supporting the execution of software resources 314, which will be described in more detail below.

[0149] Figure 3B The storage system 306 depicted in FIG. 3 also includes software resources 314 that, when executed by processing resources 312 within the storage system 306, can perform a number of tasks. The software resources 314 can include, for example, one or more computer program instruction modules that, when executed by processing resources 312 within the storage system 306, are used to implement various data protection techniques. Such data protection techniques can be implemented, for example, by system software executing on computer hardware within the storage system, by a cloud service provider, or otherwise. Such data protection techniques can include data archiving, data backup, data replication, data snapshots, data and database cloning, and other data protection techniques.

[0150] Software resources 314 may also include software for implementing software defined storage ('SDS'). In this example, software resources 314 may include one or more computer program instruction modules that, when executed, provide for policy-based provisioning and management of data storage, independent of the underlying hardware. Such software resources 314 may be used to implement storage virtualization to separate storage hardware from the software that manages the storage hardware.

[0151] The software resources 314 may also include software for facilitating and optimizing I / O operations directed to the storage system 306. For example, the software resources 314 may include software modules that perform various data reduction techniques such as, for example, data compression, data deduplication, and others. The software resources 314 may include software modules that intelligently group I / O operations together to facilitate better use of the underlying storage resources 308, software modules that perform data migration operations to migrate from within the storage system, and software modules that perform other functions. Such software resources 314 may be embodied as one or more software containers or in many other ways.

[0152] To explain further, Figure 3C An example of a cloud-based storage system 318 according to some embodiments of the present disclosure is presented. Figure 3C In the example depicted in FIG. 1 , the cloud-based storage system 318 is entirely created in a cloud computing environment 316, such as, for example, Amazon Web Services ('AWS'). TM 、Microsoft Azure TM 、Google CloudPlatform TM , IBM Cloud TM , Oracle Cloud TM The cloud-based storage system 318 may be used to provide services similar to those that may be provided by the storage systems described above.

[0153] Figure 3C 318 includes two cloud computing instances 320, 322, each for supporting the execution of a storage controller application 324, 326. For example, the cloud computing instances 320, 322 may be embodied as instances of cloud computing resources (e.g., virtual machines) that may be provided by the cloud computing environment 316 to support the execution of software applications (e.g., the storage controller applications 324, 326). For example, each of the cloud computing instances 320, 322 may be executed on an Azure VM, wherein each Azure VM may include a high-speed temporary storage device that may be used as a cache (e.g., as a read cache). In one embodiment, the cloud computing instances 320, 322 may be embodied as Amazon Elastic Compute Cloud ('EC2') instances. In this example, an Amazon Machine Image ('AMI') including the storage controller applications 324, 326 may be launched to create and configure a virtual machine that may execute the storage controller applications 324, 326.

[0154] exist Figure 3CIn the example method described in the embodiment, the storage controller applications 324, 326 may be embodied as computer program instruction modules that, when executed, perform various storage tasks. For example, the storage controller applications 324, 326 may be embodied as computer program instruction modules that, when executed, perform the same tasks as described above. Figure 1A The controllers 110A, 110B in the cloud-based storage system 318 perform the same tasks, such as writing data to the cloud-based storage system 318, erasing data from the cloud-based storage system 318, retrieving data from the cloud-based storage system 318, monitoring and reporting disk utilization and performance, performing redundancy operations (such as RAID or RAID-like data redundancy operations), compressing data, encrypting data, deduplicating data, etc. The reader will understand that because there are two cloud computing instances 320, 322 each including a storage controller application 324, 326, in some embodiments, one cloud computing instance 320 can operate as a primary controller as described above, while the other cloud computing instance 322 can operate as a secondary controller as described above. The reader will understand that Figure 3C The storage controller applications 324, 326 depicted in FIG. 3 may include the same source code that executes within different cloud computing instances 320, 322 (eg, different EC2 instances).

[0155] The reader will appreciate that other embodiments that do not include primary and secondary controllers are within the scope of the present disclosure. For example, each cloud computing instance 320, 322 may operate as a primary controller for some portion of the address space supported by the cloud-based storage system 318, each cloud computing instance 320, 322 may operate as a primary controller in which the service of I / O operations directed to the cloud-based storage system 318 is divided in some other manner, etc. Indeed, in other embodiments in which cost savings may take precedence over performance requirements, there may be only a single cloud computing instance containing a storage controller application.

[0156] Figure 3C The cloud-based storage system 318 depicted in FIG. 3 includes cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338. For example, the cloud computing instances 340a, 340b, 340n may embody examples of cloud computing resources that may be provided by the cloud computing environment 316 to support the execution of software applications. Figure 3C The cloud computing examples 340a, 340b, 340n may be different from the cloud computing examples 320, 322 described above because Figure 3CThe cloud computing instances 340a, 340b, 340n of the storage controller applications 324, 326 have local storage 330, 334, 338 resources, while the cloud computing instances 320, 322 supporting the execution of the storage controller applications 324, 326 do not need to have local storage resources. For example, the cloud computing instances 340a, 340b, 340n with local storage 330, 334, 338 can be embodied as EC2 M5 instances including one or more SSDs, EC2 R5 instances including one or more SSDs, EC2 I3 instances including one or more SSDs, and so on. In some embodiments, the local storage 330, 334, 338 must be embodied as a solid-state storage device (e.g., an SSD) rather than a storage device using a hard disk drive.

[0157] exist Figure 3C In the example depicted in FIG. 1 , each of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338 may include a software daemon 328, 332, 336 that, when executed by the cloud computing instances 340a, 340b, 340n, may present itself to the storage controller application 324, 326 as if the cloud computing instances 340a, 340b, 340n were physical storage devices (e.g., one or more SSDs). In this example, the software daemons 328, 332, 336 may include computer program instructions similar to those typically included on storage devices, such that the storage controller application 324, 326 may send and receive the same commands that a storage controller would send to the storage devices. In this way, the storage controller application 324, 326 may include code that is the same (or substantially the same) as the code that would be executed by the controller in the storage system described above. In these and similar embodiments, communication between the storage controller application 324, 326 and the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338 may utilize iSCSI, NVMe over TCP, messaging, a custom protocol, or some other mechanism.

[0158] exist Figure 3CIn the example depicted in FIG. 1 , each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 may also be coupled to a block storage 342, 344, 346 provided by the cloud computing environment 316, such as (for example) as an Amazon Elastic Block Store ('EBS') volume. In this example, the block storage 342, 344, 346 provided by the cloud computing environment 316 may be utilized in a manner similar to how the NVRAM devices described above are utilized, in that a software daemon 328, 332, 336 (or some other module) executing within a particular cloud computing instance 340a, 340b, 340n may initiate writing data to its attached EBS volume as well as writing data to its local storage 330, 334, 338 resource upon receiving a request to write data. In some alternative embodiments, data may only be written to local storage 330, 334, 338 resources within a particular cloud computing instance 340a, 340b, 340n. In alternative embodiments, rather than using block storage 342, 344, 346 provided by the cloud computing environment 316 as NVRAM, actual RAM on each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 may be used as NVRAM, thereby reducing network utilization costs associated with using EBS volumes as NVRAM. In yet another embodiment, high-performance block storage resources such as one or more Azure ultra disks may be used as NVRAM.

[0159] The storage controller applications 324, 326 may be used to perform various tasks, such as deduplicating data contained in the request, compressing data contained in the request, determining where to write the data contained in the request, etc., before ultimately sending a request to write a deduplicated, encrypted, or otherwise potentially updated version of the data to one or more of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338. In some embodiments, any of the cloud computing instances 320, 322 may receive a request to read data from the cloud-based storage system 318 and may ultimately send a request to read the data to one or more of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338.

[0160] When a request to write data is received by a particular cloud computing instance 340a, 340b, 340n having a local storage 330, 334, 338, the software daemon 328, 332, 336 may be configured to write the data not only to its own local storage 330, 334, 338 resources and any appropriate block storage 342, 344, 346 resources, but the software daemon 328, 332, 336 may also be configured to write the data to a cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n. For example, the cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n may be embodied as Amazon Simple Storage Service ('S3'). In other embodiments, each cloud computing instance 320, 322 including a storage controller application 324, 326 may initiate the storage of data in the local storage 330, 334, 338 and cloud-based object storage 348 of the cloud computing instance 340a, 340b, 340n. In other embodiments, rather than using the cloud computing instances 340a, 340b, 340n and cloud-based object storage 348 with local storage 330, 334, 338 (also referred to herein as 'virtual drives') to store data, a persistent storage layer may be implemented in other ways. For example, one or more Azure ultra disks may be used to store data persistently (e.g., after the data has been written to the NVRAM layer).

[0161] While the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n may support block-level access, the cloud-based object storage 348 attached to a particular cloud computing instance 340a, 340b, 340n only supports object-based access. The software daemons 328, 332, 336 may therefore be configured to take data blocks, package those blocks into objects, and write the objects to the cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n.

[0162] Consider an example where data is written in 1MB blocks to local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by cloud computing instances 340a, 340b, 340n. In this example, assume that a user of the cloud-based storage system 318 issues a request to write data that, after being compressed and deduplicated by the storage controller application 324, 326, would result in 5MB of data needing to be written. In this example, writing data to the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n is relatively straightforward because 5 blocks of 1MB in size are written to the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n. In this example, the software daemons 328, 332, 336 may also be configured to create five objects containing distinct 1MB chunks of data. Thus, in some embodiments, the size of each object written to the cloud-based object store 348 may be the same (or nearly the same). The reader will appreciate that in this example, metadata associated with the data itself may be included in each object (e.g., the first 1MB of the object is the data, and the remainder is the metadata associated with the data). The reader will appreciate that the cloud-based object store 348 may be incorporated into the cloud-based storage system 318 to improve the durability of the cloud-based storage system 318.

[0163] In some embodiments, all data stored by the cloud-based storage system 318 may be stored in both: 1) a cloud-based object store 348; and 2) at least one of a local storage 330, 334, 338 resource or a block storage 342, 344, 346 resource utilized by the cloud computing instances 340a, 340b, 340n. In such embodiments, the local storage 330, 334, 338 resource and the block storage 342, 344, 346 resource utilized by the cloud computing instances 340a, 340b, 340n may effectively operate as a cache that generally includes all data also stored in S3, such that all reads of data may be served by the cloud computing instances 340a, 340b, 340n without requiring the cloud computing instances 340a, 340b, 340n to access the cloud-based object store 348. However, the reader will appreciate that in other embodiments, all data stored by the cloud-based storage system 318 may be stored in the cloud-based object storage 348, but not all data stored by the cloud-based storage system 318 may be stored in at least one of the local storage 330, 334, 338 resources or block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n. In this example, various strategies may be utilized to determine which subset of data stored by the cloud-based storage system 318 should reside in: 1) the cloud-based object storage 348; and 2) at least one of the local storage 330, 334, 338 resources or block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n.

[0164] One or more computer program instruction modules executing within the cloud-based storage system 318 (e.g., a monitoring module executing on its own EC2 instance) may be designed to handle the failure of one or more of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338. In this example, the monitoring module may handle the failure of one or more of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 by creating one or more new cloud computing instances having local storage, retrieving data stored on the failed cloud computing instances 340a, 340b, 340n from the cloud-based object storage 348, and storing the data retrieved from the cloud-based object storage 348 in the local storage on the newly created cloud computing instances. The reader will appreciate that many variations of this process may be implemented.

[0165] The reader will appreciate that various performance aspects of the cloud-based storage system 318 may be monitored (e.g., by a monitoring module executing in an EC2 instance) such that the cloud-based storage system 318 may be scaled up or scaled out as needed. For example, if the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 are undersized and insufficient to service I / O requests issued by users of the cloud-based storage system 318, the monitoring module may create a new, more powerful cloud computing instance (e.g., a type of cloud computing instance that includes more processing power, more memory, etc.) that includes the storage controller application so that the new, more powerful cloud computing instance may begin operating as the primary controller. Likewise, if the monitoring module determines that the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 are oversized and cost savings may be obtained by switching to a smaller, less powerful cloud computing instance, the monitoring module may create a new, less powerful (and cheaper) cloud computing instance that includes the storage controller application so that the new, less powerful cloud computing instance may begin operating as the primary controller.

[0166] The storage system described above may implement intelligent data backup techniques by which data stored in the storage system may be replicated and stored in a different location to avoid data loss in the event of an equipment failure or some other form of disaster. For example, the storage system described above may be configured to check each backup to avoid restoring the storage system to an unintended state. Consider an example in which malware infects the storage system. In this example, the storage system may include software resources 314 that may scan each backup to identify backups captured before the malware infected the storage system and those captured after the malware infected the storage system. In this example, the storage system may restore itself from a backup that does not contain the malware - or at least not restore the portion of the backup that contains the malware. In this example, the storage system may include software resources 314 that can scan each backup to identify the presence of malware (or a virus or something else unexpected), for example, by identifying write operations served by the storage system and originating from a network subnet suspected of having delivered malware, by identifying write operations served by the storage system and originating from users suspected of having delivered malware, by identifying write operations served by the storage system and checking the contents of the write operations against a fingerprint of malware, and in many other ways.

[0167] The reader will further appreciate that backups (typically in the form of one or more snapshots) may also be used to perform rapid recovery of the storage system. Consider an example where a storage system is infected with ransomware that locks users out of the storage system. In this example, software resources 314 within the storage system may be configured to detect the presence of ransomware and may further be configured to use the retained backup to recover the storage system to a point in time prior to the point in time when the ransomware infected the storage system. In this example, the presence of ransomware may be explicitly detected through the use of a software tool utilized by the system, through the use of a key (e.g., a USB drive) inserted into the storage system, or in a similar manner. Likewise, the presence of ransomware may be inferred in response to system activity satisfying a predetermined fingerprint, such as, for example, no reads or writes into the system for a predetermined period of time.

[0168] The reader will appreciate that the various components described above can be grouped into one or more optimized computing packages as a converged infrastructure. Such a converged infrastructure can include pools of computer, storage, and networking resources that can be shared by multiple applications and managed in a collective manner using policy-driven processes. Such a converged infrastructure can be implemented with a converged infrastructure reference architecture, with standalone appliances, with a software-driven hyper-convergence approach (e.g., hyper-convergence infrastructure), or in other ways.

[0169] The reader will appreciate that the storage systems described in this disclosure can be used to support various types of software applications. Indeed, the storage system can be 'application-aware' in the sense that the storage system can obtain, maintain, or otherwise access information describing connected applications (e.g., applications utilizing the storage system) to optimize the operation of the storage system based on intelligence about the applications and their utilization patterns. For example, the storage system can optimize data layout, optimize cache behavior, optimize 'QoS' tiers, or perform some other optimization designed to improve the storage performance experienced by the application.

[0170] As an example of a type of application that may be supported by the storage system described herein, storage system 306 may be used to support such applications by providing storage resources to the following applications: artificial intelligence ('AI') applications, database applications, XOps projects (e.g., DevOps projects, DataOps projects, MLOps projects, ModelOps projects, PlatformOps projects), electronic design automation tools, event-driven software applications, high-performance computing applications, simulation applications, high-speed data capture and analysis applications, machine learning applications, media production applications, media serving applications, picture archiving and communication system ('PACS') applications, software development applications, virtual reality applications, augmented reality applications, and many other types of applications.

[0171] Given the fact that storage systems include compute resources, storage resources, and a wide variety of other resources, storage systems may be well suited to support resource-intensive applications such as, for example, AI applications. AI applications may be deployed in a variety of areas, including: predictive maintenance in manufacturing and related areas; healthcare applications, such as patient data and risk analysis; retail and marketing deployments (e.g., search notifications, social media notifications); supply chain solutions; fintech solutions, such as business analytics and reporting tools; operational deployments, such as real-time analytics tools, application performance management tools, IT infrastructure management tools; and many others.

[0172] Such AI applications may enable a device to be aware of its environment and take actions that maximize its chances of success at a goal. Examples of such AI applications may include IBM Watson TM , Microsoft Oxford TM , Google DeepMind TM , Baidu Minwa TM and others.

[0173] The storage system described above may also be well suited for supporting other types of resource-intensive applications, such as, for example, machine learning applications. Machine learning applications can perform various types of data analysis to automate analytical model building. Using algorithms that iteratively learn from data, machine learning applications can enable computers to learn without being explicitly programmed. A specific area of ​​machine learning is called reinforcement learning, which involves taking appropriate actions in a specific situation to maximize rewards.

[0174] In addition to the resources already described, the storage system described above may also include a graphics processing unit ('GPU'), sometimes referred to as a visual processing unit ('VPU'). Such a GPU may be embodied as a dedicated electronic circuit that quickly manipulates and changes memory to accelerate the creation of images in a frame buffer that are desired for output to a display device. Such a GPU may be included within any of the computing devices that are part of the storage system described above, including as one of many individually scalable components of the storage system, where other examples of individually scalable components of such a storage system may include storage components, memory components, computing components (e.g., CPUs, FPGAs, ASICs), networking components, software components, and others. In addition to the GPU, the storage system described above may also include a neural network processor ('NNP') for use in various aspects of neural network processing. Such an NNP may be used in place of (or in addition to) a GPU, and it may also be independently scalable.

[0175] As described above, the storage system described herein can be configured to support artificial intelligence applications, machine learning applications, big data analysis applications, and many other types of applications. The rapid growth of such applications is driven by three technologies: deep learning (DL), GPU processors, and big data. Deep learning is a computing model that uses massively parallel neural networks inspired by the human brain. Instead of experts handcrafting software, deep learning models write their own software by learning from a large number of examples. Such GPUs can include thousands of cores, which are well suited to running algorithms that loosely represent the parallel nature of the human brain.

[0176] Advances in deep neural networks, including the development of multi-layer neural networks, have inspired a wave of new algorithms and tools for data scientists to mine their data with artificial intelligence (AI). Using improved algorithms, larger datasets, and a variety of frameworks (including open source software libraries for machine learning across a range of tasks), data scientists are addressing new use cases such as autonomous vehicles, natural language processing and understanding, computer vision, machine reasoning, strong AI, and many others. Applications of such technologies may include: machine and vehicle object detection, recognition, and avoidance; visual identification, classification, and tagging; algorithmic financial trading strategy performance management; simultaneous localization and mapping; predictive maintenance of high-value machinery; protection against cybersecurity threats, automation of expertise; image recognition and classification; question answering; robotics; text analysis (extraction, classification) and text generation and translation; and many others. Applications of AI technologies have been implemented in a wide variety of products, including, for example: Amazon Echo's voice recognition technology, which allows users to talk to their machines; Google Translate TM , which allows machine-based language translation; Spotify’s Discover Weekly, which provides recommendations for new songs and artists that users may like based on their usage and traffic analytics; Quill’s text generation product, which takes structured data and turns it into narrative stories; Chatbot, which provides real-time, context-specific answers to questions in a conversational format; and many others.

[0177] Data is at the heart of modern AI and deep learning algorithms. Before training can begin, one problem that must be solved is around collecting labeled data that is critical to training accurate AI models. A full-scale AI deployment may be required to continuously collect, clean, transform, label, and store large amounts of data. Adding additional high-quality data points translates directly into more accurate models and better insights. Data samples may undergo a series of processing steps, including (but not limited to): 1) ingesting data from external sources into the training system and storing the data in raw form; 2) cleaning and transforming data in a format that is convenient for training, including linking data samples to appropriate labels; 3) exploring parameters and models, testing quickly with smaller data sets, and iterating to converge the most promising models to advance into the production cluster; 4) performing a training phase to select several batches of random input data, including both new and older samples, and feeding those into the production GPU server for computation to update model parameters; and 5) evaluating the use of a reserved portion of the data that was not used for training in order to evaluate the model accuracy of the retained data. This life cycle can be applied to any type of parallelized machine learning, not just neural networks or deep learning. For example, a standard machine learning framework may rely on CPUs instead of GPUs, but the data ingestion and training workflows can be the same. Readers will appreciate that a single shared storage data hub creates a coordination point throughout the life cycle without the need for additional data copies during the ingestion, preprocessing, and training phases. Ingested data is rarely used for just one purpose, and shared storage gives the flexibility to train multiple different models or apply traditional analytics to the data.

[0178] The reader will appreciate that each stage in the AI ​​data pipeline may have different requirements from a data center point (e.g., a storage system or a collection of storage systems). A scale-out storage system must provide uncompromised performance for all manner of access types and patterns (from small metadata-heavy to large files, from random to sequential access patterns, and from low to high concurrency). The storage system described above can be used as an ideal AI data center point because the system can serve unstructured workloads. In the first stage, the data is ideally ingested and stored on the same data center point that will be used in subsequent stages to avoid additional data copying. The next two steps can be completed on a standard computing server that optionally includes a GPU, and then, in the fourth and final stage, the complete training production job is run on a powerful GPU-accelerated server. Typically, there is a production pipeline next to the experimental pipeline operating on the same dataset. In addition, GPU-accelerated servers can be used independently for different models or combined together to train on a larger model, or even distributed training across multiple systems. If the shared storage hierarchy is slow, then for each stage, data must be copied to local storage, resulting in wasted time staging data on different servers. The ideal data hub for an AI training pipeline provides performance similar to data stored locally on the server nodes, while also having the simplicity and performance to enable all pipeline stages to operate concurrently.

[0179] In order for the storage system described above to be used as a data center or as part of an AI deployment, in some embodiments, the storage system may be configured to provide DMA between storage devices included in the storage system and one or more GPUs used in an AI or big data analytics pipeline. One or more GPUs may be coupled to the storage system, for example, via NVMe over Fabric ('NVMe-oF'), so that bottlenecks such as the host CPU can be bypassed and the storage system (or one of the components contained therein) can directly access the GPU memory. In this example, the storage system may utilize an API hook to the GPU to transfer data directly to the GPU. For example, the GPU may be embodied as an Nvidia TM The GPU and the storage system may support GPUDirect Storage ('GDS') software or have similar proprietary software that enables the storage system to transfer data to the GPU via RDMA or similar mechanisms.

[0180] Although the preceding paragraphs discuss deep learning applications, the reader will appreciate that the storage system described herein may also be part of a distributed deep learning ('DDL') platform to support the execution of DDL algorithms. The storage system described above may also be paired with other technologies (e.g., TensorFlow, an open source software library for data flow programming across a range of tasks that can be used for machine learning applications (e.g., neural networks) to facilitate the development of such machine learning models, applications, and the like.

[0181] The storage system described above can also be used in a neuromorphic computing environment. Neuromorphic computing is a form of computing that simulates brain cells. To support neuromorphic computing, an architecture of interconnected "neurons" replaces traditional computing models with low-power signals transmitted directly between neurons to achieve more efficient computing. Neuromorphic computing can use very large scale integration (VLSI) systems containing electronic analog circuits for simulating the neurobiological architecture present in the nervous system, as well as analog, digital, mixed-mode analog / digital VLSI, and software systems that implement models of the nervous system for perception, motor control, or multi-sensory integration.

[0182] The reader will appreciate that the storage system described above may be configured to support the storage or use of blockchains and derivatives (as well as other types of data), such as, for example, IBM TM The open source blockchain and related tools that are part of the Hyperledger project, permissioned blockchains where a certain number of trusted parties are allowed to access the blockchain, blockchain products that enable developers to build their own distributed ledger projects, and others. The blockchain and storage system described in this article can be used to support on-chain storage of data as well as off-chain storage of data.

[0183] Off-chain storage of data can be implemented in various ways, and can occur when the data itself is not stored within the blockchain. For example, in one embodiment, a hash function can be utilized, and the data itself can be fed into the hash function to produce a hash value. In this example, the hash of a large number of pieces of data can be embedded within the transaction, rather than the data itself. The reader will appreciate that in other embodiments, alternatives to blockchain can be used to facilitate decentralized storage of information. For example, one alternative to blockchain that can be used is blockweave. While conventional blockchains store each transaction to achieve confirmation, blockweaves allow for secure decentralization without using the entire chain, thereby achieving low-cost on-chain storage of data. Such blockweaves can utilize consensus mechanisms based on proof of access (PoA) and proof of work (PoW).

[0184] The storage systems described above may be used alone or in combination with other computing devices to support in-memory computing applications. In-memory computing involves storing information in RAM distributed across a cluster of computers. The reader will appreciate that the storage systems described above, particularly those that are configurable with customizable amounts of processing resources, storage resources, and memory resources (e.g., those systems where the blades contain configurable amounts of each type of resource), may be configured in a manner that can provide an infrastructure capable of supporting in-memory computing. Similarly, the storage systems described above may include component parts (e.g., NVDIMMs, 3D crosspoint storage devices that provide persistent fast random access memory) that may, in effect, provide an improved in-memory computing environment over an in-memory computing environment that relies on RAM distributed across dedicated servers.

[0185] In some embodiments, the storage system described above may be configured to operate as a hybrid in-memory computing environment that includes a universal interface to all storage media (e.g., RAM, flash storage, 3D crosspoint storage). In such embodiments, users may not know the details about where their data is stored, but they can still use the same complete unified API to address the data. In such embodiments, the storage system may (in the background) move the data to the fastest tier available - including relying on various characteristics of the data or relying on some other heuristics to intelligently place the data. In this example, the storage system may even use existing products (such as Apache Ignite and GridGain) to move data between various storage tiers, or the storage system may use custom software to move data between various storage tiers. The storage system described herein may implement various optimizations to improve the performance of in-memory computing, such as (for example) making the computing as close to the data as possible.

[0186] The reader will further appreciate that in some embodiments, the storage system described above may be paired with other resources to support the applications described above. For example, an infrastructure may include primary computing in the form of servers and workstations that specialize in using general-purpose computing on graphics processing units ('GPGPUs') to accelerate deep learning applications that are interconnected into a computing engine to train parameters of a deep neural network. Each system may have Ethernet external connectivity, Infiniband external connectivity, some other form of external connectivity, or some combination thereof. In this example, the GPUs may be grouped for a single large training or used independently to train multiple models. The infrastructure may also include a storage system (such as the storage system described above) to provide, for example, a scale-out all-flash file or object storage area through which data can be accessed via high-performance protocols (such as NFS, S3, etc.). The infrastructure may also include redundant top-of-rack Ethernet switches connected to storage and computing, such as via ports in an MLAG port channel, to achieve redundancy. The infrastructure may also include additional computing in the form of white box servers, optionally with GPUs, for data ingestion, preprocessing, and model debugging. The reader will appreciate that additional infrastructure is also possible.

[0187] The reader will appreciate that the storage system described above, alone or in coordination with other computing machinery, can be configured to support other AI-related tools. For example, the storage system can use tools such as ONXX or other open neural network exchange formats that make it easier to transfer models written in different AI frameworks. Likewise, the storage system can be configured to support tools such as Amazon's Gluon that allow developers to prototype, build, and train deep learning models. In fact, the storage system described above can be part of a larger platform, such as IBM TM Private cloud data, which includes integrated data science, data engineering, and application building services.

[0188] The reader will further appreciate that the storage system described above may also be deployed as an edge solution. Such an edge solution may be in place to optimize cloud computing systems by performing data processing at the edge of the network, near the source of the data. Edge computing can push applications, data, and computing power (i.e., services) from a centralized point to the logical extreme of the network. By using an edge solution, such as the storage system described above, computing tasks can be performed using computing resources provided by such a storage system, data can be stored using the storage resources of the storage system, and cloud-based services can be accessed by using various resources (including networking resources) of the storage system. By performing computing tasks on an edge solution, storing data on an edge solution, and generally using an edge solution, consumption of expensive cloud-based resources can be avoided, and in fact, performance improvements can be experienced relative to a heavier reliance on cloud-based resources.

[0189] While many tasks can benefit from utilizing edge solutions, some specific uses may be particularly well suited for deployment in this environment. For example, devices such as drones, self-driving cars, robots, and others may require extremely fast processing—so fast, in fact, that sending data up to a cloud environment and receiving data processing support back may simply be too slow. As an additional example, some IoT devices (such as connected cameras) may not be well suited to utilizing cloud-based resources because sending data to the cloud may be impractical (not just from a privacy perspective, a security perspective, or a financial perspective) simply due to the sheer volume of data involved. Thus, many tasks that truly involve data processing, storage, or communication may be better suited to a platform that includes edge solutions, such as the storage system described above.

[0190] The storage system described above can be used alone or in combination with other computing resources as a network edge platform that combines computing resources, storage resources, networking resources, cloud technologies, network virtualization technologies, etc. As part of the network, the edge can have characteristics similar to other network facilities from customer premises and backhaul aggregation facilities to access points (PoPs) and regional data centers. The reader will understand that network workloads (such as virtual network functions (VNFs) and others) will reside on the network edge platform. Through the combination of containers and virtual machines, the network edge platform can rely on controllers and schedulers that are no longer geographically co-located with data processing resources. Functions can be split into control planes, user and data planes, or even state machines as microservices, allowing independent optimization and expansion techniques to be applied. Such user and data planes can be implemented by adding accelerators that reside in server platforms (such as FPGAs and smart NICs) and are implemented by SDN-enabled commercial silicon and programmable ASICs.

[0191] The storage system described above can also be optimized for big data analytics, including being used as part of a composable data analytics pipeline, where a containerized analytics architecture, for example, makes analytics capabilities more composable. Big data analytics can be generally described as the process of examining large and diverse data sets to discover hidden patterns, unknown correlations, market trends, customer preferences, and other useful information that can help organizations make smarter business decisions. As part of that process, semi-structured and unstructured data (such as, for example, Internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone call detail records, IoT sensor data, and other data) can be converted into a structured form.

[0192] The storage system described above may also support (including being implemented as a system interface) applications that perform tasks in response to human voice. For example, the storage system may support the execution of an intelligent personal assistant application such as (for example) Amazon's Alexa TM , Apple Siri TM , Google Voice TM , Samsung Bixby TM , Microsoft Cortana TM and others. Although the examples described in the previous sentences use voice as input, the storage system described above may also support chatbots, conversational robots, chatbots, or other applications that are artificial dialogue entities or configured to conduct conversations via auditory or textual methods. Likewise, the storage system may actually execute such an application to enable users (e.g., system administrators) to interact with the storage system via voice. Such applications are typically capable of voice interaction, music playback, making to-do lists, setting alarms, streaming podcasts, playing audiobooks, and providing weather, traffic, and other real-time information, such as news, but in embodiments according to the present disclosure, such applications may be used as interfaces for various system management operations.

[0193] The storage systems described above may also implement an AI platform to realize the vision of self-driving storage. Such AI platforms may be configured to provide global predictive intelligence by collecting and analyzing a large number of storage system telemetry data points for easy management, analysis, and support. In fact, such storage systems may be able to predict both capacity and performance, as well as generate intelligent recommendations for workload deployment, interaction, and optimization. Such AI platforms may be configured to scan all incoming storage system telemetry data against a library of problem fingerprints to predict and resolve incidents in real time before they impact customer environments, and capture hundreds of performance-related variables for predicting performance loads.

[0194] The storage system described above can support the serial or simultaneous execution of artificial intelligence applications, machine learning applications, data analysis applications, data transformations, and other tasks that can collectively form an AI ladder. This AI ladder can be effectively formed by combining such elements to form a complete data science pipeline, where there are dependencies between the elements of the AI ​​ladder. For example, AI may require some form of machine learning, machine learning may require some form of analysis, analysis may require some form of data and information system structure, and so on. Thus, each element can be viewed as a rung in the AI ​​ladder, which together can form a complete and complex AI solution.

[0195] The storage system described above may also be used alone or in combination with other computing environments to provide an AI-everywhere experience, where AI permeates broad and expansive aspects of business and life. For example, AI may play an important role in providing deep learning solutions, deep reinforcement learning solutions, general artificial intelligence solutions, autonomous vehicles, cognitive computing solutions, commercial UAVs or drones, conversational user interfaces, enterprise classification, ontology management solutions, machine learning solutions, smart dust, smart robots, smart workplaces, and many others.

[0196] The storage system described above may also be used alone or in combination with other computing environments to provide a wide range of transparent immersive experiences (including experiences using digital twins of various "things" (e.g., people, places, processes, systems, etc.) where the technology can introduce transparency between people, businesses, and things. This transparent immersive experience may be provided as augmented reality technology, connected home, virtual reality technology, brain-computer interface, human augmentation technology, nanotube electronics, volumetric displays, 4D printing technology, or others.

[0197] The storage system described above can also be used alone or in combination with other computing environments to support a variety of digital platforms. Such digital platforms may include, for example, 5G wireless systems and platforms, digital twin platforms, edge computing platforms, IoT platforms, quantum computing platforms, serverless PaaS, software-defined security, neuromorphic computing platforms, and the like.

[0198] The storage system described above may also be part of a multi-cloud environment, where multiple cloud computing and storage services are deployed in a single heterogeneous architecture. To facilitate the operation of this multi-cloud environment, DevOps tools may be deployed to enable orchestration across clouds. Similarly, continuous development and continuous integration tools may be deployed to standardize processes around continuous integration and delivery, new feature rollouts, and deployment of cloud workloads. By standardizing these processes, a multi-cloud strategy may be implemented that enables the use of the best provider for each workload.

[0199] The storage system described above can be used as part of a platform to enable the use of cryptographic anchors, which can be used to authenticate the origin and content of a product to ensure that it matches the blockchain record associated with the product. Similarly, as part of a tool suite to protect data stored on the storage system, the storage system described above can implement various cryptographic techniques and schemes, including grid cryptography. Grid cryptography can involve the construction of cryptographic primitives involving a grid, either in the construction itself or in security proofs. Unlike public key schemes such as RSA, Diffie-Hellman, or elliptic curve cryptography, which are vulnerable to attack by quantum computers, some grid-based constructions appear to be resistant to attack by both classical and quantum computers.

[0200] A quantum computer is a device that performs quantum computations. Quantum computing uses quantum mechanical phenomena, such as superposition and entanglement, to perform computations. Quantum computers are different from traditional computers based on transistors because such traditional computers require data to be encoded as binary digits (bits), each of which is always in one of two definite states (0 or 1). Unlike traditional computers, quantum computers use qubits that can be in a superposition of states. A quantum computer maintains a sequence of qubits, where a single qubit can represent one, zero, or any quantum superposition of those two qubit states. A pair of qubits can be in any quantum superposition of 4 states, and three qubits are in any superposition of 8 states. A quantum computer with n qubits can typically be in any superposition of up to 2^n different states at the same time, while a traditional computer can only be in one of these states at any one time. A quantum Turing machine is a theoretical model of this computer.

[0201] The storage system described above can also be paired with an FPGA acceleration server as part of a larger AI or ML infrastructure. Such an FPGA acceleration server may reside near the storage system described above (e.g., in the same data center) or even be incorporated into a device including one or more storage systems, one or more FPGA acceleration servers, a networking infrastructure supporting communication between one or more storage systems and one or more FPGA acceleration servers, and other hardware and software components. Alternatively, the FPGA acceleration server may reside in a cloud computing environment that can be used to perform computing-related tasks for AI and ML jobs. Any of the embodiments described above can be used together as an FPGA-based AI or ML platform. The reader will understand that in some embodiments of an FPGA-based AI or ML platform, the FPGA included in the FPGA acceleration server can be reconfigured for different types of ML models (e.g., LSTM, CNN, GRU). The ability to reconfigure the FPGA included in the FPGA acceleration server can achieve acceleration of ML or AI applications based on the optimal numerical precision and memory model used. The reader will understand that by viewing a collection of FPGA acceleration servers as an FPGA pool, any CPU in a data center can use the FPGA pool as a shared hardware microservice, rather than limiting the server to a dedicated accelerator inserted therein.

[0202] The FPGA accelerated servers and GPU accelerated servers described above can implement a computing model where, instead of keeping a small amount of data in the CPU and running a long instruction stream through it as happens in a more traditional computing model, the machine learning model and parameters are fixed to a high bandwidth single chip memory where a large amount of data flows through the high bandwidth single chip memory. For this computing model, FPGAs can be even more efficient than GPUs because FPGAs can be programmed with only the instructions needed to run such a computing model.

[0203] The storage system described above can be configured to provide parallel storage, for example, by using a parallel file system such as BeeGFS. Such a parallel file system can include a distributed metadata architecture. For example, a parallel file system can include multiple metadata servers across which metadata is distributed, and components that include services for clients and storage servers.

[0204] The system described above can support the execution of a large number of software applications. Such software applications can be deployed in various ways, including container-based deployment models. Containerized applications can be managed using various tools. For example, containerized applications can be managed using Docker Swarm, Kubernetes, and others. Containerized applications can be used to facilitate serverless, cloud-native computing deployment and management models for software applications. To support serverless, cloud-native computing deployment and management models for software applications, containers can be used as part of an event handling mechanism (e.g., AWS Lambdas) so that various events cause the containerized application to be launched to operate as an event handler.

[0205] The systems described above may be deployed in a variety of ways, including in a manner to support fifth generation ('5G') networks. 5G networks may support substantially faster data communications than previous generations of mobile communications networks, and thus may result in disaggregation of data and computing resources, as modern large-scale data centers may become less prominent and may be replaced, for example, by more local micro data centers near mobile network towers. The systems described above may be included in such local micro data centers, and may be part of or paired with multi-access edge computing ('MEC') systems. Such MEC systems may enable cloud computing capabilities and IT service environments at the edge of a cellular network. By running applications and performing related processing tasks closer to cellular clients, network congestion may be reduced, and applications may perform better.

[0206] The storage system described above may also be configured to implement NVMe partitioned namespaces. By using NVMe partitioned namespaces, the logical address space of the namespace is divided into zones. Each zone provides a logical block address range that must be written sequentially and explicitly reset before being rewritten, thereby enabling the creation of natural boundaries that expose the device and offloading the management of internal mapping tables to the host's namespace. In order to implement NVMe partitioned namespaces ('ZNS'), a ZNS SSD or some other form of partitioned block device may be utilized that uses zones to expose the namespace logical address space. In the case of aligning the zones to the internal physical properties of the device, several inefficiencies in data placement may be eliminated. In such embodiments, for example, each zone may be mapped to a separate application so that functions such as wear leveling and obsolete item collection may be performed on a per-zone or per-application basis (rather than across the entire device). In order to support ZNS, the storage controller described herein may be configured to perform wear leveling and obsolete item collection by using, for example, Linux TM Kernel partition block device interface or other tools to interact with partition block devices.

[0207] The storage system described above may also be configured to implement zoned storage in other ways, such as, for example, by using shingled magnetic recording (SMR) storage devices. In instances where zoned storage is used, a device-managed embodiment may be deployed, where the storage device hides this complexity by managing it in firmware, presenting an interface like any other storage device. Alternatively, zoned storage may be implemented via a host-managed embodiment, which relies on the operating system knowing how to handle the drive, and only writing sequentially to certain areas of the drive. Zoned storage may similarly be implemented using a host-aware embodiment, where a combination of drive-managed and host-managed implementations are deployed.

[0208] The storage system described herein may be used to form a data lake. A data lake may be operable as the first place to which an organization's data flows, where such data may be in a raw format. Metadata tags may be implemented to facilitate searching of data elements in a data lake, particularly in embodiments where the data lake contains multiple data stores in formats that may not be easily accessed or read (e.g., unstructured data, semi-structured data, structured data). Data may be downlinked from the data lake to a data warehouse, where the data may be stored in a more processed, packaged, and consumable format. The storage system described above may also be used to implement such a data warehouse. Additionally, a data mart or data hub may allow for even more easily consumable data, where the storage system described above may also be used to provide the underlying storage resources required for the data mart or data hub. In embodiments, querying a data lake may require a schema-on-read approach, where the data is applied to a plan or schema when it is pulled from a storage location, rather than when it enters the storage location.

[0209] The storage systems described herein may also be configured to implement a recovery point objective ('RPO'), which may be established by a user, by an administrator, as a system default, as part of a storage class or service that the storage system participates in delivering, or in some other manner. A "recovery point objective" refers to a target for the maximum time difference between the last update of a source data set and the last recoverable update of a replicated data set that, given some reason, can be correctly recovered from a continuously or frequently updated copy of the source data set. Updates can be correctly recovered if all updates processed on the source data set before the last recoverable update of the replicated data set are properly accounted for.

[0210] In synchronous replication, the RPO will be zero, meaning that under normal operation, all updates done on the source dataset should be present and recoverable correctly on the replica dataset. In as-close-to-synchronous replication, the RPO can be as low as a few seconds. In snapshot-based replication, the RPO can be roughly calculated as the interval between snapshots plus the time to transfer modifications between the previously transferred snapshot and the most recent snapshot to be replicated.

[0211] If updates accumulate faster than they are replicated, then the RPO may be missed. If more data to be replicated accumulates between two snapshots (for snapshot-based replication) than can be replicated between taking a snapshot and replicating the cumulative updates for that snapshot to the replica, then the RPO may be missed. Again in snapshot-based replication, if data to be replicated accumulates at a faster rate than it can be transferred in the time between subsequent snapshots, then replication may begin to fall further behind, which may extend the miss between the expected recovery point objective and the actual recovery point represented by the last correctly replicated update.

[0212] The storage systems described above may also be part of a shared-nothing storage cluster. In a shared-nothing storage cluster, each node of the cluster has local storage and communicates with other nodes in the cluster over a network, where the storage used by the cluster is (usually) provided only by the storage connected to each individual node. A collection of nodes that synchronously replicate data sets may be an instance of a shared-nothing storage cluster because each storage system has local storage and communicates with other storage systems over a network, where those storage systems (usually) do not use storage from elsewhere that they have shared access to through some kind of interconnect. In contrast, some of the storage systems described above are inherently built as shared storage clusters because there are drive racks that are shared by paired controllers. However, other storage systems described above are built as shared-nothing storage clusters because all storage is local to a particular node (e.g., a blade) and all communications are through the network that links the compute nodes together.

[0213] In other embodiments, other forms of shared-nothing storage clusters may include embodiments where any node in the cluster has a local copy of all storage it needs, and where data is mirrored to other nodes in the cluster by synchronous replication to ensure that data is not lost or lost because other nodes are also using that storage. In this embodiment, if a new cluster node needs some data, that data can be copied to the new node from other nodes that have copies of the data.

[0214] In some embodiments, a shared storage cluster based on mirror replication may store multiple copies of all cluster stored data, where each subset of the data is replicated to a specific set of nodes, and different subsets of the data are replicated to several different sets of nodes. In some variations, embodiments may store all data stored by the cluster in all nodes, while in other variations, the nodes may be partitioned so that a first set of nodes will all store the same set of data, while a second, different set of nodes will all store different sets of data.

[0215] The reader will appreciate that a RAFT-based database (e.g., etcd) may operate as a shared-nothing storage cluster in which all RAFT nodes store all data. However, the amount of data stored in a RAFT cluster may be limited so that additional copies do not consume too much storage. A cluster of container servers may also be able to replicate all data to all cluster nodes, provided that the containers are not too large and the bulk of their data (the data manipulated by the applications running in the containers) is stored elsewhere, such as in an S3 cluster or an external file server. In this instance, container storage may be provided directly by the cluster through its shared-nothing storage model, with those containers providing images of the execution environments that form part of an application or service.

[0216] To explain further, Figure 3DAn exemplary computing device 350 is illustrated that may be specifically configured to perform one or more of the processes described herein. Figure 3D , computing device 350 may include a communication interface 352, a processor 354, a storage device 356, and an input / output ("I / O") module 358 that are communicatively connected to each other via a communication infrastructure 360. Figure 3D An exemplary computing device 350 is shown in FIG. Figure 3D The components described in the description are not intended to be limiting. In other embodiments, additional or alternative components may be used. Figure 3D Components of computing device 350 are shown in FIG.

[0217] Communication interface 352 may be configured to communicate with one or more computing devices. Examples of communication interface 352 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0218] The processor 354 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 354 may perform operations by executing computer-executable instructions 362 (e.g., applications, software, code, and / or other executable data examples) stored in the storage device 356.

[0219] Storage device 356 may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and / or devices. For example, storage device 356 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 356. For example, data representing computer executable instructions 362 configured to direct processor 354 to perform any of the operations described herein may be stored within storage device 356. In some examples, the data may be arranged in one or more databases residing within storage device 356.

[0220] The I / O module 358 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 358 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 358 may include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0221] The I / O module 358 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 358 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation. In some examples, any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 350.

[0222] To explain further, Figure 3E An example of a fleet of storage systems 376 for providing storage services (also referred to herein as 'data services') is illustrated. Figure 3E The fleet of storage systems 376 depicted in FIG. 1 includes a plurality of storage systems 374a, 374b, 374c, 374d, 374n that may each be similar to the storage systems described herein. The storage systems 374a, 374b, 374c, 374d, 374n in the fleet of storage systems 376 may be embodied as the same storage system or as different types of storage systems. For example, Figure 3E The two storage systems 374a, 374n depicted in FIG. 3 are depicted as cloud-based storage systems because the resources that together form each of the storage systems 374a, 374n are provided by different cloud service providers 370, 372. For example, the first cloud service provider 370 may be Amazon AWS. TM , and the second cloud service provider 372 is Microsoft Azure TM , but in other embodiments, one or more public clouds, private clouds, or a combination thereof may be used to provide underlying resources that are used to form a particular storage system in the fleet of storage systems 376.

[0223] According to some embodiments of the present disclosure, Figure 3E The example depicted in 380 includes an edge management service 382 for delivering storage services. The storage services delivered (also referred to herein as 'data services') may include, for example, services that provide a specific amount of storage to a customer, services that provide storage to a customer according to a predetermined service level agreement, services that provide storage to a customer according to predetermined regulatory requirements, and many others.

[0224] Figure 3EThe edge management service 382 depicted in FIG. 3 may be embodied as one or more computer program instruction modules executed, for example, on computer hardware (e.g., one or more computer processors). Alternatively, the edge management service 382 may be embodied as one or more computer program instruction modules executed on a virtualized execution environment (e.g., one or more virtual machines), in one or more containers, or in some other manner. In other embodiments, the edge management service 382 may be embodied as a combination of the embodiments described above, including embodiments in which one or more computer program instruction modules contained in the edge management service 382 are distributed across multiple physical or virtual execution environments.

[0225] The edge management service 382 may operate as a gateway for providing storage services to storage customers, where the storage services utilize storage provided by one or more storage systems 374a, 374b, 374c, 374d, 374n. For example, the edge management service 382 may be configured to provide storage services to host devices 378a, 378b, 378c, 378d, 378n that execute one or more applications that consume the storage services. In this example, the edge management service 382 may operate as a gateway between the host devices 378a, 378b, 378c, 378d, 378n and the storage systems 374a, 374b, 374c, 374d, 374n, without requiring the host devices 378a, 378b, 378c, 378d, 378n to directly access the storage systems 374a, 374b, 374c, 374d, 374n.

[0226] Figure 3E The edge management service 382 exposes the storage service module 380 to Figure 3E In other embodiments, the edge management service 382 may expose the storage service module 380 to other clients of various storage services. The various storage services may be presented to clients via one or more user interfaces, via one or more APIs, or through some other mechanism provided by the storage service module 380. Thus, Figure 3E The storage service module 380 depicted in the figure may be embodied as one or more computer program instruction modules executed on physical hardware, on a virtualized execution environment, or a combination thereof, wherein the execution of such modules enables customers of the storage service to be provided with, select and access various storage services.

[0227] Figure 3E The edge management service 382 also includes a system management service module 384 . Figure 3EThe system management service module 384 includes one or more computer program instruction modules that, when executed, coordinate with the storage systems 374a, 374b, 374c, 374d, and 374n to perform various operations to provide storage services to the host devices 378a, 378b, 378c, 378d, and 378n. The system management service module 384 may be configured, for example, to perform tasks such as provisioning storage resources from the storage systems 374a, 374b, 374c, 374d, 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c, 374d, 374n, migrating data sets or workloads among the storage systems 374a, 374b, 374c, 374d, 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c, 374d, 374n, setting one or more tunable parameters (i.e., one or more configurable settings) on the storage systems 374a, 374b, 374c, 374d, 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c, 374d, 374n, and the like. For example, many of the services described below are related to embodiments in which the storage systems 374a, 374b, 374c, 374d, 374n are configured to operate in a certain manner. In such instances, the system management service module 384 may be responsible for configuring the storage systems 374a, 374b, 374c, 374d, 374n to operate in the manner described below using an API (or some other mechanism) provided by the storage systems 374a, 374b, 374c, 374d, 374n.

[0228] In addition to configuring the storage systems 374a, 374b, 374c, 374d, 374n, the edge management service 382 itself may be configured to perform various tasks required to provide various storage services. Consider an example where the storage service includes a service that, when selected and applied, causes personally identifiable information ('PII') contained in a data set to be obfuscated when the data set is accessed. In this example, the storage systems 374a, 374b, 374c, 374d, 374n may be configured to obfuscate the PII when servicing read requests directed to the data set. Alternatively, the storage systems 374a, 374b, 374c, 374d, 374n may service the reads by transmitting back data containing the PII, but the edge management service 382 itself may obfuscate the PII as the data passes through the edge management service 382 on its way from the storage systems 374a, 374b, 374c, 374d, 374n to the host devices 378a, 378b, 378c, 378d, 378n.

[0229] Figure 3EThe storage systems 374a, 374b, 374c, 374d, 374n depicted in the drawings may be embodied as described above with reference to Figures 1A to 3D One or more of the storage systems described, including variations thereof. In practice, the storage systems 374a, 374b, 374c, 374d, 374n may be used as a storage resource pool, wherein the individual components in that pool have different performance characteristics, different storage characteristics, and the like. For example, one of the storage systems 374a may be a cloud-based storage system, another storage system 374b may be a storage system that provides block storage, another storage system 374c may be a storage system that provides file storage, another storage system 374d may be a relatively high-performance storage system, and another storage system 374n may be a relatively low-performance storage system, and the like. In alternative embodiments, there may be only a single storage system.

[0230] Figure 3E The storage systems 374a, 374b, 374c, 374d, 374n depicted in FIG may also be organized into different fault domains such that a failure of one storage system 374a should be completely unrelated to a failure of another storage system 374b. For example, each of the storage systems may receive power from an independent power system, each of the storage systems may be coupled for data communications via an independent data communications network, and so on. In addition, storage systems in a first fault domain may be accessed via a first gateway, while storage systems in a second fault domain may be accessed via a second gateway. For example, the first gateway may be a first instance of the edge management service 382, ​​and the second gateway may be a second instance of the edge management service 382, ​​including embodiments in which each instance is distinct or each instance is part of a distributed edge management service 382.

[0231] As an illustrative example of available storage services, storage services may be presented to a user associated with different levels of data protection. For example, storage services may be presented to a user that, when selected and enforced, assure the user that data associated with that user will be protected such that various recovery point objectives ('RPOs') may be guaranteed. A first available storage service may ensure, for example, that a certain data set associated with a user will be protected such that any data older than 5 seconds may be recovered in the event of a primary data store failure, while a second available storage service may ensure that a data set associated with a user will be protected such that any data older than 5 minutes may be recovered in the event of a primary data store failure.

[0232] Additional instances of storage services that may be presented to a user, selected by the user, and ultimately applied to a data set associated with the user may include one or more data compliance services. Such data compliance services may be embodied as services that, for example, may provide data compliance services to customers (i.e., users) to ensure that the user's data set is managed in a manner to comply with various regulatory requirements. For example, one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner to comply with the General Data Protection Regulation ('GDPR'), one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner to comply with the Sarbanes-Oxley Act of 2002 ('SOX'), or one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner to comply with some other regulatory act. In addition, one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner to comply with certain non-governmental guidance (e.g., best practices for auditing purposes), one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner to comply with specific client or organizational requirements, and so on.

[0233] Consider an example where a specific data compliance service is designed to ensure that a user's data set is managed in a manner that complies with the requirements stated in the GDPR. While a list of all of the requirements of the GDPR can be found in the regulation itself, for illustrative purposes, the example requirements stated in the GDPR provide that a pseudonymization process must be applied to stored data in order to transform personal data in such a way that the resulting data cannot be attributed to a specific data subject without the use of additional information. For example, data encryption techniques may be applied to render the original data incomprehensible, and such data encryption techniques cannot be reversed without access to the correct decryption key. Thus, the GDPR may require that the decryption key be stored separately from the pseudonymized data. A specific data compliance service may be provided to ensure compliance with the requirements stated in this paragraph.

[0234] In order to provide this particular data compliance service, the data compliance service may be presented to the user (e.g., via a GUI) and selected by the user. In response to receiving a selection of a particular data compliance service, one or more storage service policies may be applied to a data set associated with the user to implement the particular data compliance service. For example, a storage service policy may be applied that requires that a data set be encrypted before being stored in a storage system, before being stored in a cloud environment, or before being stored elsewhere. In order to enforce this policy, provisions may be enforced that require not only that the data set be encrypted when the data set is stored, but also that the data set be encrypted before being transmitted (e.g., sending the data set to another party). In this instance, a storage service policy may also be proposed that requires that any encryption key used to encrypt the data set is not stored on the same system that stores the data set itself. The reader will understand that many other forms of data compliance services may be provided and implemented in accordance with embodiments of the present disclosure.

[0235] The storage systems 374a, 374b, 374c, 374d, and 374n in the team storage system 376 may be jointly managed by one or more team management modules. The team management module may be Figure 3E The fleet management module may perform tasks such as monitoring the health of each storage system in the fleet, initiating updates or upgrades on one or more storage systems in the fleet, migrating workloads for load balancing or other performance purposes, and many other tasks. Thus, and for many other reasons, the storage systems 374a, 374b, 374c, 374d, 374n may be coupled to each other via one or more data communication links to exchange data between the storage systems 374a, 374b, 374c, 374d, 374n.

[0236] The storage system described herein may support various forms of data replication. For example, two or more storage systems may synchronously replicate a data set between each other. In synchronous replication, different copies of a particular data set may be maintained by multiple storage systems, but all accesses (e.g., reads) to the data set should produce consistent results regardless of which storage system the access is directed to. For example, reads directed to any of the storage systems that are synchronously replicating the data set should return the same results. Thus, although updates to data set versions do not need to occur completely simultaneously, precautions must be taken to ensure consistent access to the data set. For example, if an update (e.g., a write) directed to a data set is received by a first storage system, the update can only be confirmed to be complete if all storage systems that are synchronously replicating the data set have applied the update to their copies of the data set. In this example, synchronous replication can be implemented by using I / O forwarding (e.g., a write received at a first storage system is forwarded to a second storage system), communication between storage systems (e.g., each storage system indicates that it has completed the update), or in other ways.

[0237] In other embodiments, a data set may be replicated by using checkpoints. In checkpoint-based replication (also known as 'almost synchronous replication'), a set of updates to a data set (e.g., one or more write operations directed to the data set) may occur between different checkpoints, such that the data set is updated to a particular checkpoint only when all updates to the data set prior to the particular checkpoint have been completed. Consider an example in which a first storage system stores a real-time copy of a data set that is being accessed by a data set user. In this example, assume that the data set is replicated from the first storage system to a second storage system using checkpoint-based replication. For example, the first storage system may send a first checkpoint (at time t=0) to the second storage system, followed by a first set of updates to the data set, followed by a second checkpoint (at time t=1), followed by a second set of updates to the data set, followed by a third checkpoint (at time t=2). In this example, if the second storage system has executed all updates in the first set of updates but has not yet executed all updates in the second set of updates, then the copy of the data set stored on the second storage system may be up to date up to the second checkpoint. Alternatively, if the second storage system has performed the first set of updates and all updates in the second set of updates, then the copy of the data set stored on the second storage system may be up to date until the third checkpoint. The reader will recognize that various types of checkpoints may be used (e.g., metadata-only checkpoints), that checkpoints may be distributed based on various factors (e.g., time, number of operations, RPO settings), etc.

[0238] In other embodiments, the data set may be replicated via snapshot-based replication (also referred to as 'asynchronous replication'). In snapshot-based replication, a snapshot of the data set may be sent from a replication source (e.g., a first storage system) to a replication target (e.g., a second storage system). In this embodiment, each snapshot may include the entire data set or a subset of the data set, such as (for example) only the portion of the data set that has changed since the last snapshot was sent from the replication source to the replication target. The reader will appreciate that snapshots may be sent on demand based on a policy that takes various factors (e.g., time, number of operations, RPO settings) into account, or in some other manner.

[0239] The storage systems described above may be configured, alone or in combination, to be used as continuous data protection storage. Continuous data protection storage is a feature of a storage system that records updates to a data set in such a way that a consistent image of the previous contents of the data set can be accessed at a low temporal granularity (usually on the order of seconds or even less), and extended back for a reasonable period of time (usually hours or days). These allow access to the most recent consistent point in time of the data set, and also allow access to a point in time of the data set where an event may have just occurred (e.g., causing part of the data set to be damaged or otherwise lost), while keeping close to the maximum number of updates before that event. Conceptually, they are like a sequence of snapshots of a data set that are taken very frequently and kept for a long period of time, but continuous data protection storage is typically implemented quite differently from snapshots. A storage system that implements data continuous data protection storage may also further provide a means of accessing these points in time, accessing one or more of these points in time as a snapshot or clone copy, or restoring the data set back to one of these recorded points in time.

[0240] Over time, to reduce overhead, some of the time points maintained in the continuous data protection storage may be merged with other nearby time points, essentially removing some of these time points from the storage. This may reduce the capacity required for storage updates. A limited number of these time points may also be converted into longer duration snapshots. For example, this storage may maintain a low-granularity sequence of time points dating back a few hours from the present, with some time points merged or deleted to reduce overhead by up to an additional day. Going back further into the past, some of these time points may be converted to snapshots representing consistent time point images from only every few hours.

[0241] A checkpoint may be defined as a point in time at which modifications to a data set are divided into modifications before the checkpoint and those after the checkpoint. Those modifications before the checkpoint are included in the data set represented by the checkpoint. Those modifications after the checkpoint are excluded from the data set represented by the checkpoint. A snapshot represents an instance of a checkpoint, but a checkpoint has uses other than snapshots. A checkpoint may or may not be persistent and may be used for various internal purposes, such as (for example) low RPO replication, as part of a continuous data protection implementation scheme, or for forming crash recovery points within or between transaction engines. Establishing a checkpoint for a crash-consistent snapshot across multiple storage systems may require a fully distributed quiesce, in which all storage systems involved in the checkpoint stop processing I / O operations and wait for pending operations to complete before establishing a checkpoint. The unavailability of the data set during this period may cause problems for the host that utilizes the data set, and may also cause a backlog of pending operations that must be processed when I / O operations continue. In addition, it is difficult to implement checkpoints across multiple storage systems that utilize different storage implementation schemes and / or utilize different checkpoint protocols. As set forth in detail below, various embodiments in accordance with the present disclosure provide a mechanism for coordinating snapshots of data stored across disparate storage environments in situations where full distributed quiescing of storage operations is required.

[0242] To explain further, Figure 4 A flow chart presenting an example method for coordinating snapshots of data stored across disparate storage environments. Figure 4 The example depicted in FIG. 4 may include two or more storage systems 412, 414 and a coordinator service 416, wherein the coordinator service orchestrates coordinated checkpoints for a data set 430 across the two or more storage systems. Each storage system (such as storage systems 412, 414 in this particular example) may be similar to the storage systems described above, including any combination of the storage systems described above. In fact, Figure 4 The storage systems 412, 414 depicted in the diagram may include the same, fewer, or additional components as the storage systems described above. In various embodiments, the coordinator service 416 may be embodied as computer executable instructions executed on a processing resource of one of the storage systems 412, 414, a management server, or on some other computing device.

[0243] exist Figure 4 In the example depicted in , a first portion 432 of data set 430 is stored on storage system 412 , and a second portion 434 of data set 430 is stored on storage system 414 .

[0244] In some examples, two or more storage systems 412, 414 have different implementations and storage environments (although it is not a requirement for all embodiments of the present disclosure that the storage systems 412, 414 are different). For example, the storage system environment may implement a specific type of storage, such as block storage, file system storage, object storage, and database storage. In one non-limiting example, the storage system 412 may store a block volume as a first portion 432 of the data set 430, and another storage system 414 may store a file system as a second portion 434 of the data set 430. Thus, in some examples, the storage systems 412, 414 may implement storage of data types using different storage environments. Thus, orchestrating coordinated checkpoints for the data set 430 may include coordinating local checkpoints on the storage systems 412, 414 having different storage implementations.

[0245] In some instances, when an I / O request is completed, the storage system 412, 414 signals completion by returning the result of the read request with a success code or by confirming that the write modification has been applied, for example by returning a success code. The host process that initiated the I / O access to the data set 430 can know that the modification has been applied by receiving the completion notification, or any host can know the modification by reading from the data set 430 and observing the result of the modification. Note that this read problem only occurs for reads that can read the results of writes that have not been signaled as completed. Because of this, many discussions in this patent that discuss delayed reads can generally be relaxed to cover only reads that can return data from writes that have not yet been completed. In some discussions where write processing is completely delayed, the read may not have a chance to read the problematic write that has not yet been completed, and the read may not need to be delayed at all. In some cases, I / O access to the first part 432 of the data set 430 served by the storage system 412 may be causally dependent on I / O access to the second part 434 of the data set 430 served by the storage system 414, and vice versa. For example, a modification to a first portion 432 of a data set 430 may be causally dependent on a result of a modification that has been applied to a second portion 434 of the data set 430. Thus, orchestrating coordinated checkpoints for the data set 430 may include coordinating local checkpoints on the storage systems 412, 414 such that modifications included in a local checkpoint on one storage system do not depend on modifications that have been excluded from a local checkpoint on another storage system.

[0246] Figure 4An example method includes initiating 400, by a coordinator service 416, a coordinated checkpoint of a data set across two or more storage systems storing respective portions of the data set. In some examples, initiating 400 the coordinated checkpoint includes requesting 402, by the coordinator service 416, two or more storage systems 412, 414 to prepare respective local checkpoints for a data set 430. The coordinator service 416 directs the storage systems 412, 414 to create local checkpoints for the data set 430 through a sequence of messages to the storage systems 412, 414. In some examples, the coordinator service 416 sends a prepare message 422 to the first storage system 412 and a prepare message 424 to the second storage system 414, wherein the prepare messages 422, 424 reflect that I / O operations should be suspended before establishing local checkpoints corresponding to the coordinated checkpoint on each storage system 412, 414.

[0247] Figure 4The example method also includes determining 404 by the coordinator service 416 that each of the two or more storage systems has suspended its I / O operations. Once determined, the coordinator service 416 may further send a continue (or "commit") message to the two or more systems, which indicates that all two or more storage systems have suspended their I / O processing or completion confirmation and all storage systems can now continue I / O processing. When each of the two or more storage systems 412, 414 receives this indication, each of the storage systems 412, 414 will continue I / O processing after ensuring that its corresponding local checkpoint has been established. The point between suspending I / O processing and continuing I / O processing where the storage system establishes its local checkpoint may depend on which part of the I / O processing the storage system suspends. Individual storage systems may suspend I / O completion and establish their local checkpoints before responding to the coordinator service that it has suspended I / O processing, continuing I / O completion processing after receiving the "commit" message. Alternatively, individual storage systems may pause starting or completing I / O processing and may establish a local checkpoint of the storage system at any point after the pause and before continuing I / O processing due to receiving a "commit" message. Alternatively, the storage system may pause starting I / O processing and may wait until a "commit" message is received to establish a local checkpoint of the storage system, after which it will continue I / O processing. However, the coordinator service 416 is unaware of the storage system's delays in creating local checkpoints and the specific choices for local checkpoint establishment. For the coordinator service 416, it is sufficient that all storage systems respond to the prepare messages 422, 424 to reach a state in which no further modifications that can be observed by other storage systems or hosts will be made to the data set 430. In some instances, the coordinator service 416 determines 404 that the storage systems 412, 414 are ready to establish a checkpoint by receiving confirmation ready messages 426, 428 from each of the two or more storage systems 412, 414. The confirm-ready messages 426 , 428 may indicate that the storage systems 412 , 414 have at least suspended I / O operations for the data set 430 .

[0248] Figure 4The example method also includes requesting 406 from two or more storage systems by the coordinator service 416 in response to determining that all storage systems 412, 414 have suspended their I / O operations that the two or more storage systems 412, 414 continue I / O operations when their respective local checkpoints have been established. Based on receiving both confirmation messages 426, 428, the coordinator service 416 can infer that the storage systems 412, 414 have suspended I / O operations and are preparing to establish their local checkpoints or have already established their local checkpoints. Therefore, the coordinated checkpoint embodied by the respective local checkpoints on the storage systems 412, 414 represents a version of the data set that does not include any modifications, any of which may depend on the results of any other modifications not included in the version of the data set. Therefore, the coordinated checkpoint can be safely relied on as a recovery point. Thus, in some instances, the coordinator service 416 requests 406 the storage systems to continue I / O operations when their local checkpoints have been established by sending commit messages 440, 442 to two or more storage systems 412, 414, where the commit messages 440, 442 indicate to the storage systems 412, 414 that the I / O operations can safely continue.

[0249] It should be appreciated that the coordinator service 416 does not require each storage system 412, 414 to quiesce storage operations before the coordinator service sends a request 406 to commit a local checkpoint to the storage system 412, 414. It should also be appreciated that the coordinator service 416 does not need to confirm that the checkpoint has been established before the coordinator service sends a request 406 to commit a local checkpoint to the storage system 412, 414. In addition, in some embodiments, if the coordinator service 416 does not receive all ready messages 426, 428 within some predefined timeout period, the coordinator service may instead send an abort message to the storage system 412, 414.

[0250] Figure 4 The method also includes determining 408, by the coordinator service 416 in conjunction with the two or more storage systems 412, 414, that respective local checkpoints of the two or more storage systems 412, 414 satisfy dependency requirements for a consistent coordinated snapshot of the data set 430 when storage operations are applied by the two or more storage systems to their respective portions of the data set. The coordinated checkpoint may be successfully completed if both of the two or more storage systems 412, 414 successfully complete their local checkpoints, that is, no storage system 412, 414 aborts its local checkpoint. Due to overlap in latency when I / O processing and / or I / O completion confirmation occurs, it may be guaranteed that the local checkpoints have been established such that any local checkpoint does not include any modification that depends on the result of any other modification that is excluded from another local checkpoint.

[0251] To explain further, Figure 5Another example method of coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Figure 5 The example depicted in contains many of the same elements as the figures described above. Figure 5 In the example method of , determining 408 that respective local checkpoints of two or more storage systems satisfy dependency requirements further includes determining 502 that the first storage system 412 and the second storage system 414 have established respective local checkpoints. In some examples, the coordinator service 416 receives confirmation messages 540, 542 from the two or more storage systems 412, 414 indicating that respective local checkpoints have been successfully established and the I / O operation has continued. Due to message or processing delays, the time at which the confirmation messages 540, 542 are received by the coordinator service 416 may be substantially different. In some examples, the confirmation messages 540, 542 further indicate that the storage systems 412, 414 have maintained their respective local checkpoints. For example, the storage systems may use the checkpoints to generate snapshots, wherein the snapshots are maintained in response to receiving the commit message.

[0252] In some embodiments, waiting to receive a confirmation message 540, 542 before the coordinator service 416 signals that the coordination checkpoint has been completed further facilitates the fault handling mechanism. For example, one or more storage systems 412, 414 may implement a timeout procedure so that if a commit message is not received within a certain period of time, the local checkpoint will be aborted. For example, the first storage system 412 may prepare its local checkpoint and start a counter. Due to message delivery or other delays, the second storage system 414 may confirm to the coordinator service 416 at a relatively late time that it is ready to establish its local checkpoint. If the first storage system 412 does not receive the commit message 440 within a certain period of time, the first storage system may abort its local checkpoint and continue I / O operations on the data set 430. When the first storage system 412 finally receives the commit message, it may respond to the coordinator service 416 that the local checkpoint has been aborted.

[0253] Considering an instance where it can be assumed that under reasonable circumstances, preparing a checkpoint will not take longer than X seconds and that message latency will be no greater than Y seconds, once the storage system 412 receives the prepare message 422, it can safely set a timeout of Z seconds (where Z>X+Y) so that if it does not receive the commit message 440 within Z seconds, it can locally abort the checkpoint request and respond with 'abort' when it receives the commit message 440. The coordinator service 416 can then retry the operation. The selected timeout period may need to be increased as the number of storage systems and technologies involved increases. In some examples, the timeout may be configured on the storage systems 412, 414 or set as a parameter received by the coordinator service 416. The timeout may also be tuned based on previous behavior, or a set of test round trips may be performed before sending the prepare message that will cause the I / O operation to be suspended and the coordination checkpoint to be established. It may be advantageous to perform this test before establishing a coordination checkpoint to ensure that all storage systems and technologies are running and communicating.

[0254] Figure 5 The method also includes indicating 504, by the coordinator service 416, in response to determining that the local checkpoint has been established, that the coordinated checkpoint of the data set 430 has been completed. In some instances, indicating 504 that the coordinated checkpoint has been completed can be implemented by sending a completion message 544 to the management service indicating that the coordinated checkpoint has been completed upon receiving both success messages 540, 542 from the storage systems 412, 414. In some instances, the persistence of the local checkpoint can be separated from establishing the checkpoint. It is advantageous to provide the checkpoint ready messages 440, 442 to the coordinator service 416 as quickly as possible, but the storage system may also need to indicate that the local checkpoint is persistent. In this case, additional messages indicating that the local checkpoint has been maintained can be provided to the coordinator service 416 by the storage systems 412, 414. When all the hold messages have been received from all the storage systems 412, 414 indicating that the storage systems have maintained their local checkpoints (which can then be separated from the continuation of the I / O operation), the coordinator service 416 can consider the coordinated checkpoint to be completed. If all storage systems and checkpointing techniques support separation of maintaining and establishing checkpoints relative to I / O flows, the duration of pausing I / O operations or I / O completions can be substantially reduced in some cases.

[0255] To explain further, Figure 6 Another example method of a coordinator service for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Figure 6 The example depicted in contains many of the same elements as the figures described above. However, Figure 6The example method also includes receiving 602 a request 604 to create a coordinated checkpoint for a data set 430 from a management service 618. In some examples, the coordinator service 416 receives a request 406 to create a coordinated checkpoint for a data set 430 from the management service 618. The management service 618 requests and utilizes the checkpoint. In one example, the management service 618 provides an interface through which a user (e.g., a storage administrator) configures a storage service that relies on a checkpoint. For example, the management service 618 may facilitate scheduling snapshots, configuring recovery point objectives, configuring replication services, etc. In some examples, the request 604 from the management service may be received in a message or through an API call. In response to the request, two or more storage systems 412, 414 are directed to prepare respective local checkpoints. In some embodiments, the management service 618 and the coordinator service 416 may be hosted on the same computing device.

[0256] To explain further, Figure 7 An example method of a storage system for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is presented. Figure 7 The example depicted in FIG. 7 may include two or more storage systems 712, 714 and a coordinator service 716, where in this particular example, the coordinator service 716 may coordinate checkpoints of a data set 730 across the two or more storage systems 712, 714. Each storage system 712, 714 may be similar to the storage systems described above, including any combination of the storage systems described above. In fact, Figure 4 The storage systems 712, 714 depicted in FIG. 7 may include the same, fewer, or additional components as the storage systems described above. The coordinator service 716 may be similar to the coordinator service 416 described above.

[0257] In some examples, each storage system 712, 714 stores a portion of the data set 730, wherein in this particular example, a first portion 732 of the data set 730 is stored on one storage system 712, and a second portion 734 of the data set 730 is stored on another storage system 714. Although only two storage systems and two portions of the data set 730 are illustrated, it should be appreciated that there may be any number of storage systems storing additional portions of the data set 730. As described above, in some examples, the storage systems 712, 714 may employ different storage implementations or storage environments. Thus, the data set 730 may include different types of storage objects distributed across multiple storage systems having different storage environments. For example, the storage environment of the portion 732 of the data set 730 on one storage system 712 may implement any one of block storage, file system, database, or object storage. Similarly, the storage environment of the portion 734 of the data set 730 on another storage system may also implement any one of block storage, file system, database, or object storage. However, in some examples, the respective storage environments are different. For example, the first portion 732 of the data set 730 may include a block volume stored on the storage system 712 , and the second portion 734 of the data set 730 may include part or all of a file system on the storage system 714 .

[0258] In some examples, the storage systems 712, 714 handle the processing and confirmation of I / O requests, as discussed above. For example, when a write request directed to a data set 730 is received, the write request is processed and the requested modification is applied to the data set 730. After the requested modification has been applied and maintained, the storage system confirms that the requested modification has been completed. Before indicating completion, the result of the modification should not depend on any other I / O operation. In some examples, signaling that an I / O request to modify a data set has been completed can be implemented by sending a completion confirmation message to the requestor indicating that the request has been completed. In some examples, signaling that an I / O request to read from a data set 730 has been completed can be implemented by simply returning the result of the read operation.

[0259] In some examples, checkpoint creation can be optimized taking into account that a checkpoint need only represent a reasonable crash recovery point for a data set. A reasonable crash recovery point can be expressed as a version of a data set that does not contain any modifications that may causally depend on the results of any other modifications not contained in that version of the data set. Thus, as described below, the storage system can manipulate these causal relationships by delaying the processing or completion of operations.

[0260] Figure 7The example method includes pausing 702, by a first storage system 712, an I / O operation directed to a data set 730 in response to a request 720 to prepare a local checkpoint for the data set 730, wherein the first storage system 712 stores a first portion 732 of the data set 730. For example, the request 720 may be a prepare message received from a coordinator 716, such as Figure 4 422 in the prepare message 422. In some examples, I / O operations are suspended by a storage controller of storage system 712 setting a state in storage system 712 that can be checked during I / O request processing. Whether a particular operation is included in the checkpoint may depend on where the request in progress is in its processing when this state changes. In various embodiments, suspending I / O operations may mean delaying the processing of I / O requests (e.g., by adding I / O requests to a delay queue early in the I / O processing logic before the requests are actually executed) or delaying the signaling of I / O requests even if they have been completed internally.

[0261] In some embodiments, pausing 702 I / O operations includes initiating 704 a pause in processing I / O operations. In these embodiments, once the pause is initiated, received I / O requests directed to data set 730 are not processed, which means that any modifications to data set 730 contained in those I / O requests will not be applied until processing of the I / O requests continues. In the case where the I / O request is a read request, the time to read data from the data set will be shifted forward until processing of the I / O request continues. For write operations in process during the state change, the storage system 712 may choose to delay the signaling completion of the request or provide completion confirmation. For example, a specific processing stage may be selected as a cutoff point.

[0262] In other embodiments, pausing 702 I / O operations includes initiating 706 a delay in signaling completion confirmation of the I / O operations. In these embodiments, any modifications to the data set 730 included in those I / O requests will be applied, but completion confirmation of the I / O requests will not be provided before continuing to signal completion. In the case where the I / O request is a read request, the requested data can be read, but the requested data is not returned until the I / O operation for the data set continues, so the result of the read request cannot be relied on for any future writes until continuing to signal completion. For write operations that are ongoing during the state change, the storage system 712 may choose to delay the completion of the signaling request or provide completion confirmation. For example, a specific processing stage may be selected as a cutoff point.

[0263] In yet other embodiments, pausing 702 I / O operations includes both initiating 704 a pause in processing I / O operations and initiating 706 a delay in signaling completion confirmation of I / O operations. In these embodiments, processing of new I / O requests is suspended and completion confirmation of any ongoing operations will be delayed. Such embodiments do not require additional waiting for ongoing operations to complete before continuing the coordination protocol.

[0264] Figure 7 The example method also includes establishing 708, by the first storage system 712, a local checkpoint 750 dependent upon the I / O operation being suspended. In some examples, the local checkpoint is established 708 by updating a data structure to establish a checkpoint boundary, such as where modifications received whose processing proceeds to a certain point are included in the checkpoint and where modifications whose processing has not yet proceeded to that point are excluded from the checkpoint. For example, this boundary can be embodied as a state change. As long as the state change occurs after the I / O processing is suspended and before the processing continues, the time point of that state change can be different for different threads or processes. In some examples, the local checkpoint is used to create a named snapshot of the first portion 732 of the data set 730.

[0265] In an embodiment where completion is suspended, the local checkpoint should occur when the storage system begins to delay completion. In an embodiment where all modification processing is suspended and completion signaling is delayed, the local checkpoint can be established anywhere between the suspension and continuation of I / O processing, including just before the continuation. This is because any operations that arrive at the storage system after the suspension will not be processed, which means that they are excluded from the local checkpoint. In addition, the two storage systems do not need to implement the same checkpoint establishment procedure or timing to participate in the same coordinated checkpoint. Therefore, in some examples, the two storage systems 412, 414 may implement different checkpoint establishment procedures.

[0266] Because implementations for pausing I / O operations (I / O processing or I / O completion) and establishing local checkpoints depend only on read and modify operations being paused, and further on modify operations being included in the checkpoint, these implementations are applicable to any combination of storage technologies that support time-bounded checkpoints for modify operations. If the only form of checkpoints supported by a technology is fully persistent and named snapshot operations, then at least that storage technology can be used to represent checkpoints of snapshots. Thus, for example, one or more block volumes, scale-up file systems, and scale-out file systems can be utilized to produce consistent coordinated snapshots, as long as their implementations conform to the protocol employed by the coordinator service 716 described above.

[0267] Figure 7The method also includes continuing 710 I / O operations for data set 730 in response to a request 722 to commit the local checkpoint. Once the checkpoint is established, the storage system cannot continue I / O operations until the storage system receives the request 722 to commit the checkpoint. For example, the request 722 can be a commit message received from the coordinator 716, such as Figure 4 720. The request to commit the checkpoint 722 indicates that other storage systems (e.g., storage system 714) that also store portions of the data set 730 have also paused I / O operations on the data set 730. Therefore, for coordinated checkpoints, it is safe to rely on the local checkpoint on the first storage system 712 so that the I / O operations can continue. In some embodiments, continuing 710 I / O operations on the data set 730 while I / O processing is paused may include continuing to process pending I / O operations (e.g., I / O operations received while I / O processing was paused). In other embodiments, continuing 710 I / O operations on the data set 730 while signaling completion is paused may include continuing to signal completion of requested modifications, including modifications that were applied but not signaled when completion confirmation was paused.

[0268] In some examples, continuing 710 the I / O operation for data set 730 may further include confirming the successful commit of the local checkpoint because the local checkpoint has been established and the I / O operation has been continued. For example, storage system 712 may provide a confirmation message to coordinator service 716, such as Figure 4 722. However, the storage system 712 may also employ a fault handling mechanism in which the storage system 712 operates a timer after receiving the prepare message, and if the commit request 722 is not received within a certain period of time, the timer times out. In that scenario, the storage system 712 will abort the local checkpoint and continue the I / O operation. In this scenario, if the commit request 722 is received after the timer expires and the I / O operation continues, the storage system may respond to the request 722 with a confirmation message indicating that the local checkpoint is aborted.

[0269] To explain further, Figure 8 Another example method of a storage system for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Figure 8 The example depicted in contains many of the same elements as the figures described above. However, Figure 8 The example method also includes receiving 802 an I / O request 820 directed to a data set 730 while processing of the I / O operation is suspended. In some examples, the storage system 712 receives a request to read or write data directed to a first portion 732 of the data set 730 after the storage system 712 has suspended processing of the I / O request and before continuing the I / O processing request.

[0270] Figure 8 The example method also includes delaying 804 execution of the requested I / O operation until processing of the I / O operation continues. In some examples, the storage controller of the storage system 712 recognizes that the state set for the first portion 732 of the data set 730 indicates that I / O processing is suspended, and when the state changes to indicate that I / O processing is not suspended, the I / O request 820 is queued for later processing. Therefore, in the event that the request 820 includes a request to read data, those data are not read from the first portion 732 of the data set 730 until the I / O processing continues. In the event that the request 820 includes a request to write data, the modification is not applied to the data set, the first portion 732 of the data set 730 until the I / O processing continues.

[0271] Figure 8 The example method also includes performing 806 the requested I / O operation after continuing 710 the I / O operation. In some examples, performing 806 the requested operation may include reading data in the first portion 732 of the data set 730 or applying the requested modification to the first portion 732 of the data set 730.

[0272] To explain further, Fig. 9 Another example method of a storage system for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Fig. 9 The example depicted in contains many of the same elements as the figures described above. However, Fig. 9 The example method also includes receiving 902 an I / O request 920 directed to the data set 730 while the I / O operation is suspended. In some examples, the storage controller of the storage system 712 receives the request 920 to read or write data in the first portion 732 of the data set 730 after the storage system 712 has suspended signaling completion acknowledgments of the I / O operations and before continuing to signal completion acknowledgments.

[0273] Fig. 9 The example method also includes performing 904 the requested I / O operation in response to the I / O request 920. In some examples, performing 904 the requested operation includes reading data in the first portion 732 of the data set 730 or applying the requested modification to the first portion 732 of the data set 730. Here, the modification applied to the first portion 732 of the data set 730 is not included in the checkpoint.

[0274] Fig. 9The example method also includes delaying 906 a completion acknowledgement 922 for the requested I / O operation until the completion acknowledgement continues to be signaled. In some examples, a storage controller of the storage system 712 recognizes that a state set for the first portion 732 of the data set 730 indicates that completion acknowledgement is suspended, and retains completion acknowledgement 922 for the request 920 until the state changes to indicate that completion acknowledgement is not suspended.

[0275] Fig. 9 The example method also includes signaling 908 a completion confirmation 922 after continuing 710 the I / O operation. In some examples, signaling 908 a completion confirmation 922 includes sending a completion confirmation 922 of the I / O request 920 to the request initiator after the storage system 712 continues to signal a completion confirmation for the I / O request directed to the first portion 732 of the data set 730.

[0276] To explain further, Fig.10 Another example method of a storage system for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Fig.10 The example depicted in contains many of the same elements as the figures described above. However, Fig.10 The example method also includes indicating 1002 to the coordinator that the I / O operation is suspended. In some examples, the storage system 712 provides a ready message 1026 to the coordinator service 716 confirming that the storage system has suspended the I / O operation, such as Figure 4 722. In such instances, an indication that an I / O operation is suspended is provided to the coordinator service 716 as early as possible, because the coordinator service 716 must receive this confirmation from the storage system 712 (and from other storage systems participating in the checkpoint of the data set 730) before issuing the commit request 722. The sooner the ready message is provided to the coordinator service, the less time each storage system must wait for all other storage systems to suspend I / O operations before continuing the I / O operation. In various embodiments, a storage system may indicate 1002 that an I / O operation is suspended before or after establishing 708 its local checkpoint.

[0277] In view of the above, it can be appreciated that any I / O request received by the storage system prior to receiving a commit message from the coordinator depends only on previous requests to be included in the coordinated checkpoint (the local checkpoint on any of the storage systems) or to be excluded from the storage system's local checkpoint. This can be ensured based on the following:

[0278] • Delayed requests (ie, delays in processing requests) cannot be observed and will not be included in the checkpoint.

[0279] Delayed responses (i.e., delays in confirming request completion) cannot be observed but may be included in the checkpoint (which essentially forces concurrency). If it is processed to completion internally (but not signaled externally) before the local checkpoint, then it will be included. If it is not processed internally until after the local checkpoint is established, then it is excluded. Operations that are processed internally concurrently with the checkpoint establishment may or may be included.

[0280] Requests can only depend on previous requests that were observed due to the completion of a read or modify request. However, reads must be taken into account because they can observe the results of modifications even if the completion signal of the modification operation itself is delayed.

[0281] When a commit message is received by any storage system, all storage systems must ensure that any operations they may process to include in their local checkpoints can only depend on the included modifications, by delaying the possibility of observing any modifications that could be excluded (by delaying modifications or responses to modifications and by delaying any reads that might have observed those modifications) until after any included operation might have observed them.

[0282] As described above, there are three models that can achieve this. In the first model, the storage system can respond to the prepare message with a ready response after it has arranged for all subsequent requests to be excluded and delayed. It can achieve this by ensuring that the local checkpoint is established before responding with the ready message and by delaying completion signaling. In this case, any requests received after responding with the ready message will be excluded, and because completion signaling is delayed, any modifications that make their inclusion or exclusion ambiguous due to their concurrency with the local checkpoint operation cannot be observed.

[0283] In the second model, the storage system may delay both the processing of new requests and completion signaling of previously received requests before responding with a ready message, and may then establish a local checkpoint at any time after the delay and before continuing processing before responding with a success message. In this case, any requests received after the ready message will be excluded because they will not be processed at all, and because completion signaling is delayed, the results of any modifications whose inclusion or exclusion is unclear cannot be observed.

[0284] In a third model, the storage system may begin to delay processing of new requests before responding with a ready message, without regard to delayed completion signaling, and may then wait until it receives a commit message to establish its local checkpoint, after which it continues to process delayed requests. In that case, all requests that completed before receiving the commit message will be included, and only requests that may have observed potentially excluded modifications will be delayed.

[0285] If checkpointing (determining what concurrent modifications are excluded or excluded and ensuring that result) is fast, then the first model may have minimal performance impact and may have minimal impact on existing storage system implementations because it allows the most conventional concurrent processing and affects only one point in the conventional I / O path (completion signaling). With very fast checkpoints, the third model can also be very fast, and may be simpler if the storage system implementation makes it easier to queue incoming requests than to delay completion. If checkpointing is slow, then the second model may be preferred because it can start checkpointing but does not have to complete checkpointing before unblocking the protocol. The second method essentially allows the most parallel operation of the storage system across portions of the storage data set, but using that method reduces the parallel operation of storage requests within the local storage system to a certain extent.

[0286] To explain further, Fig.11 Another example method of a storage system for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Fig.11 The example depicted in contains many of the same elements as the figures described above. However, Fig.11 The example method also includes indicating 1102 a result of the local checkpoint after establishing the local checkpoint. In some examples, the storage system 712 provides a confirmation message 1126 (e.g., Figure 4 540 in ) to confirm the successful submission of the checkpoint. However, the storage system 712 may also adopt a fault handling mechanism in which the storage system 712 operates a timer after the local checkpoint is established, and if a request 722 to submit the checkpoint is not received within a certain period of time, the timer times out. In that scenario, the storage system 712 will abort the local checkpoint and continue the I / O operation. Therefore, in some examples, indicating 1102 the result of the local checkpoint may include sending a confirmation message indicating that the local checkpoint is aborted.

[0287] To explain further, Fig.12 An example method of a storage environment 1200 is presented for coordinating snapshots of data stored across disparate storage environments in accordance with some embodiments of the present disclosure. Fig.12 The example storage environment 1200 depicted in FIG. 1 may include two or more storage systems 1262, 1264 and a coordinator service 1266, where in this particular example, the coordinator service 1266 may orchestrate coordinated checkpoints of data sets across the two or more storage systems 1262, 1264. Each storage system 1262, 1264 may be similar to any of the storage systems described above, including any combination of the storage systems described above. In fact, Figure 4The storage systems 1262, 1264 depicted in FIG. 1264 may include the same, fewer, or additional components as the storage systems described above. The orchestrator service 1266 may be similar to the orchestrator service 416 described above and may be embodied as computer executable instructions executed on processing resources of the storage systems 1262, 1264 or on separate computing devices.

[0288] As discussed above, each storage system 1262, 1264 stores a portion of a data set (not shown), where in this particular example, a first portion 1232 of the data set 1230 is stored on one storage system 1262, and a second portion 1234 of the data set 1230 is stored on another storage system 1264. Although only two storage systems and two portions of the data set 1230 are illustrated, it should be recognized that there may be any number of storage systems storing additional portions of the data set and thus participating in the checkpoint. As described above, in some examples, the storage systems 1262, 1264 may employ different storage implementations or storage environments. Thus, the data set may include different types of storage objects distributed across multiple storage systems having different storage environments. For example, the storage environment of the portion 1232 of the data set 1230 on one storage system 1262 may implement block storage, a file system, a database, or an object storage. Likewise, the storage environment of the portion 1234 of the data set 1230 on another storage system may also implement block storage, a file system, a database, or an object storage. However, in some examples, the respective storage environments are different. For example, the first portion 1232 of the data set may include a block volume stored on the storage system 1262 , and the second portion 1234 of the data set may include a file stored on the storage system 1264 .

[0289] Fig.12 The example method includes requesting 1202 by a coordinator service 1266 that two or more storage systems 1262, 1264 be used for a data set 1230 (e.g., Figure 4 Dataset 430 or Figure 7 In some instances, the coordinator service 1266 receives a checkpoint from a management service (e.g., Figure 6 The management service 618 in the coordinator service 1266 receives a create checkpoint request 1280. In response to the request, the coordinator service 1266 directs the storage systems 1262, 1264 to create a coordinated checkpoint for the data set 1230 through a sequence of messages to the storage systems 1262, 1264. In some examples, the coordinator service 1266 provides prepare messages 1240, 1242 to two or more storage systems, wherein the prepare messages 1240, 1242 request to prepare a local checkpoint for the data set 730 on the storage systems 1262, 1264.

[0290] Fig.12The method also includes pausing 1204, by the first storage system 1262, I / O operations directed to the data set 1230 in response to the request to prepare the first local checkpoint. In response to receiving the prepare message 1240 from the coordinator service 1266, the first storage system 1262 suspends 1204 I / O operations. In some examples, the storage system 1262 is described above, for example, with respect to Figure 7 The method of pausing 702 I / O operations in the description of pausing 1204 I / O operations. Fig.12 The method also includes pausing 1206, by the second storage system 1264, in response to the request to prepare the second local checkpoint, the I / O operation directed to the data set 1230. In some examples, the storage system 1264 Figure 7 The I / O operations are suspended 1206 in the manner discussed in pausing 702 I / O operations in . In some cases, for example, due to message delivery delays, the storage systems 1262, 1264 receive the prepare messages 1240, 1242 at different times. Therefore, in some examples, the storage systems 1262, 1264 suspend I / O operations for the data set 1230 at different times. Thus, one storage system 1264 may continue I / O operations for a portion 1234 of the data set 1230 while the other storage system 1262 has suspended I / O operations for its portion 1232 of the data set 1230.

[0291] Consider the case where the first storage system 1262 is at time t n The I / O operation is suspended 1204 and the second storage system 1264 is at time t n+10 An instance of pausing 1206 an I / O operation, where the period P is t n With t n+10 . In this example, during period P, storage system 1264 is active for I / O operations on its portion 1234 of data set 1230, and storage system 1262 has suspended I / O operations on its portion of the data set. If, during period P, the second storage system receives an I / O request, and the I / O request depends on the result of an earlier I / O request that has been signaled as completed by the first storage system 1262, then that dependent I / O request may be processed and signaled as completed by the second storage system 1264. In other words, the result of the earlier I / O request may be observed by the second storage system 1264. Therefore, it is safe for the later I / O request to be included in the local checkpoint of the second storage system 1264 because the later I / O request may not depend on modifications that are excluded from the local checkpoint on the first storage system 1262.

[0292] In some examples, the storage systems 1262, 1264 utilize different mechanisms to suspend I / O operations. For example, the first storage system 1262 may suspend signaling completion confirmations, while the second storage system 1264 may suspend I / O processing entirely. The storage systems 1262, 1264 do not need to coordinate on the mechanisms for suspending I / O operations or even need to know how or when the other suspends I / O operations to establish a checkpoint.

[0293] Fig.12 The example method also includes indicating 1208, by the first storage system 1262, to the coordinator service 1266 that the first storage system has suspended its I / O operations. In some examples, the first storage system 1262 sends a ready confirmation 1244 to the coordinator service 1266 indicating that it has suspended I / O operations of the first portion 1232 of the data set 12030. Fig.12 The example method also includes indicating 1210, by the second storage system 1264, to the coordinator service 1266 that the second storage system has paused its I / O operations. In some examples, the second storage system 1264 sends a ready acknowledgement 1246 to the coordinator service 1266 indicating that it has paused I / O operations for the second portion of the data set.

[0294] Fig.12 The example method also includes establishing 1212 a first local checkpoint 1270 after pausing I / O operations on the first storage system 1262. In some examples, the storage system 1262 is configured as described above with respect to Figure 7 The first local checkpoint 1270 is established 1208 in the same manner as discussed in establishing 708 the checkpoint. The local checkpoint 1270 is a checkpoint of the first portion 1232 of the data set 1230. Fig.12 The method also includes establishing 1214 a second local checkpoint 1272 after suspending the I / O operation on the second storage system 1264. In some examples, the second storage system 1264 establishes its local checkpoint 1272, as described above with respect to Figure 7 The local checkpoint 1272 is a checkpoint of the second portion 1234 of the data set 1230. The first local checkpoint 1270 does not contain any modifications that depend on results that are excluded from the second local checkpoint 1272, and vice versa, even though the storage systems 1262, 1264 did depend on each other to suspend I / O operations before the respective local checkpoints 1270, 1272 were established. This is because the excluded results on one storage system cannot be observed and cannot become a dependency of modifications on another storage system.

[0295] As discussed above, each storage system 1262, 1264 can establish its local checkpoint by applying write boundaries that determine which pending or ongoing modifications will be included in the local checkpoint and which modifications will be excluded. For read operations, read boundaries determine the pending or ongoing read operations for which read results will be returned. These boundaries can be determined based on where a particular operation is located in its processing stage. In addition, boundaries can be established on a per-thread or per-process basis so that boundaries can be applied differently to different I / O streams because different cutoff points that determine whether an I / O operation will be included or excluded can be selected for the processing stage. In some instances, one or more of the storage systems 1262, 1264 establish their checkpoints without quiescing storage operations for their portions of the data set. That is, the storage system establishes its checkpoint when at least one I / O operation for the data set is in progress or has not yet been confirmed as completed. It should be recognized that the storage systems 1262, 1264 can establish respective local checkpoints at substantially different times. For example, where storage system 1262 employs a delay in completion confirmation to suspend I / O operations, its local checkpoint may be established before or concurrently with sending its ready message; and where storage system 1264 employs a delay in I / O processing to suspend I / O operations, its local checkpoint may be established after receiving a commit message from the coordinator service. However, because it is a point in time where both storage systems 1262, 1264 suspend their I / O operations for a data set, there is a time boundary during which no storage system application can observe modifications that create a dependency for some other I / O operation.

[0296] Fig.12 The example method of also includes determining 1216 that each of the two or more storage systems 1262, 1264 has suspended its I / O operations. For example, the coordinator service can determine that all ready messages 1244, 1246 have been received from the storage systems 1262, 1264.

[0297] Fig.12 The example method also includes requesting 1218, by the coordinator service 1266, in response to determining that all two or more storage systems 1262, 1264 have suspended their I / O operations, that the two or more storage systems 1262, 1264 continue the I / O operations when their respective local checkpoints have been established. In some examples, the coordinator service 1266 requests the storage systems 1262, 1264 to commit the checkpoints, as described above with respect to Figure 4Request 406 to continue I / O operations when local checkpoints are established. For example, the coordinator service 1266 sends a commit message 1248, 1250 indicating that the storage systems 1262, 1264 should continue I / O operations when their respective local checkpoints have been established. The commit message 1248, 1250 indicates that all storage systems 1262, 1264 participating in the checkpoint have suspended their I / O operations.

[0298] Fig.12 The example method also includes continuing 1220, by the first storage system 1262, the I / O operation for the data set 1230 in response to the first request 1248 to submit the first local checkpoint. In some examples, the first storage system 1262 Figure 7 The I / O operation is continued at 1220 in the manner discussed in Continuing 710 I / O operation. In the case where I / O processing is suspended, the storage system 1262 will begin processing pending I / O requests and new requests. In the case where completion confirmation is suspended, the storage system 1262 will begin providing completion confirmation for I / O requests processed during the suspension and for new I / O requests received. Fig.12 The example method also includes continuing 1222, by the second storage system 1264, the I / O operation for the data set 1230 in response to the second request 1250 to submit the second local checkpoint. In some examples, the second storage system 1264 is described above with respect to Figure 7 The I / O operation is continued at 1222 in the manner discussed in Continuation 710 of the I / O operation. In the case where I / O processing is suspended, the storage system 1264 will begin processing pending I / O requests and new requests. In the case where completion confirmation is suspended, the storage system 1264 will begin providing completion confirmation for I / O requests that were processed or applied during the suspension period and for new I / O requests received.

[0299] Fig.12 The example method also includes indicating 1224, by the first storage system 1262, a result of the local checkpoint after establishing the first local checkpoint. In some examples, the first storage system 1262 can indicate 1224 a result of the checkpoint as discussed above with respect to indicating 1102 a result of the checkpoint. For example, the first storage system 1262 can send a message 1252 confirming that the first local checkpoint 1270 was successfully established and that the I / O operation has continued. Fig.12 The example method also includes indicating 1226, by the second storage system 1264, a commit of the local checkpoint after establishing the second local checkpoint. In some examples, the second storage system 1264 can indicate 1226 a result of the checkpoint, as discussed above. For example, the second storage system 1264 can send a message 1254 confirming that the second local checkpoint 1272 was successfully established and the I / O operation has continued.

[0300] Fig.12 The example method also includes determining 1228, by the coordinator service 1266, that two or more storage systems 1262, 1264 have established corresponding local checkpoints. In some examples, the coordinator service 1266 can determine 1228 that corresponding local checkpoints have been established, as described above with respect to Figure 5 For example, once all confirmation messages 1252, 1254 have been received from two or more storage systems 1262, 1264, the coordinator service 1266 can determine that the corresponding local checkpoint has been established.

[0301] Fig.12 The example method also includes indicating 1229, by the coordinator service 1266, that the coordinated checkpoint of the data set 1230 has been completed in response to determining that the corresponding local checkpoint has been established. In some examples, the coordinator service 1266 can indicate that the coordinated checkpoint has been completed, as described above with respect to Figure 5 The indication 504 in the coordinated checkpoint has been completed is discussed. For example, the coordinator service 1266 can provide a checkpoint completion message 1282 to the requester of the checkpoint.

[0302] Consider examples such as Fig.13 13, where epoch A is the period before either storage system 1262, 1264 suspends I / O operations and establishes a local checkpoint. In this particular example, both storage systems 1262, 1264 suspend I / O operations by continuing to process I / O requests but delaying signaling completion confirmation. For example, an I / O request 1310 for modification A is received, the modification is applied, and completion confirmation 1312 for I / O request A is provided. In epoch A, operations received on storage system 1262 that completed before the suspension (e.g., modification A) may be dependencies of any of the following operations. Operations that enter near the time when system 1262 is suspended and near the checkpoint (e.g., modification B in I / O request 1314) may be included in the checkpoint or excluded from the checkpoint, which may depend on the progress of the modifications. Technically, any modifications included in the checkpoint can be safely completed without suspension. However, excluding modifications may be safer and simpler to implement. At time t1, storage system 1262 pauses signaling completion confirmation and establishes its local checkpoint.

[0303] exist Fig.13In the example of , period B is the period between when storage system 1262 pauses to complete and establish its local checkpoint at time t1 and when storage system 1264 pauses to complete and establish its local checkpoint at time t2. None of the modifications received by storage system 1264 during period B (e.g., modification C from I / O request 1316) depend on modifications that are excluded from the local checkpoint on storage system 1262. That is, no results are returned from period B that modification C may depend on. Therefore, completion 1318 of modification C may be signaled. Fig.13 In the example of , modification C depends on modification A. Modification A is processed and returned before the checkpoint is established, so that it is included in the checkpoint and modification C can depend on it. However, modification B, which is excluded from the checkpoint, cannot be observed and cannot become a dependency. Subsequently, an I / O request 1320 containing modification D is received by storage system 1262, where modification D depends on modification A and modification C. Modification D is applied, but completion confirmation is delayed. At time t3, storage system 1264 suspends signaling completion confirmation and establishes its local checkpoint.

[0304] exist Fig.13 In the example of , period C is a period during which storage systems 1262, 1264 suspend signaling completion confirmation and establish their local checkpoints so that any new modifications will be consistently excluded, such as modification E from I / O request 1322 received by storage system 1262 and modification F from I / O request 1324 received by storage system 1264. Since the implementation only suspends completion, modifications such as modifications E and F (which may depend on modifications A and C, for example) can continue through storage systems 1262, 1264, which is one of the performance advantages over fully distributed quiescing. In addition, both storage systems indicate to the coordinator service that local checkpoints are ready.

[0305] exist Fig.13 , period D is the period between when storage system 1262 continues to signal completion acknowledgments at time t3 and when storage system 1264 continues to signal completion acknowledgments at time t4. In response to the commit message from the coordinator service, storage system 1262 continues to signal completion acknowledgments, such as completion acknowledgment 1326 for modification D, completion acknowledgment 1328 for modification E, and completion acknowledgment 1330 for modification B. Thus, modification requests to any storage system may now be dependent on any new operation or previously suspended operation on storage system 1262. For example, a new I / O request 1332 for modification G received at storage system 1262 may be dependent on modifications A, B, C, D, and E. A completion acknowledgment 1334 is provided for modification G. An I / O request 1336 for modification H received at storage system 1264 may then be dependent on modifications A, B, C, D, E, and G.

[0306] exist Fig.13 In the example of , period E is a period after which storage system 1264 continues to signal completion confirmations at time t4. In response to the commit message from the coordinator service, storage system 1264 continues to signal completion confirmations, such as completion confirmation 1338 for modification F and completion confirmation 1340 for modification H. Therefore, full normal operation continues.

[0307] To explain further, Fig.14 Another example method of a storage system for coordinating snapshots of data stored across disparate storage environments according to some embodiments of the present disclosure is set forth. Fig.14 The example depicted in Figure 4 and may include many of the same elements as described above. Fig.14 In the example of , initiating 400 a coordinated checkpoint includes requesting 1402 a local clock value from each of two or more storage systems 412, 414 by a coordinator service 416. In order to reduce the amount of time that the storage systems wait for each to suspend I / O operations, the coordinator service 416 may schedule and establish their respective local checkpoints at a point in time when all storage systems participating in the coordinated checkpoint should suspend I / O operations (without requiring complete silence). However, without some additional clock synchronization mechanism, the system clocks on the various storage systems are not synchronized to the accuracy required to establish a coordinated checkpoint. Therefore, in some embodiments, the coordinator service 416 obtains the local clock values ​​of the participating storage systems 412, 414. For example, at time T local to the coordinator service 416 c0 , the coordinator service 416 sends requests 1420 , 1421 to retrieve local clock values ​​from the storage systems 412 , 414 .

[0308] Fig.14 The example also includes measuring 1404, by the coordinator service 416, for each of the two or more storage systems 412, 414, an amount of time to receive a clock response 1422, 1423, wherein the amount of time is identified as the precision of the storage system. The coordinator service 416 uses round-trip messages to determine an approximate value of the clock of each storage system, wherein the precision value is derived from the time of the round-trip (the cumulative time between sending a request for a storage system's clock value and receiving a response indicating the storage system's clock value). The lowest precision of the storage system can be determined by sending multiple such round-trip requests and using the lower precision values ​​derived from those requests. Thus, when each storage system 412, 414 is at T c0 When the request 1420 is received some time later, it responds with a clock response message 1422, 1423 containing the local time on that storage system. The coordinator service 416 measures T c0 With T csThe difference between cs is the time of the coordinator service's clock when it receives the clock response message from storage system 's'. c0 With T cs The difference between is chosen as the precision of that storage system.

[0309] Fig.14 The example also includes identifying 1406 a relative clock value by the coordinator service for each storage system 412, 414 based on a difference between a current clock value of the coordinator service and a local clock value of the storage system included in the clock response 1422, 1423. For example, the coordinator service 416 receives a clock response 1422, 1423 from each storage system 412, 414, wherein the clock response includes a time T s0 , where 's' is the identifier of a specific storage system. This time T s0 and the time T on the coordinator service clock c Is the relative clock value for that storage system that will be used to schedule local checkpoints on that storage system.

[0310] Fig.14 The example also includes identifying 1408, by the coordinator service 416, a worst-case accuracy among two or more storage systems 412, 414. The (maximum) accuracy value determined from the group of two or more storage systems 412, 414 may be used to calculate a delay, which may be used to ensure a duration for delaying or pausing an I / O operation, which may ensure overlap of paused I / Os among the storage systems (and thus avoid causal I / O relationships within that duration). When the coordinator service 416 receives the clock response messages 1422, 1423, it knows that the clock value for each storage system was obtained at some time between the time the coordinator service 416 issued the clock request message 1420, 1421 and the time the clock response message 1422, 1423 was received. The worst case (e.g., the time when the last clock response was received) may be selected as the time granularity T of the coordinated checkpoint. G . The time granularity represents the worst-case variation in the actual time that each storage system sees its clock. Given the uncertainty that the coordinator service 416 has about the actual local clock of each storage system, this also represents the amount of time that the coordinator service 416 can use to coordinate overlapping time spans of suspended I / Os. Therefore, in some instances, the last clock response message received by the coordinator service represents the longest (or worst-case) message delivery delay among the storage systems 412, 414. Consider an instance in which a clock response message 1422 from storage system 412 is received after a clock response message 1423 from storage system 414 (where storage systems 412, 414 only store portions of a data set and therefore participate in coordinating checkpoints). In this example, at Tc0 The transmission clock request message 1420 is transmitted at T cs The amount of time between receiving clock response messages 1422 represents the granularity T of coordinating checkpoints. G (i.e., T G =T cs -T c0 ).

[0311] Fig.14 The example method also includes providing 1410, by the coordinator service 416, to each of the two or more storage systems 412, 414, a request to initiate a local checkpoint on the storage system at a future time, wherein the future time is based on a relative clock value at which the storage system receives the request, and wherein the request includes a duration based on worst-case accuracy for pausing the I / O operation. In some examples, the coordinator service determines a future point in time for each of the clock values ​​adjusted for the two or more storage systems and a delay value derived from the maximum accuracy value among the two or more storage systems. The coordinator may then send this adjusted future point in time and delay value to each of the two or more storage systems storing a portion of the data set. For example, the coordinator service 416 determines a future time for the storage system 412 to initiate its local checkpoint based on the relative clock value of the storage system 412 derived from its clock response message 1422. The coordinator service 416 sends a message to the storage system 412 including this future time and a delay value derived from the maximum accuracy value among the two or more storage systems. G Similarly, the coordinator service 416 determines the future time when the storage system 414 initiates its local checkpoint based on the relative clock value of the storage system 414 derived from its clock response message 1423. The coordinator service 416 sends a checkpoint request 1424 containing this future time and the duration of the suspended I / O operation derived from T to the storage system 414. G The checkpoint request 1425 is exported for the duration of the suspended I / O operation.

[0312] In response, each storage system waits until its clock reaches that adjusted future time point, delays I / O processing, begins delaying I / O processing, waits for a duration corresponding to the delay value, establishes a checkpoint, and then continues I / O processing. If all of this is done correctly, the storage system can respond successfully to the coordinator. If any part fails (or if the future time point has actually passed, such as because a message was not received before that time), the storage system can respond to the coordinator with an abort indication.

[0313] In one example, the coordinator service 416 determines, for each storage system, a relatively safe relative time in the future for each storage system, where the future time T sp =T s0 +2T G+e, where e is chosen as a buffer to provide assurance to the coordinator service 416 that it can deliver the message to all storage systems 412, 414 before that future time. The coordinator service sends T sp and T G . Ensure that each storage system is sp to T sp +T G The time range of has some overlap with each other storage system. Each storage system waits according to its local clock until its corresponding future time (e.g., T sp ) and as discussed above, suspend I / O operations at that time for a specified duration (e.g., T G Each storage system 412, 414 then initiates its respective local checkpoint. When the storage system reaches its future local time (e.g., T sp ) plus a delay time (e.g. T G ) and completes the establishment of its local checkpoint, it responds to the coordinator service 416, indicating that it has completed the establishment of its local checkpoint. When the coordinator service 416 receives completion messages from all storage systems, the coordinated checkpoint is considered to have been established.

[0314] Therefore, the worst-case duration for which an I / O operation is suspended by any storage system is T established before any I / O delay actually occurs. G If any storage system responds abnormally slowly, T G Perform multiple tests to try to get the value of T G For example, if three rounds of clock request / response messages are exchanged for each storage system, then the best-case round trip for each storage system can be used to calculate T G (The calculation is the maximum of the minimum response times from all storage systems.) The value 'e' should be chosen to ensure that higher than normal message delays will not normally cause any storage system to exceed its local time T sp The checkpoint request message is received later. If the checkpoint request message is still received too late, the storage system can safely respond with an abort. If any storage system does not respond with a 'done' message after a reasonable period of time, the coordinator can abandon the checkpoint and perform cleanup.

[0315] In view of the foregoing, establishing a coordinated checkpoint can be implemented using a variety of techniques through a messaging protocol. In a first model, the storage system can respond to a prepare message with a prepare response after it has arranged for all subsequent requests to be excluded and delayed. It can do this by ensuring that the local checkpoint is established before responding with prepare complete and by delaying completion signaling. In this case, any requests received after responding with prepare complete will be excluded, and because completion signaling is delayed, any modifications that make their inclusion or exclusion ambiguous due to their concurrency with the local checkpoint operation cannot be observed.

[0316] In the second model, the storage system may delay both processing of new requests and completion signaling of previously received requests before responding with a ready, and may then establish a local checkpoint at any time after the delay and before continuing processing before responding with a success message. In this case, any requests received after the ready response will be discarded because they will not be processed at all, and because completion signaling is delayed, the results of any modifications whose inclusion or exclusion is unclear cannot be observed.

[0317] In a third model, the storage system may begin to delay processing of new requests before responding with ready, without regard to delayed completion signaling, and may then wait until it receives a commit message to establish its local checkpoint, after which it continues to process delayed requests. In that case, all requests that completed before receiving the commit message will be included, and only requests that may have observed potentially excluded modifications will be delayed.

[0318] If checkpointing (determining what concurrent modifications are excluded or excluded and ensuring that result) is fast, then the first model may have minimal performance impact and may have minimal impact on existing storage system implementations because it allows the most conventional concurrent processing and affects only one point in the conventional I / O path (completion signaling). With very fast checkpoints, the third model can also be very fast, and may be simpler if the storage system implementation makes it easier to queue incoming requests than to delay completion. If checkpointing is slow, then the second model may be preferred because it can start checkpointing but does not have to complete checkpointing before unblocking the protocol. The second model essentially allows the most parallel operation of the storage system across portions of the storage data set, but using that model reduces the parallel operation of storage requests within the local storage system to some extent. Since the same guarantees are made for all three models, each storage system can use the best of the three general models without caring about the methods used by anyone in the other storage systems.

[0319] In view of the foregoing, establishing a coordinated checkpoint can also be implemented using various techniques through a clock-based protocol. In one model, the coordinator sends a message requesting a clock to all coordinating storage systems. When the coordinator receives the clock message, it knows that the clock value of each storage system is obtained at some time between the time the coordinator sends the request message and the time the response is received. The coordinator identifies the worst-case change in the actual time that each storage system views its clock and uses this to calculate a reasonable amount of safe time that can be added to each received clock value. This forms a future safe time so that each storage system can receive a message with enough time to start delaying I / O processing and initiate a local checkpoint algorithm. The coordinator provides this local time difference to each storage system, and each storage system waits until this local future time and then delays the completion of the write and performs a new read, and then initiates a local checkpoint. The storage system responds to the coordinator, indicating that it has completed establishing its local checkpoint.

[0320] In order to coordinate checkpoints and snapshots on several storage systems, local checkpoints and snapshots are related together. This may include coordinated checkpoints that are actually coordinated and successfully coordinated or coordinated with some missing segments that are considered to be truly persistent failures. Because the local checkpoints and snapshots of each storage system are local to that storage system, and because the separate technologies coordinated by the coordinator can be quite different from each other, the coordinator implementation can relate these together while allowing the implementations and internal data structures of several storage systems to be properly separated from each other, and at the same time still be related to each other by higher-level software. In some embodiments, each local storage system checkpoint is marked in some way, for example, so that it is associated with a coordinated checkpoint. In one example, the coordinator controls the checkpoint or snapshot name or identifier. In another example, the local checkpoint is marked. The advantage of tags is that they can be added or removed quite easily.

[0321] Different technologies may support different naming or labeling methods, in which case the coordinator or management layer may track the names or identifiers of local snapshots on some local storage systems and the labels of other local storage systems. Because some storage systems may create their local snapshots for an attempted coordinated snapshot, while other storage systems may fail to create, or may crash or lose network access, thereby making the status of their local checkpoints / snapshots questionable, the coordinator may handle such failures to ensure consistent naming or labeling of local snapshots. For example, if the coordinator attempts to make a coordinated snapshot again after a snapshot failed, the coordinator may coordinate replacement names or identifiers and / or resolve labels that may or may not remain associated with the local checkpoints.

[0322] As described above, a data set may be replicated by using checkpoints. In one example of checkpoint-based replication (e.g., lightweight checkpoints or metadata-only checkpoints), a set of updates to a data set (e.g., one or more write operations directed to a data set) may occur between different checkpoints, such that the data set is updated to a particular checkpoint only when all updates to the data set prior to a particular checkpoint have been completed. In another example, a data set may be replicated by snapshot-based replication in the case where a snapshot is used as a checkpoint for a data set. In snapshot-based replication, a snapshot of a data set may be sent from a replication source storage system to a replication target storage system. In this embodiment, each snapshot may include the entire data set or a subset of the data set, such as (for example), only the portion of the data set that has changed since the last snapshot was sent from the replication source to the replication target. The reader should understand that snapshots may be sent on demand, based on a policy that takes various factors (e.g., time, number of operations, RPO settings) into account, or in some other manner. The reader should further understand that various types of checkpoints (e.g., metadata-only checkpoints, snapshot checkpoints, or other types of checkpoints that will be recognized by those skilled in the art) may be used. The reader should further understand that checkpoints may be distributed based on various factors (e.g., time, number of operations, RPO settings, etc.).

[0323] When a data set is distributed across multiple storage systems, each of which stores a distinct portion of the data set, checkpoint replication requires additional coordination. For example, source checkpoints for the data set must first be coordinated across the set of source storage systems. Once coordinated source checkpoints are established to contain consistent source checkpoints for each local portion of the data set on each source storage system, replication of those local source checkpoints to the target storage systems must also be coordinated. Additionally, replica checkpoints received by the target storage systems must also be coordinated to establish a coordinated checkpoint of the replica data set distributed across the target storage systems. Fig.15 Initially, a mechanism for coordinating consistent replication checkpoints on a target storage system according to the present disclosure is described with reference to the various figures.

[0324] To explain further, Fig.15 A flow chart presenting an example method for coordinating snapshots across multiple storage systems. Fig.15 The example depicted in may include two or more source storage systems 1512 , 1514 and a source data set 1530 stored across the two or more storage systems 1512 , 1514 . Fig.15The example depicted in may also include two or more target storage systems 1513, 1515 storing a replica data set 1531 across the two or more target storage systems 1513, 1515. In this example, the replica data set 1531 is a replication target of the source data set 1530. Each storage system (such as the source storage systems 1512, 1514 and the target storage systems 1513, 1515 in this particular example) may be similar to the storage systems described above, including any combination of the storage systems described above. In fact, Fig.15 The source storage systems 1512, 1514 and target storage systems 1513, 1515 depicted in FIG. 1 may include the same, fewer, or additional components as the storage systems described above.

[0325] Fig.15 An example of includes identifying 1502 a replica data set 1531 stored across two or more target storage systems 1513, 1515, wherein the replica data set 1531 is a replication target of a source data set 1530 stored across two or more source storage systems 1512, 1514. In some examples, the target coordinator service 1517 identifies the replica data set 1531 as a data set for which the coordinator service 1517 is responsible for orchestrating a coordination target checkpoint of the replicated data set 1531 across the two or more target storage systems 1513, 1515. For example, the target coordinator service 1517 may be configured to coordinate the replica data set 1531 by a user or an administrator or some other management entity, the target coordinator service 1517 may discover the replica data set 1531 based on monitoring of the target storage systems 1513, 1515, or the target coordinator service 1517 may identify the replica data set 1531 based on registration of the replica data set with the target coordinator service 1517 by the target storage system or from a source coordinator service. As will be described in more detail below, the coordinated source checkpoints are replicated to the target storage systems 1513, 1515, and the target coordinator service 1517 coordinates the replicated checkpoints on two or more target storage systems 1513, 1515. In various embodiments, the target coordinator service 1517 may be embodied as computer executable instructions executed on a processing resource of one of the target storage systems 1513, 1515, a management server, a management software plane, a cloud-based service, or on some other computing device.

[0326] Fig.15 An example of may also include a source coordinator service 1516 that orchestrates source checkpoints for a data set 1530 across two or more source storage systems 1512, 1514. In some examples, the source coordinator service 1516 and the source storage systems 1512, 1514 are similar to those described above with respect to Figure 4The coordinator service 416 and storage systems 412, 414 described above can be configured in the same or similar manner. Figures 4 to 14 In various embodiments, the source coordinator service 1516 may be embodied as computer executable instructions executed on a processing resource of one of the source storage systems 1512, 1514, a management server, a management software plane, a cloud-based service, or on some other computing device.

[0327] exist Fig.15 , a source data set 1530 is coordinated across two or more source storage systems 1512, 1514. For example, a first portion 1532 of the source data set 1530 is stored on the source storage system 1512, and a second portion 1534 of the data set 1530 is stored on the source storage system 1514. Thus, each source storage system 1512, 1514 stores a distinct local portion 1532, 1534 of the source data set 1530. In some examples, the two or more source storage systems 1512, 1514 have different implementations and storage environments (although it is not a requirement for all embodiments of the present disclosure that the storage systems 1512, 1514 are different). For example, the storage system environment may implement a specific type of storage, such as block storage, file system storage, object storage, and database storage. In one non-limiting example, source storage system 1512 may store a block volume as a first portion 1532 of data set 1530, and another source storage system 1514 may store a file system as a second portion 1534 of data set 1530. Thus, in some examples, source storage systems 1512, 1514 may employ different storage environments to implement storage of data types.

[0328] In some examples, each source storage system 1512, 1514 is paired with a target storage system 1513, 1515 to replicate its local portion of the source data set 1530 to its paired target storage system 1513, 1515. In some implementations, the source storage systems 1512, 1514 and the target storage systems 1513, 1515 are paired with respective replication links 1580, 1582, such as the replication links described above. The replication links 1580, 1582 may represent communication paths and replication protocols for replicating data of the local portion 1532, 1534 of the source data set 1530 from the source storage systems 1512, 1514 to the local portion of the replica data set 1531 on the target storage systems 1513, 1515. In some examples, the replication links may specify a replication policy, where the replication policy may include a snapshot or may be exclusively a snapshot, specify continuous but asynchronous replication, or specify synchronous replication. A single user interface, a single workflow, may be provided to the user for replica link specification, thereby allowing specification of one or more characteristics of data replication.

[0329] Thus, in some embodiments, identifying 1502 a replica data set 1531 stored across two or more target storage systems 1513, 1515, where the replica data set 1531 is a replication target of a source data set 1530 stored across two or more source storage systems 1512, 1514, may include determining which target storage systems store respective local portions 1533, 1535 of the replica data set 1531, where the respective local portions 1532, 1534 of the source data set 1530 are replicated from the source storage systems 1512, 1514 to the respective target storage systems 1513, 1515 via respective replication links 1580, 1582. Thus, in these examples, the local portions 1533, 1535 together constitute the replica data set 1531 stored across the target storage systems 1513, 1515.

[0330] exist Fig.15, the source storage system 1512 and the target storage system 1513 form a replication pair via a replication link 1580, wherein the local portion 1533 of the replica data set 1531 on the target storage system 1513 is the replication target of the local source portion 1532 of the source data set 1530 on the source storage system 1512. Similarly, the source storage system 1514 and the target storage system 1515 form a replication pair via a replication link 1582, wherein the local portion 1535 of the replica data set 1531 on the target storage system 1515 is the replication target of the local source portion 1534 of the source data set 1530 on the source storage system 1514. In some examples, the replication links 1580, 1582 may utilize or implement different replication technologies. For example, the source storage system 1512 and the target storage system 1513 may implement snapshot-based replication via the replication link 1580, while the source storage system 1513 and the target storage system 1515 may implement lightweight checkpoint replication via the replication link 1582. In some examples, the replication frequency and timing of replicating the local portions may be different across the replication links 1580 , 1582 .

[0331] In some examples, two or more target storage systems 1513, 1515 have different implementations and storage environments (although it is not a requirement for all embodiments of the present disclosure that the target storage systems 1513, 1515 are different). For example, the storage system environment may implement a specific type of storage, such as block storage, file system storage, object storage, and database storage. In one non-limiting example, a target storage system 1513 may store a block volume as one local portion 1533 of a replica data set 1531, and another target storage system 1515 may store a file system as another local portion 1535 of the replica data set 1531. Therefore, in some examples, the source storage systems 1512, 1514 may use different storage environments to implement storage of data types.

[0332] To illustrate the flexibility of coordinating checkpoints across multiple disparate source storage systems and replicating them to multiple disparate target storage systems, there may be instances where the two source storage systems 1512, 1514 coordinating the source data set 1530 implement different storage environments relative to each other, the two source storage systems 1512, 1514 may implement different types of local checkpoints relative to each other, and the two replication links 1580, 1582 may implement different replication protocols relative to each other. However, it should be recognized that any storage environment or replication technology difference is not a requirement for all embodiments of the present disclosure.

[0333] Fig.15The example method also includes identifying 1504 identifying two or more local replication checkpoin...

Claims

1. A method comprising: monitoring a set of two or more target storage systems, wherein the two or more target storage systems are replication targets for respective portions of a source data set; associating a first coordinated tracking data set with two or more local tracking data sets maintained on the two or more target storage systems, wherein each local tracking data set is a tracking copy of a portion of the source data set; and The first coordinated tracking data set is advanced depending on a first coordinated target checkpoint.

2. The method of claim 1, wherein the first coordinated target checkpoint is associated with a set of locally replicated checkpoints replicated from a set of source storage systems; and wherein a set of local source checkpoints are associated with a coordinated source checkpoint of the data set.

3. The method of claim 2, wherein the coordinated source checkpoint is a snapshot coordinated across the set of source storage systems; and wherein the first coordinated target checkpoint is a coordinated replication snapshot.

4. The method according to claim 1, wherein a storage implementation of a first target storage target system among the two or more target storage systems is one of a block storage implementation, a file system implementation, a database implementation, and an object storage implementation; wherein the storage implementation of a second target storage system among the two or more target storage systems is one of a block storage implementation, a file system implementation, a database implementation, and an object storage implementation; and The storage implementation scheme of the first target storage system is different from the storage implementation scheme of the second target storage system.

5. The method according to claim 1, wherein the two or more target storage systems are paired with at least one of the two or more source storage systems through corresponding replication links to form a plurality of replication pairs; and Wherein each of the two or more source storage systems stores a distinct local portion of the source data set.

6. The method according to claim 1, further comprising: identifying a second coordination target checkpoint, wherein the second coordination target checkpoint is incomplete; and A second coordinated tracking data set is created depending on the second coordinated target checkpoint.

7. The method according to claim 1, further comprising: In response to a promotion event, the first coordinated tracking data set is promoted.

8. The method of claim 7, wherein the source data sets stored across the set of source storage systems maintain promotions. 9 . The method of claim 7 , wherein the source data set stored across the set of source storage systems is destaged in response to a destage event.

10. The method of claim 9, wherein in response to the degradation event, a coordinated source checkpoint is established; and wherein the coordinated source checkpoint is replicated from the set of source storage systems to the set of target storage systems.

11. The method of claim 9, wherein a target coordinator and a source coordinator coordinate the promotion of the coordination tracking data set across the set of target storage systems and the demotion of the source data set across the set of source storage systems.

12. The method according to claim 1, further comprising: Coordinate the clones of the first coordinated tracking data set.

13. An apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having computer program instructions disposed therein, which when executed by the computer processor, cause the apparatus to: monitoring a set of two or more target storage systems, wherein the two or more target storage systems are replication targets for respective portions of a source data set; associating a first coordinated tracking data set with two or more local tracking data sets maintained on the two or more target storage systems, wherein each local tracking data set is a tracking copy of a portion of the source data set; and The first coordinated tracking data set is advanced depending on a first coordinated target checkpoint.

14. The apparatus of claim 13, further comprising computer program instructions causing the apparatus to: identifying a second coordination target checkpoint, wherein the second coordination target checkpoint is incomplete; and A second coordinated tracking data set is created depending on the second coordinated target checkpoint.

15. The apparatus of claim 13, further comprising computer program instructions causing the apparatus to: In response to a promotion event, the first coordinated tracking data set is promoted.

16. The apparatus of claim 13, further comprising computer program instructions causing the apparatus to: Coordinate the clones of the first coordinated tracking data set.

17. A computer program product disposed on a computer readable medium, the computer program product comprising computer program instructions which, when executed by a processor, cause a computer to: monitoring a set of two or more target storage systems, wherein the two or more target storage systems are replication targets for respective portions of a source data set; associating a first coordinated tracking data set with two or more local tracking data sets maintained on the two or more target storage systems, wherein each local tracking data set is a tracking copy of a portion of the source data set; and The first coordinated tracking data set is advanced depending on a first coordinated target checkpoint.

18. The computer program product of claim 17, further comprising computer program instructions that cause the computer to: identifying a second coordination target checkpoint, wherein the second coordination target checkpoint is incomplete; and A second coordinated tracking data set is created depending on the second coordinated target checkpoint.

19. The computer program product of claim 17, further comprising computer program instructions that cause the computer to: In response to a promotion event, the first coordinated tracking data set is promoted.

20. The computer program product of claim 17, further comprising computer program instructions that cause the computer to: Coordinate the clones of the first coordinated tracking data set.

Images