Virtual storage system architecture

Abstract

Serving I / O operations in a virtual storage system includes: receiving, by the virtual storage system, a request to write data to the virtual storage system; storing the data in a hierarchical storage provided by one or more virtual drives of the virtual storage system; and migrating at least a portion of the data stored in the hierarchical storage from the hierarchical storage to a more persistent data storage provided by a cloud service provider.

Description

Technical Field

[0001] The present disclosure relates generally to cloud-based storage systems, and more particularly to virtual storage system architectures. Background Art

[0002] Need to service I / O operations in the virtual storage system. Summary of the invention

[0003] The present disclosure generally provides systems and methods for servicing I / O operations in a storage system. In one aspect, a method includes receiving, by a virtual storage system, a request to write data to the virtual storage system, the virtual storage system having components residing in a plurality of availability zones; storing the data in storage provided by a first storage tier of the virtual storage system at a first availability zone; and migrating at least a portion of the data stored in the first storage tier from the first storage tier at the first availability zone to a second storage tier at a cloud-based data store provided by a cloud service provider, the second storage tier being more durable than the first storage tier of the virtual storage system, wherein the components in two or more of the plurality of availability zones coordinate with each other to synchronously store the replicated data at the cloud-based data store.

[0004] On the other hand, a virtual storage system included in a cloud computing environment includes: multiple virtual drives that provide hierarchical storage for storage operations, wherein the multiple virtual drives are arranged in multiple availability zones; and one or more virtual controllers, each of which is executed in a cloud computing instance, wherein the one or more virtual controllers are configured to: receive a request from the virtual storage system to write data to the virtual storage system; store the data in the hierarchical storage provided by one or more virtual drives in a first availability zone of the virtual storage system; and migrate at least a portion of the data stored in the hierarchical storage from the hierarchical storage at the first availability zone to a more persistent data storage at a cloud-based data storage provided by a cloud service provider, wherein components within two or more of the multiple availability zones coordinate with each other to synchronously store the replicated data at the cloud-based data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A A first example system for data storage according to some implementations is illustrated.

[0006] Figure 1B A second example system for data storage according to some implementations is illustrated.

[0007] Figure 1C A third example system for data storage according to some implementations is illustrated.

[0008] Figure 1D A fourth example system for data storage is illustrated according to some implementations.

[0009] Figure 2A is a perspective diagram of a storage cluster having multiple storage nodes and internal storage coupled to each storage node to provide network attached storage according to some embodiments.

[0010] Figure 2B is a block diagram illustrating an interconnect switch coupling a plurality of storage nodes according to some embodiments.

[0011] Figure 2C is a multi-level block diagram illustrating the contents of a storage node and the contents of one of the non-volatile solid-state storage units according to some embodiments.

[0012] Figure 2D A storage server environment is shown according to some embodiments using embodiments of the storage nodes and storage units of some of the previous figures.

[0013] Figure 2E is a blade hardware block diagram illustrating the control plane, compute and storage planes, and the authority to interact with underlying physical resources according to some embodiments.

[0014] Figure 2F Depicted is a resiliency software layer in a blade of a storage cluster according to some embodiments.

[0015] Figure 2G Depicted are permissions and storage resources in blades of a storage cluster according to some embodiments.

[0016] Figure 3A A diagram of a storage system coupled for data communication with a cloud service provider is set forth in accordance with some embodiments of the present invention.

[0017] Figure 3B A diagram of a storage system according to some embodiments of the invention is illustrated.

[0018] Figure 3C An exemplary computing device is illustrated that may be specifically configured to perform one or more of the processes described herein.

[0019] Figure 3D A block diagram illustrating multiple storage systems supporting pods according to some embodiments of the present invention is set forth.

[0020] Figure 3E A flow chart illustrating an example method of servicing I / O operations for a data set synchronized across multiple storage systems according to some embodiments of the present invention is set forth.

[0021] Figure 4 An example of a cloud-based storage system according to some embodiments of the invention is described.

[0022] Figure 5 Additional cloud-based storage system examples according to some embodiments of the invention are set forth.

[0023] Figure 6 A flow chart illustrating an example method of servicing I / O operations in a cloud-based storage system is set forth.

[0024] Figure 7 A flow chart illustrating an example method of servicing I / O operations in a cloud-based storage system is set forth.

[0025] Figure 8 Flowcharts illustrating additional example methods of servicing I / O operations in a cloud-based storage system are set forth.

[0026] Fig. 9 Flowcharts illustrating additional example methods of servicing I / O operations in a cloud-based storage system are set forth.

[0027] Fig.10 Flowcharts illustrating additional example methods of servicing I / O operations in a cloud-based storage system are set forth.

[0028] Fig.11 Flowcharts illustrating additional example methods of servicing I / O operations in a cloud-based storage system are set forth.

[0029] Fig.12 An example virtual storage system architecture is illustrated according to some embodiments of the present invention.

[0030] Fig.13 Additional example virtual storage system architectures according to some embodiments of the invention are illustrated.

[0031] Fig.14 Additional example virtual storage system architectures according to some embodiments of the invention are illustrated.

[0032] Fig.15 Additional example virtual storage system architectures according to some embodiments of the invention are illustrated.

[0033] Fig.16 Additional example virtual storage system architectures according to some embodiments of the invention are illustrated.

[0034] Fig.17 Flowcharts illustrating additional example methods of servicing I / O operations in a virtual storage system are set forth.

[0035] Fig.18Flowcharts illustrating additional example methods of servicing I / O operations in a virtual storage system are set forth. DETAILED DESCRIPTION

[0036] Reference Figure 1A The initial figures describe example methods, apparatus, and products for a virtual storage system architecture according to embodiments of the present invention. Figure 1A An example system for data storage according to some implementations is illustrated. System 100 (also referred to herein as a "storage system") includes a number of elements for purposes of illustration and not limitation. It may be noted that system 100 may include the same, more, or fewer elements configured in the same or different manner in other implementations.

[0037] The system 100 includes a plurality of computing devices 164A-B. The computing devices (also referred to herein as "client devices") may be embodied as, for example, servers in a data center, workstations, personal computers, or notebooks, etc. The computing devices 164A-B may be coupled to communicate data with one or more storage arrays 102A-B via a storage area network (SAN) 158 or a local area network (LAN) 160.

[0038] SAN 158 can be implemented using various data communication structures, devices, and protocols. For example, the structure of SAN 158 can include Fibre Channel, Ethernet, Infiniband, or Serial Attached Small Computer System Interface (SAS), etc. The data communication protocols used with SAN 158 can include Advanced Technology Attachment (ATA), Fibre Channel Protocol, Small Computer System Interface (SCSI), Internet Small Computer System Interface (iSCSI), Ultra Small Computer System Interface (HyperSCSI), or Non-Volatile Memory Express (NVMe) across structures, etc. It can be noted that SAN 158 is provided for illustration and not limitation. Other data communication couplings can be implemented between computing devices 164A-B and storage arrays 102A-B.

[0039] LAN 160 may also be implemented using various structures, devices, and protocols. For example, the structure of LAN 160 may include Ethernet (802.3) or wireless (802.11), etc. The data communication protocol used in LAN 160 may include transmission control protocol (TCP), user datagram protocol (UDP), Internet protocol (IP), hypertext transfer protocol (http), wireless access protocol (WAP), handheld device transfer protocol (HDTP), session initiation protocol (SIP) or real-time protocol (RTP), etc.

[0040] The storage arrays 102A-B may provide persistent data storage for the computing devices 164A-B. In implementations, the storage array 102A may be contained in a chassis (not shown) and the storage array 102B may be contained in another chassis (not shown). The storage arrays 102A and 102B may include one or more storage array controllers 110A-D (also referred to herein as "controllers"). The storage array controllers 110A-D may be embodied as automated computing machinery modules including computer hardware, computer software, or a combination of computer hardware and software. In some implementations, the storage array controllers 110A-D may be configured to perform various storage tasks. The storage tasks may include writing data received from the computing devices 164A-B to the storage arrays 102A-B, erasing data from the storage arrays 102A-B, retrieving data from the storage arrays 102A-B and providing the data to the computing devices 164A-B, monitoring and reporting disk utilization and performance, performing redundancy operations such as redundant array of independent drives (RAID) or RAID-like data redundancy operations, compressing data, and encrypting data, etc.

[0041] The storage array controllers 110A-D may be implemented in various ways, including being implemented as a field programmable gate array (FPGA), a programmable logic chip (PLC), an application specific integrated circuit (ASIC), a system on a chip (SOC), or any computing device including discrete components such as a processing device, a central processing unit, a computer memory, or various adapters. The storage array controllers 110A-D may, for example, include a data communication adapter configured to support communication via the SAN 158 or the LAN 160. In some implementations, the storage array controllers 110A-D may be independently coupled to the LAN 160. In an implementation, the storage array controllers 110A-D may include an I / O controller, etc., which enables the storage array controllers 110A-D to be coupled to the persistent storage resources 170A-B (also referred to herein as "storage resources") through a midplane (not shown) for data communication. The persistent storage resources 170A-B mainly include any number of storage drives 171A-F (also referred to herein as "storage devices") and any number of non-volatile random access memory (NVRAM) devices (not shown).

[0042] In some implementations, the NVRAM devices of the persistent storage resources 170A-B may be configured to receive data to be stored in the storage drives 171A-F from the storage array controllers 110A-D. In some examples, the data may originate from the computing devices 164A-B. In some examples, writing the data to the NVRAM devices may be performed faster than writing the data directly to the storage drives 171A-F. In an implementation, the storage array controllers 110A-D may be configured to utilize the NVRAM devices as fast accessible buffers for data destined to be written to the storage drives 171A-F. The latency of write requests using the NVRAM devices as buffers may be improved relative to systems in which the storage array controllers 110A-D write data directly to the storage drives 171A-F. In some implementations, the NVRAM devices may be implemented using computer memory in the form of high-bandwidth, low-latency RAM. The NVRAM devices are referred to as "non-volatile" because the NVRAM devices may receive or include a unique power source that maintains the state of the RAM after the main power of the NVRAM device is lost. Such a power source may be a battery or one or more capacitors, etc. In response to a power loss, the NVRAM device may be configured to write the contents of the RAM to persistent storage such as storage drives 171A-F.

[0043] In implementations, storage drives 171A-F may refer to any device configured to persistently record data, where "persistently" or "persistent" refers to the ability of the device to maintain the recorded data after a power loss. In some implementations, storage drives 171A-F may correspond to non-disk storage media. For example, storage drives 171A-F may be one or more solid-state drives (SSDs), flash memory-based storage, any type of solid-state non-volatile memory, or any other type of non-mechanical storage device. In other implementations, storage drives 171A-F may include a mechanical or rotating hard disk such as a hard disk drive (HDD).

[0044] In some implementations, the storage array controllers 110A-D may be configured to offload device management responsibilities from the storage drives 171A-F in the storage arrays 102A-B. For example, the storage array controllers 110A-D may manage control information that may describe the status of one or more memory banks in the storage drives 171A-F. The control information may, for example, indicate that a particular memory bank has failed and should no longer be written to, that the particular memory bank contains boot code for the storage array controllers 110A-D, the number of program-erase (P / E) cycles that have been performed on the particular memory bank, the age of the data stored in the particular memory bank, and the type of data stored in the particular memory bank. In some implementations, the control information may be stored as metadata along with the associated memory bank. In other implementations, the control information for the storage drives 171A-F may be stored in one or more particular memory banks of the storage drives 171A-F selected by the storage array controllers 110A-D. The selected memory banks may be marked with an identifier indicating that the selected memory bank contains the control information. The storage array controllers 110A-D may utilize the identifier in conjunction with the storage drives 171A-F to quickly identify the memory blocks containing the control information. For example, the storage controllers 110A-D may issue a command to locate the memory blocks containing the control information. It may be noted that the control information may be so large that a portion of the control information may be stored in multiple locations, the control information may be stored in multiple locations, for example, for redundancy purposes, or the control information may be otherwise distributed across multiple memory blocks in the storage drives 171A-F.

[0045] In implementation, the storage array controllers 110A-D may offload device management responsibilities from the storage drives 171A-F of the storage array 102A-B by retrieving control information describing the state of one or more memory banks in the storage drives 171A-F from the storage drives 171A-F. Retrieving control information from the storage drives 171A-F may be performed, for example, by the storage array controllers 110A-D querying the storage drives 171A-F to obtain the location of the control information for a particular storage drive 171A-F. The storage drives 171A-F may be configured to execute instructions that enable the storage drives 171A-F to identify the location of the control information. These instructions may be executed by a controller (not shown) associated with or otherwise located on the storage drives 171A-F, and may cause the storage drives 171A-F to scan portions of the memory banks to identify the memory banks for storing the control information for the storage drives 171A-F. The storage drivers 171A-F may respond by sending a response message including the location of the control information for the storage drivers 171A-F to the storage array controllers 110A-D. In response to receiving the response message, the storage array controllers 110A-D may issue a request to read the data stored at the address associated with the location of the control information for the storage drivers 171A-F.

[0046] In other implementations, the storage array controllers 110A-D may also offload device management responsibilities from the storage drives 171A-F by performing storage drive management operations in response to receiving control information. The storage drive management operations may, for example, include operations that are typically performed by the storage drives 171A-F (e.g., a controller (not shown) associated with a particular storage drive 171A-F). The storage drive management operations may, for example, include ensuring that data is not written to failed memory blocks within the storage drives 171A-F and ensuring that data is written to memory blocks within the storage drives 171A-F in a manner that achieves adequate wear leveling, etc.

[0047] In implementation, the storage arrays 102A-B may implement two or more storage array controllers 110A-D. For example, the storage array 102A may include the storage array controller 110A and the storage array controller 110B. In a given example, a single storage array controller 110A-D (e.g., the storage array controller 110A) of the storage system 100 may be designated as having a primary status (also referred to herein as a "primary controller"), and other storage array controllers 110A-D (e.g., the storage array controller 110A) may be designated as having a secondary status (also referred to herein as a "secondary controller"). The primary controller may have certain rights, such as permission to change data in the persistent storage resources 170A-B (e.g., write data to the persistent storage resources 170A-B), etc. At least some rights of the primary controller may supersede the rights of the secondary controller. For example, when the primary controller has the right to change the permission of data in the persistent storage resources 170A-B, the secondary controller may not have the right. The status of the storage array controllers 110A-D may change. For example, storage array controller 110A may be designated as having a secondary status, while storage array controller 110B may be designated as having a primary status.

[0048] In some implementations, a primary controller such as storage array controller 110A may be used as a primary controller for one or more storage arrays 102A-B, and a secondary controller such as storage array controller 110B may be used as a secondary controller for one or more storage arrays 102A-B, for example, storage array controller 110A may be a primary controller for storage array 102A and storage array 102B, and storage array controller 110B may be a secondary controller for storage arrays 102A and 102B. In some implementations, storage array controllers 110C and 110D (also referred to as "storage processing modules") may have neither primary status nor secondary status. Storage array controllers 110C and 110D implemented as storage processing modules may be used as a communication interface between the primary and secondary controllers (e.g., storage array controllers 110A and 110B, respectively) and storage array 102B. For example, storage array controller 110A of storage array 102A may send a write request to storage array 102B via SAN 158. The write request may be received by both storage array controllers 110C and 110D of storage array 102B. Storage array controllers 110C and 110D facilitate communication, for example, sending the write request to the appropriate storage drives 171A-F. It may be noted that in some implementations, the storage processing module may be used to increase the number of storage drives controlled by the primary and secondary controllers.

[0049] In implementation, storage array controllers 110A-D are communicatively coupled to one or more storage drives 171A-F and one or more NVRAM devices (not shown) included as part of storage arrays 102A-B via a midplane (not shown). Storage array controllers 110A-D may be coupled to the midplane via one or more data communications links, and the midplane may be coupled to storage drives 171A-F and NVRAM devices via one or more data communications links. The data communications links described herein are collectively illustrated by data communications links 108A-D, and may include, for example, a peripheral component interconnect express (PCIe) bus.

[0050] Figure 1B An example system for data storage according to some implementations is illustrated. Figure 1B The storage array controller 101 shown in the example can be used with Figure 1A The storage array controllers 110A-D described above are similar. In one example, the storage array controller 101 can be similar to the storage array controller 110A or the storage array controller 110B. The storage array controller 101 includes a number of elements for purposes of illustration and not limitation. It is noted that the storage array controller 101 can include the same, more, or fewer elements configured in the same or different manner in other implementations. It is noted that the following can be included Figure 1A , to help illustrate the features of storage array controller 101.

[0051] The storage array controller 101 may include one or more processing devices 104 and a random access memory (RAM) 111. The processing device 104 (or controller 101) represents one or more general-purpose processing devices such as a microprocessor or a central processing unit. More specifically, the processing device 104 (or controller 101) may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor for implementing other instruction sets or a processor for implementing a combination of instruction sets. The processing device 104 (or controller 101) may also be one or more special-purpose processing devices such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor.

[0052] Processing device 104 may be connected to RAM 111 via data communication link 106, which may be embodied as a high-speed memory bus such as a fourth-generation double data rate (DDR4) bus. Stored in RAM 111 is an operating system 112. In some implementations, instructions 113 are stored in RAM 111. Instructions 113 may include computer program instructions for operating in a direct-mapped flash storage system. In one embodiment, a direct-mapped flash storage system is a system that directly addresses data blocks within a flash drive without address translation by a storage controller of the flash drive.

[0053] In implementation, the storage array controller 101 includes one or more host bus adapters 103A-C coupled to the processing device 104 via data communication links 105A-C. In implementation, the host bus adapters 103A-C may be computer hardware that connects a host system (e.g., a storage array controller) to other networks and storage arrays. In some examples, the host bus adapters 103A-C may be a Fibre Channel adapter for enabling the storage array controller 101 to connect to a SAN or an Ethernet adapter for enabling the storage array controller 101 to connect to a LAN, etc. The host bus adapters 103A-C may be coupled to the processing device 104 via data communication links 105A-C such as a PCIe bus, for example.

[0054] In an implementation, the storage array controller 101 may include a host bus adapter 114 coupled to an expander 115. The expander 115 may be used to attach more storage drives to the host system. The expander 115 may be, for example, a SAS expander for enabling the host bus adapter 114 to attach to storage drives in an implementation where the host bus adapter 114 is embodied as a SAS controller.

[0055] In implementation, the storage array controller 101 may include a switch 116 coupled to the processing device 104 via the data communication link 109. The switch 116 may be a computer hardware device that can create multiple endpoints from one endpoint, thereby enabling multiple devices to share one endpoint. The switch 116 may be, for example, a PCIe switch that is coupled to a PCIe bus (e.g., the data communication link 109) and presents multiple PCIe connection points to the midplane.

[0056] In an implementation, the storage array controller 101 includes a data communication link 107 for coupling the storage array controller 101 to other storage array controllers. In some examples, the data communication link 107 may be a Quick Path Interconnect (QPI) interconnect.

[0057] A conventional storage system using a conventional flash drive may implement processing across a flash drive that is part of the conventional storage system. For example, a high-level process of the storage system may initiate and control processing across the flash drive. However, the flash drive of the conventional storage system may include its own storage controller that also performs the processing. Thus, for a conventional storage system, both high-level processing (e.g., initiated by the storage system) and low-level processing (e.g., initiated by the storage controller of the storage system) may be performed.

[0058] To address various deficiencies of conventional storage systems, operations may be performed through high-level processing rather than low-level processing. For example, a flash storage system may include a flash drive that does not include a storage controller for providing processing. Thus, the operating system of the flash storage system itself may initiate and control processing. This may be achieved through a direct-mapped flash storage system that directly addresses data blocks within the flash drive without requiring address translation by the flash drive's storage controller.

[0059] The operating system of the flash storage system can identify and maintain a list of allocation units across multiple flash drives of the flash storage system. An allocation unit can be a whole erase block or multiple erase blocks. The operating system can maintain a mapping or address range for directly mapping addresses to erase blocks of the flash drives of the flash storage system.

[0060] The erase block directly mapped to the flash drive can be used to rewrite data and erase data. For example, one or more allocation units including first data and second data can be operated, wherein the first data will be maintained and the second data will no longer be used by the flash storage system. The operating system can initiate a process to write the first data to a new location in other allocation units, erase the second data, and mark the allocation unit as available for subsequent data. Therefore, the process can be performed only by the high-level operating system of the flash storage system without the need for additional low-level processing by the controller of the flash drive.

[0061] Advantages of this processing being performed only by the operating system of the flash storage system include improved reliability of the flash drives of the flash storage system, since no unnecessary or redundant write operations are performed during this processing. A possible novelty here is the concept of initiating and controlling this processing at the operating system of the flash storage system. In addition, this processing can be controlled by the operating system across multiple flash drives. This is in contrast to processing being performed by the storage controller of the flash drive.

[0062] The storage system may include two storage array controllers that share a set of drives for failover purposes, or the storage system may include a single storage array controller for providing storage services that utilize multiple drives, or the storage system may include a distributed network of storage array controllers that each have a certain number of drives or a certain amount of flash storage, where the storage array controllers in the network collaborate to provide a complete storage service and collaborate on various aspects of the storage service including storage allocation and garbage collection.

[0063] Figure 1C A third example system 117 of data storage according to some implementations is illustrated. The system 117 (also referred to herein as a "storage system") includes a number of elements for purposes of illustration and not limitation. It may be noted that the system 117 may include the same, more, or fewer elements configured in the same or different manner in other implementations.

[0064] In one embodiment, system 117 includes dual peripheral component interconnect (PCI) flash memory devices 118 with individually addressable fast write storage. System 117 may include storage controller 119. In one embodiment, storage controllers 119A-D may be a CPU, ASIC, FPGA, or any other circuit that can implement the control structure required according to the present invention. In one embodiment, system 117 includes flash memory devices (e.g., including flash memory devices 120a-n) operably coupled to various channels of storage device controller 119. Flash memory devices 120a-n may be presented to controller 119A-D as an addressable collection of flash pages, erase blocks, and / or control elements sufficient to allow storage device controller 119A-D to program and retrieve various aspects of the flash memory. In one embodiment, the storage device controllers 119A-D can operate the flash memory devices 120a-n, including storing and retrieving the data contents of a page, arranging and erasing any block, tracking statistics related to the use and reuse of flash memory pages, erase blocks and cells, tracking and predicting error codes and failures within the flash memory, controlling voltage levels associated with programming and retrieving the contents of flash memory cells, and the like.

[0065] In one embodiment, the system 117 may include a RAM 121 for storing individually addressable fast write data. In one embodiment, the RAM 121 may be one or more separate discrete devices. In another embodiment, the RAM 121 may be integrated into the storage device controller 119A-D or multiple storage device controllers. The RAM 121 may also be used for other purposes, such as temporary program memory for a processing device (e.g., a CPU) in the storage device controller 119.

[0066] In one embodiment, the system 117 may include a storage energy device 122, such as a rechargeable battery or capacitor. The storage energy device 122 may store enough energy to power the storage device controller 119, an amount of RAM (e.g., RAM 121), and an amount of flash memory (e.g., flash memory 120a-120n) for a time sufficient to write the contents of the RAM to the flash memory. In one embodiment, if the storage device controller 119A-D detects a loss of external power, the storage device controller may write the contents of the RAM to the flash memory.

[0067] In one embodiment, the system 117 includes two data communication links 123a, 123b. In one embodiment, the data communication links 123a, 123b can be PCI interfaces. In another embodiment, the data communication links 123a, 123b can be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). The data communication links 123a, 123b can be based on the Non-Volatile Memory Express (NVMe) or the NVMe across fabric (NVMf) specification, which allows external connections from other components in the storage system 117 to the storage device controllers 119A-D. It should be noted that for convenience, the data communication links can be interchangeably referred to as PCI buses here.

[0068] The system 117 may also include an external power source (not shown) which may be provided on one or both of the data communication links 123a, 123b, or may be provided separately. An alternative embodiment includes a separate flash memory (not shown) dedicated to storing the contents of the RAM 121. The storage device controllers 119A-D may present a logical device on the PCI bus which may include an addressable fast write logical device, or may be presented as different portions of the logical address space of the storage device 118 as PCI memory or persistent storage. In one embodiment, operations to store into the device are intended to store into the RAM 121. In the event of a power failure, the storage device controllers 119A-D may write the storage contents associated with the addressable fast write logical storage to a flash memory (e.g., flash memory 120a-n) for long-term persistent storage.

[0069] In one embodiment, the logic device may include some representation of some or all of the contents of flash memory devices 120a-n, wherein the representation allows a storage system (e.g., storage system 117) including storage device 118 to directly address flash memory pages and directly reprogram erase blocks from storage system components external to the storage device via a PCI bus. The representation may also allow one or more external components to control and retrieve other aspects of the flash memory, including some or all of the following: tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells across all flash memory devices; tracking and predicting error codes and failures within and across flash memory devices; controlling voltage levels associated with programming and retrieving the contents of flash memory cells; and the like.

[0070] In one embodiment, the storage energy device 122 may be sufficient to ensure completion of ongoing operations for the flash memory devices 120a-120n, and the storage energy device 122 may power the storage device controllers 119A-D and the associated flash memory devices (e.g., 120a-n) for these operations and for storing fast write RAM to flash memory. The storage energy device 122 may be used to store cumulative statistics and other parameters maintained and tracked by the flash memory devices 120a-n and / or the storage device controller 119. A separate capacitor or storage energy device (such as a smaller capacitor embedded near or within the flash memory device itself) may be used for some or all of the operations described herein.

[0071] Various schemes may be used to track and optimize the life of the stored energy components, such as adjusting voltage levels over time, partially discharging the stored energy device 122 to measure corresponding discharge characteristics, etc. If the available energy decreases over time, the effectively available capacity of the addressable fast write storage may decrease to ensure that it can be safely written based on the currently available storage energy.

[0072] Figure 1D A third example system 124 is illustrated for data storage according to some implementations. In one embodiment, the system 124 includes storage controllers 125a, 125b. In one embodiment, the storage controllers 125a, 125b are operably coupled to dual PCI storage devices 119a, 119b and 119c, 119d, respectively. The storage controllers 125a, 125b may be operably coupled (e.g., via a storage network 130) to a number of host computers 127a-n.

[0073] In one embodiment, two storage controllers (e.g., 125a and 125b) provide storage services, such as SCS block storage arrays, file servers, object servers, databases, or data analysis services. The storage controllers 125a, 125b may provide services to host computers 127a-n external to the storage system 124 through a number of network interfaces (e.g., 126a-d). The storage controllers 125a, 125b may provide integrated services or applications entirely within the storage system 124, thereby forming a converged storage and computing system. The storage controllers 125a, 125b may utilize fast write memory within or across storage devices 119a-d to log ongoing operations to ensure that operations are not lost in the event of a power failure, storage controller removal, storage controller or storage system shutdown, or some failure of one or more software or hardware components within the storage system 124.

[0074] In one embodiment, the controllers 125a, 125b operate as PCI masters for one or the other PCI bus 128a, 128b. In another embodiment, 128a and 128b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). Other storage system embodiments may enable the storage controllers 125a, 125b to operate as multi-masters for both PCI buses 128a, 128b. Optionally, a PCI / NVMe / NVMf switching infrastructure or structure may connect multiple storage controllers. Some storage system embodiments may allow storage devices to communicate directly with each other, rather than just with storage controllers. In one embodiment, the storage device controller 119a may operate under the direction of the storage controller 125a to perform operations based on data stored in RAM (e.g., Figure 1C The recalculated version of the RAM contents may be transmitted after the storage controller has determined that the operation has been fully committed across the storage system, or when the fast write memory on the device has reached a specific usage capacity, or after a specific amount of time to ensure improved data security or free up addressable fast write capacity for reuse. This mechanism may be used, for example, to avoid a second transmission from the storage controller 125a, 125b via a bus (e.g., 128a, 128b). In one embodiment, the recalculation may include compressing the data, appending indexes or other metadata, combining multiple data segments together, performing erasure code calculations, and the like.

[0075] In one embodiment, under the direction of the storage controller 125a, 125b, the storage device controller 119a, 119b may be operable to perform operations based on the RAM (e.g., Figure 1CThe memory device 125a can be used to store data in the memory 125b and store it in the memory 125b. The memory device 125a can be used to store data in the memory 125b. The memory device 125b can be used to store data in the memory 125a. The memory device 125b ...

[0076] The storage device controllers 119A-D may include mechanisms for implementing high availability primitives for use by other parts of the storage system external to the dual PCI storage device 118. For example, a reservation or exclusion primitive may be provided so that in a storage system having two storage controllers for providing high availability storage services, one storage controller may prevent another storage controller from accessing or continuing to access the storage device. This mechanism may be used, for example, in the event that one controller detects that the other controller is not functioning properly or the interconnect between the two storage controllers itself may be functioning improperly.

[0077] In one embodiment, a storage system for use with dual PCI direct mapped storage devices with individually addressable fast write storage includes a system that manages erase blocks or groups of erase blocks as allocation units for storing data representing a storage service, or for storing metadata associated with the storage service (e.g., indexes, logs, etc.), or for properly managing the storage system itself. Flash pages, which may be several kilobytes in size, may be written as data arrives or when the storage system is to retain data for a long time interval (e.g., above a defined time threshold). To commit data more quickly, or to reduce the number of writes to the flash memory device, the storage controller may first write the data to individually addressable fast write storage on one or more storage devices.

[0078] In one embodiment, the storage controllers 125a, 125b may initiate the use of erase blocks within and across storage devices (e.g., 118) based on the age and expected remaining life of the storage devices or based on other statistics. The storage controllers 125a, 125b may initiate garbage collection and data migration between storage devices based on pages that are no longer needed, and manage flash page and erase block lifespans as well as manage overall system performance.

[0079] In one embodiment, the storage system 124 may utilize mirroring and / or erasure coding schemes as part of storing data to addressable fast write storage and / or as part of writing data to allocation units associated with erase blocks. Erasure codes may be used across storage devices, and within erase blocks or allocation units, or within and across flash memory devices on a single storage device to provide redundancy against single or multiple storage device failures or to protect against internal corruption of flash memory pages caused by flash memory operations or degradation of flash memory cells. Different levels of mirroring and erasure coding may be used to recover from multiple types of failures occurring alone or in combination.

[0080] refer to Figure 2A The embodiment depicted in ~G illustrates a storage cluster that stores user data, such as user data originating from one or more users or client systems or other sources outside the storage cluster. The storage cluster distributes user data across storage nodes housed in a chassis or across multiple chassis using redundant copies of erasure coding and metadata. Erasure coding refers to a method of data protection or reconstruction in which data is stored across a set of different locations such as disks, storage nodes, or geographic locations. Flash memory is a type of solid-state memory that can be integrated with the embodiment, but the embodiment can be extended to other types of solid-state memory or other storage media including non-solid-state memory. The control of storage locations and workloads are distributed across storage locations in a cluster peer system. Tasks such as intermediary communication between different storage nodes, detecting when a storage node becomes unavailable, and balancing I / O (input and output) across different storage nodes are all handled on a distributed basis. In some embodiments, data is laid out or distributed across multiple storage nodes in data fragments or stripes that support data recovery. The ownership of data can be reassigned within the cluster regardless of the input mode and output mode. This architecture, described in more detail below, allows for storage node failures in a cluster while the system remains operational because data can be rebuilt from other storage nodes and thus remain available for input and output operations.In various embodiments, storage nodes may be referred to as cluster nodes, blades, or servers.

[0081] The storage cluster may be contained in a chassis (i.e., a box that accommodates one or more storage nodes). A mechanism (such as a power distribution bus, etc.) for supplying power to each storage node and a communication mechanism (such as a communication bus, etc.) that enables communication between storage nodes may be included in the chassis. According to some embodiments, the storage cluster may be operated as an independent system in one location. In one embodiment, the chassis contains at least two instances of both a power distribution bus and a communication bus that can be independently enabled or disabled. The internal communication bus may be an Ethernet bus, however, other technologies such as PCIe and InfiniBand are also suitable. The chassis provides a port for an external communication bus that can communicate directly or through a switch between multiple chassis and with a client system. External communication may use technologies such as Ethernet, InfiniBand, Fibre Channel, etc. In some embodiments, the external communication bus uses different communication bus technologies for inter-chassis and client communications. If a switch is deployed in a chassis or between chassis, the switch may be used as a conversion between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, the storage cluster can be accessed by a client using a proprietary interface or a standard interface, such as a Network File System (NFS), Common Internet File System (CIFS), Small Computer System Interface (SCSI), or Hypertext Transfer Protocol (HTTP). Conversion from the client protocol can occur at a switch, a chassis external communication bus, or within each storage node. In some embodiments, multiple chassis can be coupled or connected to each other through an aggregator switch. Some and / or all of the coupled or connected chassis can be designated as a storage cluster. As discussed above, each chassis can have multiple blades, each blade having a media access control (MAC) address, but in some embodiments, the storage cluster is presented to the external network as having a single cluster IP address and a single MAC address.

[0082] Each storage node may be one or more storage servers, and each storage server is connected to one or more non-volatile solid-state memory units, which may be referred to as storage units or storage devices. One embodiment includes a single storage server in each storage node and between one to eight non-volatile solid-state memory units, but this example is not meant to be limiting. The storage server may include a processor, DRAM, and an interface for an internal communication bus and power distribution for each power bus. In some embodiments, within a storage node, the interface and the storage unit share a communication bus (e.g., PCI Express). The non-volatile solid-state memory unit may directly access the internal communication bus interface through the storage node communication bus, or request the storage node to access the bus interface. The non-volatile solid-state memory unit includes an embedded CPU, a solid-state storage controller, and a certain amount of solid-state mass storage, such as between 2 and 32 terabytes (TB) in some embodiments. An embedded volatile storage medium such as DRAM and an energy storage device are included in the non-volatile solid-state memory unit. In some embodiments, the energy storage device is a capacitor, a supercapacitor, or a battery, which enables a subset of the DRAM content to be transferred to a stable storage medium in the event of power loss. In some embodiments, the non-volatile solid-state memory unit is constructed with storage-class memory, such as phase change or magnetoresistive random access memory (MRAM), which replaces DRAM and enables power retention devices to be reduced. One of the many features of storage nodes and non-volatile solid-state storage is the ability to actively rebuild data in a storage cluster. Storage nodes and non-volatile solid-state storage can determine when a storage node or non-volatile solid-state storage in a storage cluster is unreachable, regardless of whether there is an attempt to read data involving the storage node or non-volatile solid-state storage. The storage node and non-volatile solid-state storage then collaborate to recover and rebuild the data in at least part of the new location. This constitutes an active reconstruction because the system does not need to wait until the data is needed for a read access initiated from a client system that employs the storage cluster to rebuild the data. These and further details of the storage memory and its operation are discussed below.

[0083] Figure 2A1 is a perspective view of a storage cluster 161 having multiple storage nodes 150 and internal solid-state storage coupled to each storage node to provide a network attached storage or storage area network according to some embodiments. The network attached storage, storage area network or storage cluster or other storage storage may include one or more storage clusters 161, each storage cluster 161 having one or more storage nodes 150, which are in a flexible and reconfigurable arrangement of both physical components and the amount of storage storage provided thereby. The storage cluster 161 is designed to be assembled in a rack, and one or more racks can be set up and filled according to the needs of the storage storage. The storage cluster 161 has a chassis 138, which has a plurality of slots 142. It should be understood that the chassis 138 may be referred to as a housing, a box or a rack unit. In one embodiment, the chassis 138 has fourteen slots 142, but other numbers of slots are easily conceivable. For example, some embodiments have four slots, eight slots, sixteen slots, thirty-two slots, or other suitable numbers of slots. In some embodiments, each slot 142 can accommodate a storage node 150. The chassis 138 includes flaps 148 that can be used to mount the chassis 138 on a rack. The fan 144 provides air circulation for cooling the storage node 150 and its components, but other cooling components can be used, or embodiments without cooling components can be envisioned. The switch structure 146 couples the storage nodes 150 in the chassis 138 together and couples to the network for communication with the storage. In the embodiment depicted here, for illustrative purposes, the slot 142 on the left side of the switch structure 146 and the fan 144 is shown as being occupied by the storage node 150, while the slot 142 on the right side of the switch structure 146 and the fan 144 is empty and can be used to insert the storage node 150. This configuration is an example, and one or more storage nodes 150 can occupy the slot 142 in various further arrangements. In some embodiments, the storage node arrangement does not need to be continuous or adjacent. The storage node 150 is hot-swappable, which means that the storage node 150 can be inserted into or removed from the slot 142 in the chassis 138 without stopping or shutting down the system. When storage nodes 150 are inserted or removed relative to slots 142, the system automatically reconfigures to recognize and adapt to the changes. In some embodiments, reconfiguration includes restoring redundancy and / or rebalancing data or load.

[0084] Each storage node 150 may have multiple components. In the embodiment shown here, the storage node 150 includes a printed circuit board 159 populated by a CPU 156, i.e., a processor, a memory 154 coupled to the CPU 156, and a non-volatile solid-state storage 152 coupled to the CPU 156, but other mountings and / or components may be used in further embodiments. The memory 154 has instructions executed by the CPU 156 and / or data operated by the CPU 156. As further described below, the non-volatile solid-state storage 152 includes flash memory, or in further embodiments includes other types of solid-state memory.

[0085] refer to Figure 2A As described above, the storage cluster 161 is scalable, which means that it is easy to add storage capacity with non-uniform storage sizes. In some embodiments, one or more storage nodes 150 can be inserted into or removed from each chassis and the storage cluster configures itself. Plug-in storage nodes 150, whether installed in the chassis at delivery or added later, can have different sizes. For example, in one embodiment, the storage node 150 can have any multiple of 4TB, such as 8TB, 12TB, 16TB, 32TB, etc. In further embodiments, the storage node 150 can have any multiple of other storage amounts or capacities. The storage capacity of each storage node 150 is broadcast and affects the decision on how to stripe data. For maximum storage efficiency, subject to the predetermined requirement of continuous operation in the event of the loss of up to one or up to two non-volatile solid-state storage units 152 or storage nodes 150 within the chassis, the embodiment can configure itself as widely as possible in the stripes.

[0086] Figure 2B 1 is a block diagram showing a communication interconnect 173 and a power distribution bus 172 coupling a plurality of storage nodes 150. Figure 2A In some embodiments, the communication interconnect 173 may be included in or implemented with the switch fabric 146. In some embodiments, where multiple storage clusters 161 occupy a rack, the communication interconnect 173 may be included in or implemented with the top of the rack switch. Figure 2B As illustrated, storage cluster 161 is enclosed within a single chassis 138. External port 176 is coupled to storage node 150 via communication interconnect 173, while external port 174 is coupled directly to storage node. External power port 178 is coupled to power distribution bus 172. Storage node 150 may include a storage device as described in reference Figure 2A The non-volatile solid-state storage 152 may have different capacities. In addition, one or more storage nodes 150 may be such as Figure 2BOnly the storage nodes are illustrated. Permissions 168 are implemented on the non-volatile solid-state storage 152, for example as a list or other data structure stored in memory. In some embodiments, permissions are stored in the non-volatile solid-state storage 152 and supported by software executed on a controller or other processor of the non-volatile solid-state storage 152. In a further embodiment, permissions 168 are implemented on the storage node 150, for example as a list or other data structure stored in memory 154 and supported by software executed on the CPU 156 of the storage node 150. In some embodiments, permissions 168 control how data is stored in the non-volatile solid-state storage 152 and where it is stored in the non-volatile solid-state storage 152. This control helps determine which type of erasure coding scheme is applied to the data and which storage nodes 150 have which parts of the data. Each permission 168 can be assigned to the non-volatile solid-state storage 152. In various embodiments, permissions may control the range of inode numbers, segment numbers, or other data identifiers assigned to data by the file system, by the storage node 150, or by the non-volatile solid-state storage 152.

[0087] In some embodiments, each piece of data and each piece of metadata has redundancy in the system. In addition, each piece of data and each piece of metadata has an owner, which can be referred to as authority. If the authority is not accessible, for example, due to a failure of a storage node, there is an inheritance plan for how to find the data or the metadata. In various embodiments, there are redundant copies of authority 168. In some embodiments, authority 168 has a relationship with a storage node 150 and a non-volatile solid-state storage 152. Each authority 168 covering the range of data segment numbers or other identifiers of data can be assigned to a specific non-volatile solid-state storage 152. In some embodiments, the authority 168 for all such ranges is distributed on the non-volatile solid-state storage 152 of the storage cluster. Each storage node 150 has a network port for providing access to (one or more) non-volatile solid-state storage 152 of the storage node 150. In some embodiments, data can be stored in a segment associated with a segment number, and the segment number is a kind of indirection for the configuration of a RAID (Redundant Array of Independent Disks) stripe. Therefore, the assignment and use of authority 168 establishes indirection to data. According to some embodiments, indirectness may be referred to as the ability to indirectly reference data via permissions 168 in this case. A segment identifies a set of nonvolatile solid-state storage 152 that may contain data and a local identifier in the set of nonvolatile solid-state storage 152. In some embodiments, the local identifier is an offset in the device and may be reused sequentially by multiple segments. In other embodiments, the local identifier is unique to a particular segment and is never reused. The offset in the nonvolatile solid-state storage 152 is applied to locate data to be written to or read from the nonvolatile solid-state storage 152 (in the form of a RAID stripe). Data is striped across multiple units of nonvolatile solid-state storage 152, which may include or be different from nonvolatile solid-state storage 152 having permissions 168 for a particular data segment.

[0088] In the case where there is a change in where a specific data segment is located, such as during data movement or data reconstruction, the permissions 168 for the data segment should be consulted at the non-volatile solid-state storage 152 or storage node 150 with the permissions 168. In order to locate specific data, the embodiment calculates the hash value of the data segment or applies the index node number or data segment number. The output of this operation points to the non-volatile solid-state storage 152 with the permissions 168 for the specific data. In some embodiments, there are two levels for performing this operation. The first level maps an entity identifier (ID) such as a segment number, an index node number, or a directory number to a permission identifier. The mapping may include calculations such as hashes or bit masks. The second level is to map the permission identifier to a specific non-volatile solid-state storage 152, which can be done by explicit mapping. The operation is repeatable so that when the calculation is performed, the result of the calculation can be repeatedly and reliably pointed to a specific non-volatile solid-state storage 152 with the permissions 168. The operation may include a set of reachable storage nodes as input. If the non-volatile solid-state storage units reachable by the group change, the best group changes. In some embodiments, the value that is continuously maintained is the current assignment (which is always true), and the calculated value is the target assignment that the cluster will attempt to reconfigure. This calculation can be used to determine the best non-volatile solid-state storage 152 for permissions when there is a group of non-volatile solid-state storage 152 that is reachable and constitutes the same cluster. The calculation also determines a set of ordered peer non-volatile solid-state storage 152, which will also record permissions mapped to the non-volatile solid-state storage so that permissions can be determined even if the assigned non-volatile solid-state storage is unreachable. In some embodiments, if a specific permission 168 is not available, a copy or alternative permission 168 can be consulted.

[0089] refer to Figure 2A and 2B, two of the many tasks of the CPU 156 on the storage node 150 are to decompose the write data and reorganize the read data. In the case where the system has determined that the data is to be written, the permission 168 for the data is located as above. In the case where the segment ID of the data has been determined, the write request is forwarded to the non-volatile solid-state storage 152 of the host currently determined as the permission 168 determined from the segment. Then, the host CPU 156 of the storage node 150 resident with the non-volatile solid-state storage 152 and the corresponding permission 168 decomposes or slices the data and sends the data to various non-volatile solid-state storages 152. The data sent is written as a data strip according to the erasure coding scheme. In some embodiments, the request pulls the data, and in other embodiments, the data is pushed. In contrast, when reading data, the permission 168 containing the segment ID of the data is located as above. The host CPU 156 of the storage node 150 resident with the non-volatile solid-state storage 152 and the corresponding permission 168 requests data from the non-volatile solid-state storage and the corresponding storage node pointed to by the permission. In some embodiments, the data is read from flash storage as a stripe of data. The host CPU 156 of the storage node 150 then reassembles the read data, corrects any errors (if any) according to an appropriate erasure coding scheme, and forwards the reassembled data to the network. In further embodiments, some or all of these tasks may be handled in the non-volatile solid-state storage 152. In some embodiments, the segment host requests that data be sent to the storage node 150 by requesting the page from storage and then sending the data to the storage node that made the original request.

[0090] In some systems, such as in UNIX-type file systems, data is processed using index nodes or index nodes, where the index nodes or index nodes specify data structures representing objects in the file system. The object may be, for example, a file or a directory. Metadata may accompany the object as attributes such as permission data and creation timestamps, among other attributes. Segment numbers may be assigned to all or part of such objects in the file system. In other systems, data segments are processed using segment numbers assigned elsewhere. For the purposes of discussion, the distribution unit is an entity, and an entity may be a file, a directory, or a segment. That is, an entity is a unit of data or metadata stored by a storage system. Entities are grouped into groups called permissions. Each permission has a permission owner, which is a storage node with the exclusive right to update the entity in the permission. In other words, a storage node contains permissions, and the permission in turn contains entities.

[0091] According to some embodiments, a segment is a logical container for data. A segment is an address space between the media address space and the physical flash location, i.e., the data segment number is in this address space. A segment may also contain metadata that enables data redundancy to be restored (rewritten to a different flash location or device) without involving high-level software. In one embodiment, the internal format of a segment contains client data and a media map to determine the location of the data. Where applicable, each data segment is protected by breaking the segment into multiple data and parity shards, for example, to prevent memory failures and other failures. The data and parity shards are coupled to the host CPU 156 (see ) across the data and parity shards according to an erasure coding scheme. Figure 2E and 2G ) is distributed, i.e., striped, on the non-volatile solid-state storage 152. In some embodiments, the use of the term segment refers to a container and its location in the address space of the segment. According to some embodiments, the use of the term stripe refers to a group of shards that are the same as a segment, and includes how the shards are distributed along with redundancy or parity information.

[0092] A series of address space transformations occur across the entire storage system. At the top are directory entries (file names) linked to index nodes. Index nodes point to the media address space where data is logically stored. Media addresses can be mapped through a series of indirect media to disperse the load of large files, or to implement data services such as deduplication or snapshots. Media addresses can be mapped through a series of indirect media to disperse the load of large files, or to implement data services such as deduplication or snapshots. Then, the segment address is converted to a physical flash location. According to some embodiments, the physical flash location has an address range defined by the amount of flash in the system. Media addresses and segment addresses are logical containers, and in some embodiments, 128-bit or larger identifiers are used so that they are actually infinite, where the possibility of reuse is calculated to be longer than the expected life of the system. In some embodiments, addresses from logical containers are allocated in a hierarchical manner. Initially, each non-volatile solid-state storage unit 152 can be assigned a range of address space. Within the range of this assignment, non-volatile solid-state storage 152 can allocate addresses without synchronizing with other non-volatile solid-state storage 152.

[0093] Data and metadata are stored by a set of basic storage layouts optimized for different workload modes and storage devices. These layouts include multiple redundancy schemes, compression formats, and indexing algorithms. Some of these layouts store information related to permissions and permission masters, while other layouts store file metadata and file data. The redundancy scheme includes error correction codes that allow damaged bits in a single storage device (such as a NAND flash chip, etc.), erasure codes that allow failures of multiple storage nodes, and replication schemes that allow data center or regional failures. In some embodiments, low-density parity check (LDPC) codes are used in a single storage unit. In some embodiments, Reed-Solomon encoding is used in a storage cluster, and mirroring is used in a storage grid. Metadata can be stored using an ordered log-structured index (such as a log-structured merge tree, etc.), while big data cannot be stored in a log-structured layout.

[0094] In order to maintain consistency across multiple copies of an entity, storage nodes implicitly agree on two things through computation: (1) the permissions that contain the entity, and (2) the storage nodes that contain the permissions. The assignment of entities to permissions can be accomplished by pseudo-randomly assigning entities to permissions, by dividing entities into ranges based on externally generated keys, or by placing a single entity into each permission. Examples of pseudo-random schemes are linear hashing and the scalable hashing under replication (RUSH) hash family, the latter including controlled scalable hashing under replication (CRUSH). In some embodiments, pseudo-random assignment is only used to assign permissions to nodes, because the group of nodes can be changed. The group of permissions cannot be changed, so any subjective function can be applied in these embodiments. Some placement schemes automatically place permissions on storage nodes, while other placement schemes rely on explicit mapping of permissions to storage nodes. In some embodiments, a pseudo-random scheme is used to map from each permission to a set of candidate permission owners. A pseudo-random data distribution function associated with CRUSH can assign permissions to storage nodes and create a list of assigned permissions. Each storage node has a copy of the pseudo-random data distribution function and can obtain the same calculation for distributing and subsequently finding or locating the authority. In some embodiments, the pseudo-random schemes each require a reachable set of storage nodes as input to infer the same target node. Once an entity has been placed in a authority, the entity can be stored on a physical device so that expected failures will not cause unexpected data loss. In some embodiments, the rebalancing algorithm attempts to store copies of all entities within a authority in the same layout and on the same set of machines.

[0095] Examples of expected failures include equipment failures, stolen machines, data center fires, and regional disasters such as nuclear or geological events. Different failures result in different levels of acceptable data loss. In some embodiments, a stolen storage node affects neither the security nor the reliability of the system, while a regional event may result in no data loss, a few seconds or minutes of lost updates, or even complete data loss, depending on the system configuration.

[0096] In an embodiment, the placement of data for storing redundancy is independent of the placement of permissions for data consistency. In some embodiments, the storage node containing permissions does not contain any persistent storage. As an alternative, the storage node is connected to a non-volatile solid-state storage unit that does not contain permissions. The communication interconnection between the storage node and the non-volatile solid-state storage unit includes a variety of communication technologies and has non-uniform performance and fault tolerance characteristics. In some embodiments, as described above, the non-volatile solid-state storage unit is connected to the storage node via PCI Express, the storage node is connected together in a single chassis using an Ethernet backplane, and the chassis are connected together to form a storage cluster. In some embodiments, the storage cluster is connected to the client using Ethernet or Fibre Channel. If multiple storage clusters are configured into a storage grid, the Internet or other long-distance network links (such as "metropolitan scale" links or private links that do not pass through the Internet) are used to connect multiple storage clusters.

[0097] The rights owner has the exclusive right to modify the entity, migrate the entity from one non-volatile solid-state storage unit to another non-volatile solid-state storage unit, and add and remove copies of the entity. This allows the redundancy of the underlying data to be maintained. When the rights owner fails, is about to be deactivated or overloaded, the rights are transferred to the new storage node. Transient failures make it meaningful to ensure that all non-faulty machines agree on the new rights position. The uncertainty caused by transient failures can be automatically achieved by consensus protocols such as Paxos, hot-warm failover schemes, etc., via manual intervention by remote system administrators or local hardware administrators (such as by physically removing the failed machine from the cluster or pressing a button on the failed machine, etc.). In some embodiments, a consensus protocol is used, and failover is automatic. According to some embodiments, if too many failures or replication events occur in too short a time period, the system enters a self-protection mode and stops replication and data movement activities until the administrator intervenes. As rights are transferred between storage nodes and rights owners update entities in their rights, the system transmits messages between storage nodes and non-volatile solid-state storage units. Regarding persistent messages, messages with different purposes have different types. Depending on the type of message, the system maintains different ordering and persistence guarantees. As persistent messages are being processed, the messages are temporarily stored with multiple persistent and non-persistent storage hardware technologies. In some embodiments, messages are stored in RAM, NVRAM, and on NAND flash devices, and various protocols are used to efficiently use each storage medium. Latency-sensitive client requests can be kept in replicated NVRAM and then kept in NAND, while background rebalancing operations are kept directly to NAND.

[0098] Persistent messages are persistently stored before being sent. This allows the system to continue to serve client requests despite failures and component replacements. While many hardware components contain unique identifiers visible to system administrators, manufacturers, hardware supply chains, and continuous monitoring quality control infrastructure, applications running on top of infrastructure addresses virtualize the addresses. Regardless of component failures and replacements, these virtualized addresses do not change throughout the life cycle of the storage system. This allows components of the storage system to be replaced over time without reconfiguration or interruption of client request processing, i.e., the system supports non-disruptive upgrades.

[0099] In some embodiments, virtualized addresses are stored with sufficient redundancy. The continuous monitoring system associates hardware and software states with hardware identifiers. This allows for detection and prediction of failures due to faulty components and manufacturing details. In some embodiments, the monitoring system also enables proactive transfer of permissions and entities from affected devices before a failure occurs by removing components from critical paths.

[0100] Figure 2C1 is a multi-level block diagram showing the contents of a storage node 150 and the contents of a non-volatile solid-state storage 152 of the storage node 150. In some embodiments, data is communicated with respect to the storage node 150 via a network interface controller (NIC) 202. As discussed above, each storage node 150 has a CPU 156 and one or more non-volatile solid-state storage 152. Figure 2C Moving down one level in FIG. 1 , each nonvolatile solid-state storage 152 has relatively fast nonvolatile solid-state memory, such as nonvolatile random access memory (NVRAM) 204 and flash memory 206. In some embodiments, NVRAM 204 can be a component that does not require program / erase cycles (DRAM, MRAM, PCM) and can be a memory that can support being written to much more frequently than being read from the memory. Figure 2CMoving down another level in the NVRAM 204, in one embodiment, NVRAM 204 is implemented as a high-speed volatile memory such as dynamic random access memory (DRAM) 216 that is backed up by an energy reserve 218. The energy reserve 218 provides sufficient power to keep the DRAM 216 powered long enough for the contents to be transferred to the flash memory 206 in the event of a power failure. In some embodiments, the energy reserve 218 is a capacitor, supercapacitor, battery, or other device that supplies a suitable energy supply sufficient to enable the contents of the DRAM 216 to be transferred to a stable storage medium in the event of a power loss. The flash memory 206 is implemented as a plurality of flash wafers 222, which may be referred to as a package of flash wafers 222 or an array of flash wafers 222. It should be understood that the flash wafers 222 may be packaged in any of a number of ways: one wafer per package, multiple wafers per package (i.e., a multi-chip package), in a hybrid package, as a bare wafer on a printed circuit board or other substrate, as a packaged wafer, etc. In the illustrated embodiment, the non-volatile solid-state storage 152 has a controller 212 or other processor and an input-output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or the network interface controller 202 of the flash storage node 150. A flash input-output (I / O) port 220 is coupled to a flash die 222, and a direct memory access unit (DMA) 214 is coupled to the controller 212, the DRAM 216, and the flash die 222. In the illustrated embodiment, the I / O port 210, the controller 212, the DMA unit 214, and the flash I / O port 220 are implemented on a programmable logic device (PLD) 208, such as a field programmable gate array (FPGA). In this embodiment, each flash die 222 has pages organized into sixteen kB (kilobyte) pages 224, and registers 226 capable of writing or reading data with respect to the flash die 222. In further embodiments, other types of solid-state memory are used instead of or in addition to the flash memory illustrated within flash wafer 222 .

[0101] In various embodiments disclosed herein, a storage cluster 161 can be generally contrasted with a storage array. A storage node 150 is part of a collection that creates a storage cluster 161. Each storage node 150 has the data slices and calculations required to provide data. Multiple storage nodes 150 collaborate to store and retrieve data. Storage memories or storage devices as generally used in storage arrays are less involved in the processing and manipulation of data. Storage memories or storage devices in a storage array receive commands for reading, writing or erasing data. Storage memories or storage devices in a storage array do not know the larger system in which it is embedded, or what the data means. Storage memories or storage devices in a storage array may include various types of storage memories, such as RAM, solid-state drives, hard disk drives, etc. The storage unit 152 described here has multiple interfaces that are active at the same time and serve multiple purposes. In some embodiments, some functions of a storage node 150 are transferred to a storage unit 152, thereby converting the storage unit 152 into a combination of a storage unit 152 and a storage node 150. Placing calculations (relative to stored data) in a storage unit 152 brings such calculations closer to the data itself. Various system embodiments include a hierarchical structure of storage node layers with different capabilities. In contrast, in a storage array, a controller owns and knows everything about all the data that the controller manages in a shelf or storage device. In a storage cluster 161, as described herein, multiple controllers in multiple storage units 152 and / or storage nodes 150 collaborate in various ways (e.g., for erasure coding, data sharding, metadata communication and redundancy, storage capacity expansion or contraction, and data recovery, etc.).

[0102] Figure 2D A storage server environment is shown, which uses Figure 2A 1 to 3C. In this version, each storage unit 152 is located in a chassis 138 (see Figure 2A ) has a controller 212 (see Figure 2C ), a processor such as an FPGA (field programmable gate array), a flash memory 206, and an NVRAM 204 (i.e., a DRAM 216 supported by a supercapacitor, see Figure 2B and 2C ). Storage unit 152 may be implemented as a single board containing storage and may be the largest tolerable failure domain within the chassis. In some embodiments, up to two storage units 152 may fail and the device will continue without data loss.

[0103] In some embodiments, physical storage is divided into named regions based on application usage. NVRAM 204 is a continuous reserved memory block in DRAM 216 of storage unit 152 and is backed by NAND flash memory. NVRAM 204 is logically divided into multiple memory regions, two of which are written as spools (e.g., spool_region). The space within the NVRAM 204 spool is managed independently by each authority 168. Each device provides a certain amount of storage space to each authority 168. The authority 168 further manages the life cycle and allocation within the space. Examples of spools include distributed transactions or concepts. When the main power supply of storage unit 152 fails, the on-board supercapacitor provides a short duration of power retention. During this retention interval, the contents of NVRAM 204 are refreshed to flash memory 206. When the power is turned on next time, the contents of NVRAM 204 are restored from flash memory 206.

[0104] With respect to the storage unit controller, the responsibilities of the logical "controller" are distributed across the blades containing the permissions 168. This distribution of logical control is Figure 2D 1 and 2. The control plane and storage plane management are handled independently, but the components can be physically co-located on the same blade. Each authority 168 effectively acts as an independent controller. Each authority 168 provides its own data and metadata structures, its own background workers, and maintains its own life cycle.

[0105] Figure 2E is a hardware block diagram of blade 252, which shows Figure 2D Used in a storage server environment Figure 2A 150 and storage units 152 of the embodiment of the present invention. The control plane 254, the computing and storage planes 256, 258, and the authority 168 that interacts with the underlying physical resources. The control plane 254 is partitioned into multiple authorities 168, which can use the computing resources in the computing plane 256 to run on any blade 252. The storage plane 258 is partitioned into a set of devices, each of which provides access to the flash memory 206 and NVRAM 204 resources. In one embodiment, as described herein, the computing plane 256 can perform storage array controller operations on one or more devices of the storage plane 258 (e.g., storage array).

[0106] exist Figure 2EIn the compute plane 256 and storage plane 258 of the system, permissions 168 interact with underlying physical resources (i.e., devices). From the perspective of the permissions 168, their resources are striped across all physical devices. From the perspective of the device, it provides resources to all permissions 168, regardless of where the permissions happen to be running. Each permission 168 has been assigned or has been assigned one or more partitions 260 of storage memory in the storage unit 152, such as partitions 260 in the flash memory 206 and NVRAM 204. Each permission 168 uses these allocated partitions 260 belonging to it to write or read user data. Permissions can be associated with different amounts of physical storage of the system. For example, one permission 168 can have a greater number of partitions 260 or larger size partitions 260 in one or more storage units 152 than one or more other permissions 168.

[0107] Figure 2F FIG. 2 depicts the elasticity software layer in the blade 252 of the storage cluster according to some embodiments. In the elastic architecture, the elasticity software is symmetrical, that is, the computing module 270 of each blade runs Figure 2F The three identical processing layers depicted. The storage manager 274 executes read and write requests from other blades 252 for data and metadata stored in the local storage unit 152 NVRAM 204 and flash memory 206. The authority 168 completes the client request by issuing the necessary reads and writes to the blade 252 where the corresponding data or metadata resides on the storage unit 152. The endpoint 272 parses the client connection request received from the switch structure 146 monitoring software, forwards the client connection request to the authority 168 responsible for completion, and forwards the response of the authority 168 to the client. The symmetrical three-tier structure enables the storage system to have a high degree of concurrency. In these embodiments, elasticity scales out efficiently and reliably. In addition, elasticity implements a unique scale-out technology that enables work to be evenly balanced across all resources regardless of the client access pattern, and maximizes concurrency by eliminating the need for a large amount of inter-blade coordination that typically occurs with traditional distributed locking.

[0108] Still reference Figure 2F, the authorities 168 running in the compute module 270 of the blade 252 perform the internal operations required to complete the client request. One feature of resiliency is that the authorities 168 are stateless, that is, they cache active data and metadata in the DRAM of their own blade 252 for fast access, but the authority stores each update in its NVRAM 204 partition on three separate blades 252 until the update is written to the flash memory 206. In some embodiments, all storage system writes to NVRAM 204 are made three times to the partitions on the three separate blades 252. With the triply mirrored NVRAM 204 and persistent storage protected by parity and Reed-Solomon RAID checksums, the storage system can survive the concurrent failure of two blades 252 without losing data, metadata, or access to either.

[0109] Because permissions 168 are stateless, they can be migrated between blades 252. Each permission 168 has a unique identifier. In some embodiments, NVRAM 204 and flash memory 206 partitions are associated with the identifier of the permission 168, rather than with the blade 252 on which they run. Therefore, when a permission 168 migrates, the permission 168 continues to manage the same storage partition from its new location. When a new blade 252 is installed in an embodiment of the storage cluster, the system automatically rebalances the load by partitioning the storage of the new blade 252 for use by the system's permissions 168, migrating the selected permissions 168 to the new blade 252, starting endpoints 272 on the new blade 252, and including these endpoints 272 in the client connection distribution algorithm of the switch fabric 146.

[0110] The migrated permissions 168 maintain the contents of their NVRAM 204 partitions on flash 206 from their new locations, process read and write requests from other permissions 168, and complete client requests directed to them by endpoints 272. Similarly, if a blade 252 fails or is removed, the system redistributes its permissions 168 among the remaining blades 252 in the system. The redistributed permissions 168 continue to perform their original functions from their new locations.

[0111] Figure 2GDepicted are authorities 168 and storage resources in blades 252 of a storage cluster according to some embodiments. Each authority 168 is dedicated to a partition of flash memory 206 and NVRAM 204 on each blade 252. An authority 168 manages the content and integrity of its partition independently of other authorities 168. An authority 168 compresses incoming data and temporarily stores it in its NVRAM 204 partition, then merges, RAID protects, and maintains the data in storage segments in its flash memory 206 partition. As authorities 168 write data to flash memory 206, a storage manager 274 performs necessary flash conversions to optimize write performance and maximize media life. In the background, authorities 168 perform "garbage collection" or reclaim space occupied by data that has been retired by clients by rewriting data. It should be understood that because the partitions of authorities 168 are not connected, no distributed locking is required to execute clients and writes or to perform background functions.

[0112] The embodiments described herein can utilize various software, communication and / or network protocols. In addition, the configuration of hardware and / or software can be adjusted to adapt to various protocols. For example, the present embodiment can utilize active directory (ActiveDirectory), which is a database-based system for providing authentication, directory, strategy and other services in a WINDOWSTM environment. In these embodiments, LDAP (Lightweight Directory Access Protocol) is an example application protocol for querying and modifying items in directory service providers such as active directory. In some embodiments, a network lock manager (NLM) is used as a facility for providing consultation files of system V style and record locking through a network in collaboration with a network file system (NFS). The server message block (SMB) protocol (a version of which is also referred to as a common Internet file system (CIFS)) can be integrated with the storage system discussed here. SMP is operated as an application layer network protocol that is generally used to provide shared access to files, printers and serial ports and various communications between nodes on a network. SMB also provides an authenticated inter-process communication mechanism. AMAZONTM S3 (Simple Storage Service) is a web service provided by Amazon Web Service, and the system described herein can be connected to Amazon S3 through web service interfaces (REST (Representational State Transfer), SOAP (Simple Object Access Protocol) and BitTorrent). The RESTful API (Application Programming Interface) decomposes transactions to create a series of small modules. Each module handles a specific basic part of the transaction. The control or permissions provided by these embodiments (especially for object data) can include the use of access control lists (ACLs). ACLs are lists of permissions attached to objects, and ACLs specify which users or system processes are granted access to objects and which operations are allowed for given objects. The system can utilize Internet Protocol Version 6 (IPv6) as well as IPv4 as communication protocols for providing identification and positioning systems for computers on the network and routing services across the Internet. Packet routing between networked systems can include equal cost multi-path routing (ECMP), which is a routing strategy in which the next-hop packet forwarded to a single destination can occur on multiple "best paths" that are tied for first in the routing metric calculation. Multipath routing can be used in conjunction with most routing protocols because it is a per-hop decision limited to a single router. The software can support multi-tenancy, which is an architecture where a single instance of a software application serves multiple customers. Each customer may be referred to as a tenant. In some embodiments, a tenant may be given the ability to customize some parts of the application, but not the code of the application. These embodiments may maintain an audit log. An audit log is a document that records events in a computing system.In addition to recording which resources were accessed, audit log entries typically include destination and source addresses, timestamps, and user login information that complies with various regulations. These embodiments can support various key management policies, such as encryption key rotation, etc. In addition, the system can support dynamic root passwords or some dynamically changing passwords.

[0113] Figure 3A 1 illustrates a diagram of a storage system 306 coupled to communicate data with a cloud service provider 302 according to some embodiments of the present invention. Although not depicted in detail, Figure 3A The storage system 306 depicted may be similar to the storage system 306 described above. Figures 1A to 1D and Figures 2A to 2G The storage system described above is similar. In some embodiments, Figure 3A The depicted storage system 306 may be embodied as a storage system including unbalanced active / active controllers, a storage system including balanced active / active controllers, a storage system including active / active controllers that utilize less than all of the resources of each controller so that each controller has reserve resources available to support failover, a storage system including fully active / active controllers, a storage system including controllers with data set isolation, a storage system including a two-tier architecture with a front-end controller and a back-end integrated storage controller, a storage system including a scale-out cluster of dual controller arrays, and combinations of these embodiments.

[0114] exist Figure 3A In the depicted example, the storage system 306 is coupled to the cloud service provider 302 via a data communication link 304. The data communication link 304 may be embodied as a dedicated data communication link, a data communication path provided by using one or more data communication networks such as a wide area network (WAN) or a local area network (LAN), or some other mechanism capable of transmitting digital information between the storage system 306 and the cloud service provider 302. Such a data communication link 304 may be completely wired, completely wireless, or some aggregation of wired and wireless data communication paths. In such an example, digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using one or more data communication protocols. For example, digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using the Handheld Device Transfer Protocol (HDTP), the Hypertext Transfer Protocol (HTTP), the Internet Protocol (IP), the Real-time Transport Protocol (RTP), the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), the Wireless Application Protocol (WAP), or other protocols.

[0115] Figure 3AThe depicted cloud service provider 302 may be embodied, for example, as a system and computing environment that provides services to users of the cloud service provider 302 through shared computing resources via a data communication link 304. The cloud service provider 302 may provide on-demand access to a shared pool of configurable computing resources, such as computer networks, servers, storage, applications, and services. The shared pool of configurable resources may be quickly provisioned and released to users of the cloud service provider 302 with minimal management effort. Generally, users of the cloud service provider 302 are unaware of the exact computing resources utilized by the cloud service provider 302 in order to provide services. Although in many cases such cloud service providers 302 may be accessible via the Internet, readers skilled in the art will recognize that any system that abstracts the use of shared resources to provide services to users via any data communication link may be considered a cloud service provider 302.

[0116] exist Figure 3AIn the depicted example, cloud service provider 302 may be configured to provide various services to storage system 306 and users of storage system 306 by implementing various service models. For example, cloud service provider 302 may be configured to provide services to storage system 306 and users of storage system 306 by implementing an infrastructure as a service (IaaS) service model, in which cloud service provider 302 provides computing infrastructure such as virtual machines and other resources as a service to subscribers. Additionally, cloud service provider 302 may be configured to provide services to storage system 306 and users of storage system 306 by implementing a platform as a service (PaaS) service model, in which cloud service provider 302 provides a development environment to application developers. Such a development environment may include, for example, an operating system, a programming language execution environment, a database, a web server, or other components that may be utilized by application developers to develop and run software solutions on a cloud platform. In addition, the cloud service provider 302 may be configured to provide services to the storage system 306 and the users of the storage system 306 by implementing a software as a service (SaaS) service model, wherein the cloud service provider 302 provides application software, databases, and a platform for running applications to the storage system 306 and the users of the storage system 306, thereby providing on-demand software to the storage system 306 and the users of the storage system 306 and eliminating the need to install and run applications on local computers, which can simplify the maintenance and support of the applications. The cloud service provider 302 may also be configured to provide services to the storage system 306 and the users of the storage system 306 by implementing an authentication as a service (AaaS) service model, wherein the cloud service provider 302 provides authentication services that can be used to protect access to applications, data sources, or other resources. The cloud service provider 302 may also be configured to provide services to the storage system 306 and the users of the storage system 306 by implementing a storage as a service model, wherein the cloud service provider 302 provides access to its storage infrastructure for use by the storage system 306 and the users of the storage system 306. The reader will understand that the cloud service provider 302 may be configured to provide additional services to the storage system 306 and users of the storage system 306 by implementing additional service models, as the above-described service models are included for illustrative purposes only and in no way represent a limitation on the services that the cloud service provider 302 may provide, nor do they represent a limitation on the service models that the cloud service provider 302 may implement.

[0117] exist Figure 3AIn the example depicted in , the cloud service provider 302 may, for example, be embodied as a private cloud, a public cloud, or a combination of a private cloud and a public cloud. In an embodiment where the cloud service provider 302 is embodied as a private cloud, the cloud service provider 302 may be dedicated to providing services to a single organization rather than to multiple organizations. In an embodiment where the cloud service provider 302 is embodied as a public cloud, the cloud service provider 302 may provide services to multiple organizations. Public cloud and private cloud deployment models may differ and may be accompanied by various advantages and disadvantages. For example, since public cloud deployment involves sharing computing infrastructure across different organizations, such deployment may not be ideal for organizations with security issues, mission-critical workloads, and uptime requirements, etc. Although private cloud deployments can address some of these issues, private cloud deployments may require internal staff to manage the private cloud. In another optional embodiment, the cloud service provider 302 may be embodied as a mix of private cloud services and public cloud services deployed in a hybrid cloud.

[0118] Despite Figure 3A 306 and users of the storage system 306. For example, the storage system 306 may be coupled to (or even include) a cloud storage gateway. Such a cloud storage gateway may, for example, be embodied as a hardware-based or software-based application located locally with the storage system 306. Such a cloud storage gateway may operate as a bridge between local applications executed on the storage array 306 and the remote, cloud-based storage utilized by the storage array 306. By using a cloud storage gateway, an organization can move the main iSCSI or NAS to the cloud service provider 302, thereby enabling the organization to save space on its internal storage system. Such a cloud storage gateway may be configured to emulate a disk array, a block-based device, a file server, or other storage system that can convert SCSI commands, file server commands, or other appropriate commands into a REST space protocol that facilitates communication with the cloud service provider 302.

[0119] In order to enable the storage system 306 and the users of the storage system 306 to take advantage of the services provided by the cloud service provider 302, a cloud migration process may be performed during which data, applications, or other elements from the organization's local systems (or even from another cloud environment) are moved to the cloud service provider 302. In order to successfully migrate data, applications, or other elements to the cloud service provider 302 environment, middleware such as cloud migration tools may be used to bridge the gap between the cloud service provider 302 environment and the organization's environment. Such cloud migration tools may also be configured to address the potential high network costs and long transfer times associated with migrating large amounts of data to the cloud service provider 302, as well as security issues associated with migrating sensitive data to the cloud service provider 302 over a data communications network. In order to further enable the storage system 306 and the users of the storage system 306 to take advantage of the services provided by the cloud service provider 302, a cloud orchestrator may also be used to arrange and coordinate automated tasks in an effort to create a merged process or workflow. Such a cloud orchestrator may perform tasks such as configuring various components (whether these components are cloud components or internal components) and managing the interconnections between these components. Cloud orchestrators can simplify inter-component communication and connectivity to ensure that links are properly configured and maintained.

[0120] exist Figure 3A In the depicted example and as briefly described above, the cloud service provider 302 can be configured to provide services to the storage system 306 and the users of the storage system 306 by using a SaaS service model, wherein the cloud service provider 302 provides application software, databases, and a platform for running applications to the storage system 306 and the users of the storage system 306, thereby providing the storage system 306 and the users of the storage system 306 with on-demand software and eliminating the need to install and run applications on local computers, which can simplify the maintenance and support of the applications. According to various embodiments of the present invention, these applications can take a variety of forms. For example, the cloud service provider 302 can be configured to provide access to data analysis applications to the storage system 306 and the users of the storage system 306. Such data analysis applications can, for example, be configured to receive telemetry data dialed back by the storage system 306. Such telemetry data may describe various operational characteristics of the storage system 306 and may be analyzed to, for example, determine the health of the storage system 306, identify workloads executing on the storage system 306, predict when the storage system 306 will exhaust various resources, recommend configuration changes, hardware or software upgrades, workflow migrations, or other actions that may improve the operation of the storage system 306.

[0121] The cloud service provider 302 may also be configured to provide access to a virtualized computing environment to the storage system 306 and users of the storage system 306. Such a virtualized computing environment may be embodied, for example, as a virtual machine or other virtualized computer hardware platform, a virtual storage device, and virtualized computer network resources, etc. Examples of such a virtualized environment may include a virtual machine created to simulate an actual computer, a virtualized desktop environment that separates a logical desktop from a physical machine, a virtualized file system that allows unified access to different types of specific file systems, and many other environments.

[0122] To further illustrate, Figure 3B 1 illustrates a diagram of a storage system 306 according to some embodiments of the present invention. Although not depicted in detail, Figure 3B The storage system 306 depicted may be similar to the storage system 306 described above. Figures 1A to 1D and Figures 2A to 2G The storage system described is similar in that the storage system may include many of the components described above.

[0123] Figure 3BThe depicted storage system 306 may include storage resources 308, which may be embodied in a variety of forms. For example, in some embodiments, the storage resources 308 may include nano RAM or other forms of non-volatile random access memory utilizing carbon nanotubes deposited on a substrate. In some embodiments, the storage resources 308 may include 3D crosspoint non-volatile memory in combination with a stackable cross-grid data access array, where bit storage is based on changes in body resistance. In some embodiments, the storage resources 308 may include flash memory, including single-level cell (SLC) NAND flash memory, multi-level cell (MLC) NAND flash memory, triple-level cell (TLC) NAND flash memory, and quad-level cell (QLC) NAND flash memory, among others. In some embodiments, the storage resources 308 may include non-volatile magnetoresistive random access memory (MRAM) that stores data using magnetic storage elements, including spin transfer torque (STT) MRAM. In some embodiments, the example storage resources 308 may include non-volatile phase change memory (PCM), which may have the ability to accommodate multiple bits in a single cell because the cell can achieve multiple different intermediate states. In some embodiments, storage resources 308 may include quantum memory that allows for storage and retrieval of photon quantum information. In some embodiments, example storage resources 308 may include resistive random access memory (ReRAM), where data is stored by changing the resistance across a dielectric solid state material. In some embodiments, storage resources 308 may include storage class memory (SCM), where solid-state non-volatile memory may be manufactured at high density using some combination of sub-lithographic patterning techniques, multiple bits per cell, and multiple device layers, etc. The reader will appreciate that the above-described storage system may utilize other forms of computer memory and storage devices, including DRAM, SRAM, EEPROM, general purpose memory, and many other memories. Figure 3B The depicted storage resources 308 may be embodied in various form factors including, but not limited to, dual in-line memory modules (DIMMs), non-volatile dual in-line memory modules (NVDIMMs), M.2, U.2, and the like. Figure 3BThe depicted storage resources 308 may include various forms of storage class memory (SCM). The SCM can effectively treat fast, non-volatile memory (e.g., NAND flash) as an extension of DRAM, so that the entire data set can be treated as an in-memory data set that is completely located in DRAM. The SCM may include, for example, non-volatile media such as NAND flash. Such NAND flash can be accessed using NVMe, where NVMe can use the PCIe bus as its transport bus, thereby providing relatively low access latency compared to earlier protocols. In fact, network protocols for SSDs in all-flash arrays may include NVMe (ROCE, NVME TCP), Fibre Channel (NVMe FC), and InfiniBandwidth (iWARP) using Ethernet, among others, which make it possible to treat fast, non-volatile memory as an extension of DRAM. In view of the fact that DRAM is typically byte-addressable and fast, non-volatile memory such as NAND flash is block-addressable, a controller software / hardware stack may be required to convert block data into bytes stored in the media. Examples of media and software that may be used as SCM may include, for example, 3D XPoint, Intel Memory Drive Technology, and Samsung's Z-SSD, among others.

[0124] Figure 3B The depicted example storage system 306 can implement various storage architectures. For example, a storage system according to some embodiments of the present invention can utilize block storage, where data is stored in blocks, and each block is essentially used as a separate hard drive. A storage system according to some embodiments of the present invention can utilize object storage, where data is managed as an object. Each object can include the data itself, a variable amount of metadata, and a globally unique identifier, where object storage can be implemented at multiple levels (e.g., device level, system level, interface level). A storage system according to some embodiments of the present invention utilizes file storage, where data is stored in a hierarchical structure. Such data can be stored in files and folders and presented to both the system storing the data and the system retrieving the data in the same format. Figure 3B The depicted example storage system 306 may be embodied as a storage system in which additional storage resources may be added by using a scale-up model, additional storage resources may be added by using a scale-out model, or some combination thereof. In a scale-up model, additional storage may be added by adding additional storage devices. However, in a scale-out model, additional storage nodes may be added to a cluster of storage nodes, wherein such storage nodes may include additional processing resources and additional network resources, among other things.

[0125] Figure 3B The depicted storage system 306 also includes communication resources 310, which may help facilitate data communication between components within the storage system 306 and data communication between the storage system 306 and computing devices outside the storage system 306. The communication resources 310 can be configured to utilize a variety of different protocols and data communication structures to facilitate data communication between components within the storage system and data communication between the storage system and computing devices outside the storage system. For example, the communication resources 310 may include Fibre Channel (FC) technologies such as FC structures and FC protocols, which can transmit SCSI commands across FC networks. The communication resources 310 may also include FC over Ethernet (FCoE) technologies, through which FC frames are encapsulated and transmitted across Ethernet networks. The communication resources 310 may also include InfiniBand (IB) technologies, in which a switch fabric topology is utilized to facilitate transmission between channel adapters. The communication resources 310 may also include NVM Express (NVMe) technologies and NVMe over fabric (NVMeoF) technologies, through which non-volatile storage media attached via a PCI Express (PCIe) bus may be accessed. The communication resources 310 may also include: a mechanism for accessing storage resources 308 within the storage system 306 using serial attached SCSI (SAS), a serial ATA (SATA) bus interface for connecting storage resources 308 within the storage system 306 to a host bus adapter within the storage system 306, Internet Small Computer System Interface (iSCSI) technology for providing block-level access to storage resources 308 within the storage system 306, and other communication resources that can help facilitate data communications between components within the storage system 306 and data communications between the storage system 306 and computing devices outside the storage system 306. Figure 3B The depicted storage system 306 also includes processing resources 312, which can facilitate the execution of computer program instructions and other computing tasks within the storage system 306. The processing resources 312 can include one or more application specific integrated circuits (ASICs) customized for a particular purpose and one or more central processing units (CPUs). The processing resources 312 can also include one or more digital signal processors (DSPs), one or more field programmable gate arrays (FPGAs), one or more systems on a chip (SoCs), or other forms of processing resources 312. The storage system 306 can utilize the storage resources 312 to perform a variety of tasks, including but not limited to supporting the execution of software resources 314, which will be described in more detail below. Figure 3BThe depicted storage system 306 also includes software resources 314 that can perform various tasks when executed by the processing resources 312 within the storage system 306. The software resources 314 can, for example, include one or more computer program instruction modules that, when executed by the processing resources 312 within the storage system 306, facilitate the execution of various data protection techniques to maintain the integrity of the data stored within the storage system. The reader will understand that such data protection techniques can be executed, for example, by system software executed on computer hardware within the storage system, by a cloud service provider, or otherwise. Such data protection techniques can, for example, include data archiving techniques that move data that is no longer actively used to a separate storage device or separate storage system for long-term retention, data backup techniques in which data stored in the storage system can be copied and stored in different locations to avoid data loss in the event of a device failure or some other form of disaster in the storage system, data replication techniques in which data stored in the storage system is copied to another storage system so that the data can be accessed via multiple storage systems, data snapshot techniques that capture the state of data within the storage system at different points in time, data and database cloning techniques that can create duplicate copies of data and databases, and other data protection techniques. By using this data protection technology, business continuity and disaster recovery goals can be met because the failure of the storage system may not result in the loss of data stored in the storage system.

[0126] Software resources 314 may also include software that facilitates implementation of software defined storage (SDS). In such an example, software resources 314 may include one or more computer program instruction modules that, when executed, facilitate policy-based provisioning and management of data storage independent of underlying hardware. Such software resources 314 may facilitate implementation of storage virtualization to separate storage hardware from software that manages the storage hardware.

[0127] Software resources 314 may also include software that helps facilitate and optimize I / O operations, where the I / O operations are directed to storage resources 308 in storage system 306. For example, software resources 314 may include software modules that perform various data reduction techniques such as data compression and data deduplication. Software resources 314 may include software modules that intelligently group I / O operations together to facilitate better use of underlying storage resources 308, software modules that perform data migration operations to migrate from within the storage system, and software modules that perform other functions. Such software resources 314 may be embodied as one or more software containers or in many other ways.

[0128] The reader will appreciate that the presence of these software resources 314 can provide an improved user experience of the storage system 306, an expansion of the functionality supported by the storage system 306, and a number of other benefits. Consider the specific example of software resources 314 that implement data backup techniques in which data stored in the storage system can be copied and stored in different locations to avoid data loss in the event of a device failure or some other form of disaster. In such an example, the system described herein can perform backup operations more reliably (and in a manner that is less burdensome to the user) relative to interactive backup management systems that require a high degree of user interaction and provide less robust automation and feature sets, etc.

[0129] The above-mentioned storage system can perform intelligent data backup technology in which the data stored in the storage system can be copied and stored in different locations to avoid data loss in the event of a device failure or some other form of disaster. For example, the above-mentioned storage system can be configured to check each backup to avoid restoring the storage system to an undesirable state. Consider the example of malware infecting the storage system. In such an example, the storage system may include software resources 314, which can scan each backup to identify backups captured before the malware infects the storage system and backups captured after the malware infects the storage system. In such an example, the storage system can restore itself from backups that do not include malware, or at least not restore the portion of the backup that contains malware. In such an example, the storage system may include software resources 314, which can scan each backup to, for example, identify write operations served by the storage system and from a network subnet suspected of having delivered malware (or a virus or some other undesirable thing), by identifying write operations served by the storage system and from users suspected of having delivered malware, by identifying write operations served by the storage system and checking the content of the write operations for the fingerprint map of the malware, and adopting many other ways to identify the presence of malware.

[0130] The reader will further appreciate that backups (typically in the form of one or more snapshots) may also be used to perform rapid recovery of the storage system. Consider an example in which a storage system is infected with ransomware that locks users out of the storage system. In such an example, software resources 314 within the storage system may be configured to detect the presence of ransomware, and may be further configured to use retained backups to restore the storage system to a point in time prior to the point in time when the ransomware infected the storage system. In such an example, the presence of ransomware may be explicitly detected using a software tool utilized by the system, by using a key (e.g., a USB drive) inserted into the storage system, or in a similar manner. Similarly, the presence of ransomware may be inferred in response to system activity satisfying, for example, a predetermined fingerprint profile (such as, for example, no reads or writes into the system within a predetermined time period, etc.).

[0131] The reader will understand that Figure 3B The various components depicted may be grouped into one or more optimized computing packages as a converged infrastructure. Such a converged infrastructure may include pools of computer, storage, and network resources that may be shared by multiple applications and managed in a collective manner using policy-driven processing. Such a converged infrastructure may minimize compatibility issues between components within the storage system 306 while also reducing various costs associated with establishing and operating the storage system 306. Such a converged infrastructure may be implemented using a converged infrastructure reference architecture, using separate appliances, using a software-driven hyper-convergence approach (e.g., a hyper-converged infrastructure), or in other ways.

[0132] The reader will understand that Figure 3B The depicted storage system 306 can help support various types of software applications. For example, the storage system 306 can help support artificial intelligence (AI) applications, database applications, DevOps projects, electronic design automation tools, event-driven software applications, high-performance computing applications, simulation applications, high-speed data capture and analysis applications, machine learning applications, media production applications, media serving applications, picture archiving and communication system (PACS) applications, software development applications, virtual reality applications, augmented reality applications, and many other types of applications by providing storage resources to these applications.

[0133] The above-mentioned storage system can be operated to support various applications. In view of the fact that the storage system includes computing resources, storage resources and various other resources, the storage system may be very suitable for supporting, for example, resource-intensive applications, such as AI applications, etc. Such AI applications can enable the device to perceive its environment and take actions that maximize the possibility of success on a certain goal. Examples of these AI applications may include IBM Watson, Microsoft Oxford, Google DeepMind and Baidu Minwa, etc. The above-mentioned storage system may also be very suitable for supporting, for example, other types of resource-intensive applications, such as machine learning applications, etc. Machine learning applications can perform various types of data analysis to automate the construction of analytical models. Using algorithms that learn from data iteratively, machine learning applications can enable computers to achieve learning without explicit programming. A specific field of machine learning is called reinforcement learning, which involves taking appropriate actions in specific situations to maximize rewards. Reinforcement learning can be used to find the best behavior or path that a specific software application or machine should take in a specific situation. The difference between reinforcement learning and other fields of machine learning (e.g., supervised learning, unsupervised learning) is that for reinforcement learning, there is no need to present the correct input / output pair, nor is there a need to explicitly correct suboptimal actions.

[0134] In addition to the resources already described, the storage system described above may also include a graphics processing unit (GPU), sometimes also referred to as a visual processing unit (VPU). Such a GPU may be embodied as a specialized electronic circuit that rapidly manipulates and changes memory to accelerate the creation of images in a frame buffer intended for output to a display device. Such a GPU may be included in any computing device that is part of the storage system described above (including as one of many separately scalable components of the storage system), where other examples of separately scalable components of such a storage system may include storage components, memory components, computing components (e.g., CPU, FPGA, ASIC), network components, and software components, among others. In addition to the GPU, the storage system described above may also include a neural network processor (NNP) for various aspects of neural network processing. Such an NNP may be used as a replacement (or in addition) for a GPU and may also be independently scalable.

[0135] As described above, the storage system described here can be configured to support artificial intelligence applications, machine learning applications, big data analysis applications, and many other types of applications. The rapid growth of such applications is driven by three technologies: deep learning (DL), GPU processors, and big data. Deep learning is a computational model that uses massively parallel neural networks inspired by the human brain. Instead of expert handcrafted software, deep learning models write their own software by learning from a large number of examples. GPUs are modern processors with thousands of cores that are well suited for running algorithms that loosely represent the parallel nature of the human brain.

[0136] The development of deep neural networks has unleashed a wave of new algorithms and tools that allow data scientists to exploit their data using artificial intelligence (AI). With improved algorithms, larger datasets, and a variety of frameworks, including open source software libraries for machine learning across a range of tasks, data scientists are tackling new use cases such as autonomous vehicles, natural language processing and understanding, computer vision, machine reasoning, and strong AI. Applications of this technology can include: machine and vehicle object detection, recognition, and avoidance; visual recognition, classification, and tagging; algorithmic financial trading strategy performance management; simultaneous localization and mapping; predictive maintenance of high-value machinery; protection against cybersecurity threats; automation of expertise; image recognition and classification; question answering; robotics; and text analysis (extraction, classification) and text generation and translation; among others. Examples of applications of AI technology that have been implemented in a wide range of products include Amazon Echo’s voice recognition technology that allows users to talk to their machines, Google Translate that allows machine-based language translation, and more. TM , Spotify's Discover Weekly, which provides recommendations about new songs and artists that users may like based on user usage and business analysis, Quill's text generation service that takes structured data and converts it into a narrative story, and chatbots that provide real-time, context-specific answers to questions in a conversational format. In addition, AI may affect various industries and sectors. For example, AI solutions can be used in health care to record clinical logs, patient files, research data, and other inputs to generate possible treatment options for doctors to explore. Similarly, retailers can also use AI solutions to personalize consumer recommendations based on digital footprints of individual behavior, archival data, or other data.

[0137] However, training deep neural networks requires both high-quality input data and a lot of computation. GPUs are massively parallel processors capable of operating on large amounts of data simultaneously. When combined into multi-GPU clusters, high-throughput pipelines may be required to feed input data from storage to the compute engines. Deep learning is more than just building and training models. There is also the entire data pipeline that must be designed for the scale, iteration, and experimentation required for data science teams to succeed.

[0138] Data is at the heart of modern AI and deep learning algorithms. Before training can begin, one problem that must be solved revolves around collecting labeled data that is critical to training accurate AI models. Full-scale AI deployments may be required to continuously collect, clean, transform, label, and store large amounts of data. Adding additional high-quality data points directly translates into more accurate models and better insights. Data samples can go through a series of processing steps, including but not limited to: 1) ingesting data from external sources into the training system and storing the data in raw form; 2) cleaning the data and transforming the data in a format that is convenient for training, including linking data samples to appropriate labels; 3) exploring parameters and models, testing quickly with smaller data sets, and iterating to converge on the most promising model to enter the production cluster; 4) performing a training phase to select random batches of input data (both new and old samples) and feed these input data to production GPU servers for computation to update model parameters; and 5) evaluation, including using a reserved portion of the data not used in training to evaluate the model accuracy on the reserved data. This life cycle can apply to any type of parallelized machine learning, not just neural networks or deep learning. For example, a standard machine learning framework may rely on CPUs instead of GPUs, but the data ingestion and training workflows may be the same. The reader will understand that a single shared storage data center creates a coordination point throughout the life cycle without the need for additional data copies between ingestion, preprocessing, and training stages. Ingested data is rarely used for just one purpose, and shared storage grants the flexibility to train multiple different models or apply traditional analytics to the data.

[0139] The reader will understand that each stage in the AI ​​data pipeline can have different requirements from the data center (e.g., a storage system or a collection of storage systems). The scale-out storage system must provide uncompromising performance for all types of access and patterns (from small metadata levels to large files, from random access patterns to sequential access patterns, and from low concurrency to high concurrency). The above storage systems can serve as ideal AI data centers because these systems can serve unstructured workloads. In the first stage, ideally, the data is ingested and stored in the same data center that the subsequent stages will use to avoid excessive data copying. The next two steps can be completed on a standard computing server that optionally includes a GPU, and then in the fourth and final stage, the complete training production job is run on a powerful GPU-accelerated server. Typically, there is a production pipeline next to the experimental pipeline that operates on the same data set. In addition, GPU-accelerated servers can be used independently for different models, or combined together to train a larger model, or even across multiple systems for distributed training. If the shared storage layer is slow, the data must be copied to the local storage of each stage, resulting in a waste of time to stage the data to different servers. The ideal data center for an AI training pipeline provides performance similar to data stored locally on server nodes, while also having the simplicity and performance to enable all pipeline stages to operate concurrently.

[0140] Data scientists are committed to improving the practicality of training models through various methods such as: more data, better data, smarter training, and deeper models. In many cases, teams of data scientists will share the same dataset and work in parallel to produce new and improved training models. Typically, teams of data scientists will work on the same shared dataset concurrently during these stages. Multiple concurrent workloads of data processing, experiments, and full-scale training layer the needs of multiple access modes on the storage layer. In other words, storage cannot just meet large file reads, but must deal with a mixture of large and small file reads and writes. Finally, as multiple data scientists explore datasets and models, it may be crucial to store data in its native format to provide each user with the flexibility to transform, clean, and use data in a unique way. The above storage system can provide a natural shared storage home for the dataset, which has data protection redundancy (for example, by using RAID6) and the performance required as a common access point for multiple developers and multiple experiments. Using the above storage system can avoid the need to carefully copy subsets of data for local work, thereby saving both engineering and GPU acceleration server usage time. As the original dataset and the desired transformation are constantly updated and changed, the burden of these copies continues to grow.

[0141] The reader will understand that the fundamental reason for the surge in success of deep learning is the continued improvement of models with larger dataset sizes. In contrast, classical machine learning algorithms such as logistic regression stop improving in accuracy at smaller dataset sizes. As such, the separation of compute and storage resources can also allow for the separate scaling of each layer, thereby avoiding many of the complexities inherent in managing both together. Scale-out storage systems must be able to scale easily as dataset sizes grow or new datasets are considered. Similarly, if more concurrent training is required, additional GPUs or other compute resources can be added without worrying about their internal storage. In addition, the above storage systems can make it easier to build, operate, and grow AI systems due to the random read bandwidth provided by the storage system, the ability of the storage system to read small files (50KB) at random and high rates (meaning that no additional effort is required to aggregate individual data points to generate larger, storage-friendly files), the ability of the storage system to scale capacity and performance as the dataset grows or throughput requirements grow, the ability of the storage system to support files or objects, the ability of the storage system to adjust performance for large or small files (i.e., without the need for the user to provide a file system), the ability of the storage system to support non-disruptive upgrades of hardware and software even during production model training, and many other reasons.

[0142] Small file performance of the storage layer can be critical, as many types of inputs, including text, audio, or images, are natively stored as small files. If the storage layer does not handle small files well, additional steps will be required to pre-process and group samples into larger files. Storage built on spinning disks that relies on SSDs as a cache layer may not meet the required performance. Since training with randomized batches of inputs produces more accurate models, the entire dataset must be accessible with full performance. SSD caches only provide high performance for a small subset of the data and will be ineffective in hiding the latency of spinning drives.

[0143] Although the previous paragraphs discuss deep learning applications, readers will understand that the storage system described here can also be part of a distributed deep learning (DDL) platform to support the execution of DDL algorithms. Distributed deep learning can be used to significantly accelerate deep learning using distributed computing on GPUs (or other forms of accelerators or computer program instruction executors), so that parallelization can be achieved. In addition, the output of training machine learning and deep learning models (such as fully trained machine learning models, etc.) can be used for various purposes and can be used in combination with other tools. For example, a trained machine learning model can be used in combination with tools such as Core ML to integrate various machine learning model types into applications. In fact, the trained model can be run through the Core ML converter tool and inserted into a custom application that can be deployed on a compatible device. The above-mentioned storage system can also be paired with other technologies such as TensorFlow (an open source software library used for data flow programming across a series of tasks that can be used for machine learning applications such as neural networks) to facilitate the development of such machine learning models and applications.

[0144] The reader will further appreciate that as AI becomes available for mass consumption, the above-described systems may be deployed in a variety of ways to support the democratization of AI. The democratization of AI may include, for example, the ability to provide AI as a platform as a service, the growth of artificial general intelligence provision, the proliferation of autonomous level 4 and autonomous level 5 vehicles, the availability of autonomous mobile robots, and the development of conversational AI platforms. For example, the above-described systems may be deployed in a cloud environment, an edge environment, or other environment that helps support the democratization of AI. As part of the democratization of AI, there may be a move from narrow AI, which encompasses a wide range of machine learning solutions that target specific tasks, to artificial general intelligence, in which the use of machine learning is expanded to handle a wide range of use cases, and general artificial intelligence, like humans, can perform essentially any intelligent task that humans can perform and can dynamically learn.

[0145] The above storage system can also be used in a neuromorphic computing environment. Neuromorphic computing is a form of computing that simulates brain cells. To support neuromorphic computing, the architecture of interconnected "neurons" replaces the traditional computing model, directly transmitting low-power signals between neurons for more efficient computing. Neuromorphic computing can utilize very large scale integration (VLSI) systems containing electronic analog circuits to simulate the neurobiological architecture present in the nervous system, as well as analog, digital, mixed-mode analog / digital VLSI, and software systems for implementing models of the nervous system for perception, motor control, or multi-sensor integration.

[0146] The reader will understand that the storage system described above may be configured to support the storage or use of a blockchain (among other types of data). Such a blockchain may be embodied as a growing list of records (called blocks) that are linked and protected using encryption. Each block in the blockchain may contain, among other things, a hash pointer, a timestamp, and transaction data as a link to the previous block. The blockchain may be designed to resist modification of the data and may be used as an open distributed ledger that can efficiently and in a verifiable and permanent manner record transactions between two parties. This makes the blockchain potentially suitable for recording events, medical records, and other record management activities (such as identity management and transaction processing, etc.). In addition to supporting the storage and use of blockchain technology, the storage system described above may also support the storage and use of derivatives, such as open source blockchains and related tools as part of the IBMTM Hyperledger project, permissioned blockchains where a certain number of trusted parties are allowed to access the blockchain, and blockchain products that enable developers to build their own distributed ledger projects, etc. The reader will understand that blockchain technology may affect a variety of industries and sectors. For example, blockchain technology can be used as a blockchain-based contract in real estate matters. The use of blockchain-based contracts can eliminate the need for third parties and enable self-executing actions when conditions are met. Similarly, a universal health record can be created by aggregating and placing an individual's health history on a blockchain ledger for access and update by any health care provider or licensed health care provider.

[0147] The reader will understand that the use of blockchain is not limited to financial transactions and contracts, etc. In fact, blockchain can be used to achieve decentralized aggregation, sorting, timestamping and archiving of any type of information (including structured data, correspondence, documents or other data). By using blockchain, participants can verifiably and permanently agree on exactly what data was entered, when it was entered, and by whom without relying on trusted intermediaries. For example, SAP's recently launched blockchain platform (which supports multi-chain and hyperledger fabric) targets a wide range of supply chain and other non-financial applications.

[0148] One way to record data using a blockchain is to embed the data directly into the transaction. Each blockchain transaction can be digitally signed by one or more parties, copied to multiple nodes, sorted and time-stamped by the consensus algorithm of the chain, and permanently stored in a tamper-proof manner. Therefore, any data within the transaction will be stored by each node in the same but independent manner together with proof of who wrote the data and when the data was written. Users of the chain can retrieve the information at any time in the future. This type of storage may be referred to as on-chain storage. However, when trying to store very large data sets, on-chain storage may not be particularly practical. For this reason, according to embodiments of the present invention, the blockchain and storage system described herein can be used to support on-chain storage of data and off-chain storage of data. Off-chain storage of data can be implemented in various ways and can occur when the data itself is not stored in the blockchain. For example, in one embodiment, a hash function can be utilized, and the data itself can be fed into the hash function to generate a hash value. In such an example, a hash of a large amount of data can be embedded in the transaction instead of the data itself. Each hash can be used as a commitment to its input data, where the data itself is stored outside the blockchain. The reader will understand that any blockchain participant who needs off-chain data cannot reproduce the data based on the hash of the data, but if the data can be retrieved in other ways, the on-chain hash is used to confirm who created the data and when it was created. As with conventional on-chain data, the hash may be embedded in a digitally signed transaction, and the transaction consensus is included in the chain. The reader will understand that in other embodiments, alternatives to blockchains can be used to facilitate decentralized storage of information. For example, one alternative to blockchains that can be used is a blockweave. While traditional blockchains store each transaction to achieve verification, blockweaves allow for secure decentralization without using the entire chain, thereby achieving low-cost on-chain data storage. Such a blockweave can utilize a consensus mechanism based on proof of access (PoA) and proof of work (PoW). While typical PoW systems rely only on the previous block to generate each consecutive block, the PoA algorithm can include data from a randomly selected previous block. Combined with the blockweave data structure, miners do not need to store all blocks (forming the blockchain), but can store any previous blocks that form a weave of blocks (blockweave). This can increase levels of scalability, speed, and low cost, and reduce the cost of data storage, in part because: miners do not need to store all blocks, greatly reducing the amount of electricity consumed during the mining process; because power consumption decreases as the network scales; and because the blocknet requires less and less hashing power to reach consensus as data is added to the system. In addition, the blocknet can be deployed in a decentralized storage network, creating incentives in the blocknet to encourage rapid data sharing.This decentralized storage network can also utilize block shadowing technology, where nodes only send a minimal block "shadow" to other nodes that allows peers to reconstruct the full block, rather than sending the full block itself.

[0149] The above-mentioned storage system can be used alone or in combination with other computing devices to support in-memory computing applications. In-memory computing involves storing information in RAM distributed across computer clusters. In-memory computing helps commercial customers including retailers, banks and utilities to quickly detect patterns, analyze massive amounts of data instantly and perform their operations quickly. The reader will understand that the above-mentioned storage system, especially the storage system that can be configured with customizable amounts of processing resources, storage resources and memory resources (for example, these systems where the blade contains configurable amounts of various types of resources) can be configured in a manner that provides an infrastructure that can support in-memory computing. Similarly, the above-mentioned storage system can include component parts (for example, NVDIMM, 3D crosspoint storage that provides persistent fast random access memory), which can actually provide an improved in-memory computing environment compared to the in-memory computing environment that relies on RAM distributed across dedicated servers.

[0150] In some embodiments, the storage system described above may be configured to operate as a hybrid in-memory computing environment that includes a common interface for all storage media (e.g., RAM, flash storage, 3D crosspoint storage). In such an embodiment, users may not know the details about where their data is stored, but they can still use the same complete unified API to address the data. In such an embodiment, the storage system can (in the background) move the data to the fastest available tier, including intelligently placing the data based on various characteristics of the data or based on some other heuristics. In such an example, the storage system can even utilize existing products such as Apache Ignite and GridGain to move data between different storage tiers, or the storage system can utilize custom software to move data between different storage tiers. The storage system described here can implement various optimizations to, for example, improve the performance of in-memory computing, such as performing computing as close to the data as possible.

[0151] The reader will further understand that in some embodiments, the above-mentioned storage system can be paired with other resources to support the above-mentioned applications. For example, an infrastructure can include primary computing in the form of servers and workstations, which are dedicated to general computing on graphics processing units (GPGPUs) to accelerate deep learning applications interconnected to computing engines to train the parameters of deep neural networks. Each system can have an Ethernet external connection, an infinite bandwidth external connection, some other form of external connection, or some combination thereof. In such an example, GPUs can be grouped for a single large training, or used independently to train multiple models. The infrastructure can also include a storage system (such as the above-mentioned storage system, etc.) to, for example, provide an out-of-scale all-flash file or object storage that can access data via high-performance protocols such as NFS and S3. The infrastructure can also, for example, include redundant top-of-rack Ethernet switches that are connected to storage and computing via ports in the MLAG port channel to achieve redundancy. The infrastructure can also include additional computing in the form of white box servers (optionally using GPUs) for data ingestion, preprocessing, and model debugging. The reader will understand that there may be more infrastructures.

[0152] The reader will appreciate that the above-described system may be more suitable for the above-described applications relative to other systems that may, for example, include a distributed direct attached storage (DDAS) solution deployed in a server node. Such a DDAS solution may be constructed to handle large, less sequential accesses, but may be less able to handle small random accesses. The reader will further appreciate that the above-described storage system may be used to provide a platform for the above-described applications that is more preferred than utilizing cloud-based resources because the storage system may be included in a more secure, more locally and internally managed, more robust on-site or internal infrastructure in terms of feature set and performance, or the platform is more preferred than utilizing cloud-based resources as part of the platform to support the above-described applications. For example, services built on platforms such as IBM's Watson may require an enterprise to distribute personal user information such as financial transaction information or identifiable patient records to other institutions. As such, for various technical reasons and various business reasons, cloud-based provision of AI as a service may be less desirable than internally managed and provided AI as a service supported by storage systems such as the above-described storage systems.

[0153] The reader will understand that the above-mentioned storage system can be configured to support other AI-related tools alone or in coordination with other computing machinery. For example, the storage system can utilize tools such as ONXX or other open neural network exchange formats that make it easier to transfer models written in different AI frameworks. Similarly, the storage system can also be configured to support tools such as Amazon's Gluon that allow developers to prototype, build and train deep learning models. In fact, the above-mentioned storage system can be part of a larger platform such as IBMTM Cloud Private for Data (IBMTM Private Cloud for Data), which includes integrated data science, data engineering and application building services. Such a platform can seamlessly collect, organize, protect and analyze data across enterprises, and use a single solution to simplify hybrid data management, unified data governance and integration, data science and business analysis.

[0154] The reader will further appreciate that the storage system described above may also be deployed as an edge solution. Such an edge solution may appropriately optimize a cloud computing system by performing data processing at the edge of the network close to the data source. Edge computing may push applications, data, and computing power (i.e., services) away from a centralized point to the logical end of the network. By using edge solutions such as the storage system described above, computing resources provided by such a storage system may be used to perform computing tasks, data may be stored using the storage resources of the storage system, and cloud-based services may be accessed by using various resources of the storage system (including network resources). By performing computing tasks on edge solutions, storing data related to edge solutions, and generally utilizing edge solutions, consumption of expensive cloud-based resources may be avoided, and in fact, performance may be improved relative to a greater reliance on cloud-based resources.

[0155] While many tasks may benefit from the utilization of edge solutions, some specific uses may be particularly well suited for deployment in such environments. For example, devices such as drones, autonomous cars, and robots may require extremely fast processing, so fast in fact that sending data up to a cloud environment and back to receive data processing support may simply be too slow. Similarly, machines such as locomotives and gas turbines that generate large amounts of information through the use of a wide range of data generating sensors can benefit from the fast data processing capabilities of edge solutions. As an additional example, some IoT devices (such as connected cameras, etc.) may not be suitable for utilizing cloud-based resources because sending data to the cloud may be impractical (not just from a privacy perspective, a security perspective, or a financial perspective) simply due to the sheer amount of data involved. For this reason, many tasks that are truly related to data processing, storage, or communication may be more suitable for platforms that include edge solutions (such as the storage systems mentioned above, etc.).

[0156] Consider the specific example of inventory management in a warehouse, distribution center, or similar location. Large inventory, warehousing, shipping, order fulfillment, manufacturing, or other operations have a large amount of inventory on inventory shelves and high-resolution digital cameras that generate a firehose of big data. All of this data can be input into an image processing system that can reduce the amount of data to a firehose of small data. All of the small data can be stored in an internally deployed storage. The internally deployed storage at the edge of the facility can be coupled to the cloud, for example, for external reporting, real-time control, and cloud storage. The results of image processing can be used for inventory management so that inventory can be tracked on the shelf and replenished, moved, shipped, modified with new products, or discontinued / obsolete products can be deleted. The above scenario is a prime candidate for an embodiment of the configurable processing and storage system described above. A combination of compute blades and offload blades suitable for image processing, possibly with deep learning on offload FPGAs or (one or more) offload custom blades, can receive a firehose of big data from all digital cameras and generate a firehose of small data. Then, all of the small data can be stored by a storage node, which operates with a storage unit that best processes the data stream with a combination of any type of storage blade. This is an example of storage and function acceleration and integration. Depending on external communication needs with the cloud and external processing in the cloud, and depending on the reliability of network connectivity and cloud resources, the system can be sized for storage and compute management with bursty workloads and variable conductivity reliability. Additionally, depending on other inventory management aspects, the system can be configured for scheduling and resource management in a hybrid edge / cloud environment.

[0157] The storage system described above can be used alone or in combination with other computing resources as a network edge platform that combines computing resources, storage resources, network resources, cloud technology, and network virtualization technology. As part of the network, the edge can have similar characteristics to other network facilities from customer locations and backhaul aggregation facilities to points of presence (PoP) and regional data centers. The reader will understand that network workloads such as virtual network functions (VNFs) will reside on the network edge platform. Enabled by a combination of containers and virtual machines, the network edge platform can rely on controllers and schedulers that are no longer located in the same geographical location as data processing resources. Functions as microservices can be divided into control planes, user planes, and data planes, or even state machines, allowing independent optimization and expansion techniques to be applied. Such user planes and data planes can be enabled by increased accelerators (accelerators residing on server platforms such as FPGAs and smart NICs) and commercial silicon and programmable ASICs enabled by SDN. The storage system described above can also be optimized for big data analysis. Big data analysis can generally be described as the process of examining large and diverse data sets to discover hidden patterns, unknown correlations, market trends, customer preferences, and other useful information that can help organizations make more informed business decisions. Big data analytics applications enable data scientists, predictive modelers, statisticians, and other analytics professionals to analyze growing volumes of structured transactional data, additional data in other forms that are often not exploited by traditional business intelligence (BI) and analytics programs. As part of this processing, for example, semi-structured and unstructured data (such as Internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone call detail records, IoT sensor data, and other data) can be converted into a structured form. Big data analytics is a form of advanced analytics that involves the use of complex applications of elements such as predictive models, statistical algorithms, and what-if analysis based on high-performance analytical systems.

[0158] The above-mentioned storage system can also support (including being implemented as a system interface) applications that perform tasks in response to human language. For example, the storage system can support the execution of intelligent personal assistant applications, such as Amazon's Alexa, AppleSiri, Google Voice, Samsung Bixby and Microsoft Cortana, etc. Although the example described in the previous sentence uses voice as input, the above-mentioned storage system can also support chatbots, conversational robots, rappers, or artificial dialogue entities or other applications configured to conduct dialogues via auditory or text methods. Similarly, the storage system can actually execute such applications so that users (such as system administrators, etc.) can interact with the storage system via voice. Although in an embodiment according to the present invention, this application can be used as an interface for various system management operations, this application is generally capable of voice interaction, music playback, making a to-do list, setting alarms, streaming podcasts, playing audio books, and providing weather, traffic and other real-time information (such as news, etc.).

[0159] The storage systems described above may also implement an AI platform to realize the vision of self-driving storage. Such an AI platform may be configured to provide global predictive intelligence by collecting and analyzing a large number of storage system telemetry data points for easy management, analysis, and support. In fact, such a storage system may be able to predict both capacity and performance, as well as generate intelligent recommendations related to workload deployment, interaction, and optimization. Such an AI platform may be configured to scan all incoming storage system telemetry data against a library of problem fingerprints to predict and resolve incidents in real time before they impact the customer environment, and capture hundreds of performance-related variables for predicting performance loads.

[0160] The storage system described above can support serialization or simultaneous execution of artificial intelligence applications, machine learning applications, data analysis applications, data transformations, and other tasks that can collectively form an AI ladder. This AI ladder can be effectively formed by combining these elements to form a complete data science pipeline, where there are dependencies between the elements of the AI ​​ladder. For example, AI may require some form of machine learning to occur, machine learning may require some form of analysis to occur, and analysis may require some form of data and information architecture design to occur, and so on. As such, the elements can be viewed as rungs in the AI ​​ladder that can collectively form a complete and complex AI solution.

[0161] The above storage system can also be used alone or in combination with other computing environments to provide an AI ubiquitous experience, where AI permeates a wide range of business and life. For example, AI can play an important role in providing deep learning solutions, deep reinforcement learning solutions, artificial general intelligence solutions, autonomous vehicles, cognitive computing solutions, commercial UAVs or drones, conversational user interfaces, enterprise classifications, ontology management solutions, machine learning solutions, smart dust, smart robots, and smart workplaces. The above storage system can also be used alone or in combination with other computing environments to provide a wide range of transparent immersive experiences, where technology can introduce transparency between people, businesses, and things. This transparent immersive experience can be provided as augmented reality technology, connected homes, virtual reality technology, brain-computer interfaces, human augmentation technology, nanotube electronics, volumetric displays, or 4D printing technology. The above storage system can also be used alone or in combination with other computing environments to support various digital platforms. Such digital platforms can include, for example, 5G wireless systems and platforms, digital twin platforms, edge computing platforms, IoT platforms, quantum computing platforms, serverless PaaS, software-defined security, and neuromorphic computing platforms.

[0162] The reader will appreciate that some transparent immersive experiences may involve the use of digital twins of various “things” such as people, places, processes, and systems. Such digital twins and other immersive technologies may change the way humans interact with technology as conversational platforms, augmented reality, virtual reality, and mixed reality provide more natural and immersive interactions with the digital world. In fact, digital twins may be connected to the real world, perhaps even in real time, to understand the state of things or systems and respond to changes, etc. Because digital twins incorporate a wealth of information about individual assets and groups of assets (and may even provide control over those assets), digital twins can communicate with each other to obtain a digital factory model of multiple connected digital twins.

[0163] The above-mentioned storage system can also be part of a multi-cloud environment, in which multiple cloud computing and storage services are deployed in a single heterogeneous architecture. In order to facilitate the operation of this multi-cloud environment, DevOps tools can be deployed to achieve cross-cloud orchestration. Similarly, continuous development and continuous integration tools can be deployed to standardize the processing around continuous integration and delivery, new feature launches, and provision of cloud workloads. By standardizing these processes, a multi-cloud strategy can be implemented, which enables the use of the best provider for each workload. In addition, application monitoring and visibility tools can be deployed to move application workloads around different clouds, identify performance issues, and perform other tasks. In addition, security and compliance tools can also be deployed to ensure compliance with security requirements and government regulations, etc. This multi-cloud environment can also include tools for application delivery and intelligent workload management to ensure efficient application delivery and help guide workloads across distributed and heterogeneous architectures, as well as tools that make the deployment and maintenance of packaged and customized applications in the cloud easy and achieve portability between clouds. A multi-cloud environment can similarly include tools for data portability.

[0164] The above storage system can be used as part of a platform to implement the use of cryptographic anchors that can be used to authenticate the origin and content of a product to ensure that the product matches the blockchain record associated with the product. This cryptographic anchor can take many forms, including, for example, as edible ink, mobile sensors, and microchips. Similarly, as part of a tool kit to protect data stored on the storage system, the above storage system can implement various encryption techniques and schemes, including lattice cryptography. Lattice cryptography can involve constructing cryptographic primitives involving lattices in the construction itself or in security proofs. Unlike public key systems such as RSA, Diffie-Hellman, or elliptic curve cryptography, which are vulnerable to quantum computer attacks, some lattice-based constructions appear to be resistant to attacks by both classical and quantum computers.

[0165] A quantum computer is a device that performs quantum computing. Quantum computing is computing that uses quantum mechanical phenomena such as superposition and entanglement. Quantum computers are different from traditional computers based on transistors because such traditional computers require data to be encoded as binary digits (bits), each of which is always in one of two definite states (0 or 1). In contrast to traditional computers, quantum computers use qubits, which can be in a superposition of states. A quantum computer maintains a sequence of qubits, in which one qubit can represent 1, 0, or any quantum superposition of the states of these two qubits. A qubit pair can be in any quantum superposition of 4 states, and three qubits can be in any superposition of 8 states. A quantum computer with n qubits can generally be in any superposition of up to 2^n different states at the same time, while a traditional computer can only be in one of these states at any one time. A quantum Turing machine is a theoretical model of such a computer.

[0166] The storage system described above can also be paired with an FPGA acceleration server that is part of a larger AI or ML infrastructure. Such an FPGA acceleration server can reside near the storage system described above (e.g., in the same data center), or even be incorporated into a device including one or more storage systems, one or more FPGA acceleration servers, a network infrastructure that supports communication between one or more storage systems and one or more FPGA acceleration servers, and other hardware and software components. Alternatively, the FPGA acceleration server can reside in a cloud computing environment that can be used to perform computation-related tasks for AI and ML jobs. Any of the above embodiments can be used together as an FPGA-based AI or ML platform. The reader will understand that in some embodiments of an FPGA-based AI or ML platform, the FPGA contained in the FPGA acceleration server can be reconfigured for different types of ML models (e.g., LSTM, CNN, GRU). The ability to reconfigure the FPGA contained in the FPGA acceleration server can be based on the optimal numerical accuracy and the memory model being used to achieve acceleration of ML or AI applications. The reader will understand that by treating a collection of FPGA acceleration servers as an FPGA pool, any CPU in a data center can utilize the FPGA pool as a shared hardware microservice, rather than limiting the server to a dedicated accelerator inserted therein.

[0167] The FPGA accelerated servers and GPU accelerated servers described above can implement a computing model in which machine learning models and parameters are fixed to high-bandwidth on-chip memory and a large amount of data is streamed through the high-bandwidth on-chip memory, rather than retaining a small amount of data in the CPU and running a long instruction stream on the CPU as happens in a more traditional computing model. For this computing model, FPGAs may even be more efficient than GPUs because FPGAs can be programmed with only the instructions required to run this computing model.

[0168] The above-mentioned storage system can be configured to provide parallel storage, for example, by using a parallel file system such as BeeGFS. Such a parallel file system may include a distributed metadata architecture. For example, a parallel file system may include a plurality of metadata servers distributed with metadata, and components comprising services for clients and storage servers. By using a parallel file system, file content may be distributed on a plurality of storage servers using striping at the directory level, and metadata may be distributed on a plurality of metadata servers, wherein each server stores a portion of a complete file system tree. The reader will appreciate that, in some embodiments, the storage server and the metadata server may run in user space on an existing local file system. In addition, client services, metadata servers, or hardware servers do not require dedicated hardware, because metadata servers, storage servers, and even client services may run on the same machine.

[0169] The reader will appreciate that, in part due to the emergence of many of the above technologies, including mobile devices, cloud services, social networks, and big data analytics, an information technology platform may be needed to integrate all of these technologies and drive new business opportunities by rapidly delivering revenue generation (generating products, services, and experiences) rather than just providing technologies to automate internal business processes. An information technology organization may need to balance the resources and investments required to keep core legacy systems up and running, while also integrating technologies to build an information technology platform that can provide speed and flexibility in areas such as developing big data, managing unstructured data, and working with cloud applications and services. One possible embodiment of such an information technology platform is a composable infrastructure that includes a fluid resource pool, such as the many systems described above that can meet the changing needs of applications by allowing the combination and recombination of blocks of disaggregated computing, storage, and structural infrastructure. Such a composable infrastructure may also include a single management interface to eliminate complexity, and a unified API to discover, search, inventory, configure, provision, update, and diagnose the composable infrastructure.

[0170] The above-mentioned system can support the execution of a wide range of software applications. Such software applications can be deployed in a variety of ways (including container-based deployment models). Containerized applications can be managed using various tools. For example, containerized applications can be managed using Docker Swarm (a kind of aggregation and scheduling tool used by Docker containers, which enables IT administrators and developers to establish and manage clusters of Docker nodes as a single virtual system). Similarly, containerized applications can be managed by using Kubernetes (a container orchestration system for automating the deployment, expansion and management of containerized applications). Kubernetes can be executed on, for example, operating systems (such as Red Hat Enterprise Linux, Ubuntu Server, and SUSE Linux Enterprise Server). In these examples, the master node can assign tasks to workers / subordinate nodes. Kubernetes can include a group of components (e.g., kubelet, kube-proxy, cAdvisor) that manage a single node, and a group of components (e.g., etcd, API server, scheduler, control manager) that form a control plane. Various controllers (e.g., replication controllers, daemon set controllers) can drive the state of a Kubernetes cluster by managing a set of pods including one or more containers deployed on a single node. Containerized applications can be used to facilitate serverless, cloud native computing deployment and management models for software applications. To support serverless, cloud native computing deployment and management models for software applications, containers can be used as part of an event handling mechanism (e.g., AWS Lambdas), so that various events cause containerized applications to spin up to operate as event handlers.

[0171] The above-mentioned system can be deployed in various ways, including in a way to support the fifth generation (5G) network. 5G networks may support much faster data communications than previous generations of mobile communication networks, which may result in the decomposition of data and computing resources, because modern large-scale data centers may become less prominent and may be replaced by more local micro data centers close to mobile network towers. The above-mentioned system may be included in such a local micro data center, and may be part of or paired with a multi-access edge computing (MEC) system. Such a MEC system can enable cloud computing capabilities and IT service environments at the edge of the cellular network. By running applications and performing related processing tasks closer to cellular customers, network congestion may be reduced and applications may perform better. MEC technology is designed to be implemented at cellular base stations or other edge nodes, and can flexibly and quickly deploy new applications and services for customers. MEC can also allow cellular operators to open their radio access networks (RAN) to authorized third parties (such as application developers and content providers). In addition, edge computing and micro data centers can greatly reduce the cost of smartphones that work with 5G networks, because customers may not need devices with such intensive processing power and expensive necessary components.

[0172] The reader will appreciate that 5G networks are likely to generate more data than previous generations of networks, particularly given that the high network bandwidth provided by 5G networks may enable 5G networks to process volumes and types of data that were not feasible for previous generations of networks (e.g., sensor data from self-driving cars, data generated using AR / VR technology). In these examples, the scalability provided by the above system may be very valuable due to increased data volumes and increased adoption of emerging technologies, etc.

[0173] To further illustrate, Figure 3C An exemplary computing device 350 is illustrated that may be specifically configured to perform one or more of the processes described herein. Figure 3C As shown, computing device 350 may include a communication interface 352, a processor 354, a storage device 356, and an input / output (I / O) module 358 that are communicatively coupled to each other via a communication infrastructure 360. Figure 3C An exemplary computing device 350 is shown in FIG. Figure 3C The illustrated components are not intended to be limiting. Additional or alternative components may be used in other embodiments. Figure 3C Components of computing device 350 are shown.

[0174] The communication interface 352 may be configured to communicate with one or more computing devices. Examples of the communication interface 352 include, but are not limited to, a wired network interface (such as a network interface card, etc.), a wireless network interface (such as a wireless network interface card, etc.), a modem, an audio / video connection, and any other suitable interface. The processor 354 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 354 may operate by executing computer-executable instructions 362 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 356.

[0175] Storage device 356 may include one or more data storage media, devices or configurations, and may employ any type, form and combination of data storage media and / or devices. For example, storage device 356 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data including data described herein may be temporarily and / or permanently stored in storage device 356. For example, data representing computer executable instructions 362 configured to direct processor 354 to perform any operation described herein may be stored in storage device 356. In some examples, the data may be arranged in one or more databases residing in storage device 356.

[0176] The I / O module 358 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 358 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 358 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0177] The I / O module 358 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O module 358 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation. In some examples, any system, computing device, and / or other components described herein may be implemented by a computing device 350.

[0178] To further illustrate, Figure 3DA block diagram illustrating multiple storage systems (311-402, 311-404, 311-406) supporting pods according to some embodiments of the present invention is set forth. Although not depicted in detail, Figure 3D The storage system (311-402, 311-404, 311-406) depicted can be compared with the above reference Figures 1A to 1D , Figures 2A to 2G , Figure 3A-3B or any combination thereof. Figure 3D The depicted storage systems (311-402, 311-404, 311-406) may include the same, fewer, or additional components as compared to the storage systems described above.

[0179] exist Figure 3D In the depicted example, the storage systems (311-402, 311-404, 311-406) are each depicted as having at least one computer processor (311-408, 311-410, 311-412), computer memory (311-414, 311-416, 311-418), and computer storage (311-420, 311-422, 311-424). Although in some embodiments, computer memory (311-414, 311-416, 311-418) and computer storage (311-420, 311-422, 311-424) may be part of the same hardware device, in other embodiments, computer memory (311-414, 311-416, 311-418) and computer storage (311-420, 311-422, 311-424) may be part of different hardware devices. In this particular example, the distinction between computer memory (311-414, 311-416, 311-418) and computer storage (311-420, 311-422, 311-424) may be that the computer memory (311-414, 311-416, 311-418) is physically proximate to the computer processor (311-408, 311-410, 311-412) and may store computer program instructions executed by the computer processor (311-408, 311-410, 311-412), whereas the computer storage (311-420, 311-422, 311-424) is embodied as non-volatile storage for storing user data and metadata describing the user data, etc. Referring to FIG. Figure 1AIn the example, for example, the computer processors (311-408, 311-410, 311-412) and computer memories (311-414, 311-416, 311-418) of a particular storage system (311-402, 311-404, 311-406) may reside within one or more of the controllers (110A-110D), while additional storage devices (171A-171F) may be used as computer storage (311-420, 311-422, 311-424) within the particular storage system (311-402, 311-404, 311-406).

[0180] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may be attached to one or more pods (311-430, 311-432) in accordance with some embodiments of the invention. Figure 3D The depicted pods (311-430, 311-432) may each include a data set (311-426, 311-428). For example, a first pod (311-430) to which three storage systems (311-402, 311-404, 311-406) have been attached includes a first data set (311-426), while a second pod (311-432) to which two storage systems (311-404, 311-406) have been attached includes a second data set (311-428). In such an example, when a particular storage system is attached to a pod, the pod's data set is copied to the particular storage system and then kept up to date as the data set is modified. A storage system may be removed from a pod, resulting in the data set no longer being kept up to date on the removed storage system. In Figure 3D In the depicted example, any storage system that is active for a pod (that is an up-to-date, operational, non-failed component of a non-failed pod) can receive and process requests to modify or read a dataset for the pod.

[0181] exist Figure 3DIn the depicted example, each pod (311-430, 311-432) may also include a set of managed objects and management operations, as well as a set of access operations to modify or read a data set (311-426, 311-428) associated with a particular pod (311-430, 311-432). In such an example, the management operations may modify or query the managed objects equivalently through any storage system. Likewise, the access operations to read or modify the data set may operate equivalently through any storage system. In such an example, while each storage system stores a separate copy of the data set as an appropriate subset of the data set stored and advertised for use by that storage system, the operations to modify the managed objects or data set performed and completed by either storage system are reflected in subsequent management object queries to the pod or subsequent access operations to read the data set.

[0182] The reader will appreciate that pods can enable more capabilities than just a cluster synchronously replicating a data set. For example, pods can be used to implement tenants whereby data sets are securely separated from each other in some way. Pods can also be used to implement a virtual array or virtual storage system where each pod is presented as a unique storage entity on a network (e.g., a storage area network or an Internet Protocol network) with a separate address. In the case of multiple storage system pods that implement a virtual storage system, all physical storage systems associated with the pods can present themselves as the same storage system in some way (e.g., as if multiple physical storage systems were no different from multiple network ports of a single storage system).

[0183] The reader will appreciate that a pod can also be a management unit representing a collection of volumes, file systems, object / analytic storage, snapshots, and other management entities, where management changes made to any one storage system (e.g., name change, property change, export or permission of a portion of the management pod's data set) are automatically reflected to all active storage systems associated with the pod. Additionally, a pod can also be a data collection and data analysis unit, where performance and capacity metrics are presented in a way that is aggregated across all active storage systems of the pod or calls for data collection and analysis for each pod separately, or perhaps for each pod, the contribution of each attached storage system to incoming content and performance.

[0184] One model of pod membership can be defined as a list of storage systems and subsets of that list where a storage system is considered to be in sync with a pod. A storage system can be considered to be in sync with a pod if it is within recovery that has the same idle content as the last written copy of the data set associated with the pod. The idle content is the content after any ongoing modifications have been completed without having to process new modifications. This is sometimes referred to as "crash-recoverable" consistency. Recovery of a pod performs processing to reconcile differences in applying concurrent updates to the in-sync storage systems in the pod. Recovery can resolve any inconsistencies between storage systems upon completion of concurrent modifications that have been requested for individual members of the pod but not signaled to any requester as having completed successfully. A storage system that is listed as a pod member but not listed as in sync with a pod can be described as "detached" from the pod. A storage system that is listed as a pod member, in sync with a pod, and currently available to actively serve data for the pod is "online" to the pod.

[0185] Each storage system member of a pod may have its own copy of membership, including which storage systems it last knew were in sync, and which storage systems it last knew comprised the entire pod membership set. In order to be online for the pod, a storage system must consider itself to be in sync for the pod, and must communicate with all other storage systems it considers to be in sync for the pod. If a storage system cannot determine that it is in sync and communicating with all other storage systems that are in sync, then the storage system must stop processing new incoming requests for the pod (or must complete the request with an error or exception) until the storage system can determine that it is in sync and communicating with all other storage systems that are in sync. A first storage system may conclude that a second paired storage system should detach, which would allow the first storage system to continue because it is now in sync with all storage systems currently in the list. However, the second storage system must be prevented from alternatively concluding that the first storage system should detach and the second storage system continues to operate. This would result in a "split brain" condition, which could result in the risk of irreconcilable data sets, data set corruption, or application corruption. Situations where it may be necessary to determine how to proceed without communicating with a paired storage system may occur when a storage system is operating normally and then notices a loss of communications, when a storage system is currently recovering from some prior failure, when a storage system is restarting or resuming from a temporary power loss or interruption in recovery communications, when a storage system is switching operations from one storage system controller set to another storage system controller set for whatever reason, or during or after any combination of these or other kinds of events. In fact, whenever a storage system associated with a pod cannot communicate with all known non-detached members, the storage system can wait briefly until communications can be established, go offline and continue to wait, or the storage system can determine in some way that it is safe to detach the non-communicating storage system without the risk of a split-brain resulting from the non-communicating storage system coming to another point of view, and then continue. If a safe detach can be performed quickly enough, the storage system can remain online to the pod with only a slight delay and without causing application interruption to applications that can make requests to the storage system that remains online.

[0186] An example of such a situation is where a storage system may know that it is out of date. This may occur, for example, when a first storage system is first added to a pod that is already associated with one or more storage systems, or when the first storage system is reconnected to another storage system and discovers that the other storage system has marked the first storage system as detached. In this case, the first storage system simply waits until it is connected to some other set of storage systems that are synchronized with the pod.

[0187] The model needs to account to some extent for how storage systems are added to or removed from a pod or a synced pod member list. Since each storage system will have its own copy of the list, and since two independent storage systems cannot update their local copies at exactly the same time, and since local copies are available on restart or in various failure scenarios, care must be taken to ensure that transient inconsistencies do not cause problems. For example, if one storage system is synced with respect to a pod, and a second storage system is added, then if the second storage system is updated to first list the two storage systems as synced, and then if the two storage systems fail and restart, the second storage system may start and wait to connect to the first storage system, and the first storage system may not know that it should or can wait for the second storage system. If the second storage system responds to the inability to connect to the first storage system by detaching the processing of the first storage system, it may successfully complete a process that the first storage system is not aware of, resulting in a split brain. Because of this, it may be necessary to ensure that storage systems do not inappropriately disagree about whether storage systems can choose to detach processing without communicating.

[0188] One way to ensure that storage systems do not inappropriately disagree about whether storage systems can choose to detach without communicating is to ensure that when a new storage system is added to a pod's list of in-sync members, the new storage system is first stored as being a detached member (perhaps it was added as a in-sync member). Existing in-sync storage systems can then store locally that the new storage system is a member of the in-sync pod, and then the new storage system stores that fact locally. If there is a set of reboots or network outages before the new storage system stores its in-sync status, the original storage system may have detached the new storage system due to not communicating, but the new storage system will wait. Removing a communicating storage system from a pod may require a reverse version of this change: first, the storage system being removed is stored as no longer in sync, then the storage system will keep storing the storage system being removed as no longer in sync, and then all storage systems remove the storage system being removed from their pod membership lists. Depending on the implementation, the intermediate persistent detached state may not be required. Whether caution is required in local copies of membership lists may depend on the model storage systems use for mutual monitoring or for verifying their membership. If a consensus model is used for both, or if an external system (or external distributed or clustered system) is used to store and verify pod membership, inconsistencies in the locally stored membership list may not matter. In the event of a communication failure or failure of one or more storage systems in a pod, or when a storage system starts (or fails over to a secondary controller) and cannot communicate with the pod's paired storage system, and one or more storage systems decide that it is time to separate one or more paired storage systems, some algorithm or mechanism must be used to decide that it is safe to do so and complete the separation. One means of resolving separation is to use a majority (or quorum) model for membership. In the case of three storage systems, as long as two are communicating, they can agree to separate the third storage system that is not communicating, but the third storage system cannot separate the other two storage systems by itself. Confusion may arise when storage systems communicate inconsistently. For example, storage system A may be communicating with storage system B but not C, while storage system B may be communicating with both A and C. Therefore, A and B can separate C, or B and C can separate A, but more communication between pod members may be required to resolve this issue. When adding and removing storage systems, the swarm membership model needs to be considered. For example, if a fourth storage system is added, the "majority" among the storage systems is now 3. Transitioning from three storage systems (requiring two storage systems for a majority) to a pod that includes a fourth storage system (requiring three storage systems for a majority) may require something similar to the model described previously to carefully add storage systems to the sync list. For example, a fourth storage system may start in an attached state, but not yet attached, in which case that fourth storage system will not cause a vote for the swarm.Once in this state, each of the original three pod members can be updated to be aware of the fourth member and the new requirement for a majority of the three storage systems to detach the fourth. Removing a storage system from a pod may also move the storage system to a locally stored "detached" state before updating the other pod members. A variation on this is to use a distributed consensus mechanism such as PAXOS or RAFT to implement any membership changes or handle detach requests. Another means of managing membership transitions is to use an external system outside the storage system itself to handle pod membership. In order to become online for the pod, the storage system must first contact the external pod membership system to verify that it is synchronized with the pod. Any storage system that is online for the pod should then maintain communication with the pod membership system and should wait or go offline in the event of a loss of communication. The external pod membership manager can be implemented as a highly available cluster using a variety of cluster tools such as Oracle RAC, Linux HA, VERITAS Cluster Server, or IBM's HACMP. An external pod membership manager can also use a distributed configuration tool such as Etcd or Zookeeper, or a reliable distributed database such as Amazon's DynamoDB.

[0189] exist Figure 3DIn the depicted example, the depicted storage system (311-402, 311-404, 311-406) may receive a request to read a portion of a data set (311-426, 311-428) and locally process the request to read the portion of the data set, according to some embodiments of the invention. The reader will appreciate that, while a request to modify (e.g., a write operation) a data set (311-426, 311-428) requires coordination among the storage systems (311-402, 311-404, 311-406) in the pod because the data set (311-426, 311-428) should be consistent across all storage systems (311-402, 311-404, 311-406) in the pod, responding to a request to read a portion of a data set (311-426, 311-428) does not require similar coordination among the storage systems (311-402, 311-404, 311-406). As such, a particular storage system receiving a read request can locally service the read request by reading a portion of the data set (311-426, 311-428) stored within the storage system's storage device without synchronous communication with other storage systems in the pod. Read requests received by a storage system for a replicated data set in a replicated cluster are expected to avoid any communication in the vast majority of cases (at least when received by a storage system running within the cluster, where the cluster is nominally running). Such reads are typically processed simply by reading from the local copy of the cluster data set, without further interaction with other storage systems in the cluster.

[0190] The reader will understand that the storage system can take steps to ensure read consistency so that a read request will return the same result regardless of which storage system processes the read request. For example, the contents of a cluster data set produced by any set of updates received by any set of storage systems in the cluster should remain consistent across the cluster, at least at any time when the updates are idle (all previous modification operations have been indicated as completed and no new update requests are received and processed in any way). More specifically, instances of a cluster data set across a set of storage systems can only differ by updates that have not yet completed. This means, for example, that any two write requests that overlap within a volume block range, or any combination of write requests and overlapping snapshots, compare writes, or virtual block range copies must produce consistent results on all copies of the data set. Two operations should not produce the same result as if they occurred in one order on one storage system in the replicated cluster and in a different order on another storage system.

[0191] In addition, read requests can be time-ordered. For example, if a read request is received and completed on a replicated cluster, and the read request is followed by another read request for an overlapping address range received by the replicated cluster, and one or both of these reads overlap in time and volume address range with a modification request received by the replicated cluster (regardless of whether either the read or the modification is received by the same storage system or a different storage system in the replicated cluster), then if the first read reflects the result of the update, the second read should also reflect the result of the update, rather than returning the data before the update. If the first read does not reflect the update, the second read may reflect the update, or may not reflect the update. This ensures that the "time" of the data segment cannot roll back between the two read requests.

[0192] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) can also detect an interruption in data communications with one or more other storage systems and determine whether a particular storage system should remain in the pod. An interruption in data communications with one or more other storage systems may occur for various reasons. For example, an interruption in data communications with one or more other storage systems may occur because one of the storage systems fails, because a network interconnect fails, or because of some other reason. An important aspect of a synchronous replication cluster is to ensure that any failure handling does not result in unrecoverable inconsistencies or any inconsistencies in responses. For example, if the network between two storage systems fails, at most one of the storage systems can continue to process new incoming I / O requests for the pod. And if one storage system continues to process, the other storage system cannot process any new requests (including read requests) to be completed.

[0193] exist Figure 3DIn the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also determine whether a particular storage system should remain in a pod in response to detecting an interruption in data communications with one or more other storage systems. As described above, in order to be "online" as part of a pod, a storage system must consider itself to be in sync with the pod and must communicate with all other storage systems that it considers to be in sync with the pod. If a storage system cannot determine that it is in sync and is communicating with all other storage systems that are in sync, the storage system may stop processing new incoming requests to access data sets (311-426, 311-428). As such, a storage system may determine whether a particular storage system should remain online as part of a pod, for example, by determining whether it can communicate with all other storage systems it considers to be in sync with the pod (e.g., via one or more test messages), by determining that all other storage systems it considers to be in sync with the pod also consider the storage system to be attached to the pod, by a combination of both steps in which a particular storage system must confirm that it can communicate with all other storage systems it considers to be in sync with the pod and that all other storage systems it considers to be in sync with the pod also consider the storage system to be attached to the pod, or by some other mechanism.

[0194] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also make data sets on the particular storage system accessible for management and data set operations in response to determining that the particular storage system should remain in a pod. The storage system may make data sets (311-426, 311-428) on the particular storage system accessible for management and data set operations, for example, by accepting requests to access versions of data sets (311-426, 311-428) stored on the storage system and processing those requests, by accepting and processing management operations associated with data sets (311-426, 311-428) issued by a host or an authorized administrator, by accepting and processing management operations associated with data sets (311-426, 311-428) issued by one of the other storage systems, or in some other manner.

[0195] However, in Figure 3DIn the depicted example, the depicted storage systems (311-402, 311-404, 311-406) can render a data set on a particular storage system inaccessible for management and data set operations in response to determining that the particular storage system should not be maintained in a pod. The storage system can render a data set (311-426, 311-428) on a particular storage system inaccessible for management and data set operations, for example, by denying a request to access a version of the data set (311-426, 311-428) stored by the storage system, by denying a management operation associated with the data set (311-426, 311-428) issued by a host or other authorized administrator, by denying a management operation associated with the data set (311-426, 311-428) issued by one of the other storage systems in the pod, or in some other manner.

[0196] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also detect that the disruption in data communications with one or more other storage systems has been repaired and make data sets on the particular storage system accessible for management and data set operations. The storage system may detect that the disruption in data communications with one or more other storage systems has been repaired, for example, by receiving a message from the one or more other storage systems. In response to detecting that the disruption in data communications with the one or more other storage systems has been repaired, the storage system may make data sets (311-426, 311-428) on the particular storage system accessible for management and data set operations once the previously detached storage system is resynchronized with the storage system that remains attached to the pod.

[0197] exist Figure 3DIn the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also be offline from the pod, such that management and data set operations are no longer allowed for the particular storage system. The depicted storage systems (311-402, 311-404, 311-406) may be offline from the pod, such that management and data set operations are no longer allowed for the particular storage system, for various reasons. For example, the depicted storage systems (311-402, 311-404, 311-406) may also be offline due to some failure of the storage system itself, due to updates or other maintenance being performed on the storage system, due to communication failures, or for many other reasons. In such an example, the depicted storage systems (311-402, 311-404, 311-406) can subsequently update the data set on the particular storage system to include all updates to the data set since the particular storage system was offline and online again relative to the pod, such that the particular storage system enables management and data set operations, as described in more detail in the resynchronization section included below.

[0198] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also identify a target storage system for asynchronously receiving a data set, wherein the target storage system is not one of the multiple storage systems to which the data set is synchronously replicated. Such a target storage system may, for example, represent a backup storage system, a storage system that utilizes synchronous replication of a data set, and the like. In fact, synchronous replication may be utilized to distribute copies of a data set closer to certain server racks to obtain better local read performance. One such scenario is where smaller top-of-rack storage systems are symmetrically replicated to larger storage systems that are centrally located in a data center or campus, and where these larger storage systems are more carefully managed for reliability, or connected to external networks for asynchronous replication or backup services.

[0199] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also identify a portion of the data set that is not being asynchronously replicated by any other storage system to the target storage system, and asynchronously replicate the portion of the data set that is not being asynchronously replicated by any other storage system to the target storage system, wherein two or more storage systems together replicate the entire data set to the target storage system. In this manner, the work associated with asynchronously replicating a particular data set may be split among the members of the pod such that each storage system in the pod is only responsible for asynchronously replicating a subset of the data set to the target storage system.

[0200] exist Figure 3DIn the depicted example, the depicted storage systems (311-402, 311-404, 311-406) may also be detached from the pod, such that the particular storage system detached from the pod is no longer included in the set of storage systems for synchronous replication of the data set. Figure 3D Storage system (311-404) from Figure 3D If the pod (311-430) shown is detached, the pod (311-430) will only include the storage system (311-402, 311-406) as the storage system that will synchronously replicate the data set (311-426) included in the pod (311-430). In such an example, detaching a storage system from a pod can also include removing the data set from the specific storage system detached from the pod. Figure 3D Storage system (311-404) from Figure 4 In the example of pod (311-430) separation shown, the data set (311-426) included in the pod (311-430) may be deleted or otherwise removed from the storage system (311-404).

[0201] The reader will appreciate that a pod model that can be further supported can implement multiple unique management capabilities. In addition, the pod model itself introduces some problems that can be solved by the implementation. For example, when a storage system is offline for a pod (such as due to an interconnect failure and another storage system of the pod wins in the mediation), but is otherwise running, it may still be desirable or necessary to access the offline pod's data set on the offline storage system. One solution may be to simply enable the pod in some separation mode and allow access to the data set. However, this solution may be dangerous and may make the metadata and data of the pod more difficult to mediate when the storage system regains communication. In addition, there may still be separate paths for the host to access the offline storage system and the storage system that is still online. In this case, the host can issue I / O to the two storage systems even if they are no longer synchronized because the host sees the target port reporting a volume with the same identifier and the host I / O driver assumes that the host sees other paths to the same volume. This can cause quite serious data corruption because the reads and writes issued to the two storage systems are no longer consistent, even if the host assumes that they are consistent. As a variation of this, in a cluster application (such as a shared storage cluster database, etc.), the cluster application running on one host may be reading or writing to one storage system, and the same cluster application running on another host may be reading or writing to a "separate" storage system, but the two instances of the cluster application are communicating with each other based on the assumption that the data sets they each see are completely consistent for completed writes. Since they are inconsistent, this assumption is violated, and the application's data set (e.g., database) may eventually become corrupted very quickly.

[0202] One way to address both of these issues is to allow an offline pod, or a snapshot of an offline pod, to be copied to a new pod with a new volume, where the new volume has a new identity sufficient so that the host I / O drivers and cluster applications do not confuse the copied volume as the same as a volume that is still online on another storage system. Since each pod maintains a complete copy of the data set (which is crash consistent, but may be slightly different from the copy of the pod's data set on another storage system), and since each pod has an independent copy of all data and metadata required to operate on the pod's contents, it is a simple matter of making a virtual copy of some or all volumes or snapshots in a pod a new volume in the new pod. For example, in a logical zone map implementation, all that is required is to define a new volume in the new pod, where the new volume references the logical zone map in the copy pod associated with the pod's volume or snapshot, and the logical zone map is marked as a copy when written. Similar to how a volume snapshot copied to a new volume can be implemented, the new volume should be treated as a new volume. The volume may have the same administrative name, but in the new pod namespace. However, they should have different base identifiers and different logical unit identifiers than the original volumes.

[0203] In some cases, virtual network isolation techniques can be used in a manner such as by creating a virtual LAN in the case of an IP network or a virtual SAN in the case of a Fibre Channel network: Isolation of volumes presented to some interface can be ensured to be inaccessible from host network interfaces or host SCSI initiator ports that might also see the original volume. In such cases, it may be safe to provide a volume copy with the same SCSI or other storage identifier as the original volume. This can be used, for example, in situations where an application expects to see a specific set of storage identifiers so as not to run under excessive load when reconfigured.

[0204] Some of the techniques described here can also be used outside of the context of an active failure to test readiness to handle a failure. Disaster recovery configurations often require readiness testing (sometimes called "fire drills"), where frequent and repeated testing is considered necessary to ensure that most or all aspects of the disaster recovery plan are correct and take into account any recent changes to applications, data sets, or changes to equipment. Readiness testing should be non-intrusive to current production operations (including replication). In many cases, actual operations cannot actually be invoked on an active configuration, but a good way to approach is to use storage operations to copy production data sets, and then perhaps couple this with the use of virtual networks to create an isolated environment containing all the data that is considered necessary for important applications that must be successfully established in the event of a disaster. Making such a copy of a synchronously replicated (or even asynchronously replicated) data set available within the site (or collection of sites) where the disaster recovery readiness testing process is expected to take place, and then starting important applications against that data set to ensure that it can be up and running is a powerful tool because it helps to ensure that important parts of the application data set are not left out of the disaster recovery plan. If necessary and practical, this can be coupled with a virtual isolated network (perhaps with an isolated collection of physical or virtual machines) to get as close to a real-world disaster recovery takeover scenario as possible. In fact, copying a pod (or set of pods) to another pod as a point-in-time image of the pod's dataset instantly creates an isolated dataset that contains all of the copied elements and can then be operated on essentially the same as the original pod, and allows it to be isolated from the original pod to a single site (or sites). Moreover, these are fast operations, and they can be torn down and easily repeated, allowing the test to be repeated as frequently as desired.

[0205] Some enhancements can be made to get closer to perfect disaster recovery testing. For example, in conjunction with isolated networks, SCSI logical unit identities or other types of identities can be copied to the target pod so that the test server, virtual machine, and application see the same identity. In addition, the management environment of the server can be configured to respond to requests from a specific set of virtual networks to respond to requests and operations on the original pod name, so the script does not need to use test variations through alternative "test" versions of object names. Further enhancements can be used in the case where the host-side server infrastructure that will be taken over in the event of a disaster takeover can be used during testing. This includes the case where the disaster recovery data center is fully stocked with alternative server infrastructure, where the alternative server infrastructure will generally not be used until the disaster indicates to do so. It also includes the case where the infrastructure can be used for non-critical operations (for example, running analysis on production data, or simply supporting application development or other functions that may be important, but can be stopped when needed to obtain more critical functions). Specifically, host definitions and configurations and the server infrastructure that will use them can be set up because they will be used in actual disaster recovery takeover events and tested as part of disaster recovery takeover testing, where the tested volumes are connected to these host definitions from virtual pod copies used to provide snapshots of data sets. Then, from the perspective of the storage systems involved, these host definitions and configurations used for testing, as well as the volume-to-host connection configurations used during testing, can be reused when an actual disaster takeover event is triggered, greatly reducing the configuration differences between the test configuration and the actual configuration that would be used in the event of a disaster recovery takeover.

[0206] In some cases, it may make sense to move volumes out of a first pod and into a new second pod that includes only those volumes. Pod membership and high availability and recovery characteristics can then be adjusted individually, and management of the two resulting pod data sets can then be isolated from each other. What can be done in one direction should also be possible in the other direction. To some extent, it makes sense to take two pods and merge them into one so that the volumes in each of the original two pods will track each other's storage system membership and high availability and recovery characteristics and events. Both of these operations can be done safely and with reasonably minimal or no disruption to running applications by relying on the features that are suggested for changing the broker or group properties of the pods discussed in the previous section. For example, through the broker, the broker of a pod can be changed using a sequence of steps that include the following steps, in which each storage system in the pod is changed to depend on both the first broker and the second broker, and then each is changed to depend only on the second broker. If a failure occurs in the middle of the sequence, some storage systems may depend on both the first and second brokers, but in any case recovery and failure handling will not result in some storage systems depending only on the first broker and others depending only on the second broker. The swarm can be handled similarly by temporarily depending on a win against both the first and second swarm models to continue recovery. This may result in the availability of the pod in the face of the failure being dependent on additional resources for a very short period of time, thereby reducing potential availability, but the period is short and the reduction in availability tends to be small. For the broker, if the change in the broker parameters is simply a change in the keys used by the broker, and the brokered service used is the same, the potential reduction in availability is even less, because it now depends only on two calls to the same service versus one call to that service, rather than on separate calls to two separate services.

[0207] The reader will note that changing the swarm model can be quite complex. An additional step may be required where the storage system will participate in the second swarm model, but will not depend on winning in the second swarm model, followed by a step that also depends on the second swarm model. This may be necessary to account for the fact that if only one system processed the change to depend on the swarm model, it would never win in the swarm because there would never be a majority. Utilizing this model to appropriately change the high availability parameters (broker relationships, swarm model, takeover preferences), safe procedures can be created for these operations to split a pod in two or join two pods into one. This may require adding another capability: linking a second pod to a first pod for high availability, such that if the two pods include compatible high availability parameters, the second pod linked to the first pod can depend on the first pod to determine and incentivize processing and operations related to separation, offline state and synchronization state, and recovery and resynchronization actions.

[0208] To split a pod into two (an operation to move some volumes into the newly created pod), a distributed operation can be formed that can be described as forming a second pod into which a set of volumes previously in the first pod are moved, copying high availability parameters from the first pod to the second pod to ensure they are compatible for linking, and linking the second pod to the first pod for high availability. This operation can be encoded as a message and should be implemented by each storage system in the pod in such a way that the storage system ensures that the operation occurs entirely on that storage system, or not at all if processing is interrupted by a failure. Once all in-sync storage systems of both pods have processed this operation, the storage systems can then process subsequent operations that change the second pod so that it is no longer linked to the first pod. As with other changes to the high availability characteristics of a pod, this involves first changing each in-sync storage system to rely on both the previous model (which was high availability linked to the first pod) and the new model (which is now independent high availability itself). In the case of a broker or swarm, this means that the storage system processing such a change will first depend on the broker or swarm being implemented appropriately for the first pod, and will additionally depend on a new separate broker (e.g., a new broker key) or swarm being implemented for the second pod before the second pod can proceed after a failure requiring the broker or test swarm. As previously described with respect to changing the swarm model, an intermediate step may set up the storage system to participate in the swarm of the second pod before the step where the storage system participates in and depends on the swarm of the second pod. Once all synchronized storage systems have processed the change to depend on the new parameters of the broker or swarm for both the first and second pods, the split is complete.

[0209] Incorporating the second pod into the first pod is essentially the reverse. First, the second pod must be adjusted to be compatible with the first pod by having the same list of storage systems and by having compatible high availability models. This may involve a collection of steps, such as those described elsewhere in this document to add or remove storage systems or to change the mediator or group model, etc. Depending on the implementation, it may only be necessary to achieve the same list of storage systems. Incorporation continues by processing operations on each synchronized storage system to link the second pod to the first pod for high availability. Each storage system that processes the operation will then depend on the first pod for high availability and then on the second pod for high availability. Once all synchronized storage systems of the second pod have processed the operation, the storage system will then process subsequent operations to eliminate the link between the second pod and the first pod, migrate volumes from the second pod to the first pod, and delete the second pod. As long as the implementation allows host or application dataset modification or read operations to be correctly directed to the volume by identity, and as long as the identity is maintained as appropriate for the storage protocol or storage model (for example, as long as the logical unit identifier of the volume and the use of the target port used to access the volume are maintained in the case of SCSI), host or application dataset access can be maintained throughout these operations.

[0210] Migrating volumes between pods can be problematic. If the pods have the same set of sync-membership storage systems, this can be straightforward: temporarily suspend operations on the volumes being migrated, switch control of operations on those volumes to the software and fabric controlling the new pod, and then restart operations. This allows for seamless migration, and applications can continue to function normally except for a very brief pause in operations, provided that the network and ports between the pods are migrated correctly. Depending on the implementation, pausing operations may not even be necessary, or may be system-internal so that the pause in operations has no effect. Copying volumes between pods with different sync-memberships is more of a problem. If the target pod of the copy has a subset of the sync members from the source pod, this is not a problem: the member storage systems can be safely discarded without further work. However, if the target pod adds a sync-member storage system to a volume on the source pod, the added storage system must be synchronized to include the contents of the volume before it can be used. Until synchronization occurs, this makes the copied volume significantly different from a synchronized volume, because failure handling is different, and request handling from member storage systems that are not yet synchronized either does not work, must be forwarded, or will not be as fast because reads must cross the interconnect. Additionally, the internal implementation will have to handle some volumes that are synchronizing and preparing for failures, and other volumes that are not synchronizing.

[0211] There are other issues related to operational reliability in the face of failures. Coordinating the migration of volumes between multiple storage system pods is a distributed operation. If pods are the unit of failure handling and recovery, and if brokers or swarms or any means are used to avoid split-brain scenarios, then volumes are switched from one pod to another with a specific set of state and configurations and relationships for failure handling, recovery, brokers, and swarms, then the storage systems in the pods must carefully coordinate changes related to the processing for any volume. Operations cannot be distributed atomically between storage systems, but must be staged in some way. The broker and swarm model essentially provides pods with tools for achieving distributed transaction atomicity, but this may not extend to inter-pod operations without adding to the implementation.

[0212] Even for two pods that share the same first and second storage systems, consider a simple migration of a volume from a first pod to a second pod. To some extent, the storage systems will coordinate to define that the volume is now in the second pod and is no longer in the first pod. If there is no inherent mechanism for transactional atomicity across the storage system of the two pods, then in the event of a network failure that results in a failure handling to detach the storage system from the two pods, the native implementation may leave the volume in the first pod on the first storage system and the second pod on the second storage system. If the pods individually determine which storage system successfully detaches the other storage system, the result may be to detach the other storage system from the same storage system for both pods, in which case the result of the volume migration recovery should be consistent, otherwise it may result in different storage systems for the two pods being detached from each other. If the first storage system detaches the second storage system for the first pod, and the second storage system detaches the first storage system for the second pod, then the recovery may result in the volume being recovered to the first pod on the first storage system and the second pod on the second storage system, and then the volume running on both storage systems and exported to the host and storage applications. If, instead, the first storage system is detached from the second storage system for the first pod, and the second storage system is detached from the first storage system for the second pod, recovery may result in the volume being dropped from the second pod due to the first storage system and from the first pod due to the second storage system, causing the volume to disappear completely. Things become more complicated if the pods between which the volume is migrated are on different sets of storage systems. A solution to these problems may be to use an intermediate pod, and the techniques for splitting and joining pods described previously. The intermediate pod may never be presented as a visible managed object associated with the storage system. In this model, the volume to be moved from the first pod to the second pod is first split from the first pod into a new intermediate pod using the split operation described previously. The storage system membership of the pod can then be adjusted to match the storage system membership by adding or removing storage systems relative to the intermediate pod as needed. The intermediate pod can then be joined with the second pod.

[0213] To further illustrate, Figure 3E A flowchart illustrating an example method of servicing I / O operations for a data set (311-42) synchronized across multiple storage systems (311-38, 311-40) according to some embodiments of the present invention is set forth. Although depicted in less detail, Figure 3E The storage system (311-38, 311-40) described above can be used in conjunction with the above reference Figures 1A to 1D , Figures 2A to 2G , Figure 3A-3B or any combination thereof. Figure 3E The depicted storage system may include the same, fewer, or additional components as compared to the storage system described above.

[0214] Figure 3E The data sets (311-42) depicted in the diagram may be embodied as, for example, the contents of a particular volume, the contents of a particular share of a volume, or any other collection of one or more data elements. The data sets (311-42) may be synchronized across multiple storage systems (311-38, 311-40) such that each storage system (311-38, 311-40) retains a local copy of the data set (311-42). In the examples described herein, such data sets (311-42) are synchronously replicated across the storage systems (311-38, 311-40) in a manner such that the data sets (311-42) may be accessed by any storage system (311-38, 311-40) with the following performance characteristics: such that no one storage system in the cluster operates substantially better than any other storage system in the cluster, at least as long as the cluster and the particular storage system being accessed are nominally operating. In such a system, modifications to a data set (311-42) should be performed on copies of the data set residing on various storage systems (311-38, 311-40) in a manner such that accessing the data set (311-42) on any storage system (311-38, 311-40) will produce consistent results. For example, a write request issued to a data set must be serviced on all storage systems (311-38, 311-40), or not serviced on a storage system (311-38, 311-40) that is nominally operational at the start of the write and remains nominally operational during the completion of the write. Similarly, some groups of operations (e.g., two write operations to the same location within the data set) must be performed in the same order on all storage systems (311-38, 311-40), or other steps, as described in more detail below, must be taken so that the data set is ultimately the same on all storage systems (311-38, 311-40). Modifications to the data set (311-42) need not be performed at exactly the same time, but some actions (e.g., issuing acknowledgments for write requests to the data set, enabling read access to locations within the data set targeted by write requests that have not yet completed on both storage systems) may be delayed until the copies of the data set on each storage system (311-38, 311-40) have been modified.

[0215] exist Figure 3EIn the depicted example method, designating one storage system (311-40) as a "leader" and another storage system (311-38) as a "follower" may be a corresponding relationship of the respective storage systems for the purpose of synchronously replicating a particular data set across the storage systems. In such an example, and as will be described in more detail below, the leader storage system (311-40) may be responsible for performing some processing of incoming I / O operations and passing such information to the follower storage system (311-38) or performing other tasks not required by the follower storage system (311-40). The leader storage system (311-40) may be responsible for performing tasks not required by the follower storage system (311-38) for all incoming I / O operations, or alternatively, the leader-follower relationship may be specific to only a subset of I / O operations received by either storage system. For example, the leader-follower relationship may be specific to I / O operations directed to a first volume, a first set of volumes, a first set of logical addresses, a first set of physical addresses, or some other logical or physical descriptor. In this manner, a first storage system may serve as the leader storage system for I / O operations for a first set of volumes (or other descriptors), while a second storage system may serve as the leader storage system for I / O operations for a second set of volumes (or other descriptors). Figure 3E The depicted example method depicts an embodiment in which synchronization of multiple storage systems (311-38, 311-40) occurs in response to a request (311-04) to modify a data set (311-42) received by a leader storage system (311-40), but synchronization of multiple storage systems (311-38, 311-40) may also be performed in response to a request (311-04) to modify a data set (311-42) received by a follower storage system (311-38), as will be described in more detail below.

[0216] Figure 3E The depicted example method includes receiving (311-06) by a leader storage system (311-40) a request (311-04) to modify a data set (311-42). The request (311-04) to modify a data set (311-42) may be embodied, for example, as a request to write data to a location within the storage system (311-40) that contains data included in the data set (311-42), a request to write data to a volume that contains data included in the data set (311-42), a request to take a snapshot of the data set (311-42), a virtual range copy, an UNMAP operation that essentially represents the deletion of some portion of the data in the data set (311-42), a transformation that modifies the data set (311-42) (rather than changing a portion of the data within the data set), or some other operation that results in a change to some portion of the data included in the data set (311-42). Figure 3EIn the depicted example method, a request (311-04) for modifying a data set (311-42) is issued by a host (311-02), which may be embodied as, for example, an application executing on a virtual machine, an application executing on a computing device connected to a storage system (311-40), or some other entity configured to access the storage system (311-40).

[0217] Figure 3E The depicted example method also includes generating (311-08) information (311-10) describing modifications to a data set (311-42) by the leader storage system (311-40). The leader storage system (311-40) may generate (311-08) information (311-10) describing modifications to a data set (311-42), for example, by determining ordering with any other operations in progress, by determining appropriate results for overlapping modifications (e.g., appropriate results for two requests to modify the same storage location), and by computing any distributed state changes, such as common elements of metadata across all members of a pod (e.g., all storage systems spanned by a synchronously replicated data set), etc. The information (311-10) describing modifications to a data set (311-42) may be embodied as, for example, system-level information describing an I / O operation to be performed by a storage system. The leader storage system (311-40) may generate (311-08) information (311-10) describing the modification to the data set (311-42) by processing the request (311-04) to modify the data set (311-42) just enough to figure out what should happen in order to service the request (311-04) to modify the data set (311-42). For example, the leader storage system (311-40) may determine whether some ordering of the request (311-04) to modify the data set (311-42) needs to be performed relative to other requests to modify the data set (311-42), or some other steps, as described in more detail below, must be taken to produce equivalent results on each storage system (311-38, 311-40).

[0218] Consider an example where the request (311-04) to modify the data set (311-42) is embodied as a request to copy a block from a first address range in the data set (311-42) to a second address range in the data set (311-42). In such an example, assume that three other write operations (write A, write B, write C) are directed to the first address range in the data set (311-42). In such an example, if the leader storage system (311-40) services write A and write B (but not write C) before copying the block from the first address range in the data set (311-42) to the second address range in the data set (311-42), then the follower storage system (311-38) must also service write A and write B (but not write C) before copying the block from the first address range in the data set (311-42) to the second address range in the data set (311-42) to produce a consistent result. As such, when the leader storage system (311-40) generates (311-08) information (311-10) describing a modification to a data set (311-42), in this example, the leader storage system (311-40) can generate information (e.g., sequence numbers for Write A and Write B) that identify other operations that must be completed before the follower storage system (311-38) can process the request (311-04) to modify the data set (311-42).

[0219] Consider an additional example of two requests (e.g., Write A and Write B) for overlapping portions of a data set (311-42). In such an example, if the leader storage system (311-40) services Write A and then services Write B, and the follower storage system (311-38) services Write B and then services Write A, then the data set (311-42) will be inconsistent across the two storage systems (311-38, 311-40). As such, when the leader storage system (311-40) generates (311-08) information (311-10) describing modifications to the data set (311-42), in this example, the leader storage system (311-40) can generate information identifying the order in which the requests should be executed (e.g., sequence numbers for Write A and Write B). Alternatively, rather than generating information (311-10) describing modifications to a data set (311-42) that requires intermediate actions from individual storage systems (311-38, 311-40), the leader storage system (311-40) may generate (311-08) information (311-10) describing modifications to a data set (311-42) that includes information identifying appropriate results for both requests. For example, if write B logically follows (and overlaps) write A, then the end result must be that the data set (311-42) includes the portion of write B that overlaps with write A, rather than the portion of write A that overlaps with write B. Such a result may be facilitated by merging the results in memory and writing such merged results to the data set (311-42), rather than strictly requiring a particular storage system (311-38, 311-40) to perform write A and then subsequently perform write B. The reader will appreciate that more subtle cases involve snapshots and virtual address range copies.

[0220] The reader will further understand that the correct results of any operation must be committed to a recoverable point before the operation can be confirmed. However, multiple operations can be committed together, or operations can be partially committed if recovery will ensure correctness. For example, a snapshot can be committed locally with a recorded dependency on the expected writes of A and B, but A or B themselves may not be committed. If the lost I / O cannot be recovered from the other array, the snapshot cannot be confirmed and the recovery may end in a rollback snapshot. In addition, if write B overlaps with write A, the leader may "order" B to be after A, but A may actually be discarded, and then the operation to write A will only wait for B. Writes A, B, C, and D that are coupled with snapshots between A, B and C, D can be submitted and / or confirmed together in some or all parts, as long as recovery does not result in inconsistent snapshots across arrays, and as long as the later operation is confirmed not to be completed before the earlier operation has been persisted to a point that is guaranteed to be recoverable.

[0221] Figure 3E The depicted example method also includes sending (311-12) information (311-10) describing the modification to the data set (311-42) from the leader storage system (311-40) to the follower storage system (311-38). Sending (311-12) the information (311-10) describing the modification to the data set (311-42) from the leader storage system (311-40) to the follower storage system (311-38) can be performed, for example, by the leader storage system (311-40) sending one or more messages to the follower storage system (311-38). The leader storage system (311-40) can also send an I / O payload (311-14) of the request (311-04) to modify the data set (311-42) in the same message or in one or more different messages. For example, when a request (311-04) to modify a data set (311-42) is embodied as a request to write data to the data set (311-42), the I / O payload (311-14) may be embodied as data to be written to storage within the follower storage system (311-38). In such an example, because the request (311-04) to modify the data set (311-42) is received (311-06) by the leader storage system (311-40), the follower storage system (311-38) has not yet received the I / O payload (311-14) associated with the request (311-04) to modify the data set (311-42). Figure 3E In the depicted example method, information (311-10) describing modifications to a data set (311-42) and an I / O payload (311-14) associated with a request (311-04) to modify the data set (311-42) can be sent (311-12) from a leader storage system (311-40) to a follower storage system (311-38) via one or more data communication networks coupling the leader storage system (311-40) to the follower storage system (311-38), via one or more dedicated data communication links (e.g., a first link for sending the I / O payload and a second link for sending the information describing the modifications to the data set), or via some other mechanism.

[0222] Figure 3EThe depicted example method also includes receiving (311-16), by the follower storage system (311-38), information (311-10) describing the modification to the data set (311-42). The follower storage system (311-38) may receive (311-16) the information (311-10) describing the modification to the data set (311-42) and the I / O payload (311-14) from the leader storage system (311-40), for example, via one or more messages sent from the leader storage system (311-40) to the follower storage system (311-38). One or more messages can be sent from the leader storage system (311-40) to the follower storage system (311-38) via one or more dedicated data communication links between the two storage systems (311-38, 311-40) by the leader storage system (311-40) using RDMA or a similar mechanism or otherwise writing messages to a predetermined memory location (e.g., the location of a queue) on the follower storage system (311-38).

[0223] In one embodiment, a follower storage system (311-38) may receive (311-16) information (311-10) and an I / O payload (311-14) describing modifications to a data set (311-42) from a leader storage system (311-40) using SCSI requests (writes from a sender to a receiver, or reads from a receiver to a sender) as a communication mechanism. In such an embodiment, a SCSI write request is used to encode the information to be sent (which includes any data and metadata), and the SCSI write request may be delivered to a specific pseudo device or through a specifically configured SCSI network, or through any other agreed upon addressing mechanism. Alternatively, the model may also issue a set of open SCSI read requests from a receiver to a sender using a specific device, a specifically configured SCSI network, or other agreed upon mechanism. In response to one or more of these open SCSI requests, the encoded information including the data and metadata will be delivered to the receiver. This model may be implemented over a Fibre Channel SCSI network, which is typically deployed as a "dark fiber" storage network infrastructure between data centers. This model also allows the same network lines to be used for host-to-remote array multipathing and bulk array-to-array communications. Figure 3E The depicted example method also includes processing (311-18), by the follower storage system (311-38), a request (311-04) to modify the data set (311-42). Figure 3EIn the depicted example method, a follower storage system (311-38) can process (311-18) a request (311-04) to modify a data set (311-42) by modifying the contents of one or more storage devices (e.g., NVRAM devices, SSDs, HDDs) included in the follower storage system (311-38) based on information (311-10) describing the modification to the data set (311-42) and an I / O payload (311-14) received from a leader storage system (311-40). Consider the following example, in which the request (311-04) to modify the data set (311-42) is embodied as a write operation to a volume included in the data set (311-42), and the information (311-10) describing the modification to the data set (311-42) indicates that the write operation can only be performed after a previously issued write operation has been processed. In such an example, processing (311-18) a request (311-04) to modify a data set (311-42) can be performed by the follower storage system (311-38) first verifying that a previously issued write operation has been processed on the follower storage system (311-38) and then writing an I / O payload (311-14) associated with the write operation to one or more storage devices included in the follower storage system (311-38). In such an example, for example, when the I / O payload (311-14) has been submitted to persistent storage within the follower storage system (311-38), the request (311-04) to modify the data set (311-42) can be considered to have been completed and successfully processed.

[0224] Figure 3E The depicted example method also includes acknowledging (311-20), by the follower storage system (311-38), to the leader storage system (311-40), completion of the request (311-04) to modify the data set (311-42). Figure 3E In the depicted example method, confirmation (311-20) by a follower storage system (311-38) to a leader storage system (311-40) of completion of a request (311-04) to modify a data set (311-42) may be performed by the follower storage system (311-38) sending a confirmation (311-22) message to the leader storage system (311-40). Such a message may include, for example, information identifying the specific request (311-04) to modify a data set (311-42) that was completed and any additional information useful in the follower storage system (311-38) confirming (311-20) completion of the request (311-04) to modify a data set (311-42). Figure 3EIn the depicted example method, a confirmation (311-22) message is issued by the follower storage system (311-38) to the leader storage system (311-40) to illustrate the completion of the request (311-04) to modify the data set (311-42) to the leader storage system (311-40).

[0225] Figure 3E The depicted example method also includes processing (311-24), by the leader storage system (311-40), a request (311-04) to modify the data set (311-42). Figure 3E In the depicted example method, a leader storage system (311-40) may process (311-24) a request (311-04) to modify a data set (311-42) by modifying the contents of one or more storage devices (e.g., NVRAM devices, SSDs, HDDs) included in the leader storage system (311-40) based on information (311-10) describing the modification to the data set (311-42) and an I / O payload (311-14) received as part of the request (311-04) to modify the data set (311-42). Consider the following example, in which the request (311-04) to modify the data set (311-42) is embodied as a write operation to a volume included in the data set (311-42), and the information (311-10) describing the modification to the data set (311-42) indicates that the write operation can only be performed after a previously issued write operation has been processed. In such an example, processing (311-24) a request (311-04) to modify a data set (311-42) can be performed by the leader storage system (311-40) first verifying that a previously issued write operation has been processed by the leader storage system (311-40) and then writing an I / O payload (311-14) associated with the write operation to one or more storage devices included in the leader storage system (311-40). In such an example, for example, when the I / O payload (311-14) has been submitted to persistent storage within the leader storage system (311-40), the request (311-04) to modify the data set (311-42) can be considered to have been completed and successfully processed.

[0226] Figure 3EThe depicted example method also includes receiving (311-26) an indication from a follower storage system (311-38) that the follower storage system (311-38) has processed the request (311-04) to modify the data set (311-42). In this example, the indication that the follower storage system (311-38) has processed the request (311-04) to modify the data set (311-42) is embodied as a confirmation (311-22) message sent from the follower storage system (311-38) to the leader storage system (311-40). The reader will understand that although many of the above steps are depicted and described as occurring in a particular order, a particular order is not actually required. In fact, because the follower storage system (311-38) and the leader storage system (311-40) are independent storage systems, each storage system can perform some of the above steps in parallel. For example, the follower storage system (311-38) may receive (311-16) information (311-10) describing a modification to a dataset (311-42), process (311-18) the request (311-04) to modify the dataset (311-42), or confirm (311-20) the completion of the request (311-04) to modify the dataset (311-42) before the leader storage system (311-40) has processed (311-24) the request (311-04) to modify the dataset (311-42). Alternatively, the leader storage system (311-40) may have processed (311-24) the request (311-04) to modify the dataset (311-42) before the follower storage system (311-38) has received (311-16) information (311-10) describing the modification to the dataset (311-42), processed (311-18) the request (311-04) to modify the dataset (311-42), or confirmed (311-20) the completion of the request (311-04) to modify the dataset (311-42).

[0227] Figure 3E The depicted example method also includes confirming (311-34), by the leader storage system (311-40), completion of the request (311-04) to modify the data set (311-42). Figure 3E In the depicted example method, confirmation (311-34) of completion of the request (311-04) to modify the data set (311-42) may be performed using one or more confirmation (311-36) messages sent from the leader storage system (311-40) to the host (311-02), or via some other suitable mechanism. Figure 3EIn the depicted example method, the leader storage system (311-40) may determine (311-28) whether a request (311-04) to modify a data set (311-42) has been processed (311-18) by a follower storage system (311-38) before confirming (311-34) completion of the request (311-04) to modify a data set (311-42). The leader storage system (311-40) can determine (311-28) whether the request (311-04) to modify the data set (311-42) has been processed (311-18) by the follower storage system (311-38), for example, by determining whether the leader storage system (311-40) has received a confirmation message or other message from the follower storage system (311-38) indicating that the request (311-04) to modify the data set (311-42) has been processed (311-18) by the follower storage system (311-38). In such an example, if the leader storage system (311-40) affirmatively (311-30) determines that the request (311-04) to modify the dataset (311-42) has been processed (311-18) by the follower storage system (311-38) and also processed (311-24) by the leader storage system (311-38), the leader storage system (311-40) can proceed by confirming (311-34) the completion of the request (311-04) to modify the dataset (311-42) to the host (311-02) that initiated the request (311-04) to modify the dataset (311-42). However, if the leader storage system (311-40) determines that the request (311-04) to modify the data set (311-42) has not yet (311-32) been processed (311-18) by the follower storage system (311-38) or has not yet been processed (311-24) by the leader storage system (311-38), the leader storage system (311-40) may not yet have acknowledged (311-34) the modification of the data set (311-42) to the host (311-02) that initiated the request (311-04) to modify the data set (311-42). -42), this is because when the request (311-04) to modify the dataset (311-42) has been successfully processed on all storage systems (311-38, 311-40) spanning the synchronously replicated dataset (311-42), the leader storage system (311-40) can confirm (311-34) the completion of the request (311-04) to modify the dataset (311-42) to the host (311-02) that initiated the request (311-04) to modify the dataset (311-42).

[0228] The reader will understand that Figure 3EIn the depicted example method, sending (311-12) information (311-10) describing modifications to a data set (311-42) from a leader storage system (311-40) to a follower storage system (311-38), and confirming (311-20) completion of a request (311-04) to modify a data set (311-42) by the follower storage system (311-38) to the leader storage system (311-40) can be performed using a single round-trip message transfer. For example, a single round-trip message transfer can be used by using Fibre Channel as a data interconnect. Typically, the SCSI protocol is used in conjunction with Fibre Channel. Such interconnects are typically provided between data centers because some older replication technologies can be constructed to essentially replicate data as SCSI transactions over a Fibre Channel network. In addition, historically Fibre Channel SCSI infrastructures have had less overhead and lower latency than Ethernet and TCP / IP based networks. In addition, when Fibre Channel is used to connect data centers internally to block storage arrays, the Fibre Channel network can be extended to other data centers so that when a local storage array fails, hosts in one data center can switch to access storage arrays in remote data centers.

[0229] SCSI can be used as a general communication mechanism, even though it is generally designed to be used with block storage protocols for storing and retrieving data in block-oriented volumes (or tapes). For example, a SCSI read (SCSI READ) or a SCSI write (SCSI WRITE) can be used to deliver or retrieve message data between storage controllers in a paired storage system. A typical implementation of a SCSI write requires two message round trips: the SCSI initiator sends a SCSI CDB describing a SCSI write operation, the SCSI target receives the CDB, and the SCSI target sends a "ready to receive" message to the SCSI initiator. The SCSI initiator then sends data to the SCSI target, and when the SCSI write is completed, the SCSI target responds to the SCSI initiator with a successful completion. On the other hand, a SCSI read request requires only one round trip: the SCSI initiator sends a SCSI CDB describing a SCSI read operation, the SCSI target receives the CDB and responds with data, and then responds with a successful completion. As a result, over distance, SCSI reads result in half the distance-related latency compared to SCSI writes. Therefore, it may be faster for a data communication receiver to receive a message using a SCSI read request than for a message sender to send data using a SCSI write request. Using SCSI reads only requires the message sender to operate as a SCSI target and the message receiver to operate as a SCSI initiator. The message receiver can send a certain number of SCSI CDB read requests to any message sender, and the message sender will respond to one of the outstanding CDB read requests when message data is available. Since the SCSI subsystem may time out if a read request is outstanding for too long (e.g., 10 seconds), a read request should be responded to within a few seconds even if there is no message data to send.

[0230] As described in the SCSI stream command standard from the T10 Technical Committee of the International Information Technology Standards Committee, SCSI tape requests support variable response data, which can be more flexible for returning variable-sized message data. The SCSI standard also supports an immediate mode for SCSI write requests, which can allow a single round-trip SCSI write command. The reader will understand that many of the embodiments described below also utilize single round-trip messaging.

[0231] To further illustrate, Figure 4 An example of a cloud-based storage system (403) according to some embodiments of the present invention is described. Figure 4In the depicted example, for example, the cloud-based storage system (403) is created entirely in a cloud computing environment (402) such as Amazon Web Service (AWS), Microsoft Azure, Google Cloud Platform, IBM Cloud, and Oracle Cloud. The cloud-based storage system (403) can be used to provide services similar to those that may be provided by the storage systems described above. For example, the cloud-based storage system (403) can be used to provide block storage services to users of the cloud-based storage system (403), the cloud-based storage system (403) can be used to provide storage services to users of the cloud-based storage system (403) by using solid-state storage, and so on.

[0232] Figure 4 The depicted cloud-based storage system (403) includes two cloud computing instances (404, 406), each of which is used to support the execution of a storage controller application (408, 410). The cloud computing instances (404, 406) may, for example, be embodied as instances of cloud computing resources (e.g., virtual machines) that the cloud computing environment (402) may provide to support the execution of software applications such as the storage controller application (408, 410). In one embodiment, the cloud computing instances (404, 406) may be embodied as Amazon Elastic Compute Cloud (EC2) instances. In such an example, an Amazon Machine Image (AMI) including the storage controller application (408, 410) may be launched to create and configure a virtual machine that can execute the storage controller application (408, 410).

[0233] exist Figure 4 In the example method depicted, the storage controller application (408, 410) may be embodied as a computer program instruction module that, when executed, performs various storage tasks. For example, the storage controller application (408, 410) may be embodied as a computer program instruction module that, when executed, performs the same functions as the controller ( Figure 1A110A, 110B) in the cloud-based storage system (403), such as writing data received from a user of the cloud-based storage system (403) to the cloud-based storage system (403), erasing data from the cloud-based storage system (403), retrieving data from the cloud-based storage system (403) and providing the data to a user of the cloud-based storage system (403), monitoring and reporting disk utilization and performance, performing redundancy operations such as redundant array of independent drives (RAID) or RAID-like data redundancy operations, compressing data, encrypting data, and deduplicating data. The reader will understand that since there are two cloud computing instances (404, 406) each including a storage controller application (408, 410), in some embodiments, one cloud computing instance (404) can operate as a primary controller as described above, while the other cloud computing instance (406) can operate as a secondary controller as described above. In such an example, in order to save costs, the cloud computing instance (404) operating as the primary controller may be deployed on a relatively high-performance and relatively expensive cloud computing instance, while the cloud computing instance (406) operating as the secondary controller may be deployed on a relatively low-performance and relatively cheap cloud computing instance. The reader will understand that Figure 4 The depicted storage controller applications (408, 410) may include the same source code that executes in different cloud computing instances (404, 406).

[0234] Consider an example where the cloud computing environment (402) is embodied as AWS and the cloud computing instances are embodied as EC2 instances. In such an example, AWS provides many types of EC2 instances. For example, AWS provides a set of general-purpose EC2 instances that include different levels of memory and processing power. In such an example, the cloud computing instance (404) operating as a primary controller may be deployed on an instance type having a relatively large amount of memory and processing power, while the cloud computing instance (406) operating as a secondary controller may be deployed on an instance type having a relatively small amount of memory and processing power. In such an example, upon a failover event that switches the roles of the primary and secondary controllers, a double failover may actually be performed such that: 1) a first failover event occurs, wherein the cloud computing instance (406) that was previously operating as a secondary controller begins to operate as a primary controller; and 2) a third cloud computing instance (not shown) that is an instance type having a relatively large amount of memory and processing power is spun up using a copy of the storage controller application, wherein the third cloud computing instance begins to operate as a primary controller, and the cloud computing instance (406) that was initially operating as a secondary controller begins to operate as a secondary controller again. In such an example, the cloud computing instance (404) that was previously operating as the primary controller may be terminated. The reader will appreciate that in alternative embodiments, the cloud computing instance (404) that was operating as the secondary controller after the failover event may continue to operate as the secondary controller, while the cloud computing instance (406) that was operating as the primary controller after the failover event may be terminated once a third cloud computing instance (not shown) assumes the primary role.

[0235] The reader will appreciate that while the above-described embodiments relate to embodiments in which one cloud computing instance (404) operates as a primary controller and a second cloud computing instance (406) operates as a secondary controller, other embodiments are also within the scope of the present invention. For example, each cloud computing instance (404, 406) may operate as a primary controller for a portion of the address space supported by the cloud-based storage system (403), each cloud computing instance (404, 406) may operate as a primary controller that partitions the service of I / O operations for the cloud-based storage system (403) in some other manner, and so on. In fact, in other embodiments where cost savings may take precedence over performance requirements, there may be only a single cloud computing instance that includes a storage controller application. In such an example, a controller failure may require more time to recover because a new cloud computing instance that includes the storage controller application will need to be spun up rather than having the created cloud computing instance assume the role of servicing I / O operations that would otherwise be handled by the failed cloud computing instance.

[0236] Figure 4The depicted cloud-based storage system ( 403 ) includes cloud computing instances ( 424 a , 424 b , 424 n ) with local storage ( 414 , 418 , 422 ). Figure 4 The depicted cloud computing instances ( 424 a , 424 b , 424 n ) may, for example, be embodied as instances of cloud computing resources that the cloud computing environment ( 402 ) may provide to support execution of a software application. Figure 4 The cloud computing instances (424a, 424b, 424n) may be different from the cloud computing instances (404, 406) described above because Figure 4 The cloud computing instances (424a, 424b, 424n) of the storage controller application (408, 410) have local storage (414, 418, 422) resources, while the cloud computing instances (404, 406) supporting the execution of the storage controller application (408, 410) do not need to have local storage resources. The cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) can be embodied, for example, as EC2 M5 instances including one or more SSDs, EC2 R5 instances including one or more SSDs, and EC2 I3 instances including one or more SSDs, etc. In some embodiments, the local storage (414, 418, 422) must be embodied as solid-state storage (e.g., SSD) rather than storage utilizing a hard disk drive.

[0237] exist Figure 4 In the depicted example, cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) can each include a software daemon (412, 416, 420) that, when executed by the cloud computing instances (424a, 424b, 424n), can present themselves to the storage controller application (408, 410) as if the cloud computing instances (424a, 424b, 424n) were physical storage devices (e.g., one or more SSDs). In such an example, the software daemons (412, 416, 420) can include computer program instructions similar to computer program instructions typically included on a storage device so that the storage controller application (408, 410) can send and receive the same commands that the storage controller would send to the storage device. In this manner, the storage controller application (408, 410) can include code that is the same (or substantially the same) as the code that the controller in the storage system described above would execute. In these and similar embodiments, communication between the storage controller application (408, 410) and cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) may utilize iSCSI, NVMe over TCP, messaging, a custom protocol, or some other mechanism.

[0238] exist Figure 4 In the depicted example, each of the cloud computing instances (424a, 424b, 424n) having local storage (414, 418, 422) can also be coupled to block storage (426, 428, 430) provided by the cloud computing environment (402). The block storage (426, 428, 430) provided by the cloud computing environment (402) can be embodied, for example, as Amazon Elastic Block Store (EBS) volumes. For example, a first EBS volume (426) can be coupled to a first cloud computing instance (424a), a second EBS volume (428) can be coupled to a second cloud computing instance (424b), and a third EBS volume (430) can be coupled to a third cloud computing instance (424n). In such an example, the block storage (426, 428, 430) provided by the cloud computing environment (402) can be utilized in a manner similar to how the NVRAM device described above is utilized, because the software daemon (412, 416, 420) (or some other module) executing in a particular cloud computing instance (424a, 424b, 424n) upon receiving a request to write data can initiate the writing of data to its attached EBS volume and the writing of data to its local storage (414, 418, 422) resources. In some optional embodiments, data can only be written to the local storage (414, 418, 422) resources within a particular cloud computing instance (424a, 424b, 424n). In an alternative embodiment, actual RAM on each cloud computing instance (424a, 424b, 424n) with local storage (414, 418, 422) may be used as NVRAM instead of using block storage (426, 428, 430) provided by the cloud computing environment (402) as NVRAM, thereby reducing network utilization costs associated with using EBS volumes as NVRAM.

[0239] exist Figure 4In the depicted example, cloud computing instances (424a, 424b, 424n) having local storage (414, 418, 422) may be utilized by cloud computing instances (404, 406) supporting execution of storage controller applications (408, 410) to service I / O operations for a cloud-based storage system (403). Consider an example in which a first cloud computing instance (404) executing the storage controller application (408) is operating as a primary controller. In such an example, the first cloud computing instance (404) executing the storage controller application (408) may receive a request from a user of the cloud-based storage system (403) to write data to the cloud-based storage system (403) (directly or indirectly via a secondary controller). In such an example, a first cloud computing instance (404) executing a storage controller application (408) may perform various tasks, such as deduplicating data included in the request, compressing data included in the request, and determining where to write the data included in the request, before ultimately sending a request to write a deduplicated, encrypted, or otherwise updatable version of the data to one or more cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422). In some embodiments, any of the cloud computing instances (404, 406) may receive a request to read data from a cloud-based storage system (403) and may ultimately send the request to read the data to one or more cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422).

[0240] The reader will understand that when a cloud computing instance (424a, 424b, 424n) having local storage (414, 418, 422) receives a request to write data, the software daemon (412, 416, 420) or some other computer program instruction module executing on the particular cloud computing instance (424a, 424b, 424n) may be configured not only to write the data to its own local storage (414, 418, 422) resources and any suitable block storage (426, 428, 430) provided by the cloud computing environment (402), but the software daemon (412, 416, 420) or some other computer program instruction module executing on the particular cloud computing instance (424a, 424b, 424n) may also be configured to write the data to a cloud-based object storage (432) attached to the particular cloud computing instance (424a, 424b, 424n). The cloud-based object storage (432) attached to a particular cloud computing instance (424a, 424b, 424n) may be embodied, for example, as an Amazon Simple Storage Service (S3) storage accessible to the particular cloud computing instance (424a, 424b, 424n). In other embodiments, the cloud computing instances (404, 406) each including a storage controller application (408, 410) may initiate the storage of data in the local storage (414, 418, 422) and the cloud-based object storage (432) of the cloud computing instance (424a, 424b, 424n).

[0241] The reader will appreciate that the software daemons (412, 416, 420) or other computer program instruction modules for writing data to block storage (e.g., local storage (414, 418, 422) resources) and also writing data to cloud-based object storage (432) can be executed on dissimilar types of processing units (e.g., different types of cloud computing instances, cloud computing instances containing different processing units). In fact, the software daemons (412, 416, 420) or other computer program instruction modules for writing data to block storage (e.g., local storage (414, 418, 422) resources) and also writing data to cloud-based object storage (432) can be migrated between different types of cloud computing instances as needed.

[0242] The reader will appreciate that, as described above, the cloud-based storage system (403) can be used to provide block storage services to users of the cloud-based storage system (403). Although the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n) can support block-level access, the cloud-based object storage (432) attached to a particular cloud computing instance (424a, 424b, 424n) only supports object-based access. To address this issue, the software daemon (412, 416, 420) or some other computer program instruction module being executed on the particular cloud computing instance (424a, 424b, 424n) can be configured to obtain data blocks, package the data blocks into objects, and write the objects to the cloud-based object storage (432) attached to the particular cloud computing instance (424a, 424b, 424n). Consider an example where data is written in 1MB chunks to local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by cloud computing instances (424a, 424b, 424n). In such an example, assume that a user of the cloud-based storage system (403) issues a request to write data that, after being compressed and deduplicated by the storage controller application (408, 410), results in 5MB of data needing to be written. In such an example, writing data to the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n) is relatively simple as five blocks of 1MB in size are written to the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). In such an example, the software daemon (412, 416, 420) or some other module of computer program instructions being executed on a particular cloud computing instance (424a, 424b, 424n) may be configured to: 1) create a first object including a first 1MB of data and write the first object to a cloud-based object store (432); 2) create a second object including a second 1MB of data and write the second object to the cloud-based object store (432); 3) create a third object including a third 1MB of data and write the third object to the cloud-based object store (432); and so on. As such, in some embodiments, the size of each object written to the cloud-based object store (432) may be the same (or nearly the same). The reader will appreciate that in such an example, metadata associated with the data itself may be included with each object (e.g., the first 1MB of the object is the data, and the remainder is metadata associated with the data).

[0243] The reader will appreciate that cloud-based object storage (432) can be incorporated into the cloud-based storage system (403) to improve the persistence of the cloud-based storage system (403). Continuing with the above example where the cloud computing instances (424a, 424b, 424n) are EC2 instances, the reader will appreciate that EC2 instances are only guaranteed to have 99.9% monthly uptime, and data stored in the local instance storage only persists during the life cycle of the EC2 instance. As such, relying on cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) as the sole source of persistent data storage in the cloud-based storage system (403) may result in a relatively unreliable storage system. Similarly, EBS volumes are designed for 99.999% availability. As such, even relying on EBS as a persistent data store in the cloud-based storage system (403) may result in a less-persistent storage system. However, Amazon S3 is designed to provide 99.9999999999% durability, meaning that a cloud-based storage system (403) that may incorporate S3 into its storage pool may be significantly more durable than other options.

[0244] The reader will appreciate that while a cloud-based storage system (403) that may incorporate S3 into its storage pool is significantly more durable than other options, utilizing S3 as the primary storage pool may result in a storage system with relatively slow response times and relatively long I / O latencies. As such, Figure 4 The depicted cloud-based storage system (403) not only stores data in S3, but the cloud-based storage system (403) also stores data in local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by cloud computing instances (424a, 424b, 424n), so that read operations can be served from the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by cloud computing instances (424a, 424b, 424n), thereby reducing read latency when users of the cloud-based storage system (403) attempt to read data from the cloud-based storage system (403).

[0245] In some embodiments, all data stored by the cloud-based storage system (403) can be stored in both: 1) a cloud-based object store (432), and 2) at least one of local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). In these embodiments, the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n) can effectively operate as a cache that generally includes all data also stored in S3, so that all data reads can be served by the cloud computing instances (424a, 424b, 424n) without requiring the cloud computing instances (424a, 424b, 424n) to access the cloud-based object store (432). However, the reader will appreciate that in other embodiments, all data stored by the cloud-based storage system (403) may be stored in the cloud-based object storage (432), but less data than all data stored by the cloud-based storage system (403) may be stored in at least one of the local storage (414, 418, 422) resources and the block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). In such examples, various policies may be utilized to determine which subset of data stored by the cloud-based storage system (403) should reside in: 1) the cloud-based object storage (432), and 2) at least one of the local storage (414, 418, 422) resources and the block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n).

[0246] As described above, when the cloud computing instances (424a, 424b, 424n) with local storage are embodied as EC2 instances, the cloud computing instances (424a, 424b, 424n) with local storage can only be guaranteed to have a monthly uptime of 99.9%, and the data stored in the local instance storage only persists during the life cycle of each cloud computing instance (424a, 424b, 424n) with local storage. As such, one or more computer program instruction modules (e.g., a monitoring module executing on its own EC2 instance) executing within the cloud-based storage system (403) can be designed to handle failures of one or more cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422). In such an example, the monitoring module can handle the failure of one or more cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) by creating one or more new cloud computing instances with local storage, retrieving data stored on the failed cloud computing instances (424a, 424b, 424n) from the cloud-based object storage (432), and storing the data retrieved from the cloud-based object storage (432) in the local storage on the newly created cloud computing instances. The reader will appreciate that many variations of this process can be implemented.

[0247] Consider an example where all cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) fail. In such an example, the monitoring module can create a new cloud computing instance with local storage, wherein a high-bandwidth instance type is selected that allows for a maximum data transfer rate between the newly created high-bandwidth cloud computing instance with local storage and the cloud-based object storage (432). The reader will appreciate that selecting an instance type that allows for a maximum data transfer rate between the new cloud computing instance and the cloud-based object storage (432) enables the new high-bandwidth cloud computing instance to re-converge with data from the cloud-based object storage (432) as quickly as possible. Once the new high-bandwidth cloud computing instance is re-converged with data from the cloud-based object storage (432), a cheaper low-bandwidth cloud computing instance can be created, data can be migrated to the cheaper low-bandwidth cloud computing instance, and the high-bandwidth cloud computing instance can be terminated.

[0248] The reader will appreciate that in some embodiments, the number of new cloud computing instances created may significantly exceed the number of cloud computing instances required to locally store all data stored by the cloud-based storage system (403). The number of new cloud computing instances created may significantly exceed the number of cloud computing instances required to locally store all data stored by the cloud-based storage system (403) to more quickly pull data from the cloud-based object store (432) to the new cloud computing instances because each new cloud computing instance may retrieve (in parallel) a portion of the data stored by the cloud-based storage system (403). In these embodiments, once the data stored by the cloud-based storage system (403) is pulled into the newly created cloud computing instances, the data may be consolidated into a subset of the newly created cloud computing instances, and the excess newly created cloud computing instances may be terminated.

[0249] Consider an example where 1,000 cloud computing instances are needed to locally store all valid data that users of a cloud-based storage system (403) have written to the cloud-based storage system (403). In such an example, assume that all 1,000 cloud computing instances fail. In such an example, the monitoring module may cause the creation of 100,000 cloud computing instances, each of which is responsible for retrieving a different 1 / 100,000 chunk of the valid data that users of the cloud-based storage system (403) have written to the cloud-based storage system (403) from the cloud-based object storage (432) and locally storing a different chunk of the data set it retrieved. In such an example, because each of the 100,000 cloud computing instances can retrieve data from the cloud-based object storage (432) in parallel, the cache layer can be restored 100 times faster than an embodiment in which the monitoring module only creates 1,000 replacement cloud computing instances. In such an example, over time, data stored locally in 100,000 cloud computing instances may be consolidated into 1,000 cloud computing instances, and the remaining 99,000 cloud computing instances may be terminated.

[0250] The reader will appreciate that various performance aspects of the cloud-based storage system (403) may be monitored (e.g., by a monitoring module executing in an EC2 instance) so that the cloud-based storage system (403) may be scaled up or out as needed. Consider an example in which the monitoring module monitors the performance of the cloud-based storage system (403) via communications with one or more cloud computing instances (404, 406) each used to support execution of a storage controller application (408, 410), via monitoring communications between cloud computing instances (404, 406, 424a, 424b, 424n), via monitoring communications between cloud computing instances (404, 406, 424a, 424b, 424n) and a cloud-based object store (432), or in some other manner. In such an example, assume that the monitoring module determines that the cloud computing instance (404, 406) used to support the execution of the storage controller application (408, 410) is too small and insufficient to service I / O requests issued by users of the cloud-based storage system (403). In such an example, the monitoring module can create a new, more powerful cloud computing instance (e.g., a type of cloud computing instance that includes more processing power, more memory, etc.) that includes the storage controller application, so that the new, more powerful cloud computing instance can begin operating as the primary controller. Similarly, if the monitoring module determines that the cloud computing instance (404, 406) used to support the execution of the storage controller application (408, 410) is too large and costs can be saved by switching to a smaller, less powerful cloud computing instance, the monitoring module can create a new, less powerful (and less expensive) cloud computing instance that includes the storage controller application, so that the new, less powerful cloud computing instance can begin operating as the primary controller.

[0251] Consider an example where the monitoring module determines that the utilization of local storage collectively provided by the cloud computing instances (424a, 424b, 424n) has reached a predetermined utilization threshold (e.g., 95%) as an additional example of dynamically adjusting the size of the cloud-based storage system (403). In such an example, the monitoring module can create additional cloud computing instances with local storage to expand the local storage pool provided by the cloud computing instances. Optionally, the monitoring module can create one or more new cloud computing instances with a larger amount of local storage than the existing cloud computing instances (424a, 424b, 424n), so that data stored in the existing cloud computing instances (424a, 424b, 424n) can be migrated to the one or more new cloud computing instances, and the existing cloud computing instances (424a, 424b, 424n) can be terminated, thereby expanding the local storage pool provided by the cloud computing instances. Similarly, if the local storage pool provided by the cloud computing instances is unnecessarily large, data can be consolidated and some cloud computing instances can be terminated.

[0252] The reader will appreciate that the cloud-based storage system (403) can be automatically sized up or down by the monitoring module applying a set of predetermined rules that may be relatively simple or relatively complex. In fact, the monitoring module may not only take into account the current state of the cloud-based storage system (403), but the monitoring module may also apply predictive policies based on, for example, observed behavior (e.g., from 10 pm until 6 am every day, the storage system is relatively lightly used) and predetermined fingerprints (e.g., each time the virtual desktop infrastructure adds 100 virtual desktops, the number of IOPS for the storage system increases by X). In such an example, the dynamic expansion of the cloud-based storage system (403) can be based on current performance indicators, predicted workloads, and many other factors (including combinations thereof).

[0253] The reader will further appreciate that because the cloud-based storage system (403) can scale dynamically, the cloud-based storage system (403) can operate in an even more dynamic manner. Consider the example of garbage collection. In traditional storage systems, the amount of storage is fixed. Because of this, at some point, the storage system may be forced to perform garbage collection because the amount of available storage has become so limited that the storage system is on the verge of running out of storage. In contrast, the cloud-based storage system (403) described herein can always "add" additional storage (e.g., by adding more cloud computing instances with local storage). Because the cloud-based storage system (403) described herein can always "add" additional storage, the cloud-based storage system (403) can make more intelligent decisions about when to perform garbage collection. For example, the cloud-based storage system (403) can implement a policy that only performs garbage collection when the number of IOPS served by the cloud-based storage system (403) is below a certain level. In some embodiments, given that the size of a cloud-based storage system (403) is not limited in the same way that traditional storage systems are limited, other system-level features (e.g., deduplication, compression) may also be turned off and on in response to system load. The reader will appreciate that embodiments of the present invention address issues with block storage services provided by some cloud computing environments, as some cloud computing environments only allow one cloud computing instance to connect to a block storage volume at a time. For example, in Amazon AWS, only a single EC2 instance can connect to an EBS volume. By using EC2 instances with local storage, embodiments of the present invention may provide multi-connection capabilities, where multiple EC2 instances may be connected to another EC2 instance (driver instance) with local storage. In such embodiments, the driver instance may include software executed within the driver instance that allows the driver instance to support I / O directed to a specific amount from each connected EC2 instance. As such, some embodiments of the present invention may be embodied as a multi-connection block storage service that may not include Figure 4 All components depicted.

[0254] In some embodiments, particularly in embodiments where the cloud-based object storage (432) resource is embodied as Amazon S3, the cloud-based storage system (403) may include one or more modules (e.g., computer program instruction modules executed on an EC2 instance) that are configured to ensure that when the local storage of a particular cloud computing instance is re-merged with data from S3, the appropriate data is actually in S3. This problem occurs largely because S3 implements an eventual consistency model, in which when an existing object is overwritten, reads of the object will eventually (but not necessarily immediately) become consistent and will eventually (but not necessarily immediately) return the rewritten version of the object. To address this problem, in some embodiments of the present invention, objects in S3 are never overwritten. Instead, a traditional "rewrite" would result in the creation of a new object (including an updated version of the data) and the eventual deletion of an old object (including a previous version of the data).

[0255] In some embodiments of the present invention, as part of an attempt to never (or almost never) rewrite an object, when data is written to S3, the resulting object can be marked with a serial number. In some embodiments, these serial numbers can be maintained elsewhere (e.g., in a database) so that at any point in time, the serial number associated with the latest version of a certain data can be known. In this way, it is possible to determine whether S3 has the latest version of a certain data by only reading the serial number associated with the object without actually reading the data from S3. The ability to make this judgment may be particularly important in the event of a crash of a cloud computing instance with local storage, because it is not desirable to re-integrate the local storage of a replacement cloud computing instance with outdated data. In fact, since the cloud-based storage system (403) does not need to access the data to verify its validity, the data can remain encrypted and access fees can be avoided.

[0256] exist Figure 4In the depicted example and as described above, the cloud computing instances (404, 406) used to support the execution of the storage controller application (408, 410) can operate in a primary / secondary configuration, wherein one of the cloud computing instances (404, 406) used to support the execution of the storage controller application (408, 410) is responsible for writing data to the local storage attached to the cloud computing instance (424a, 424b, 424n) with local storage (414, 418, 422). However, in such an example, because each cloud computing instance (404, 406) used to support the execution of the storage controller application (408, 410) can access the cloud computing instance (424a, 424b, 424n) with local storage, both cloud computing instances (404, 406) used to support the execution of the storage controller application (408, 410) can service requests to read data from the cloud-based storage system (403). To further illustrate, Figure 5 An example of an additional cloud-based storage system (502) according to some embodiments of the present invention is described. Figure 5 In the depicted example, for example, the cloud-based storage system (502) is created entirely in a cloud computing environment (402) such as AWS, Microsoft Azure, Google Cloud Platform, IBM Cloud, and Oracle Cloud. The cloud-based storage system (502) can be used to provide services similar to those that may be provided by the storage systems described above. For example, the cloud-based storage system (502) can be used to provide block storage services to users of the cloud-based storage system (502), the cloud-based storage system (502) can be used to provide storage services to users of the cloud-based storage system (502) by using solid-state storage, and so on.

[0257] Figure 5 The depicted cloud-based storage system (502) may be used in conjunction with Figure 4 The depicted cloud-based storage system (403) operates in a somewhat similar manner, as Figure 5 The depicted cloud-based storage system (502) includes a storage controller application (506) executing in a cloud computing instance (504). Figure 5In the depicted example, the cloud computing instance (504) executing the storage controller application (506) is a cloud computing instance (504) having local storage (508). In such an example, data written to the cloud-based storage system (502) can be stored in the local storage (508) of the cloud computing instance (504), and can also be stored in the cloud-based object storage (510) in the same manner as described above using the cloud-based object storage (432). For example, in some embodiments, the storage controller application (506) can be responsible for writing data to the local storage (508) of the cloud computing instance (504), while the software daemon (512) can be responsible for ensuring that the data is written to the cloud-based object storage (510) in the same manner as described above using the cloud-based object storage (432). In other embodiments, the same entity (e.g., the storage controller application) can be responsible for writing data to the local storage (508) of the cloud computing instance (504), and also responsible for ensuring that the data is written to the cloud-based object storage (510) in the same manner as described above using the cloud-based object storage (432).

[0258] The reader will understand that Figure 5 The depicted cloud-based storage system (502) may represent a Figure 4 A cheaper, more robust version of the depicted cloud-based storage system. In yet another alternative embodiment, Figure 5 The depicted cloud-based storage system (502) may include additional cloud computing instances with local storage that support execution of the storage controller application (506) so that if the cloud computing instance (504) executing the storage controller application (506) fails, failover may occur. Likewise, in other embodiments, Figure 5 The depicted cloud-based storage system ( 502 ) may include additional cloud computing instances with local storage to expand the amount of local storage provided by the cloud computing instances in the cloud-based storage system ( 502 ).

[0259] The reader will understand that the above references Figure 4 Many of the failure scenarios described will also apply to Figure 5 The depicted cloud-based storage system (502). Figure 5 The depicted cloud-based storage system (502) can be dynamically scaled up and down in a similar manner as described above. The performance of various system-level tasks can also be determined by Figure 5 The depicted cloud-based storage system ( 502 ) performs in an intelligent manner as described above.

[0260] The reader will appreciate that in an effort to increase the resiliency of the cloud-based storage system described above, various components may be located in different availability zones. For example, a first cloud computing instance that supports the execution of a storage controller application may be located in a first availability zone, while a second cloud computing instance that also supports the execution of a storage controller application may be located in a second availability zone. Similarly, cloud computing instances with local storage may be distributed across multiple availability zones. In fact, in some embodiments, an entire second cloud-based storage system may be created in a different availability zone, wherein the data in the original cloud-based storage system is replicated (synchronously or asynchronously) to the second cloud-based storage system, such that if the entire cloud-based storage system fails, a replacement cloud-based storage system (the second cloud-based storage system) may be introduced in a negligible amount of time.

[0261] The reader will appreciate that the cloud-based storage system described herein can be used as part of a storage system fleet. In fact, the cloud-based storage system described herein can be paired with a locally deployed storage system. In such an example, data stored in the internal storage can be copied (synchronously or asynchronously) to the cloud-based storage system, and vice versa.

[0262] To further illustrate, Figure 6 A flow chart illustrating an example method for servicing I / O operations in a cloud-based storage system (604) is set forth. Although not depicted in detail, Figure 6 The depicted cloud-based storage system ( 604 ) may be similar to the cloud-based storage systems described above, and may be supported by a cloud computing environment ( 602 ).

[0263] Figure 6 The depicted example method includes receiving (606), by a cloud-based storage system (604), a request to write data to the cloud-based storage system (604). For example, the request to write data may be received from an application executing in the cloud computing environment, by a user of the storage system that is communicatively coupled to the cloud computing environment, and otherwise. In such an example, the request may include data to be written to the cloud-based storage system (604). In other embodiments, the request to write data to the cloud-based storage system (604) may occur at boot time when the cloud-based storage system (604) is being booted.

[0264] Figure 6The depicted example method also includes deduplicating the data (608). Data deduplication is a data reduction technique used to eliminate duplicate copies of duplicate data. The cloud-based storage system (604) can deduplicate the data (608), for example, by comparing one or more portions of the data to data already stored in the cloud-based storage system (604), by comparing fingerprints of one or more portions of the data to fingerprints of data already stored in the cloud-based storage system (604), or in other ways. In such examples, duplicate data can be removed and replaced with a reference to an existing copy of the data already stored in the cloud-based storage system (604).

[0265] Figure 6 The depicted example method also includes compressing the data (610). Data compression is a data reduction technique that encodes information using fewer bits than the original representation. The cloud-based storage system (604) can compress the data (610) by applying one or more data compression algorithms to the data (at this point, the data may not include data already stored in the cloud-based storage system (604)).

[0266] Figure 6 The depicted example method also includes encrypting the data (612). Data encryption is a technique that involves converting data from a readable format into an encoded format that can only be read or processed after the data is decrypted. The cloud-based storage system (604) can encrypt the data (which at this point may have been deduplicated and compressed) using an encryption key (612). The reader will understand that although Figure 6 The depicted embodiment involves deduplicating data (608), compressing data (610), and encrypting data (612), but other embodiments exist that perform fewer of these steps, and there are embodiments that perform the same or fewer numbers of steps in a different order.

[0267] Figure 6 The depicted example method also includes storing (614) the data in a block storage of the cloud-based storage system (604). As described in more detail above, storing (614) the data in a block storage of the cloud-based storage system (604) can be performed, for example, by storing (616) the data in solid-state storage, such as local storage (e.g., SSD) of one or more cloud computing instances. In such an example, the data can be dispersed across the local storage of many cloud computing instances along with parity data to implement RAID or RAID-like data redundancy.

[0268] Figure 6The depicted example method also includes storing (618) the data in an object store of the cloud-based storage system (604). Storing (618) the data in the object store of the cloud-based storage system may include creating (620) one or more equal-sized objects, wherein each equal-sized object includes a different chunk of data. In such an example, since each object includes data and metadata, the data portio...

Claims

1. A method for servicing I / O operations in a virtual storage system, the method comprising: receiving, by the virtual storage system, a request to write data to the virtual storage system, the virtual storage system having components residing in a plurality of availability zones; storing the data in storage provided by a first storage tier of the virtual storage system at a first availability zone; as well as Migrating at least a portion of the data stored in the first storage tier from the first storage tier at the first availability zone to a second storage tier at a cloud-based data store provided by a cloud service provider, the second storage tier being more durable than the first storage tier of the virtual storage system, wherein components within two or more of the plurality of availability zones coordinate with each other to synchronously store the replicated data at the cloud-based data store.

2. The method according to claim 1, wherein: The first storage tier includes a hierarchical memory that provides transactional consistency and write confirmation.

3. The method according to claim 1, wherein: The first storage layer includes virtual drivers provided by a virtual driver server of the virtual storage system, and wherein the second storage layer includes object storage provided by the cloud service provider, the cloud service provider providing object storage independent of the virtual storage system.

4. The method according to claim 2, wherein: In response to detecting a condition for transferring data from the staging storage to the virtual drive provided by the cloud service provider, the at least a portion of the data stored in the staging storage is migrated.

5. The method according to claim 2, wherein: The hierarchical storage includes a plurality of virtual drivers provided by a virtual driver server.

6. The method according to claim 5, wherein: The virtual drive server includes corresponding local storage.

7. The method according to claim 5, wherein: The virtual drive server provides a virtual drive as block-based data storage.

8. The method according to claim 1, wherein: A request to write data to the virtual storage system is received by one or more virtual controllers running within a virtual machine, container, or bare metal server.

9. The method according to claim 2, wherein: The hierarchical storage is provided by a plurality of virtual drive servers, each of which includes both a virtual controller and a local storage.

10. The method according to claim 2, wherein: The at least a portion of the data stored in the first storage tier is stored in the hierarchical storage and is deduplicated, encrypted, or compressed prior to being migrated from the hierarchical storage to the second storage tier.

11. The method according to claim 2, wherein: The hierarchical storage of the virtual storage system is characterized by low read latency relative to the second storage tier provided by the cloud service provider.

12. A virtual storage system included in a cloud computing environment, the virtual storage system comprising: a plurality of virtual drives providing hierarchical storage for storage operations, wherein the plurality of virtual drives are arranged in a plurality of availability zones; as well as One or more virtual controllers, each virtual controller is executed in a cloud computing instance, wherein the one or more virtual controllers are configured to: receiving, by the virtual storage system, a request to write data into the virtual storage system; storing the data in a hierarchical storage provided by one or more virtual drives in a first availability zone of the virtual storage system; and Migrate at least a portion of the data stored in the tiered storage from the tiered storage at the first availability zone to a more persistent data storage at a cloud-based data storage provided by a cloud service provider, wherein components within two or more of the multiple availability zones coordinate with each other to synchronously store the replicated data at the cloud-based data storage.

13. The virtual storage system according to claim 12, wherein: Local storage for a given virtual drive server is provided to one or more virtual drives.

14. The virtual storage system according to claim 12, wherein: A first subset of virtual controllers is located in the first availability zone, and wherein a second subset of virtual controllers is located in a second availability zone.

15. The virtual storage system according to claim 12, wherein: A first subset of virtual drives is located in the first availability zone, and wherein a second subset of virtual drives is located in a second availability zone.

16. The virtual storage system according to claim 15, wherein: The cloud-based data storage is an object storage service provided by the cloud service provider.

17. The virtual storage system according to claim 14, wherein: In response to detecting a condition for transferring data from the staging storage to a cloud-based data storage provided by the cloud service provider, the at least a portion of the data stored within the staging storage is migrated.

18. The virtual storage system according to claim 12, wherein: The hierarchical storage includes a plurality of virtual drivers provided by a virtual driver server.

19. The virtual storage system according to claim 12, wherein: The plurality of virtual drives include both corresponding virtual controllers and corresponding local storage.

20. The virtual storage system according to claim 12, wherein: The hierarchical storage is provided by a plurality of virtual drive servers, each of which includes both a virtual controller and a local storage.

Images