Data recovery in virtual storage systems

Abstract

Data recovery in a virtual storage system includes detecting a data loss within a data set within storage provided by a first storage tier of the virtual storage system, wherein recovery data for the data set is stored in a second storage tier, determining a recovery point for the data set from which a consistent version of the data set is recoverable from the recovery data stored in the second storage tier up to the recovery point, and recovering the consistent version of the data set within the storage provided by the first storage tier of the virtual storage system.

Description

Attached Figure Description

[0001] Figure 1A This section presents a first example system based on some implementation of data storage.

[0002] Figure 1B This section presents a second example system based on some implementation of data storage.

[0003] Figure 1C This example demonstrates a third instance system based on some implementation of data storage.

[0004] Figure 1D This example demonstrates a fourth sample system based on some implementation of data storage.

[0005] Figure 2A This is a perspective view of a storage cluster having multiple storage nodes and internal storage coupled to each storage node to provide network-attached storage, according to some embodiments.

[0006] Figure 2B This is a block diagram illustrating an interconnect switch coupling multiple storage nodes according to some embodiments.

[0007] Figure 2C This is a multi-level block diagram illustrating the contents of a storage node and one of the contents of a non-volatile solid-state storage cell according to some embodiments.

[0008] Figure 2D This illustrates a storage server environment using some of the storage nodes and storage units shown in the previous figures, according to some embodiments.

[0009] Figure 2E This is a blade hardware block diagram illustrating the control plane, compute and storage plane, and permissions for interacting with underlying physical resources according to some embodiments.

[0010] Figure 2F A resilient software layer in the blades of a storage cluster according to some embodiments is described.

[0011] Figure 2G The permissions and storage resources in the blades of a storage cluster according to some embodiments are described.

[0012] Figure 3A A diagram illustrating a storage system coupled to communicate data with a cloud service provider according to some embodiments of the present invention is provided.

[0013] Figure 3B A diagram illustrating a storage system according to some embodiments of the present invention is provided.

[0014] Figure 3C Examples of exemplary computing devices may be specifically configured to perform one or more of the processes described herein.

[0015] Figure 3D Block diagrams illustrating several storage systems supporting pods according to some embodiments of the present invention are presented.

[0016] Figure 3E A flowchart illustrating example methods for service I / O operations on datasets synchronized across multiple storage systems, according to some embodiments of the present invention, is provided.

[0017] Figure 4 Examples of cloud-based storage systems according to some embodiments of the present invention are illustrated.

[0018] Figure 5 Examples of additional cloud-based storage systems according to some embodiments of the present invention are illustrated.

[0019] Figure 6 A flowchart illustrating an example method for servicing I / O operations in a cloud-based storage system is provided.

[0020] Figure 7 A flowchart illustrating an example method for servicing I / O operations in a cloud-based storage system is provided.

[0021] Figure 8 A flowchart illustrating additional example methods for servicing I / O operations in a cloud-based storage system is provided.

[0022] Figure 9 A flowchart illustrating additional example methods for servicing I / O operations in a cloud-based storage system is provided.

[0023] Figure 10 A flowchart illustrating additional example methods for servicing I / O operations in a cloud-based storage system is provided.

[0024] Figure 11 A flowchart illustrating additional example methods for servicing I / O operations in a cloud-based storage system is provided.

[0025] Figure 12 An example virtual storage system architecture is illustrated according to some embodiments of the present invention.

[0026] Figure 13 Additional example virtual storage system architectures are illustrated according to some embodiments of the present invention.

[0027] Figure 14 Additional example virtual storage system architectures are illustrated according to some embodiments of the present invention.

[0028] Figure 15 Additional example virtual storage system architectures are illustrated according to some embodiments of the present invention.

[0029] Figure 16 Additional example virtual storage system architectures are illustrated according to some embodiments of the present invention.

[0030] Figure 17 A flowchart illustrating additional example methods for data recovery in a virtual storage system according to some embodiments of the present invention is provided. Detailed Implementation

[0031] Reference Figure 1A The accompanying drawings at the beginning illustrate example methods, apparatus, and products for data recovery in a virtual storage system according to embodiments of the present invention. Figure 1A This example illustrates a data storage system based on some implementation. System 100 (also referred to herein as a "storage system") includes multiple elements for illustrative and not limiting purposes. It may be noted that system 100 may include the same, more, or fewer elements configured in the same or different ways in other implementations.

[0032] System 100 includes multiple computing devices 164A-B. These computing devices (also referred to herein as "client devices") can be, for example, servers, workstations, personal computers, or laptops in a data center. The computing devices 164A-B can be coupled to communicate with one or more storage arrays 102A-B via a Storage Area Network (SAN) 158 or a Local Area Network (LAN) 160.

[0033] SAN 158 can be implemented using various data communication architectures, devices, and protocols. For example, the architecture of SAN 158 can include Fibre Channel, Ethernet, Infiniband, or Serial Attached Small Computer System Interface (SAS). Data communication protocols used with SAN 158 can include Advanced Technology Attachment (ATA), Fibre Channel, Small Computer System Interface (SCSI), Internet Small Computer System Interface (iSCSI), HyperSCSI, or cross-architecture Non-Volatile Memory Standard (NVMe). It should be noted that SAN 158 is provided for illustration and not limitation. Other data communication couplings can be implemented between computing devices 164A-B and storage arrays 102A-B.

[0034] LAN 160 can also be implemented using various architectures, devices, and protocols. For example, the architecture of LAN 160 can include Ethernet (802.3) or wireless (802.11). The data communication protocols used in LAN 160 can include Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), Wireless Access Protocol (WAP), Handheld Device Transfer Protocol (HDTP), Session Initiation Protocol (SIP), or Real-Time Protocol (RTP), etc.

[0035] Storage arrays 102A-B can provide persistent data storage for computing devices 164A-B. In an implementation, storage array 102A may be contained in a chassis (not shown), and storage array 102B may be contained in another chassis (not shown). Storage arrays 102A and 102B may include one or more storage array controllers 110A-D (also referred to herein as "controllers"). Storage array controllers 110A-D may be embodied as automated computing machine modules comprising computer hardware, computer software, or a combination of computer hardware and software. In some implementations, storage array controllers 110A-D may be configured to perform various storage tasks. Storage tasks may include writing data received from computing devices 164A-B to storage arrays 102A-B, erasing data from storage arrays 102A-B, retrieving data from storage arrays 102A-B and providing the data to computing devices 164A-B, monitoring and reporting disk utilization and performance, performing redundancy operations such as Redundant Array of Independent Drives (RAID) or RAID-like data redundancy operations, compressing data, and encrypting data.

[0036] The storage array controllers 110A-D can be implemented in various ways, including as a field-programmable gate array (FPGA), a programmable logic chip (PLC), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), or any computing device including discrete components such as processing devices, central processing units, computer memory, or various adapters. The storage array controllers 110A-D may, for example, include data communication adapters configured to support communication via SAN 158 or LAN 160. In some implementations, the storage array controllers 110A-D may be independently coupled to LAN 160. In implementations, the storage array controllers 110A-D may include I / O controllers, etc., that couple the storage array controllers 110A-D to persistent storage resources 170A-B (also referred to herein as "storage resources") via a midplane (not shown) for data communication. The persistent storage resources 170A-B primarily include any number of storage drives 171A-F (also referred to herein as "storage devices") and any number of non-volatile random access memory (NVRAM) devices (not shown).

[0037] In some implementations, the NVRAM devices of persistent storage resources 170A-B can be configured to receive data to be stored in storage drives 171A-F from storage array controllers 110A-D. In some examples, the data may originate from computing devices 164A-B. In some examples, writing data to the NVRAM device can be performed faster than writing data directly to storage drives 171A-F. In implementations, storage array controllers 110A-D can be configured to utilize the NVRAM device as a fast-access buffer for data destined to be written to storage drives 171A-F. The latency of write requests using the NVRAM device as a buffer may be improved compared to a system where storage array controllers 110A-D directly write data to storage drives 171A-F. In some implementations, the NVRAM device can be implemented using computer memory in the form of high-bandwidth, low-latency RAM. NVRAM devices are referred to as "non-volatile" because they can receive or include a single power source that maintains the state of the RAM after the main power supply of the NVRAM device is consumed. Such a power source can be a battery or one or more capacitors, etc. In response to power consumption, the NVRAM device can be configured to write the contents of RAM to persistent storage such as memory drives 171A~F.

[0038] In implementations, storage drives 171A-F can refer to any device configured to persistently record data, where "persistently" or "persistently" means the device's ability to retain the recorded data after power loss. In some implementations, storage drives 171A-F can correspond to non-disk storage media. For example, storage drives 171A-F can be one or more solid-state drives (SSDs), flash memory-based storage, any type of solid-state non-volatile memory, or any other type of non-mechanical storage device. In other implementations, storage drives 171A-F can include mechanical or spinning hard disks such as hard disk drives (HDDs).

[0039] In some implementations, memory array controllers 110A-D can be configured to offload device management responsibilities from memory drives 171A-F in memory arrays 102A-B. For example, memory array controllers 110A-D can manage control information describing the status of one or more memory blocks in memory drives 171A-F. The control information may, for example, indicate that a particular memory block has failed and should no longer be written to; the particular memory block contains the startup code of memory array controllers 110A-D, the number of program / erase (P / E) cycles performed on the particular memory block, the age of the data stored in the particular memory block, and the type of data stored in the particular memory block, etc. In some implementations, the control information can be stored as metadata along with the associated memory blocks. In other implementations, the control information for memory drives 171A-F can be stored in one or more specific memory blocks of memory drives 171A-F selected by memory array controllers 110A-D. The selected memory blocks may be marked with identifiers indicating that the selected memory blocks contain control information. The memory array controllers 110A-D can utilize this identifier in conjunction with memory drives 171A-F to quickly identify memory blocks containing control information. For example, the memory controllers 110A-D can issue commands to locate memory blocks containing control information. It can be noted that the control information may be so large that a portion of the control information may be stored in multiple locations, the control information may be stored in multiple locations for redundancy purposes, or the control information may otherwise be distributed across multiple memory blocks in the memory drives 171A-F.

[0040] In implementation, the memory array controllers 110A-D can offload device management responsibility from the memory drives 171A-F by retrieving control information describing the state of one or more memory blocks in the memory drives 171A-F of the memory arrays 102A-B. Retrieving control information from the memory drives 171A-F can be performed, for example, by the memory array controllers 110A-D querying the memory drives 171A-F to obtain the location of control information for a specific memory drive 171A-F. The memory drives 171A-F can be configured to execute instructions that enable them to identify the location of the control information. These instructions can be executed by a controller (not shown) associated with or otherwise located on the memory drives 171A-F, and can cause the memory drives 171A-F to scan portions of each memory block to identify the memory blocks used to store the control information for the memory drives 171A-F. Storage drives 171A-F can respond by sending a response message containing the location of control information for storage drives 171A-F to storage array controllers 110A-D. Upon receiving the response message, storage array controllers 110A-D can issue a request to read data stored at the address associated with the location of the control information for storage drives 171A-F.

[0041] In other implementations, the storage array controllers 110A-D can also offload device management responsibility from the storage drives 171A-F by performing storage drive management operations in response to receiving control information. Storage drive management operations may include, for example, operations typically performed by the storage drives 171A-F (e.g., a controller (not shown) associated with a particular storage drive 171A-F). Storage drive management operations may include, for example, ensuring that data is not written to faulty memory blocks within the storage drives 171A-F and ensuring that data is written to memory blocks within the storage drives 171A-F in a manner that achieves sufficient wear leveling.

[0042] In implementation, storage arrays 102A-B can implement two or more storage array controllers 110A-D. For example, storage array 102A may include storage array controller 110A and storage array controller 110B. In a given instance, a single storage array controller 110A-D (e.g., storage array controller 110A) of storage system 100 may be designated as having a primary state (also referred to herein as the "master controller"), and other storage array controllers 110A-D (e.g., storage array controller 110A) may be designated as having a secondary state (also referred to herein as the "secondary controller"). The master controller may have specific rights, such as permission to modify data in persistent storage resources 170A-B (e.g., to write data to persistent storage resources 170A-B). At least some rights of the master controller may supersede the rights of the secondary controller. For example, when the master controller has the right to modify data in persistent storage resources 170A-B, the secondary controller may not have that right. The state of storage array controllers 110A-D can be changed. For example, storage array controller 110A can be designated to have a secondary state, while storage array controller 110B can be designated to have a primary state.

[0043] In some implementations, a primary controller, such as storage array controller 110A, can be used as the primary controller for one or more storage arrays 102A-B, and a secondary controller, such as storage array controller 110B, can be used as an auxiliary controller for one or more storage arrays 102A-B. For example, storage array controller 110A can be the primary controller for both storage arrays 102A and 102B, and storage array controller 110B can be an auxiliary controller for both storage arrays 102A and 102B. In some implementations, storage array controllers 110C and 110D (also referred to as "storage processing modules") may have neither a primary nor an auxiliary status. Storage array controllers 110C and 110D, implemented as storage processing modules, can serve as communication interfaces between the primary and auxiliary controllers (e.g., storage array controllers 110A and 110B, respectively) and storage array 102B. For example, storage array controller 110A of storage array 102A can send write requests to storage array 102B via SAN 158. Write requests can be received by both storage array controllers 110C and 110D of storage array 102B. Storage array controllers 110C and 110D facilitate communication, for example, sending write requests to the appropriate storage drives 171A-F. It can be noted that in some implementations, the storage processing module can be used to increase the number of storage drives controlled by the master and slave controllers.

[0044] In implementation, memory array controllers 110A-D are communicatively coupled to one or more memory drives 171A-F and one or more NVRAM devices (not shown) included as part of memory arrays 102A-B via a midplane (not shown). Memory array controllers 110A-D may be coupled to the midplane via one or more data communication links, and the midplane may be coupled to memory drives 171A-F and NVRAM devices via one or more data communication links. The data communication links described herein are collectively exemplified by data communication links 108A-D and may include, for example, a Fast Peripheral Component Interconnect (PCIe) bus.

[0045] Figure 1B Example systems for data storage based on some implementations are shown. Figure 1B The illustrated memory array controller 101 can be used with... Figure 1A The described memory array controllers 110A-D are similar. In one example, memory array controller 101 may be similar to memory array controller 110A or memory array controller 110B. Memory array controller 101 includes multiple elements for illustrative and not limiting purposes. It can be noted that memory array controller 101 may include the same, more, or fewer elements configured in the same or different manner in other implementations. It can be noted that the following may include... Figure 1A The components are shown in the figure to help illustrate the features of the storage array controller 101.

[0046] The memory array controller 101 may include one or more processing devices 104 and random access memory (RAM) 111. The processing device 104 (or controller 101) represents one or more general-purpose processing devices such as a microprocessor or central processing unit. More specifically, the processing device 104 (or controller 101) may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor for implementing other instruction sets or a combination of instruction sets. The processing device 104 (or controller 101) may also be one or more special-purpose processing devices such as an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), or a network processor.

[0047] Processing device 104 can be connected to RAM 111 via data communication link 106, which can be embodied as a high-speed memory bus such as a fourth-generation double data rate (DDR4) bus. Operating system 112 is stored in RAM 111. In some implementations, instructions 113 are stored in RAM 111. Instructions 113 may include computer program instructions for operating in a direct-mapped flash memory storage system. In one embodiment, a direct-mapped flash memory storage system is a system that directly addresses data blocks within a flash drive without address translation performed by the flash drive's memory controller.

[0048] In implementation, the storage array controller 101 includes one or more host bus adapters 103A-C coupled to the processing device 104 via data communication links 105A-C. In implementation, the host bus adapters 103A-C may be computer hardware that connects a host system (e.g., the storage array controller) to other networks and storage arrays. In some examples, the host bus adapters 103A-C may be Fibre Channel adapters for enabling the storage array controller 101 to connect to a SAN or Ethernet adapters for enabling the storage array controller 101 to connect to a LAN, etc. The host bus adapters 103A-C may, for example, be coupled to the processing device 104 via data communication links 105A-C such as a PCIe bus.

[0049] In an implementation, the storage array controller 101 may include a host bus adapter 114 coupled to an extender 115. The extender 115 can be used to attach the host system to more storage drives. The extender 115 may be, for example, a SAS extender used in an implementation where the host bus adapter 114 is embodied as a SAS controller to enable the host bus adapter 114 to be attached to storage drives.

[0050] In implementation, the storage array controller 101 may include a switch 116 coupled to the processing device 104 via a data communication link 109. The switch 116 may be a computer hardware device capable of creating multiple endpoints from a single endpoint, thereby enabling multiple devices to share a single endpoint. The switch 116 may be, for example, a PCIe switch coupled to a PCIe bus (e.g., data communication link 109) and presenting multiple PCIe connection points to a midplane.

[0051] In implementation, the storage array controller 101 includes a data communication link 107 for coupling the storage array controller 101 to other storage array controllers. In some examples, the data communication link 107 may be a Fast Path Interconnect (QPI) interconnect.

[0052] Traditional storage systems using conventional flash drives can implement processing across flash drives that are part of the traditional storage system. For example, high-level processing of the storage system can initiate and control processing across flash drives. However, the flash drives in a traditional storage system may include their own storage controllers that also perform this processing. Therefore, for a traditional storage system, both high-level processing (e.g., initiated by the storage system) and low-level processing (e.g., initiated by the storage system's storage controller) can be performed.

[0053] To address the various shortcomings of traditional storage systems, operation can be performed using high-level processing rather than low-level processing. For example, a flash storage system can include flash drives that do not contain a storage controller for providing processing. Therefore, the operating system of the flash storage system itself can initiate and control processing. This can be achieved through a direct-mapped flash storage system, which directly addresses data blocks within the flash drive without the address translation performed by the flash drive's storage controller.

[0054] The operating system of a flash memory storage system can identify and maintain a list of allocation units across multiple flash drives in the flash memory storage system. An allocation unit can be an entire erase block or multiple erase blocks. The operating system can maintain a mapping or address range used to directly map addresses to erase blocks in the flash drives of the flash memory storage system.

[0055] Erasable blocks directly mapped to the flash drive can be used to rewrite and erase data. For example, one or more allocation units containing first and second data can be operated on, where the first data is retained while the second data is no longer used by the flash storage system. The operating system can initiate processes to write the first data to a new location within another allocation unit, erase the second data, and mark the allocation unit as available for subsequent data. Therefore, this processing can be performed solely by the high-level operating system of the flash storage system, without requiring additional low-level processing from the flash drive's controller.

[0056] The advantages of this processing, performed solely by the operating system of the flash memory storage system, include improved reliability of the flash drives because no unnecessary or redundant write operations are performed during the process. A potentially novel aspect here is the concept of initiating and controlling this processing at the operating system level of the flash memory storage system. Furthermore, this processing can be controlled by the operating system across multiple flash drives. This contrasts with processing performed by the storage controller of the flash drives.

[0057] The storage system may include two storage array controllers sharing a set of drives for failover purposes, or the storage system may include a single storage array controller for providing storage services utilizing multiple drives, or the storage system may include a distributed network of storage array controllers, each having a number of drives or a amount of flash storage, wherein the storage array controllers in the network cooperate to provide complete storage services and cooperate in various aspects of storage services, including storage allocation and garbage collection.

[0058] Figure 1C A third example system 117 is illustrated according to some implementation of data storage. System 117 (also referred to herein as a “storage system”) includes multiple elements for illustrative and non-limiting purposes. It may be noted that system 117 may include the same, more or fewer elements configured in the same or different ways in other implementations.

[0059] In one embodiment, system 117 includes a dual peripheral component interconnect (PCI) flash memory device 118 having individually addressable fast write storage. System 117 may include a memory device controller 119. In one embodiment, memory device controllers 119a-d may be a CPU, ASIC, FPGA, or any other circuitry capable of implementing the control architecture required by the present invention. In one embodiment, system 117 includes flash memory devices (e.g., flash memory devices 120a-n) operatively coupled to various channels of memory device controller 119. Flash memory devices 120a-n may be presented to memory device controller 119a-d as an addressable set of flash pages, erase blocks, and / or control elements sufficient to allow memory device controller 119a-d to program and retrieve aspects of the flash memory. In one embodiment, storage device controllers 119a-d can operate flash memory devices 120a-n, including storing and retrieving data content of pages, arranging and erasing any blocks, tracking statistics related to the use and reuse of flash memory pages, erased blocks and cells, tracking and predicting error codes and faults within the flash memory, controlling voltage levels associated with programming and retrieving the contents of flash memory cells, and so on.

[0060] In one embodiment, system 117 may include RAM 121 for storing individually addressable, fast-write data. In one embodiment, RAM 121 may be one or more separate discrete devices. In another embodiment, RAM 121 may be integrated into storage device controllers 119a-d or multiple storage device controllers. RAM 121 may also be used for other purposes, such as temporary program memory for processing devices (e.g., CPU) in storage device controller 119.

[0061] In one embodiment, system 117 may include an energy storage device 122, such as a rechargeable battery or capacitor. The energy storage device 122 can store enough energy to power the storage device controller 119, a certain amount of RAM (e.g., RAM 121), and a certain amount of flash memory (e.g., flash memory devices 120a-120n) for a period sufficient to write the contents of the RAM to the flash memory. In one embodiment, if the storage device controller 119a-d detects a loss of external power, the storage device controller can write the contents of the RAM to the flash memory.

[0062] In one embodiment, system 117 includes two data communication links 123a and 123b. In one embodiment, data communication links 123a and 123b may be PCI interfaces. In another embodiment, data communication links 123a and 123b may be based on other communication standards (e.g., HyperTransport, Infinite Bandwidth, etc.). Data communication links 123a and 123b may be based on the Non-Volatile Memory Standard (NVMe) or the Cross-Architecture NVMe (NVMf) specification, which allows external connections from other components in storage system 117 to storage device controllers 119a-d. It should be noted that, for convenience, the data communication links may be interchangeably referred to herein as PCI buses.

[0063] System 117 may also include an external power supply (not shown), which may be located on one or both of data communication links 123a, 123b, or may be located separately. An alternative embodiment includes a separate flash memory (not shown) dedicated to storing the contents of RAM 121. Storage device controllers 119a-d may present logical devices on a PCI bus, which may include addressable fast write logic devices or different portions of the logical address space of storage device 118, which may be presented as PCI memory or persistent storage. In one embodiment, operations to be stored in the device are intended to be stored in RAM 121. In the event of a power failure, storage device controllers 119a-d may write storage contents associated with addressable fast write logic storage to flash memory (e.g., flash memory devices 120a-n) for long-term persistent storage.

[0064] In one embodiment, the logic device may include a representation of some or all of the contents of flash memory devices 120a-n, wherein the representation allows a storage system (e.g., storage system 117) including storage device 118 to directly address flash memory pages and directly reprogram erase blocks from storage system components outside the storage device via a PCI bus. The representation may also allow one or more external components to control and retrieve other aspects of the flash memory, including some or all of the following: tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells across all flash memory devices; tracking and predicting error codes and faults within and across flash memory devices; controlling voltage levels associated with programming and retrieving the contents of flash cells; and so on.

[0065] In one embodiment, the energy storage device 122 may be sufficient to ensure the completion of ongoing operations for the flash memory devices 120a-120n. The energy storage device 122 may power the memory device controllers 119a-d and the associated flash memory devices (e.g., 120a-n) for these operations and for storing fast writes from RAM to flash memory. The energy storage device 122 may be used to store accumulated statistics and other parameters saved and tracked by the flash memory devices 120a-n and / or the memory device controller 119. For some or all of the operations described herein, separate capacitors or energy storage devices (such as smaller capacitors embedded in or near the flash memory devices) may be used.

[0066] Various methods can be used to track and optimize the lifespan of storage energy components, such as adjusting voltage levels over time and partially discharging the storage energy device 122 to measure the corresponding discharge characteristics. If available energy decreases over time, the effective available capacity of the addressable fast write storage may decrease to ensure that it can be safely written based on the currently available storage energy.

[0067] Figure 1D A third example system 124 based on some implementations of data storage is illustrated. In one embodiment, system 124 includes storage controllers 125a, 125b. In one embodiment, storage controllers 125a, 125b are operatively coupled to dual PCI storage device controllers 119a, 119b and 119c, 119d, respectively. Storage controllers 125a, 125b may be operatively coupled (e.g., via storage network 130) to a number of host computers 127a~n.

[0068] In one embodiment, two storage controllers (e.g., 125a and 125b) provide storage services such as SCS block storage arrays, file servers, object servers, databases, or data analytics services. Storage controllers 125a and 125b can provide services to host computers 127a-n outside the storage system 124 via a number of network interfaces (e.g., 126a-d). Storage controllers 125a and 125b can provide integrated services or applications entirely within the storage system 124, forming a converged storage and computing system. Storage controllers 125a and 125b can utilize fast write memory within or across storage device controllers 119a-d to record ongoing operations, ensuring that operations are not lost in the event of power failure, storage controller removal, shutdown of the storage controllers or storage system, or some failure of one or more software or hardware components within the storage system 124.

[0069] In one embodiment, controllers 125a and 125b operate as a PCI master of one or more PCI buses 128a and 128b. In another embodiment, 128a and 128b may be based on other communication standards (e.g., HyperTransport, Infinite Bandwidth, etc.). Other storage system embodiments may allow storage controllers 125a and 125b to operate as multiple masters of both PCI buses 128a and 128b. Optionally, a PCI / NVMe / NVMe switching infrastructure or structure may connect multiple storage controllers. Some storage system embodiments may allow storage devices to communicate directly with each other, rather than only with storage controllers. In one embodiment, storage device controller 119a may operate under the instruction of storage controller 125a to perform operations based on data stored in RAM (e.g., ...). Figure 1C The data in RAM 121 is synthesized and transferred to the flash memory device. For example, a recalculated version of the RAM contents may be transferred after the memory controller has determined that the operation has been fully committed across the memory system, or after a certain amount of time has been reached in the fast write memory on the device, or after a certain amount of time has elapsed, to ensure improved data security or to free up addressable fast write capacity for reuse. This mechanism can, for example, be used to avoid a second transfer from memory controllers 125a, 125b via the bus (e.g., 128a, 128b). In one embodiment, recalculation may include compressing data, adding indexes or other metadata, combining multiple data segments together, performing erase code calculations, etc.

[0070] In one embodiment, under the instruction of storage controllers 125a, 125b, storage device controllers 119a, 119b can be operable without involving storage controllers 125a, 125b, according to RAM (e.g., Figure 1C The data stored in RAM 121 is used to calculate data and transfer data to other storage devices. This operation can be used to mirror stored data from one controller 125a to another controller 125b, or it can be used to offload compression, data aggregation, and / or erasure encoding calculations and transfers to storage devices to reduce the load on the storage controller or the storage controller interfaces 129a, 129b to the PCI buses 128a, 128b.

[0071] Storage device controllers 119a-d may include mechanisms for implementing high availability primitives for use by other parts of the storage system outside the dual PCI storage device 118. For example, reservation or exclusion primitives may be provided, allowing one storage controller in a storage system with two storage controllers for providing high availability storage services to prevent the other storage controller from accessing or continuing to access the storage device. This mechanism may be used, for example, if one controller detects that the other controller is malfunctioning or that the interconnect between the two storage controllers itself may be malfunctioning.

[0072] In one embodiment, a storage system used with dual PCI direct-mapped storage devices having individually addressable fast write storage includes a system that manages erase blocks or groups of erase blocks as allocation units for storing data representing the storage service, or for storing metadata associated with the storage service (e.g., indexes, logs, etc.), or for appropriately managing the storage system itself. Flash pages, which may be several kilobytes in size, can be written when data arrives or when the storage system needs to retain data for a long time interval (e.g., above a defined time threshold). To commit data faster or to reduce the number of writes to the flash memory device, the storage controller may first write data to individually addressable fast write storage on one or more storage devices.

[0073] In one embodiment, storage controllers 125a and 125b may initiate the use of erase blocks within and across storage devices (e.g., 118) based on the age and expected remaining lifetime of the storage device or based on other statistics. Storage controllers 125a and 125b may also initiate garbage collection and data migration between storage devices based on pages that are no longer needed, and manage flash page and erase block lifetimes, as well as overall system performance.

[0074] In one embodiment, storage system 124 may utilize mirroring and / or erasure coding schemes as part of storing data into addressable, fast-write storage and / or as part of writing data to allocation units associated with erase blocks. Erasure codes may be used across storage devices, within erase blocks or allocation units, or within flash memory devices on a single storage device and across flash memory devices on a single storage device, to provide redundancy or protection against single or multiple storage device failures to prevent internal damage to flash memory pages due to flash memory operation or deterioration of flash memory cells. Different levels of mirroring and erasure coding can be used to recover from multiple types of failures occurring individually or in combination.

[0075] refer to Figure 2A The embodiments described in ~G exemplify a storage cluster that stores user data, such as user data originating from one or more user or client systems or other sources outside the storage cluster. The storage cluster uses erase coding and redundant copies of metadata to distribute user data across storage nodes housed within a chassis or across multiple chassis. Erase coding refers to a method of data protection or reconstruction in which data is stored across a set of different locations such as disks, storage nodes, or geographic locations. Flash memory is a type of solid-state memory that can be integrated with the embodiments, but the embodiments can be extended to other types of solid-state memory or other storage media including non-solid-state memory. Control of storage locations and workloads are distributed across storage locations in a cluster peer-to-peer system. Tasks such as mediating communication between different storage nodes, detecting when a storage node becomes unavailable, and balancing I / O (input and output) across different storage nodes are all handled on a distributed basis. In some embodiments, data is laid out or distributed across multiple storage nodes in data segments or stripes that support data recovery. Ownership of data can be reassigned within the cluster regardless of input and output modes. The architecture described in more detail below allows the system to remain operational even if storage nodes in the cluster fail, because data can be reconstructed from other storage nodes and thus remain available for input and output operations. In various embodiments, storage nodes may be referred to as cluster nodes, blades, or servers.

[0076] A storage cluster may be contained within a chassis (i.e., a enclosure that houses one or more storage nodes). Mechanisms for powering the individual storage nodes (such as power distribution buses) and communication mechanisms enabling communication between storage nodes (such as communication buses) may also be included within the chassis. According to some embodiments, the storage cluster may operate as a standalone system in one location. In one embodiment, the chassis contains at least two instances of both the power distribution bus and the communication bus, which can be independently enabled or disabled. The internal communication bus may be an Ethernet bus; however, other technologies such as PCIe and wireless bandwidth are equally suitable. The chassis provides ports for an external communication bus that enables communication directly or via a switch between multiple chassis and with client systems. External communication may use technologies such as Ethernet, wireless bandwidth, Fibre Channel, etc. In some embodiments, the external communication bus uses different communication bus technologies for inter-chassis and client communication. If a switch is deployed within or between chassis, the switch may act as a converter between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, the storage cluster can be accessed by clients using proprietary or standard interfaces such as Network File System (NFS), Common Internet File System (CIFS), Small Computer System Interface (SCSI), or Hypertext Transfer Protocol (HTTP). The translation from the client protocol can occur at the switch, the external communication bus of the chassis, or within each storage node. In some embodiments, multiple chassis can be coupled or connected to each other via an aggregator switch. Chassis that are partially and / or fully coupled or connected can be designated as a storage cluster. As discussed above, each chassis can have multiple blades, each with a Media Access Control (MAC) address; however, in some embodiments, the storage cluster is presented to the external network as having a single cluster IP address and a single MAC address.

[0077] Each storage node can be one or more storage servers, and each storage server is connected to one or more non-volatile solid-state memory (NSS) cells, which may be referred to as storage cells or storage devices. One embodiment includes a single storage server in each storage node and between one and eight NSS cells; however, this example is not intended to be limiting. The storage server may include a processor, DRAM, and interfaces for an internal communication bus, as well as power distribution for each power bus. In some embodiments, within a storage node, interfaces and storage cells share a communication bus (e.g., PCI Express). NSS cells can directly access the internal communication bus interface via the storage node's communication bus, or request access to the bus interface from the storage node. The NSS cell includes an embedded CPU, a solid-state storage controller, and a certain amount of solid-state mass storage, such as between 2 and 32 terabytes (TB) in some embodiments. The NSS cell includes embedded volatile storage media such as DRAM and energy storage devices. In some embodiments, the energy storage devices are capacitors, supercapacitors, or batteries, which enable the transfer of a subset of DRAM content to a stable storage medium in the event of power loss. In some embodiments, non-volatile solid-state memory cells are constructed using storage-type memories such as phase-change or magnetoresistive random access memory (MRAM), which replace DRAM and enable power-saving devices to be reduced in size.

[0078] One of the many characteristics of storage nodes and non-volatile solid-state storage (NSS) is the ability to proactively rebuild data within a storage cluster. Storage nodes and NSS can determine when a storage node or NSS in a storage cluster is unreachable, regardless of whether there are attempts to read data involving that storage node or NSS. The storage node and NSS then cooperate to recover and rebuild the data in at least a partially new location. This constitutes proactive rebuilding because the system does not need to wait until a read access initiated from a client system employing the storage cluster requires the data before rebuilding it. These and further details of storage devices and their operation are discussed below.

[0079] Figure 2AThis is a perspective view of a storage cluster 161 having multiple storage nodes 150 and internal solid-state memory coupled to each storage node to provide network-attached storage or a storage area network, according to some embodiments. Network-attached storage, a storage area network, or a storage cluster or other storage memory may include one or more storage clusters 161, each storage cluster 161 having one or more storage nodes 150 in a flexible and reconfigurable arrangement of both the physical components and the amount of storage memory provided therefrom. Storage clusters 161 are designed to be mounted in racks, and one or more racks can be set up and filled as needed for the storage memory. Storage cluster 161 has a chassis 138 having multiple slots 142. It should be understood that chassis 138 may be referred to as a housing, enclosure, or rack unit. In one embodiment, chassis 138 has fourteen slots 142, but other numbers of slots are readily contemplated. For example, some embodiments have four slots, eight slots, sixteen slots, thirty-two slots, or other suitable numbers of slots. In some embodiments, each slot 142 may accommodate one storage node 150. The chassis 138 includes flaps 148 for mounting the chassis 138 onto a rack. A fan 144 provides air circulation for cooling the storage nodes 150 and their components; however, other cooling components may be used, or embodiments without cooling components are conceivable. A switch structure 146 couples the storage nodes 150 together within the chassis 138 and to a network for communication with the memory. In the embodiment depicted herein, for illustrative purposes, slot 142 to the left of the switch structure 146 and fan 144 is shown occupied by a storage node 150, while slot 142 to the right of the switch structure 146 and fan 144 is empty and available for insertion of a storage node 150. This configuration is an example, and in various further arrangements, one or more storage nodes 150 may occupy slot 142. In some embodiments, the storage node arrangement does not need to be contiguous or adjacent. Storage node 150 is hot-swappable, meaning it can be inserted into or removed from slot 142 in chassis 138 without stopping or shutting down the system. When storage node 150 is inserted or removed relative to slot 142, the system automatically reconfigures to recognize and adapt to the change. In some embodiments, reconfiguration includes restoring redundancy and / or rebalancing data or load.

[0080] Each storage node 150 may have multiple components. In the embodiment shown herein, storage node 150 includes a printed circuit board 159 filled with a CPU 156 (processor), a memory 154 coupled to the CPU 156, and non-volatile solid-state storage 152 coupled to the CPU 156; however, other mounting components and / or other components may be used in further embodiments. The memory 154 has instructions executed by the CPU 156 and / or data operated by the CPU 156. As further explained below, the non-volatile solid-state storage 152 includes flash memory, or in further embodiments, other types of solid-state storage.

[0081] refer to Figure 2A As described above, storage cluster 161 is scalable, meaning that storage capacity with non-uniform storage sizes can be easily added. In some embodiments, one or more storage nodes 150 can be inserted into or removed from chassis and the storage cluster self-configured. Inserted storage nodes 150, whether delivered to the chassis or added later, can have different sizes. For example, in one embodiment, storage nodes 150 can have any multiple of 4 TB, such as 8 TB, 12 TB, 16 TB, 32 TB, etc. In further embodiments, storage nodes 150 can have any multiple of other storage amounts or capacities. The storage capacity of each storage node 150 is broadcast and influences the decision of how data is striped. For maximum storage efficiency, subject to predetermined requirements of continuous operation even with the loss of up to one or more non-volatile solid-state drives 152 or storage nodes 150 within the chassis, embodiments can self-configure as extensively as possible within stripes.

[0082] Figure 2B This is a block diagram illustrating the communication interconnect 173 and power distribution bus 172 coupling multiple storage nodes 150. (See reference...) Figure 2A In some embodiments, the communication interconnect 173 may be included in or implemented with the switch structure 146. In some embodiments, where multiple storage clusters 161 occupy a rack, the communication interconnect 173 may be included on top of or with the rack switch. Figure 2B As illustrated, storage cluster 161 is enclosed within a single chassis 138. External port 176 is coupled to storage node 150 via communication interconnect 173, while external port 174 is directly coupled to the storage node. External power port 178 is coupled to power distribution bus 172. Storage node 150 may include, as shown in the reference... Figure 2A The non-volatile solid-state storage 152 can vary in size and capacity. Additionally, one or more storage nodes 150 can be as follows: Figure 2BThe illustration only considers the storage nodes. Permission 168 is implemented on non-volatile solid-state storage 152, for example as a list or other data structure stored in memory. In some embodiments, permissions are stored within non-volatile solid-state storage 152 and supported by software executing on the controller or other processor of non-volatile solid-state storage 152. In a further embodiment, permission 168 is implemented on storage node 150, for example as a list or other data structure stored in memory 154 and supported by software executing on the CPU 156 of storage node 150. In some embodiments, permission 168 controls how and where data is stored in non-volatile solid-state storage 152. This control helps determine which type of erasure coding scheme is applied to the data and which storage nodes 150 have which portions of the data. Each permission 168 may be assigned to non-volatile solid-state storage 152. In various embodiments, each permission can control the range of inode numbers, segment numbers, or other data identifiers assigned to data by the file system, storage node 150, or non-volatile solid-state storage 152.

[0083] In some embodiments, each piece of data and each piece of metadata is redundant in the system. Additionally, each piece of data and each piece of metadata has an owner, which may be referred to as permissions. If a permission becomes unreachable, for example, due to a storage node failure, there is a succession plan for how to find the data or metadata. In various embodiments, redundant copies of permission 168 exist. In some embodiments, permission 168 has a relationship with storage node 150 and non-volatile solid-state storage 152. Permissions 168 covering a range of data segment numbers or other identifiers of the data may be assigned to a specific non-volatile solid-state storage 152. In some embodiments, permissions 168 for all such ranges are distributed across the non-volatile solid-state storage 152 of the storage cluster. Each storage node 150 has a network port for providing access to one or more non-volatile solid-state storage 152s of that storage node 150. In some embodiments, data may be stored in segments associated with segment numbers, and these segment numbers are an indirection to the configuration of a RAID (Redundant Array of Independent Disks) stripe. Therefore, the assignment and use of permission 168 establishes indirection to the data. According to some embodiments, indirection may be referred to as the ability to indirectly reference data via permission 168 in this context. A segment identifier may comprise a set of non-volatile solid-state drives 152 containing the data and a local identifier within that set of non-volatile solid-state drives 152. In some embodiments, the local identifier is an offset within the device and may be reused sequentially by multiple segments. In other embodiments, the local identifier is unique for a particular segment and is never reused. The offset in the non-volatile solid-state drives 152 is used to locate data to be written to or read from the non-volatile solid-state drives 152 (in the form of RAID striping). Data is striped across multiple cells of non-volatile solid-state drives 152, which may include or differ from non-volatile solid-state drives 152 having permission 168 for a particular data segment.

[0084] In cases where the location of a specific data segment changes, such as during data movement or data reconstruction, the permission 168 for that data segment should be looked up at the non-volatile solid-state storage 152 or storage node 150 that has the permission 168. To locate specific data, embodiments calculate a hash value of the data segment or apply an inode number or data segment number. The output of this operation points to the non-volatile solid-state storage 152 that has the permission 168 for that specific data. In some embodiments, there are two levels for performing this operation. The first level maps an entity identifier (ID), such as a segment number, inode number, or directory number, to a permission identifier. This mapping may include calculations such as hashes or bitmasks. The second level maps the permission identifier to a specific non-volatile solid-state storage 152, which can be done through explicit mapping. This operation is repeatable, such that when calculations are performed, the results of the calculations reliably and repeatedly point to the specific non-volatile solid-state storage 152 that has the permission 168. This operation may include a set of reachable storage nodes as input. If the set of reachable non-volatile solid-state storage units changes, the optimal set changes. In some embodiments, the persistent value is the current assignment (which is always true), and the calculated value is the target assignment that the cluster will attempt to reconfigure. This calculation can be used to determine the optimal non-volatile solid-state storage 152 for permissions when there is a set of reachable non-volatile solid-state storage 152 that constitutes the same cluster. The calculation also determines an ordered set of peer non-volatile solid-state storage 152 that also records permissions for non-volatile solid-state storage mappings, so that permissions can be determined even if the assigned non-volatile solid-state storage is unreachable. In some embodiments, if a particular permission 168 is unavailable, a copy or alternative permission 168 can be consulted.

[0085] refer to Figure 2A and 2BTwo of the many tasks of the CPU 156 on storage node 150 are breaking down written data and reassembling read data. When the system determines that data needs to be written, the permission 168 for that data is positioned as described above. With the segment ID of the data determined, the write request is forwarded to the non-volatile solid-state storage 152 of the host currently identified as having permission 168 from the segment. The host CPU 156 residing on the non-volatile solid-state storage 152 and the corresponding permission 168 of the storage node 150 then breaks down or fragments the data and sends it to the various non-volatile solid-state storage units 152. The sent data is written as data stripes according to the erase encoding scheme. In some embodiments, data is requested to be pulled; in other embodiments, data is pushed. Conversely, when reading data, the permission 168 containing the segment ID of the data is positioned as described above. The host CPU 156 residing on the non-volatile solid-state storage 152 and the corresponding permission 168 of the storage node 150 requests data from the non-volatile solid-state storage and the corresponding storage node pointed to by the permission. In some embodiments, the data is read as a data stripe from flash storage. The host CPU 156 of storage node 150 then reassembles the read data, corrects any errors (if any) according to an appropriate erasure encoding scheme, and forwards the reassembled data to the network. In further embodiments, some or all of these tasks may be processed in non-volatile solid-state storage 152. In some embodiments, a segment host requests data to be sent to storage node 150 by requesting a page from storage and then sending the data to the storage node that made the original request.

[0086] In some systems, such as UNIX-like file systems, inodes or index nodes are used to process data, where an inode or index node specifies a data structure representing an object in the file system. An object can be, for example, a file or directory. Metadata may accompany the object as attributes such as permission data and creation timestamps, as well as other attributes. Segment numbers can be assigned to all or part of such objects in the file system. In other systems, segment numbers assigned elsewhere are used to process data segments. For the purposes of this discussion, the unit of distribution is an entity, and an entity can be a file, directory, or segment. That is, an entity is a unit of data or metadata stored by the storage system. Entities are grouped into groups called permissions. Each permission has a permission owner, which is a storage node with the exclusive right to update the entities within the permission. In other words, a storage node contains permissions, and those permissions, in turn, contain entities.

[0087] According to some embodiments, a segment is a logical container for data. A segment is an address space between the media address space and the physical flash memory location, i.e., the data segment number resides in that address space. A segment may also contain metadata, which enables data redundancy to be recovered (rewritten to a different flash memory location or device) without involving high-level software. In one embodiment, the internal format of a segment contains client data and a media mapping to determine the location of the data. Where applicable, segments are protected by breaking them down into multiple data and parity shards, for example, to prevent memory failures and other faults. Depending on the erasure coding scheme, the data and parity shards are coupled across the host CPU 156 (see [link to relevant documentation]). Figure 2E and 2G The non-volatile solid-state storage 152 is distributed, i.e., striped. In some embodiments, the term segment is used to refer to a container and its location in the segment's address space. According to some embodiments, the term stripe is used to refer to a group of fragments that are the same as segments, and includes how the fragments are distributed along with redundancy or parity information.

[0088] A series of address space translations occur across the entire storage system. At the top are directory entries (filenames) linked to inodes. Inodes point to the address space of the media in which the data is logically stored. Media addresses can be mapped through a series of indirect media to distribute the load of large files or to implement data services such as deduplication or snapshots. The segment addresses are then translated into physical flash memory locations. According to some embodiments, the physical flash memory locations have address ranges defined by the amount of flash memory in the system. Media addresses and segment addresses are logical containers and, in some embodiments, use identifiers of 128 bits or more, making them virtually unlimited, where the probability of reuse is calculated to be longer than the expected lifespan of the system. In some embodiments, addresses from logical containers are allocated in a hierarchical manner. Initially, each non-volatile solid-state storage 152 may be assigned a range of address space. Within this assigned range, the non-volatile solid-state storage 152 can allocate addresses without synchronization with other non-volatile solid-state storage 152.

[0089] Data and metadata are stored using a set of basic storage layouts optimized for different workload patterns and storage devices. These layouts incorporate multiple redundancy schemes, compression formats, and indexing algorithms. Some layouts store information related to permissions and permission control, while others store file metadata and file data. The redundancy schemes include error correction codes that allow for damaged bits within a single storage device (such as NAND flash memory chips), erase codes that allow for failures of multiple storage nodes, and replication schemes that allow for data center or region failures. In some embodiments, low-density parity-check (LDPC) codes are used within a single storage cell. In some embodiments, Reed-Solomon coding is used within the storage cluster, and mirroring is used within the storage grid. Metadata can be stored using indexes of ordered log structures (such as merge trees of log structures), whereas large amounts of data cannot be stored in log-structured layouts.

[0090] To maintain consistency across multiple replicas of an entity, storage nodes implicitly agree on two things through computation: (1) the permissions containing the entity, and (2) the storage node containing the permissions. Entity-to-permission assignment can be accomplished by pseudo-randomly assigning entities to permissions, by dividing entities into ranges based on externally generated keys, or by placing individual entities into permissions. Examples of pseudo-random schemes are linear hashes and the RUSH family of hashes, the latter including Controlled Replication-Based Scalable Hash (CRUSH). In some embodiments, pseudo-random assignment is used only to assign permissions to nodes because the set of nodes can change. The set of permissions cannot change, so any subjective functionality can be applied in these embodiments. Some placement schemes automatically place permissions on storage nodes, while others rely on an explicit mapping of permissions to storage nodes. In some embodiments, a pseudo-random scheme is used to map permissions to a set of candidate permission owners. A pseudo-random data distribution function associated with CRUSH can assign permissions to storage nodes and create a list of assigned permissions. Each storage node has a copy of the pseudo-random data distribution function and can obtain the same computations used for distribution and subsequent lookup or location of permissions. In some embodiments, each pseudo-random scheme requires a set of reachable storage nodes as input to infer the same target node. Once an entity has been placed in the permissions, it can be stored on the physical device so that anticipated failures will not cause unintended data loss. In some embodiments, the rebalancing algorithm attempts to store copies of all entities within the permissions in the same layout and on the same set of machines.

[0091] Examples of anticipated failures include equipment failure, stolen machines, data center fires, and regional disasters such as nuclear events or geological events. Different failures result in different levels of acceptable data loss. In some embodiments, stolen storage nodes do not affect the security or reliability of the system, while regional events may result in no data loss, lost updates for seconds or minutes, or even complete data loss, depending on the system configuration.

[0092] In some embodiments, the placement of redundant data is independent of the placement of permissions for data consistency. In some embodiments, the storage node containing permissions does not contain any persistent storage. Alternatively, the storage node is connected to a non-volatile solid-state storage unit that does not contain permissions. The communication interconnects between storage nodes and non-volatile solid-state storage units encompass various communication technologies and have non-uniform performance and fault tolerance characteristics. In some embodiments, as described above, the non-volatile solid-state storage units are connected to the storage nodes via PCI Express, the storage nodes are connected together within a single chassis using an Ethernet backplane, and the chassis are connected together to form a storage cluster. In some embodiments, the storage clusters are connected to clients using Ethernet or Fibre Channel. If multiple storage clusters are configured in a storage grid, the Internet or other long-distance network links (such as “metro-scale” links or private links that do not cross the Internet) are used to connect the multiple storage clusters.

[0093] The permission owner has exclusive rights to modify entities, migrate entities from one non-volatile solid-state storage unit to another, and add and remove copies of entities. This allows for the maintenance of redundancy in the underlying data. When the permission owner fails, is about to be deactivated, or is overloaded, permissions are transferred to a new storage node. Transient failures make it meaningful to ensure that all non-faulty machines agree on the new permission location. Uncertainty caused by transient failures can be automatically addressed through consensus protocols such as Paxos, hot-to-warm failover schemes, or through manual intervention by a remote system administrator or a local hardware administrator (such as by physically removing the faulty machine from the cluster or pressing a button on the faulty machine). In some embodiments, a consensus protocol is used, and failover is automatic. According to some embodiments, if too many failures or replication events occur within too short a time period, the system enters a self-protection mode and stops replication and data movement activities until administrator intervention.

[0094] As permissions are transferred between storage nodes and permission owners update entities within their permissions, the system transmits messages between storage nodes and non-volatile solid-state storage units. Regarding persistent messages, messages with different purposes have different types. Depending on the message type, the system maintains different ordering and persistence guarantees. As a persistent message is being processed, it is temporarily stored using multiple persistent and non-persistent storage hardware technologies. In some embodiments, messages are stored in RAM, NVRAM, and NAND flash memory devices, and various protocols are used to efficiently utilize each storage medium. Latency-sensitive client requests can be held in replicated NVRAM and then in NAND, while background rebalancing operations are held directly to NAND.

[0095] Persistent messages are persistently stored before being sent. This allows the system to continue serving client requests despite failures and component replacements. While many hardware components contain unique identifiers visible to system administrators, manufacturers, the hardware supply chain, and ongoing quality control infrastructure, applications running on top of these infrastructure addresses virtualize those addresses. These virtualized addresses remain unchanged throughout the storage system's lifecycle, regardless of component failures and replacements. This allows components of the storage system to be replaced over time without requiring reconfiguration or interruption of client request processing; i.e., the system supports non-disruptive upgrades.

[0096] In some embodiments, virtualized addresses are stored with sufficient redundancy. The continuous monitoring system correlates hardware and software status with hardware identifiers. This allows for the detection and prediction of failures due to faulty components and manufacturing details. In some embodiments, the monitoring system also enables the proactive transfer of authority and entities from affected devices before a failure occurs by removing components from the critical path.

[0097] Figure 2C This is a multi-level block diagram illustrating the contents of storage node 150 and the contents of non-volatile solid-state storage 152 of storage node 150. In some embodiments, data is communicated relative to storage node 150 via network interface controller (NIC) 202. As discussed above, each storage node 150 has a CPU 156 and one or more non-volatile solid-state storage units 152. Figure 2C Moving down one level, each non-volatile solid-state storage 152 has a relatively fast non-volatile solid-state memory, such as non-volatile random access memory (NVRAM) 204 and flash memory 206. In some embodiments, NVRAM 204 can be a component that does not require programming / erasing cycles (DRAM, MRAM, PCM) and can be a memory capable of supporting writes much more frequently than reads from this memory. Figure 2CMoving down another level, NVRAM 204 is implemented in one embodiment as a high-speed volatile memory such as Dynamic Random Access Memory (DRAM) 216, backed up by an energy reserve 218. The energy reserve 218 provides sufficient power to maintain power to the DRAM 216 for a sufficient period of time in the event of a power failure to transfer content to the flash memory 206. In some embodiments, the energy reserve 218 is a capacitor, supercapacitor, battery, or other device that provides a suitable energy supply sufficient to allow the content of the DRAM 216 to be transferred to a stable storage medium in the event of power loss. The flash memory 206 is implemented as a plurality of flash memory wafers 222, which may be referred to as a package of flash memory wafers 222 or an array of flash memory wafers 222. It should be understood that the flash memory wafers 222 can be packaged in any of the following ways: one wafer per package, multiple wafers per package (i.e., multi-chip package), in a hybrid package, as bare wafers on a printed circuit board or other substrate, as packaged wafers, etc. In the illustrated embodiment, the non-volatile solid-state storage 152 has a controller 212 or other processor and an input / output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or network interface controller 202 of the flash memory storage node 150. The flash input / output (I / O) port 220 is coupled to the flash memory wafer 222, and the direct memory access unit (DMA) 214 is coupled to the controller 212, DRAM 216, and flash memory wafer 222. In the illustrated embodiment, the I / O port 210, controller 212, DMA unit 214, and flash I / O port 220 are implemented on a programmable logic device (PLD) 208 (e.g., a field-programmable gate array (FPGA)). In this embodiment, each flash memory wafer 222 has pages organized as sixteen-kB (kilobyte) pages 224 and registers 226 capable of writing or reading data relative to the flash memory wafer 222. In a further embodiment, other types of solid-state memory are used instead of the flash memory illustrated in flash memory wafer 222 or as an addition to the flash memory illustrated in flash memory wafer 222.

[0098] In the various embodiments disclosed herein, storage cluster 161 can generally be contrasted with a storage array. Storage nodes 150 are part of a collection that creates storage cluster 161. Each storage node 150 possesses the data slices and computations required to provide data. Multiple storage nodes 150 collaborate to store and retrieve data. Storage memories or storage devices, as generally used in storage arrays, are less involved in data processing and manipulation. Storage memories or storage devices in a storage array receive commands for reading, writing, or erasing data. Storage memories or storage devices in a storage array are unaware of the larger system they are embedded in or what the data means. Storage memories or storage devices in a storage array can include various types of storage memories, such as RAM, solid-state drives, hard disk drives, etc. The non-volatile solid-state storage 152 described herein has multiple interfaces that are active simultaneously and serve multiple purposes. In some embodiments, some functions of storage nodes 150 are offloaded to non-volatile solid-state storage 152, thereby transforming non-volatile solid-state storage 152 into a combination of non-volatile solid-state storage 152 and storage nodes 150. Placing computation (relative to the stored data) in non-volatile solid-state storage 152 brings this computation closer to the data itself. Various system embodiments include a hierarchical structure with storage node layers possessing different capabilities. In contrast, in a storage array, the controller owns and knows everything related to all the data managed by the controller in the shelving or storage device. In storage cluster 161, as described herein, multiple controllers in multiple non-volatile solid-state storage units 152 and / or storage nodes 150 cooperate in various ways (e.g., for erasure coding, data fragmentation, metadata communication and redundancy, storage capacity expansion or contraction, and data recovery, etc.).

[0099] Figure 2D This illustrates a storage server environment that uses... Figure 2A An embodiment of storage node 150 and non-volatile solid-state storage 152 in ~C. In this version, each non-volatile solid-state storage 152 is located in chassis 138 (see...). Figure 2A The PCIe (Peripheral Component Interconnect Express) board in the ) has a controller such as 212 (see Figure 2C Processors such as FPGAs (Field-Programmable Gate Arrays), flash memory 206, and NVRAM 204 (i.e., DRAM 216 supported by supercapacitors, see...) Figure 2B and 2C The non-volatile solid-state storage 152 can be implemented as a single board containing the storage and can be the largest tolerable fault domain within the chassis. In some embodiments, up to two non-volatile solid-state storage units 152 may fail, and the device will continue without data loss.

[0100] In some embodiments, physical storage is partitioned into named regions based on application purpose. NVRAM 204 is a contiguous reserved memory block within DRAM 216 of non-volatile solid-state storage 152 and is backed by NAND flash memory. NVRAM 204 is logically partitioned into multiple memory regions, two of which are written as spools (e.g., spool_regions). The space within the NVRAM 204 spool is managed independently by respective permissions 168. Each device provides a certain amount of storage space to each permission 168. The permission 168 further manages the lifecycle and allocation within that space. Examples of spooling include distributed transactions or concepts. When the main power supply to non-volatile solid-state storage 152 fails, an onboard supercapacitor provides a short-duration power hold. During this hold interval, the contents of NVRAM 204 are flushed to flash memory 206. Upon the next power-on, the contents of NVRAM 204 are restored from flash memory 206.

[0101] Regarding the storage unit controller, the responsibility of the logical "controller" is distributed across each blade, containing 168 permissions. This distribution of logical control... Figure 2D The diagram shows a host controller 242, a middleware controller 244, and one or more storage unit controllers 246. The management of the control plane and the storage plane is handled independently, but components can physically coexist on the same blade. Each authority 168 effectively functions as an independent controller. Each authority 168 provides its own data and metadata structure, its own background workers, and maintains its own lifecycle.

[0102] Figure 2E This is a hardware block diagram of the blade 252, which is shown in Figure 2D Use in storage server environment Figure 2A The ~C storage node 150 and non-volatile solid-state storage 152 embodiment includes a control plane 254, compute and storage planes 256 and 258, and permissions 168 for interacting with underlying physical resources. The control plane 254 is partitioned into multiple permissions 168 that can utilize compute resources in the compute plane 256 to run on any blade 252. The storage plane 258 is partitioned into a set of devices, each providing access to flash memory 206 and NVRAM 204 resources. In one embodiment, as described herein, the compute plane 256 can act as a storage array controller over one or more devices of the storage plane 258 (e.g., a storage array).

[0103] exist Figure 2EIn the compute plane 256 and storage plane 258, permission 168 interacts with the underlying physical resources (i.e., devices). From the perspective of permission 168, its resources are striped across all physical devices. From the perspective of a device, it provides resources to all permissions 168, regardless of where the permission is actually running. Each permission 168 has been allocated or has always been allocated one or more partitions 260 of storage memory in non-volatile solid-state storage 152, such as partitions 260 in flash memory 206 and NVRAM 204. Each permission 168 uses these allocated partitions 260 belonging to it to write or read user data. Permissions can be associated with different amounts of physical storage in the system. For example, a permission 168 may have a larger number or larger size of partitions 260 in one or more non-volatile solid-state storage 152 compared to one or more other permissions 168.

[0104] Figure 2F A resilient software layer in blade 252 of a storage cluster according to some embodiments is depicted. In the resilient architecture, the resilient software is symmetrical, i.e., the compute modules 270 of each blade run... Figure 2F The three depicted processing layers are identical. Storage manager 274 executes read and write requests from other blades 252 for data and metadata stored in local non-volatile solid-state storage 152, NVRAM 204, and flash memory 206. Authority 168 fulfills client requests by issuing the necessary read and write requests to the blade 252 residing on the non-volatile solid-state storage 152. Endpoint 272 parses client connection requests received from the monitoring software of switch architecture 146, forwards the client connection requests to authority 168 responsible for completion, and forwards the response of authority 168 to the client. The symmetrical three-tier architecture enables high concurrency in the storage system. In these embodiments, elasticity scales out efficiently and reliably. Furthermore, elasticity implements a unique scaling-out technique that allows for uniform balancing across all resources regardless of client access patterns and maximizes concurrency by eliminating the significant need for inter-blade coordination that typically occurs using traditional distributed locking.

[0105] Still referencing Figure 2FThe permissions 168, running in the compute module 270 of blade 252, perform the internal operations required to fulfill the client request. A characteristic of this resilience is that the permissions 168 are stateless; they cache active data and metadata in the DRAM of their own blade 252 for fast access, but each permission stores each update in an NVRAM 204 partition on its three separate blades 252 until the update is written to flash memory 206. In some embodiments, all storage system writes to NVRAM 204 are performed three times on the partitions on the three separate blades 252. Utilizing triple-mirrored NVRAM 204 and persistent storage protected by parity and Reed-Solomon RAID checksums, the storage system can survive concurrent failures of two blades 252 without losing data, metadata, or access to either.

[0106] Because permission 168 is stateless, it can be migrated between blades 252. Each permission 168 has a unique identifier. In some embodiments, NVRAM 204 and flash memory 206 partitions are associated with the identifier of permission 168, rather than with the blade 252 on which they run. Therefore, when permission 168 is migrated, it continues to manage the same storage partition from its new location. In embodiments where a new blade 252 is installed in the storage cluster, the system automatically rebalances the load by partitioning the storage of the new blade 252 for use by the system's permissions 168, migrating the selected permissions 168 to the new blade 252, launching endpoints 272 on the new blade 252, and including these endpoints 272 in the client connection distribution algorithm of the switch architecture 146.

[0107] The migrated permissions 168 maintain the contents of their NVRAM 204 partitions on flash memory 206 from their new locations, process read and write requests from other permissions 168, and fulfill client requests to them from endpoint 272. Similarly, if blade 252 fails or is removed, the system redistributes its permissions 168 across the remaining blades 252 in the system. The redistributed permissions 168 continue to perform their original functions from their new locations.

[0108] Figure 2GThe diagram depicts permissions 168 and storage resources within blades 252 of a storage cluster according to some embodiments. Each permission 168 is specifically responsible for partitioning the flash memory 206 and NVRAM 204 on its respective blade 252. Permission 168 manages the content and integrity of its partitions independently of other permissions 168. Permission 168 compresses incoming data and temporarily stores it in its NVRAM 204 partition, then merges, RAID-protects, and maintains the data in a storage segment within its flash memory 206 partition. As permission 168 writes data to flash memory 206, storage manager 274 performs necessary flash conversions to optimize write performance and maximize media lifetime. In the background, permission 168 performs "garbage collection" or reclaims space occupied by data discarded by clients by rewriting the data. It should be understood that because the partitions of permission 168 are not contiguous, distributed locking is not required for client and write or background functions.

[0109] The embodiments described herein can utilize various software, communication, and / or network protocols. Furthermore, the hardware and / or software configurations can be adjusted to accommodate various protocols. For example, this embodiment can utilize Active Directory, which is used in Windows... TM Database-based systems provide authentication, directories, policies, and other services in the environment. In these embodiments, LDAP (Lightweight Directory Access Protocol) is an example application protocol for querying and modifying entries in directory service providers such as Active Directory. In some embodiments, a Network Lock Manager (NLM) is used to work in conjunction with the Network File System (NFS) to provide System V-style advisory documents and facilities for locking records over the network. The Server Message Block (SMB) protocol (one version of which is also known as the Common Internet File System (CIFS)) can be integrated with the storage systems discussed herein. SMP operates as an application-layer network protocol commonly used to provide shared access to files, printers, and serial ports, as well as various communications between nodes on a network. SMB also provides an authenticated inter-process communication mechanism. AMAZON TMS3 (Simple Storage Service) is a web service provided by Amazon Web Services, and the systems described herein can be integrated with Amazon S3 via web service interfaces (REST (Representational State Transfer), SOAP (Simple Object Access Protocol), and BitTorrent). RESTful APIs (Application Programming Interfaces) break down transactions into a series of small modules. Each module handles a specific, fundamental part of the transaction. The control or permission provided by these embodiments (especially for object data) may include the use of Access Control Lists (ACLs). An ACL is a list of permissions attached to an object, specifying which users or systems are granted access to the object and what operations are permitted for a given object. Systems may utilize Internet Protocol version 6 (IPv6) and IPv4 as communication protocols used to provide identification and location systems for computers on a network and to route traffic across the Internet. Packet routing between networked systems may include Equal Cost Multipath Routing (ECMP), a routing policy where next-hop packets forwarded to a single destination can occur on multiple "best paths" that are tied for first place in the routing metric calculation. Multipath routing can be used in conjunction with most routing protocols because it is limited to per-hop decisions by a single router. The software can support multi-tenancy, an architecture where a single instance of a software application serves multiple clients. Each client can be referred to as a tenant. In some embodiments, tenants may be given the ability to customize parts of the application, but not the application's code. These embodiments can maintain audit logs. Audit logs are documents that record events in a computing system. In addition to recording which resources were accessed, audit log entries typically include destination and source addresses, timestamps, and user login information conforming to various regulations. These embodiments can support various key management strategies, such as cryptographic key rotation. Additionally, the system can support dynamic root ciphers or some dynamically changing cipher.

[0110] Figure 3A A diagram illustrates a storage system 306 coupled to a cloud service provider 302 for data communication, according to some embodiments of the present invention. Although depicted in limited detail, Figure 3A The described storage system 306 can be compared with the above reference. Figures 1A-1D and Figures 2A-2G The storage system described above is similar. In some embodiments, Figure 3AThe described storage system 306 can be embodied as a storage system including an unbalanced active / active controller, a storage system including a balanced active / active controller, a storage system including an active / active controller that utilizes less than all of the resources of each controller so that each controller has reserve resources available to support failover, a storage system including a fully active / active controller, a storage system including controllers with data set isolation, a storage system including a two-tier architecture with a front-end controller and a back-end integrated storage controller, a storage system including a scale-out cluster of dual-controller arrays, and combinations of these embodiments.

[0111] exist Figure 3A In the depicted example, storage system 306 is coupled to cloud service provider 302 via data communication link 304. Data communication link 304 may be embodied as a dedicated data communication link, a data communication path provided by using one or more data communication networks such as a wide area network (WAN) or local area network (LAN), or some other mechanism capable of transmitting digital information between storage system 306 and cloud service provider 302. Such data communication link 304 may be entirely wired, entirely wireless, or some aggregation of wired and wireless data communication paths. In such an example, one or more data communication protocols may be used to exchange digital information between storage system 306 and cloud service provider 302 via data communication link 304. For example, handheld device transfer protocol (HDTP), hypertext transfer protocol (HTTP), Internet Protocol (IP), real-time transfer protocol (RTP), transmission control protocol (TCP), user datagram protocol (UDP), wireless application protocol (WAP), or other protocols may be used to exchange digital information between storage system 306 and cloud service provider 302 via data communication link 304.

[0112] Figure 3A The depicted cloud service provider 302 can be embodied, for example, as a system and computing environment that provides services to users of the cloud service provider 302 via a data communication link 304 through shared computing resources. The cloud service provider 302 can provide on-demand access to a shared pool of configurable computing resources such as computer networks, servers, storage, applications, and services. This shared pool of configurable resources can be quickly provided and released to users of the cloud service provider 302 with minimal management effort. Generally, users of the cloud service provider 302 are unaware of the exact computing resources utilized by the cloud service provider 302 to provide the service. Although in many cases such a cloud service provider 302 may be accessible via the Internet, those skilled in the art will recognize that any system that extracts the use of shared resources to provide services to users via any data communication link can be considered a cloud service provider 302.

[0113] exist Figure 3A In the depicted example, cloud service provider 302 can be configured to provide various services to storage system 306 and its users by implementing various service models. For example, cloud service provider 302 can be configured to provide services to storage system 306 and its users by implementing an Infrastructure as a Service (IaaS) model, whereby cloud service provider 302 provides subscribers with computing infrastructure such as virtual machines and other resources as a service. Alternatively, cloud service provider 302 can be configured to provide services to storage system 306 and its users by implementing a Platform as a Service (PaaS) model, whereby cloud service provider 302 provides application developers with a development environment. This development environment may, for example, include an operating system, a programming language execution environment, a database, a web server, or other components that application developers can utilize to develop and run software solutions on the cloud platform. Furthermore, cloud service provider 302 can be configured to provide services to storage system 306 and its users by implementing a Software as a Service (SaaS) model. This model provides application software, databases, and a platform for running applications to storage system 306 and its users, thereby providing on-demand software and eliminating the need to install and run applications on local computers. This simplifies application maintenance and support. Cloud service provider 302 can also be configured to provide services to storage system 306 and its users by implementing an Authentication as a Service (AaaS) model. This model provides authentication services that can be used to protect access to applications, data sources, or other resources. Finally, cloud service provider 302 can be configured to provide services to storage system 306 and its users by implementing a Storage as a Service model. This model provides access to its storage infrastructure for use by storage system 306 and its users. The reader will understand that cloud service provider 302 may be configured to provide additional services to storage system 306 and users of storage system 306 by implementing additional service models, since the above service models are included for illustrative purposes only and in no way imply any limitation on the services that cloud service provider 302 may provide, nor any limitation on the service models that cloud service provider 302 may implement.

[0114] exist Figure 3AIn the examples depicted, cloud service provider 302 may be embodied as, for example, a private cloud, a public cloud, or a combination of private and public clouds. In an embodiment where cloud service provider 302 is embodied as a private cloud, cloud service provider 302 may be dedicated to providing services to a single organization rather than multiple organizations. In an embodiment where cloud service provider 302 is embodied as a public cloud, cloud service provider 302 may provide services to multiple organizations. Public and private cloud deployment models may differ and may come with various advantages and disadvantages. For example, because public cloud deployment involves sharing computing infrastructure across different organizations, such deployment may not be ideal for organizations with security concerns, mission-critical workloads, and uptime requirements. While private cloud deployment can address some of these issues, it may require internal staff to manage the private cloud. In another alternative embodiment, cloud service provider 302 may be embodied as a hybrid of private cloud services and public cloud services deployed in a hybrid cloud environment.

[0115] Despite Figure 3A While not explicitly described, the reader will understand that additional hardware and software components may be required to facilitate the delivery of cloud services to storage system 306 and its users. For example, storage system 306 may be coupled to (or even include) a cloud storage gateway. Such a cloud storage gateway may, for example, be embodied as a hardware-based or software-based application located locally alongside storage system 306. This cloud storage gateway can operate as a bridge between local applications running on storage system 306 and the remote, cloud-based storage utilized by storage system 306. By using a cloud storage gateway, organizations can move their primary iSCSI or NAS to cloud service provider 302, thereby saving space on their internal storage systems. Such a cloud storage gateway can be configured to emulate disk arrays, block-based devices, file servers, or other storage systems that can translate SCSI commands, file server commands, or other suitable commands into REST space protocols that facilitate communication with cloud service provider 302.

[0116] To enable storage system 306 and its users to utilize the services provided by cloud service provider 302, a cloud migration process can be performed, during which data, applications, or other elements from the organization's on-premises systems (or even from another cloud environment) are moved to cloud service provider 302. To successfully migrate data, applications, or other elements to the environment of cloud service provider 302, middleware such as cloud migration tools can be used to bridge the gap between the environment of cloud service provider 302 and the organizational environment. Such cloud migration tools can also be configured to address the potentially high network costs and long transfer times associated with migrating large amounts of data to cloud service provider 302, as well as the security issues associated with migrating sensitive data to cloud service provider 302 over data communication networks. To further enable storage system 306 and its users to utilize the services provided by cloud service provider 302, a cloud orchestrator can be used to deploy and coordinate automated tasks to create merged processes or workflows. Such a cloud orchestrator can perform tasks such as configuring various components (whether cloud or on-premises) and managing the interconnections between these components. Cloud orchestrators can simplify communication and connectivity between components to ensure that links are configured and maintained correctly.

[0117] exist Figure 3A In the illustrated example and as briefly described above, cloud service provider 302 can be configured to provide services to storage system 306 and its users using a SaaS service model. Cloud service provider 302 provides storage system 306 and its users with application software, databases, and a platform for running applications, thereby providing on-demand software and eliminating the need to install and run applications on local computers. This simplifies application maintenance and support. According to various embodiments of the invention, these applications can take many forms. For example, cloud service provider 302 can be configured to provide storage system 306 and its users with access to a data analytics application. Such a data analytics application can, for example, be configured to receive telemetry data callbacked by storage system 306. This telemetry data can describe various operational characteristics of the storage system 306 and can be analyzed to, for example, determine the health status of the storage system 306, identify the workloads performed on the storage system 306, predict when the storage system 306 will exhaust various resources, recommend configuration changes, hardware or software upgrades, workflow migrations, or other actions that may improve the operation of the storage system 306.

[0118] The cloud service provider 302 can also be configured to provide access to a virtualized computing environment to the storage system 306 and its users. This virtualized computing environment can, for example, be embodied in a virtual machine or other virtualized computer hardware platform, a virtual storage device, and virtualized computer network resources. Examples of such a virtualized environment may include virtual machines created to emulate a physical computer, a virtualized desktop environment that separates the logical desktop from the physical machine, a virtualized file system that allows unified access to different types of specific file systems, and many other environments.

[0119] To further illustrate, Figure 3B A diagram illustrating a storage system 306 according to some embodiments of the present invention is provided. Although the depiction is not very detailed, Figure 3B The described storage system 306 can be compared with the above reference. Figures 1A-1D and Figures 2A-2G The storage system described above is similar, as it may include many of the components mentioned above.

[0120] Figure 3BThe depicted storage system 306 may include storage resource 308, which may be embodied in a variety of forms. For example, in some embodiments, storage resource 308 may include nano-RAM or other forms of non-volatile random access memory utilizing carbon nanotubes deposited on a substrate. In some embodiments, storage resource 308, in conjunction with a stackable cross-grid data access array, may include 3D cross-point non-volatile memory, where bit storage is based on changes in bulk resistance. In some embodiments, storage resource 308 may include flash memory, including single-cell (SLC) NAND flash, multi-cell (MLC) NAND flash, three-cell (TLC) NAND flash, and four-cell (QLC) NAND flash, etc. In some embodiments, storage resource 308 may include non-volatile magnetoresistive random access memory (MRAM) that stores data using magnetic storage elements, including spin-transfer torque (STT) MRAM. In some embodiments, example storage resource 308 may include non-volatile phase-change memory (PCM) that may have the ability to accommodate multiple bits in a single cell, since the cell can realize multiple different intermediate states. In some embodiments, storage resource 308 may include a quantum memory that allows for the storage and retrieval of photonic quantum information. In some embodiments, example storage resource 308 may include resistive random access memory (ReRAM), wherein data is stored by varying the resistance across a dielectric solid-state material. In some embodiments, storage resource 308 may include storage-class memory (SCM), wherein the solid-state non-volatile memory may be fabricated at high density using some combination of sub-lithographic patterning techniques, multiple bits per cell, and multiple device layers, etc. The reader will understand that the storage system described above may utilize other forms of computer memory and storage devices, including DRAM, SRAM, EEPROM, general-purpose memory, and many other memories. Figure 3B The described storage resource 308 can be represented in various form factors, including but not limited to dual in-line memory modules (DIMMs), non-volatile dual in-line memory modules (NVDIMMs), M.2 and U.2, etc.

[0121] Figure 3BThe depicted storage resource 308 can include various forms of storage class memory (SCM). SCM can effectively treat fast, non-volatile memory (e.g., NAND flash) as an extension of DRAM, allowing the entire dataset to be viewed as an in-memory dataset entirely within DRAM. SCM can, for example, include non-volatile media such as NAND flash. Such NAND flash can be accessed using NVMe, which can use the PCIe bus as its transport bus, thus providing relatively low access latency compared to earlier protocols. In fact, network protocols used for SSDs in all-flash arrays can include NVMe using Ethernet (ROCE, NVMe TCP), Fibre Channel (NVMe FC), and Infinite Bandwidth (iWARP), which allow fast, non-volatile memory to be treated as an extension of DRAM. Given that DRAM is typically byte-addressable and fast, non-volatile memory such as NAND flash is block-addressable, a controller software / hardware stack may be required to translate block data into bytes stored in the medium. Examples of media and software that can be used as SCMs include, for example, 3D XPoint, Intel Memory Drive Technology, and Samsung's Z-SSD.

[0122] Figure 3B The illustrated example storage system 306 can implement various storage architectures. For example, a storage system according to some embodiments of the invention can utilize block storage, where data is stored in blocks, and each block essentially functions as a separate hard disk drive. A storage system according to some embodiments of the invention can utilize object storage, where data is managed as objects. Each object may include the data itself, variable metadata, and a globally unique identifier, where object storage can be implemented at multiple levels (e.g., device level, system level, interface level). A storage system according to some embodiments of the invention utilizes file storage, where data is stored in a hierarchical structure. Such data can be stored in files and folders and presented in the same format to both the system storing the data and the system retrieving the data.

[0123] Figure 3B The illustrated example storage system 306 can be embodied as a storage system in which additional storage resources can be added using a scale-up model, a scale-out model, or some combination thereof. In the scale-up model, additional storage can be added by adding additional storage devices. However, in the scale-out model, additional storage nodes can be added to a cluster of storage nodes, where such storage nodes may include additional processing resources and additional network resources, etc.

[0124] Figure 3B The depicted storage system 306 also includes communication resources 310, which may facilitate data communication between components within the storage system 306 and between the storage system 306 and computing devices outside the storage system 306. Communication resources 310 can be configured to utilize various protocols and data communication architectures to facilitate data communication between components within the storage system and between the storage system and computing devices outside the storage system. For example, communication resources 310 may include Fibre Channel (FC) technologies such as FC architectures and FC protocols, which can transmit SCSI commands across FC networks. Communication resources 310 may also include FC over Ethernet (FCoE) technologies, through which FC frames are encapsulated and transmitted across Ethernet networks. Communication resources 310 may also include Infinite Bandwidth (IB) technologies, in which switch architecture topologies are utilized to facilitate transmission between channel adapters. Communication resources 310 may also include NVM Express (NVMe) technologies and cross-architecture NVMe (NVMeoF) technologies, through which non-volatile storage media attached via the PCI Express (PCIe) bus can be accessed. Communication resources 310 may also include: a mechanism for accessing storage resources 308 within storage system 306 using Serial Attached SCSI (SAS); a Serial ATA (SATA) bus interface for connecting storage resources 308 within storage system 306 to a host bus adapter within storage system 306; Internet Small Computer System Interface (iSCSI) technology for providing block-level access to storage resources 308 within storage system 306; and other communication resources that may facilitate data communication between components within storage system 306 and data communication between storage system 306 and computing devices outside storage system 306.

[0125] Figure 3B The depicted storage system 306 also includes processing resources 312 that can facilitate the execution of computer program instructions and other computational tasks within the storage system 306. Processing resources 312 may include one or more application-specific integrated circuits (ASICs) and one or more central processing units (CPUs) customized for a particular purpose. Processing resources 312 may also include one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), one or more system-on-a-chip (SoC) or other forms of processing resources 312. Storage system 306 can utilize processing resources 312 to perform various tasks, including but not limited to supporting the execution of software resources 314, which will be described in more detail below.

[0126] Figure 3B The depicted storage system 306 also includes software resource 314, which, when executed by processing resource 312 within storage system 306, can perform various tasks. Software resource 314 may, for example, include one or more computer program instruction modules that, when executed by processing resource 312 within storage system 306, facilitate the implementation of various data protection techniques to maintain the integrity of the data stored within the storage system. The reader will understand that such data protection techniques can be implemented, for example, through system software running on the computer hardware within the storage system, through a cloud service provider, or otherwise. Such data protection techniques may include, for example, data archiving techniques that move data no longer actively used to a separate storage device or separate storage system for long-term retention; data backup techniques that allow data stored in the storage system to be copied and stored in different locations to prevent data loss in the event of storage system failure or some other form of disaster; data replication techniques that copy data stored in one storage system to another storage system so that the data can be accessed via multiple storage systems; data snapshot techniques that capture the state of data within the storage system at different points in time; data and database cloning techniques that can create duplicate copies of data and databases; and other data protection techniques. By using this data protection technology, business continuity and disaster recovery goals can be met, because a failure of the storage system may not result in the loss of the data stored in the storage system.

[0127] Software resource 314 may also include software that facilitates the implementation of software-defined storage (SDS). In such an example, software resource 314 may include one or more computer program instruction modules that, when executed, facilitate the policy-based provisioning and management of data storage independent of the underlying hardware. Such software resource 314 can facilitate storage virtualization to separate storage hardware from the software that manages the storage hardware.

[0128] Software resource 314 may also include software that facilitates and optimizes I / O operations targeting storage resources 308 in storage system 306. For example, software resource 314 may include software modules that perform various data reduction techniques such as data compression and data deduplication. Software resource 314 may include software modules that intelligently group I / O operations to facilitate better use of the underlying storage resources 308, software modules that perform data migration operations for migration from within the storage system, and software modules that perform other functions. Such software resource 314 may be embodied as one or more software containers or in many other ways.

[0129] The reader will understand that the existence of these software resources 314 can provide an improved user experience for the storage system 306, an expansion of the functionality supported by the storage system 306, and many other benefits. Consider a concrete example of software resource 314 that performs data backup technology where data stored in the storage system can be copied and stored in different locations to avoid data loss in the event of equipment failure or some other form of disaster. In such an example, the system described herein can perform backup operations more reliably (and in a less burdensome manner for the user) compared to interactive backup management systems that require a high degree of user interaction and provide less robust automation and feature sets, etc.

[0130] The aforementioned storage system can implement intelligent data backup technology, allowing data stored in the storage system to be copied and stored in different locations to avoid data loss in the event of device failure or some other form of disaster. For example, the aforementioned storage system can be configured to inspect each backup to prevent the storage system from being restored to an undesirable state. Consider an example of a storage system infected with malware. In such an example, the storage system may include software resource 314 that can scan each backup to identify backups captured before and after malware infection. In such an example, the storage system can restore itself from backups that do not contain malware, or at least not restore the portion of the backup containing malware. In such an example, the storage system may include software resource 314 that can scan each backup to identify the presence of malware, for example, by identifying write operations served by the storage system and originating from a network subnet suspected of having delivered malware (or a virus or some other undesirable substance); by identifying write operations served by the storage system and originating from users suspected of having delivered malware; by identifying write operations served by the storage system and examining the content of the write operations against a malware fingerprint; and by employing many other methods.

[0131] Readers will further understand that backups (typically in the form of one or more snapshots) can also be used for rapid recovery of storage systems. Consider an example where a storage system is infected with ransomware that locks users out of the system. In such an example, software resource 314 within the storage system can be configured to detect the presence of ransomware and can be further configured to restore the storage system to a point in time prior to the time the ransomware infected the storage system using a retained backup. In such an example, the presence of ransomware can be explicitly detected using software tools exploited by the system, by using a key inserted into the storage system (e.g., a USB drive), or in a similar manner. Similarly, the presence of ransomware can be inferred in response to system activity satisfying, for example, a predetermined fingerprint (such as no reads or writes to the system within a predetermined time period).

[0132] The reader will understand, Figure 3B The various components depicted can be grouped into one or more optimized compute packages as a converged infrastructure. This converged infrastructure can include pools of computing, storage, and networking resources that can be shared by multiple applications and managed collectively using policy-driven processing. This converged infrastructure can minimize compatibility issues between components within storage system 306, while also reducing the various costs associated with establishing and operating storage system 306. This converged infrastructure can be implemented using a converged infrastructure reference architecture, using individual devices, using a software-driven hyperconverged approach (e.g., hyperconverged infrastructure), or otherwise.

[0133] The reader will understand, Figure 3B The described storage system 306 can help support a wide variety of software applications. For example, storage system 306 can help support applications such as artificial intelligence (AI) applications, database applications, DevOps projects, electronic design automation tools, event-driven software applications, high-performance computing applications, simulation applications, high-speed data capture and analysis applications, machine learning applications, media production applications, media service applications, picture archiving and communication system (PACS) applications, software development applications, virtual reality applications, augmented reality applications, and many other types of applications.

[0134] The aforementioned storage systems can operate to support a wide range of applications. Given that storage systems encompass computing resources, storage resources, and various other resources, they may be well-suited to support resource-intensive applications, such as AI applications. Such AI applications enable devices to perceive their environment and take actions that maximize the probability of success at a given goal. Examples of these AI applications include IBM Watson, Microsoft Oxford, Google DeepMind, and Baidu Minwa. The aforementioned storage systems are also well-suited to support other types of resource-intensive applications, such as machine learning applications. Machine learning applications can perform various types of data analysis to automate the construction of analytical models. Using algorithms that iteratively learn from data, machine learning applications enable computers to learn without explicit programming. A specific area of ​​machine learning is called reinforcement learning, which involves taking appropriate actions in a given situation to maximize rewards. Reinforcement learning can be used to find the optimal behavior or path that a particular software application or machine should take in a given situation. Reinforcement learning differs from other areas of machine learning (e.g., supervised learning, unsupervised learning) in that, with reinforcement learning, there is no need to present correct input / output pairs or explicitly correct suboptimal actions.

[0135] In addition to the resources already described, the aforementioned storage system may also include a graphics processing unit (GPU), sometimes also referred to as a visual processing unit (VPU). Such a GPU can be embodied as specialized electronic circuitry that rapidly manipulates and modifies memory to accelerate the creation of images intended for output to a frame buffer on a display device. This GPU can be included in any computing device that is part of the aforementioned storage system (including as one of many separately expandable components of the storage system), wherein other examples of separately expandable components of the storage system may include storage components, memory components, computing components (e.g., CPU, FPGA, ASIC), network components, and software components, etc. Besides the GPU, the aforementioned storage system may also include a neural network processor (NNP) for various aspects of neural network processing. This NNP can be used as a replacement (or addition) to the GPU and can also be independently expandable.

[0136] As described above, the storage system described herein can be configured to support artificial intelligence applications, machine learning applications, big data analytics applications, and many other types of applications. The rapid growth of these applications is driven by three technologies: deep learning (DL), GPU processors, and big data. Deep learning is a computational model that leverages massively parallel neural networks inspired by the human brain. Instead of software handcrafted by experts, deep learning models write their own software by learning from a vast number of examples. GPUs are modern processors with thousands of cores, well-suited for running algorithms that loosely represent the parallel nature of the human brain.

[0137] The development of deep neural networks has spurred a wave of new algorithms and tools, enabling data scientists to leverage artificial intelligence (AI) to mine their data. Utilizing improved algorithms, larger datasets, and various frameworks (including open-source software libraries for machine learning across a range of tasks), data scientists are tackling new use cases such as autonomous vehicles, natural language processing and understanding, computer vision, machine reasoning, and strong AI. Applications of this technology can include: machine and vehicle object detection, recognition, and avoidance; visual recognition, classification, and labeling; performance management of algorithmic financial trading strategies; simultaneous localization and mapping; predictive maintenance of high-value machinery; prevention of cybersecurity threats; automation of expertise; image recognition and classification; question answering; robotics; and text analysis (extraction, classification) and text generation and translation; and more. Applications of AI technology already implemented in a wide range of products include Amazon Echo's speech recognition technology that allows users to converse with their machines; Google Translate™ that enables machine-based language translation; Spotify's Discover Weekly, which provides recommendations for new songs and artists users might like based on user usage and business analytics; Quill's text generation service that acquires structured data and transforms it into narrative stories; and chatbots that provide real-time, context-specific answers to questions in a conversational format. Furthermore, AI can impact various industries and sectors. For example, AI solutions can be used in healthcare to record clinical logs, patient files, research data, and other inputs to generate potential treatment options for doctors to explore. Similarly, retailers can use AI solutions to personalize consumer recommendations based on digital footprints of individual behavior, archival data, or other data.

[0138] However, training deep neural networks requires both high-quality input data and massive computation. GPUs are massively parallel processors capable of operating on large amounts of data simultaneously. When combined into multi-GPU clusters, high-throughput pipelines may be needed to feed input data from storage to the compute engine. Deep learning is not just about building and training models. There is also an entire data pipeline that must be designed for the scale, iterations, and experimentation required for a data science team to succeed.

[0139] Data is central to modern AI and deep learning algorithms. One crucial issue that must be addressed before training can begin revolves around collecting labeled data essential for training accurate AI models. This may require a full-scale AI deployment to continuously collect, clean, transform, label, and store massive amounts of data. Adding additional high-quality data points directly translates into more accurate models and better insights. Data samples can undergo a series of processing steps, including but not limited to: 1) ingesting data from external sources into the training system and storing it in its raw form; 2) cleaning the data and transforming it into a training-friendly format, including linking data samples to appropriate labels; 3) exploring parameters and models, conducting rapid testing with smaller datasets, and iterating to converge to the model most promising for production clusters; 4) performing a training phase to select random batches of input data (including both new and old samples) and feeding this input data to production GPU servers for computation to update model parameters; and 5) evaluation, including using a retained portion of the data not used during training to evaluate the model's accuracy against the retained data. This lifecycle can be applied to any type of parallelized machine learning, not just neural networks or deep learning. For example, a standard machine learning framework may rely on a CPU instead of a GPU, but the data ingestion and training workflows can remain the same. The reader will understand that a single shared storage data center creates a coordination point throughout the entire lifecycle without requiring additional copies of data between ingestion, preprocessing, and training stages. The ingested data is rarely used for a single purpose, and shared storage provides the flexibility to train multiple different models or apply traditional analytics to the data.

[0140] Readers will understand that each stage in an AI data pipeline can have different requirements from the data center (e.g., a storage system or a collection of storage systems). Scaling out storage systems must provide uncompromising performance for all types and patterns of access (from small metadata volumes to large files, from random access to sequential access, and from low to high concurrency). The aforementioned storage systems can serve as ideal AI data centers because they can serve unstructured workloads. In the first stage, ideally, data is ingested and stored in the same data center that subsequent stages will use to avoid excessive data copying. The next two steps can be done on standard compute servers that optionally include GPUs, and then in the fourth and final stages, complete training production jobs are run on powerful GPU-accelerated servers. Typically, a production pipeline exists alongside the experimental pipeline that operates on the same dataset. Furthermore, GPU-accelerated servers can be used independently for different models, or combined to train a larger model, or even span multiple systems for distributed training. If the shared storage layer is slow, data must be copied to local storage at each stage, resulting in wasted time tiering data across different servers. An ideal data center for an AI training pipeline offers performance similar to data stored locally on server nodes, while also providing simplicity and performance to enable all pipeline-level concurrent operations.

[0141] Data scientists strive to improve the usability of trained models through various methods, including more data, better data, smarter training, and deeper models. In many cases, teams of data scientists will share the same dataset and work in parallel to produce new and improved trained models. Typically, teams will work concurrently on the same shared dataset throughout these phases. Multiple concurrent workloads of data processing, experimentation, and full-scale training layer the demands on multiple access patterns on the storage tier. In other words, storage cannot simply accommodate large file reads but must handle a mix of reads and writes of files of varying sizes. Finally, as multiple data scientists explore datasets and models, storing data in its native format to provide the flexibility for each user to transform, clean, and use data in a unique way can be crucial. The storage systems described above can provide a natural shared storage home for datasets, offering data protection redundancy (e.g., through the use of RAID 6) and the performance necessary to serve as a common access point for multiple developers and experiments. Using these storage systems avoids the need to carefully copy subsets of data for local work, saving both engineering and GPU-accelerated server usage time. The burden of these copies grows as the original dataset and desired transformations are continuously updated and changed.

[0142] Readers will understand that the fundamental reason for the surge in success of deep learning is the continuous improvement of models with large dataset sizes. In contrast, classic machine learning algorithms, such as Logistic Regression, stop improving accuracy at smaller dataset sizes. For this reason, the separation of computational and storage resources also allows for the independent scaling of each layer, avoiding many of the complexities inherent in managing both together. As dataset sizes grow, or new datasets are considered, outward-scaling storage systems must be able to scale easily. Similarly, if more concurrent training is needed, additional GPUs or other computational resources can be added without worrying about their internal storage. Furthermore, storage systems offer greater random read bandwidth, the ability to read small files (50 KB) at high random rates (meaning no extra effort is needed to aggregate individual data points to generate larger, storage-friendly files), the ability to scale capacity and performance as datasets or throughput requirements increase, the ability to support files or objects, the ability to tune performance for large or small files (i.e., without requiring a user-defined file system), the ability to support non-disruptive hardware and software upgrades even during production model training, and many other reasons, making it easier to build, operate, and scale AI systems.

[0143] The performance of the storage layer for small files can be critical, as many types of input (including text, audio, or images) are natively stored as small files. If the storage layer doesn't handle small files well, additional steps are needed to preprocess the samples and group them into larger files. Storage built on a spinning disk that relies on SSDs as a caching layer may not provide the required performance. Since training with random batches of input yields more accurate models, the entire dataset must be accessible at full performance. SSD caches only provide high performance for small subsets of the data and are ineffective in hiding the latency of spinning drives.

[0144] While the preceding paragraphs discussed deep learning applications, the reader will understand that the storage system described here can also be part of a distributed deep learning (DDL) platform to support the execution of DDL algorithms. Distributed deep learning can be used to significantly accelerate deep learning on GPUs (or other forms of accelerators or computer program instruction executors) utilizing distributed computing, enabling parallelization. Furthermore, the output of training machine learning and deep learning models (such as fully trained machine learning models) can be used for various purposes and can be combined with other tools. For example, trained machine learning models can be used with tools such as Core ML to integrate various machine learning model types into applications. In fact, trained models can be run using Core ML converter tools and plugged into custom applications that can be deployed on compatible devices. The aforementioned storage system can also be paired with other technologies such as TensorFlow (an open-source software library used for dataflow programming across a range of tasks applicable to machine learning applications such as neural networks) to facilitate the development of such machine learning models and applications.

[0145] Readers will further understand that as AI becomes available for mass consumption, the aforementioned systems can be deployed in various ways to support the democratization of AI. This democratization can include, for example, providing the capability of AI as a platform-as-a-service, the growth of artificial general intelligence (AGI), the proliferation of autonomous Level 4 and Level 5 vehicles, the availability of autonomous mobile robots, and the development of conversational AI platforms. For instance, these systems can be deployed in cloud environments, edge environments, or other environments that contribute to supporting the democratization of AI. As part of this democratization, a shift may occur from narrow AI to AGI, which encompasses a broad range of machine learning solutions targeted at specific tasks. In AGI, the use of machine learning is expanded to handle a wide range of use cases, and AGI, like humans, can essentially perform any intelligent task that humans can perform and that can learn dynamically.

[0146] The aforementioned storage systems can also be used in neuromorphic computing environments. Neuromorphic computing is a form of computation that simulates brain cells. To support neuromorphic computing, an architecture of interconnected "neurons" replaces traditional computational models, directly transmitting low-power signals between neurons for more efficient computation. Neuromorphic computing can utilize very large-scale integrated (VLSI) systems containing electronic analog circuitry to simulate the neurobiological architecture present in the nervous system, as well as analog, digital, and mixed-mode analog / digital VLSI, and software systems for implementing models of the nervous system for sensing, motor control, or multi-sensor integration.

[0147] The reader will understand that the storage system described above can be configured to support the storage or use of a blockchain (among other types of data). This blockchain can be represented as a growing list of records (called blocks) linked together and protected using cryptography. Each block in the blockchain can contain hash pointers as links to previous blocks, timestamps, and transaction data. Blockchains can be designed to resist data modification and can serve as open, distributed ledgers that efficiently and verifiably record transactions between two parties. This makes blockchains potentially suitable for recording events, medical records, and other record management activities such as identity management and transaction processing. In addition to supporting the storage and use of blockchain technology, the storage system described above can also support the storage and use of derivatives, such as those used by IBM. TM The Hyperledger project includes open-source blockchains and related tools, permissioned blockchains that allow access to a limited number of trusted parties, and blockchain products that enable developers to build their own distributed ledger projects. Readers will understand that blockchain technology can impact a wide range of industries and sectors. For example, blockchain technology can be used in real estate transactions as a blockchain-based contract, eliminating the need for third parties and enabling self-executing actions under certain conditions. Similarly, a universal health record can be created by aggregating and placing an individual's health history onto a blockchain ledger for access and updating by any healthcare provider or licensed healthcare provider.

[0148] Readers will understand that the uses of blockchain are not limited to financial transactions and contracts. In fact, blockchain can be used to decentralized aggregate, sort, timestamp, and archive any type of information, including structured data, correspondences, documents, or other data. By using blockchain, participants can verifiably and permanently agree on exactly what data was entered, when it was entered, and by whom, without relying on a trusted intermediary. For example, SAP's recently launched blockchain platform (which supports multi-chain and hyperledger structures) targets a wide range of supply chain and other non-financial applications.

[0149] One way to record data using blockchain is to directly embed each data point within a transaction. Each blockchain transaction can be digitally signed by one or more parties, replicated across multiple nodes, sorted and timestamped using the chain's consensus algorithm, and permanently stored in a tamper-proof manner. Therefore, any data within a transaction will be stored by each node in the same but independent way, along with proof of who wrote the data and when it was written. Users of the chain can retrieve this information at any time in the future. This type of storage can be called on-chain storage. However, on-chain storage may not be particularly practical when attempting to store very large datasets. Therefore, according to embodiments of the present invention, the blockchain and storage system described herein can be used to support both on-chain and off-chain storage of data.

[0150] Off-chain storage of data can be implemented in various ways and can occur even when the data itself is not stored on the blockchain. For example, in one embodiment, a hash function can be utilized, and the data itself can be fed into the hash function to generate a hash value. In such an example, the hash of a large amount of data can be embedded within a transaction instead of the data itself. Each hash can be used as a commitment to its input data, where the data itself is stored off-chain. The reader will understand that any blockchain participant who needs off-chain data cannot reproduce the data based on its hash, but if the data can be retrieved in other ways, the on-chain hash is used to confirm who created the data and when it was created. As with regular on-chain data, hashes can be embedded in digitally signed transactions whose consensus is included on-chain.

[0151] Readers will understand that alternatives to blockchain can be used in other embodiments to facilitate decentralized storage of information. For example, one usable alternative to blockchain is blockweave. While traditional blockchains store each transaction for verification, blockweave allows for secure decentralization without using the entire chain, enabling low-cost on-chain data storage. This blockweave can leverage consensus mechanisms based on Proof-of-Access (PoA) and Proof-of-Work (PoW). While typical PoW systems rely solely on the previous block to generate consecutive blocks, PoA algorithms can incorporate data from randomly selected previous blocks. Combined with the blockweave data structure, miners do not need to store all the blocks (forming the blockchain) but can instead store any previous blocks of the weave (blockweave) that forms the blocks. This improves scalability, speed, and cost, and reduces the cost of data storage, in part: because miners do not need to store all blocks, significantly reducing the power consumed during mining processing; because power consumption decreases as the network expands; and because blockweave requires less and less hashing power to reach consensus as data is added to the system. Furthermore, block weaving can be deployed in distributed storage networks, creating incentive mechanisms to encourage rapid data sharing. Such distributed storage networks can also utilize block shadowing, where nodes send only a minimum block "shadow" to other nodes that allow peers to reconstruct the complete block, rather than sending the full block itself.

[0152] The aforementioned storage systems can be used alone or in combination with other computing devices to support in-memory computing applications. In-memory computing involves storing information in RAM distributed across computer clusters. In-memory computing helps business customers, including retailers, banks, and utilities, to quickly detect patterns, instantly analyze massive amounts of data, and perform their operations rapidly. The reader will understand that the aforementioned storage systems, particularly those configurable with customizable amounts of processing, storage, and memory resources (e.g., blade systems containing various types of configurable resources), can be configured to provide the infrastructure to support in-memory computing. Similarly, the aforementioned storage systems can include components (e.g., NVDIMMs, 3D cross-point storage providing persistent, fast random access memory) that can effectively provide an improved in-memory computing environment compared to in-memory computing environments that rely on RAM distributed across dedicated servers.

[0153] In some embodiments, the storage system described above can be configured to operate as a hybrid in-memory computing environment that includes a common interface to all storage media (e.g., RAM, flash storage, 3D cross-linked storage). In such embodiments, users may not know the details of where their data is stored, but they can still address the data using the same complete, unified API. In such embodiments, the storage system can (in the background) move data to the fastest available tier, including intelligently placing data based on various characteristics of the data or some other heuristic. In such examples, the storage system can even leverage existing products such as Apache Ignite and GridGain to move data between different storage tiers, or the storage system can utilize custom software to move data between different storage tiers. The storage system described herein can implement various optimizations to, for example, improve the performance of in-memory computing, such as performing computations as close to the data as possible.

[0154] As the reader will further understand, in some embodiments, the storage system described above can be paired with other resources to support the applications described above. For example, an infrastructure may include primary computing in the form of servers and workstations dedicated to general-purpose computing on graphics processing units (GPGPUs) to accelerate deep learning applications interconnected to a computing engine for training parameters of deep neural networks. Each system may have Ethernet external connectivity, unlimited bandwidth external connectivity, some other form of external connectivity, or some combination thereof. In such an example, GPUs may be grouped for a single large training session or used independently to train multiple models. The infrastructure may also include storage systems (such as those described above) to provide, for example, scalable all-flash file or object storage that can access data via high-performance protocols such as NFS and S3. The infrastructure may also include, for example, redundant top-rack Ethernet switches connected to storage and computing via ports in MLAG port channels for redundancy. The infrastructure may also include additional computing in the form of white-box servers (optionally utilizing GPUs) for data ingestion, preprocessing, and model debugging. The reader will understand that there may be many more infrastructures.

[0155] Readers will understand that the system described above may be better suited for the applications described above compared to other systems, such as those that include Distributed Direct Attach Storage (DDAS) solutions deployed in server nodes. Such DDAS solutions can be built to handle large, fewer sequential accesses, but may be less capable of handling small random accesses. Readers will further understand that the storage system described above can be used to provide a platform for the applications described above, which is preferable to utilizing cloud-based resources, because the storage system can be included in a more secure, more locally and internally managed, and more robust on-premises or internal infrastructure in terms of feature set and performance, or the platform is preferable to utilizing cloud-based resources as part of a platform to support the applications described above. For example, services built on platforms such as IBM's Watson may require enterprises to distribute personal user information such as financial transaction information or identifiable patient records to other institutions. Therefore, for various technical and business reasons, providing AI-as-a-Service based in the cloud may be less desirable than AI-as-a-Service supported and provided internally by storage systems such as the storage system described above.

[0156] Readers will understand that the aforementioned storage system can be configured, either independently or in conjunction with other computing machines, to support other AI-related tools. For example, the storage system can utilize tools that facilitate the transfer of models written in different AI frameworks, such as ONXX or other open neural network exchange formats. Similarly, the storage system can be configured to support tools such as Amazon's Gluon, which allows developers to prototype, build, and train deep learning models. In fact, the aforementioned storage system could be, for example, IBM's... TM Cloud Private for Data (IBM) TM This is part of a larger platform, such as a private cloud, that integrates data science, data engineering, and application building services. Such a platform can seamlessly collect, organize, protect, and analyze data across the enterprise, and leverage a single solution to simplify hybrid data management, unified data governance and integration, data science, and business analytics.

[0157] Readers will further understand that the aforementioned storage systems can also be deployed as edge solutions. Such edge solutions can appropriately optimize cloud computing systems by performing data processing at the network edge, close to the data source. Edge computing pushes applications, data, and computing power (i.e., services) away from the central point to the logical end of the network. By using edge solutions such as the storage systems described above, computing tasks can be performed using the computing resources provided by such storage systems, data can be stored using the storage resources of the storage systems, and cloud-based services can be accessed using various resources of the storage systems, including network resources. By performing computing tasks on edge solutions, storing data related to edge solutions, and generally utilizing edge solutions, expensive cloud-based resources can be avoided, and in fact, performance can be improved relative to a greater reliance on cloud-based resources.

[0158] While many tasks may benefit from the use of edge solutions, certain applications may be particularly well-suited for deployment in such environments. For example, devices such as drones, autonomous vehicles, and robots may require extremely fast processing speeds; in fact, such speeds make sending data up to the cloud and receiving it back for processing support seem incredibly slow. Similarly, machines such as locomotives and gas turbines, which generate vast amounts of information using a wide range of data-generating sensors, can benefit from the rapid data processing capabilities of edge solutions. As an additional example, some IoT devices (such as connected cameras) may not be well-suited for utilizing cloud-based resources because sending data to the cloud may be impractical simply due to the sheer volume of data involved (not only from a privacy, security, or financial perspective). Therefore, many tasks truly related to data processing, storage, or communication may be better suited to platforms that include edge solutions (such as the aforementioned storage systems).

[0159] Consider specific examples of inventory management in warehouses, distribution centers, or similar locations. Large-scale inventory, warehousing, shipping, order fulfillment, manufacturing, or other operations involve large amounts of inventory on shelves and high-resolution digital cameras generating a stream of large amounts of data (firehose). All of this data can be fed into an image processing system that can reduce the data volume to a stream of small data. All of this small data can be stored in on-premises storage. On-premises storage at the facility edge can be coupled to the cloud, for example, for external reporting, real-time control, and cloud storage. The results of image processing can be used for inventory management, enabling inventory to be tracked on shelves and replenished, moved, shipped, modified with new products, or discontinued / obsolete products, etc. The above scenario is a primary candidate for embodiments of the configurable processing and storage systems described above. A combination of computational blades and unloading blades suitable for image processing, possibly combined with deep learning regarding unloading FPGAs or (one or more) custom unloading blades, can receive a stream of large data from all digital cameras and generate a stream of small data. All of this small data can then be stored by storage nodes that operate with storage units that optimally process the data stream, regardless of the type of storage blade combination. This is an example of storage and functionality acceleration and integration. Depending on external communication needs with the cloud and external processing within the cloud, and depending on network connectivity and cloud resource reliability, the system can be scalable for storage and compute management based on bursty workloads and variable conductance reliability. Additionally, depending on other inventory management aspects, the system can be configured for scheduling and resource management in hybrid edge / cloud environments.

[0160] The aforementioned storage systems can be used alone or in combination with other computing resources as a network edge platform that combines computing resources, storage resources, network resources, cloud technologies, and network virtualization technologies. As part of the network, the edge, from customer locations and backhaul aggregation facilities to points of presence (PoPs) and regional data centers, possesses characteristics similar to other network infrastructures. Readers will understand that network workloads such as Virtual Network Functions (VNFs) will reside on the network edge platform. Enabled through a combination of containers and virtual machines, the network edge platform can rely on controllers and schedulers that are no longer geographically located alongside data processing resources. Functions as microservices can be segmented into control plane, user plane, and data plane, or even state machines, allowing for independent optimization and scaling techniques. This user plane and data plane can be enabled through added accelerators (residing on server platforms such as FPGAs and smart NICs) and commercial silicon and programmable ASICs enabled by SDN.

[0161] The aforementioned storage systems can also be optimized for big data analytics. Big data analytics can generally be described as the process of examining large and diverse datasets to discover hidden patterns, unknown correlations, market trends, customer preferences, and other useful information that can help organizations make more informed business decisions. Big data analytics applications enable data scientists, predictive modelers, statisticians, and other analytics professionals to analyze ever-growing volumes of structured transactional data, traditional business intelligence (BI), and other forms of additional data that are not typically explored by analytics programs. As part of this process, semi-structured and unstructured data, such as internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone call details, IoT sensor data, and other data, can be transformed into structured forms. Big data analytics is a form of advanced analytics that involves complex applications utilizing elements such as predictive models, statistical algorithms, and hypothesis testing based on high-performance analytics systems.

[0162] The aforementioned storage system can also support (including implementation as a system interface) applications that respond to human language to perform tasks. For example, the storage system can support intelligent personal assistant applications such as Amazon's Alexa, Apple's Siri, Google Voice, Samsung Bixby, and Microsoft Cortana. While the example described in the previous sentence utilizes voice as input, the aforementioned storage system can also support chatbots, conversational bots, talkers, or artificial dialogue entities or other applications configured to converse via auditory or text methods. Similarly, the storage system can actually execute such applications to enable users (such as system administrators) to interact with the storage system via voice. Although in embodiments of the invention, such applications can serve as interfaces for various system management operations, they generally enable voice interaction, music playback, creating to-do lists, setting alarms, streaming podcasts, playing audiobooks, and providing weather, traffic, and other real-time information (such as news).

[0163] The aforementioned storage system can also implement an AI platform to realize the vision of self-driven storage. This AI platform can be configured to provide global predictive intelligence by collecting and analyzing vast amounts of storage system telemetry data points, enabling easy management, analysis, and support. In fact, such a storage system can be capable of predicting both capacity and performance, as well as generating intelligent recommendations related to workload deployment, interaction, and optimization. This AI platform can be configured to scan all incoming storage system telemetry data against a problem fingerprint library to predict and resolve events in real time before they impact the customer's environment, and capture hundreds of performance-related variables for predicting performance loads.

[0164] The aforementioned storage system can support the serialization or simultaneous execution of artificial intelligence applications, machine learning applications, data analysis applications, data transformation, and other tasks that can collectively form an AI ladder. This AI ladder can be effectively formed by combining these elements to create a complete data science pipeline, where dependencies exist between the elements. For example, AI may require some form of machine learning, machine learning may require some form of analysis, and analysis may require some form of data and information architecture design, and so on. Therefore, each element can be viewed as a step in an AI ladder that can collectively form a complete and complex AI solution.

[0165] The aforementioned storage systems can also be used, either alone or in combination with other computing environments, to deliver ubiquitous AI experiences, permeating a wide range of business and life. For example, AI can play a significant role in providing deep learning solutions, deep reinforcement learning solutions, artificial general intelligence solutions, autonomous vehicles, cognitive computing solutions, commercial UAVs or drones, conversational user interfaces, enterprise classification, ontology management solutions, machine learning solutions, smart dust, smart robots, and smart workplaces. These storage systems can also be used, either alone or in combination with other computing environments, to deliver a wide range of transparent immersive experiences, where technology can introduce transparency between people, businesses, and things. Such transparent immersive experiences can be provided as augmented reality, connected homes, virtual reality, brain-computer interfaces, human augmentation technologies, nanotube electronics, volumetric displays, or 4D printing technologies. Furthermore, these storage systems can also be used, either alone or in combination with other computing environments, to support various digital platforms. These digital platforms can include, for example, 5G wireless systems and platforms, digital twin platforms, edge computing platforms, IoT platforms, quantum computing platforms, serverless PaaS, software-defined security, and neuromorphic computing platforms.

[0166] Readers will understand that some transparent immersive experiences can involve digital twins of various “things” such as people, places, processes, and systems. Such digital twins, along with other immersive technologies, can transform how humans interact with technology, as conversational platforms, augmented reality, virtual reality, and mixed reality offer more natural and immersive interactions with the digital world. In fact, digital twins may connect to the real world, even in real time, to understand the state of things or systems and respond to changes. Because digital twins incorporate vast amounts of information related to individual assets and asset groups (and may even provide control over these assets), they can communicate with each other to create a digital factory model of multiple interconnected digital twins.

[0167] The aforementioned storage system can also be part of a multi-cloud environment, where multiple cloud computing and storage services are deployed across a single heterogeneous architecture. To facilitate the operation of such a multi-cloud environment, DevOps tools can be deployed for cross-cloud orchestration. Similarly, continuous development and continuous integration tools can be deployed to standardize processes surrounding continuous integration and delivery, new feature rollout, and the provisioning of cloud workloads. By standardizing these processes, a multi-cloud strategy can be implemented, enabling the utilization of the best providers for each workload. Furthermore, application monitoring and visibility tools can be deployed to monitor application workloads moving across different clouds, identify performance issues, and perform other tasks. Additionally, security and compliance tools can be deployed to ensure compliance with security requirements and government regulations. This multi-cloud environment can also include tools for application delivery and intelligent workload management to ensure efficient application delivery and help bootstrap workloads across distributed and heterogeneous architectures, as well as tools to facilitate the deployment and maintenance of packaged and customized applications in the cloud and achieve inter-cloud portability. A multi-cloud environment can similarly include tools for data portability.

[0168] The aforementioned storage system can be used as part of a platform to implement cryptographic anchors, which can be used to authenticate the origin and content of a product to ensure that the product matches the blockchain record associated with it. Such cryptographic anchors can take many forms, including as edible ink, mobile sensors, and microchips. Similarly, as part of a toolkit for protecting the data stored on the storage system, the aforementioned storage system can implement various cryptographic techniques and schemes, including lattice cryptography. Lattice cryptography can involve constructing cryptographic primitives involving lattices in the construction itself or in security proofs. Unlike public-key cryptosystems such as RSA, Diffie-Hellman, or elliptic curve cryptography, which are vulnerable to quantum computer attacks, some lattice-based constructions appear to be resistant to attacks from both classical and quantum computers.

[0169] A quantum computer is a device that performs quantum computing. Quantum computing is computation that utilizes quantum mechanical phenomena such as superposition and entanglement. Quantum computers differ from conventional transistor-based computers because conventional computers require data to be encoded as binary digits (bits), each of which is always in one of two definite states (0 or 1). In contrast to conventional computers, quantum computers use qubits, which can be in a superposition of states. A quantum computer maintains a sequence of qubits, where one qubit can represent 1, 0, or any quantum superposition of those two states. A pair of qubits can be in any quantum superposition of four states, and three qubits can be in any superposition of eight states. A quantum computer with n qubits can generally be in any superposition of up to 2^n different states simultaneously, while a conventional computer can only be in one of these states at any given time. The quantum Turing machine is a theoretical model of this type of computer.

[0170] The aforementioned storage systems can also be paired with FPGA acceleration servers as part of a larger AI or ML infrastructure. Such FPGA acceleration servers can reside near the aforementioned storage systems (e.g., in the same data center) or even be incorporated into a device that includes one or more storage systems, one or more FPGA acceleration servers, network infrastructure supporting communication between the one or more storage systems and one or more FPGA acceleration servers, and other hardware and software components. Alternatively, the FPGA acceleration servers can reside within a cloud computing environment that can be used to perform computationally relevant tasks for AI and ML jobs. Any of the above embodiments can be used collectively as an FPGA-based AI or ML platform. The reader will understand that in some embodiments of an FPGA-based AI or ML platform, the FPGAs contained within the FPGA acceleration servers can be reconfigured for different types of ML models (e.g., LSTM, CNN, GRU). The ability to reconfigure the FPGAs contained within the FPGA acceleration servers can accelerate ML or AI applications based on optimal numerical accuracy and the memory model being used. The reader will understand that by viewing the collection of FPGA acceleration servers as an FPGA pool, any CPU in the data center can utilize the FPGA pool as a shared hardware microservice, rather than limiting servers to dedicated accelerators plugged into it.

[0171] The aforementioned FPGA-accelerated servers and GPU-accelerated servers can implement computational models where machine learning models and parameters are pinned to high-bandwidth on-chip memory and streamed in large amounts via high-bandwidth on-chip memory, rather than holding a small amount of data in the CPU and running long instruction streams on the CPU as occurs in more traditional computational models. For this type of computational model, FPGAs may even be more efficient than GPUs because FPGAs can be programmed using only the instructions required to run such a computational model.

[0172] The aforementioned storage system can be configured to provide parallel storage, for example, by using a parallel file system such as BeeGFS. This parallel file system can include a distributed metadata architecture. For example, a parallel file system can include multiple metadata servers distributing metadata, as well as components containing services for clients and storage servers. By using a parallel file system, file content can be striped across multiple storage servers at the directory level, and metadata can be distributed across multiple metadata servers, each storing a portion of the complete file system tree. The reader will understand that in some embodiments, the storage servers and metadata servers can run in user space on an existing local file system. Furthermore, client services, metadata servers, or hardware servers do not require dedicated hardware, as the metadata server, storage server, and even client services can run on the same machine.

[0173] Readers will understand that, in part, the emergence of many of the aforementioned technologies, including mobile devices, cloud services, social networks, and big data analytics, may necessitate an IT platform to integrate all these technologies and drive new business opportunities by rapidly delivering revenue (generating products, services, and experiences) rather than simply providing technologies to automate internal business processes. IT organizations may need to balance the resources and investments required to keep core legacy systems up and running while simultaneously integrating technologies to build an IT platform that provides speed and flexibility in areas such as developing big data, managing unstructured data, and working with cloud applications and services. One possible embodiment of such an IT platform is a composable infrastructure that includes fluid resource pools, such as the many systems described above that can meet the evolving needs of applications by allowing the combination and recombination of blocks of decomposed computing, storage, and structural infrastructure. This composable infrastructure may also include a single management interface to eliminate complexity and a unified API for discovering, searching, inventorying, configuring, providing, updating, and diagnosing the composable infrastructure.

[0174] The systems described above can support the execution of a wide range of software applications. These applications can be deployed in various ways, including container-based deployment models. Containerized applications can be managed using a variety of tools. For example, containerized applications can be managed using Docker Swarm (a clustering and scheduling tool for Docker containers, enabling IT administrators and developers to build and manage clusters of Docker nodes as a single virtual system). Similarly, containerized applications can be managed using Kubernetes (a container orchestration system for automating the deployment, scaling, and management of containerized applications). Kubernetes can run on operating systems such as Red Hat Enterprise Linux, Ubuntu Server, and SUSE Linux Enterprise Server. In these examples, the master node can assign tasks to worker / subordinate nodes. Kubernetes can include a set of components that manage individual nodes (e.g., kubelet, kube-proxy, cAdvisor) and a set of components that form the control plane (e.g., etcd, API server, scheduler, control manager). Various controllers (e.g., replication controllers, daemon set controllers) can drive the state of a Kubernetes cluster by managing a set of pods, including one or more containers deployed on a single node. Containerized applications can be used to facilitate serverless, cloud-native compute deployment and management models for software applications. To support these models, containers can be used as part of event handling mechanisms (e.g., AWS Lambdas), causing various events to cause containerized applications to spin up and operate as event handlers.

[0175] The aforementioned systems can be deployed in various ways, including to support fifth-generation (5G) networks. 5G networks may support data communication much faster than previous generations of mobile communication networks, potentially leading to a fragmentation of data and computing resources. This is because modern large-scale data centers may become less prominent and may be replaced by more localized micro data centers closer to mobile network towers. The systems described above can be incorporated into such localized micro data centers and can be part of or paired with multi-access edge computing (MEC) systems. Such MEC systems can enable cloud computing capabilities and IT service environments at the edge of cellular networks. By running applications and performing relevant processing tasks closer to cellular customers, network congestion may be reduced, and applications may perform better. MEC technology is designed to be implemented at cellular base stations or other edge nodes and allows for the flexible and rapid deployment of new applications and services for customers. MEC can also allow cellular operators to open their radio access networks (RANs) to authorized third parties, such as application developers and content providers. Furthermore, edge computing and micro data centers can significantly reduce the cost of smartphones working with 5G networks, as customers may not need devices with such intensive processing power and expensive necessary components.

[0176] Readers will understand that 5G networks may generate significantly more data than previous generations, especially considering the high bandwidth they offer, which may enable 5G to handle volumes and types of data that were infeasible with previous generations (e.g., sensor data from autonomous vehicles, data generated using AR / VR technologies). In these examples, the scalability offered by these systems can be extremely valuable due to the increased data volume and the growing adoption of emerging technologies.

[0177] To further illustrate, Figure 3C An exemplary computing device 350 may be specifically configured to perform one or more of the processes described herein. For example... Figure 3C As shown, the computing device 350 may include a communication interface 352, a processor 354, a storage device 356, and an input / output (I / O) module 358 that are communicatively connected to each other via a communication infrastructure 360. Although Figure 3C An exemplary computing device 350 is shown, but Figure 3C The components illustrated are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 3C The components of the computing device 350 shown.

[0178] The communication interface 352 can be configured to communicate with one or more computing devices. Examples of the communication interface 352 include, but are not limited to, wired network interfaces (such as network interface cards), wireless network interfaces (such as wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0179] Processor 354 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. Processor 354 can operate by executing computer-executable instructions 362 (e.g., applications, software, code, and / or other executable data instances) stored in storage device 356.

[0180] Storage device 356 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, storage device 356 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data including the data described herein may be stored temporarily and / or permanently in storage device 356. For example, data representing computer-executable instructions 362 configured to boot processor 354 to perform any of the operations described herein may be stored within storage device 356. In some examples, data may be arranged in one or more databases residing within storage device 356.

[0181] I / O module 358 may include one or more I / O modules configured to receive user input and provide user output. I / O module 358 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 358 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0182] I / O module 358 may include one or more means for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 358 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be served by a particular implementation. In some examples, any of the systems, computing devices, and / or other components described herein may be implemented via computing device 350.

[0183] To further illustrate, Figure 3D Block diagrams illustrating several storage systems (311-402, 311-404, 311-406) supporting pods according to some embodiments of the present invention are presented. Although depicted in limited detail, Figure 3D The described storage systems (311-402, 311-404, 311-406) are comparable to those in the above references. Figures 1A-1D , Figures 2A-2G , Figures 3A-3B Similar to the storage system described above, or any combination thereof. In fact, Figure 3D The described storage systems (311-402, 311-404, 311-406) may include the same, fewer, or additional components as described above.

[0184] exist Figure 3D In the examples depicted, the storage systems (311-402, 311-404, 311-406) are each depicted as having at least one computer processor (311-408, 311-410, 311-412), computer memory (311-414, 311-416, 311-418), and computer storage (311-420, 311-422, 311-424). Although in some embodiments the computer memory (311-414, 311-416, 311-418) and computer storage (311-420, 311-422, 311-424) may be part of the same hardware device, in other embodiments the computer memory (311-414, 311-416, 311-418) and computer storage (311-420, 311-422, 311-424) may be part of different hardware devices. In this specific example, the difference between computer memory (311-414, 311-416, 311-418) and computer storage (311-420, 311-422, 311-424) might be that computer memory (311-414, 311-416, 311-418) is physically close to computer processor (311-408, 311-410, 311-412) and can store computer program instructions executed by computer processor (311-408, 311-410, 311-412), while computer storage (311-420, 311-422, 311-424) is embodied as non-volatile storage for storing user data and metadata describing user data, etc. (Refer to above) Figure 1AFor example, the computer processors (311-408, 311-410, 311-412) and computer memories (311-414, 311-416, 311-418) of a particular storage system (311-402, 311-404, 311-406) may reside in one or more of the controllers (110A to 110D), while additional storage drives (171A to 171F) may be used as computer storage (311-420, 311-422, 311-424) within the particular storage system (311-402, 311-404, 311-406).

[0185] exist Figure 3D In the examples depicted, according to some embodiments of the invention, the depicted storage system (311-402, 311-404, 311-406) may be attached to one or more pods (311-430, 311-432). Figure 3D Each of the depicted pods (311-430, 311-432) may include datasets (311-426, 311-428). For example, a first pod (311-430) to which three storage systems (311-402, 311-404, 311-406) have been attached includes a first dataset (311-426), while a second pod (311-432) to which two storage systems (311-404, 311-406) have been attached includes a second dataset (311-428). In such an example, when a particular storage system is attached to a pod, the pod's dataset is copied to that particular storage system and then remains up-to-date as the dataset is modified. A storage system can be removed from a pod, causing the dataset to no longer remain up-to-date on the removed storage system. Figure 3D In the example depicted, any storage system active for the pod (which is the latest, operational, and non-faulty component of the non-faulty pod) can receive and process requests to modify or read the pod's dataset.

[0186] exist Figure 3DIn the depicted example, each pod (311-430, 311-432) may also include a set of managed objects and management operations, as well as a set of access operations for modifying or reading the dataset (311-426, 311-428) associated with a particular pod (311-430, 311-432). In such an example, management operations can equivalently modify or query managed objects through any storage system. Similarly, access operations for reading or modifying datasets can be equivalently performed through any storage system. In such an example, although each storage system stores a separate copy of the dataset as an appropriate subset of the dataset stored and advertised for use by that storage system, subsequent managed object queries of pods or subsequent access operations reading datasets reflect the operations performed and completed through any storage system for modifying managed objects or datasets.

[0187] Readers will understand that pods offer far more capabilities than simply replicating datasets synchronously in a cluster. For example, pods can be used to implement tenants, thereby securely separating datasets from each other in a certain way. Pods can also be used to implement virtual arrays or virtual storage systems, where each pod is presented as a unique storage entity on a network (e.g., a storage area network or an Internet Protocol network) with a separate address. In the case of multi-storage system pods implementing virtual storage systems, all physical storage systems associated with a pod can present themselves as the same storage system in a certain way (e.g., as if multiple physical storage systems were no different from multiple network ports of a single storage system).

[0188] Readers will understand that a pod can also be a management unit representing a collection of volumes, file systems, object / analysis storage, snapshots, and other management entities, where management changes to any storage system (e.g., name changes, attribute changes, exporting or licensing of a portion of the dataset managing the pod) are automatically reflected in all active storage systems associated with the pod. Additionally, a pod can also be a data collection and analysis unit, where performance and capacity metrics are presented either aggregated across all active storage systems of the pod or by calling for data collection and analysis separately for each pod, or possibly by presenting the contribution of each additional storage system to incoming content and performance for each pod.

[0189] A model for pod membership can be defined as a list of storage systems and a subset of that list, where storage systems are considered to be synchronized with the pod. A storage system is considered to be synchronized with the pod if it is at least within a recovery that has the same free content as the last write copy of the dataset associated with the pod. Free content is what has been completed after any ongoing modifications have been made without processing any new modifications. This is sometimes referred to as “crash-recoverable” consistency. Pod recovery performs processing to mediate differences in concurrent updates applied to synchronized storage systems within the pod. Recovery can resolve any inconsistencies between storage systems when they have been requested by the various members of the pod, but without signaling to any requester that a concurrent modification has been successfully completed. Storage systems listed as pod members but not listed as synchronized with the pod can be described as “detached” from the pod. Storage systems listed as pod members, synchronized with the pod, and currently available to actively serve the data of the pod are “online” for the pod.

[0190] Each storage system member of a pod can have its own copy of its membership, including which storage systems it last knew were synchronized, and which storage systems it last knew comprised the entire pod membership set. To be online to the pod, a storage system must consider itself synchronized to the pod and must communicate with all other storage systems it considers synchronized to the pod. If a storage system cannot determine that it is synchronized and is communicating with all other synchronized storage systems, then the storage system must stop processing new incoming requests from the pod (or must complete the request with an error or exception) until the storage system can determine that it is synchronized and is communicating with all other synchronized storage systems. A first storage system might conclude that a second paired storage system should detach, which would allow the first storage system to continue because it is now synchronized with all existing storage systems in the list. However, it is necessary to prevent a second storage system from alternatively concluding that the first storage system should detach and continuing to operate. This would lead to a "split-brain" situation, which could result in dangerous situations such as unresolvable datasets, dataset corruption, or application corruption.

[0191] Situations may arise where it is necessary to determine how to continue without communicating with the paired storage system when: a storage system is operating normally and then notices a loss of communication; a storage system is currently recovering from some previous failure; a storage system is restarting or restarting from a temporary power loss or communication interruption; a storage system is switching operations from one storage system controller set to another for whatever reason; or during or after any combination of these or other kinds of events. In fact, whenever a storage system associated with a pod cannot communicate with all known non-detached members, the storage system may briefly wait until communication can be established, go offline and continue waiting, or the storage system may somehow determine that detaching the non-communicating storage system is safe without the risk of a split brain due to the non-communicating storage system taking a different view, and then continue. If safe detachment can be performed quickly enough, the storage system can remain online for the pod with only a slight delay and without causing application interruption for applications that can request access from the online storage system.

[0192] An example of this scenario is when a storage system may know it is outdated. This could happen, for example, when the first storage system is first added to a pod that is already associated with one or more storage systems, or when the first storage system reconnects to another storage system and discovers that the other storage system has marked the first storage system as detached. In this case, the first storage system simply waits until it connects to another set of storage systems that is synchronized with the pod.

[0193] This model needs to consider, to some extent, how storage systems are added to or removed from the list of members of a pod or a synchronized pod. Since each storage system will have its own copy of the list, and since two independent storage systems cannot update their local copies perfectly simultaneously, and since local copies are available upon restart or in various failure scenarios, care must be taken to ensure that transient inconsistencies do not cause problems. For example, if a storage system is synchronized with the pod and a second storage system is added, and then if the second storage system is updated to list both storage systems as synchronized first, and then if both storage systems fail and restart, the second storage system might start and wait to connect to the first storage system, while the first storage system might not know that it should or can wait for the second storage system. If the second storage system responds to the inability to connect to the first storage system by detaching its processing from the first storage system, it might successfully perform processing that the first storage system is unaware of, resulting in a split brain. Therefore, it may be necessary to ensure that storage systems do not inappropriately disagree on whether they can choose to detach their processing when not communicating.

[0194] One way to ensure that storage systems don't inappropriately disagree on whether they can choose to detach when not communicating is to ensure that when a new storage system is added to the pod's synchronized member list, the new storage system is first stored as a detached member (perhaps it was added as a synchronized member). Then, existing synchronized storage systems can locally store that the new storage system is a synchronized pod member, and the new storage system stores that fact locally. If there is a set of restarts or network outages before the new storage system stores its synchronized state, the original storage systems might detach the new storage system due to lack of communication, but the new storage system will wait. Removing a communicating storage system from a pod might require the reverse version of this change: first, the storage system being removed stores that it is no longer synchronized, then the storage system will keep storing that the storage system being removed is no longer synchronized, and then all storage systems remove the removed storage system from their pod membership lists. Depending on the implementation, an intermediate persistent detached state may not be necessary. Whether caution is needed regarding local copies of the membership list may depend on the intended use of the model storage systems for mutual monitoring or for verifying their membership. If a consensus model is used for two pods, or if an external system (or an external distributed or clustered system) is used to store and verify pod membership, inconsistencies in the locally stored membership list may be irrelevant.

[0195] In the event of a communication failure or the failure of one or more storage systems within the pod, or when a storage system starts up (or fails over to a secondary controller) and is unable to communicate with its paired storage systems in the pod, and it is time for one or more storage systems to decide to detach one or more paired storage systems, an algorithm or mechanism must be employed to determine whether it is safe to do so and to complete the detachment. One approach to resolving detachment is to use a majority (or quorum) model for membership. In the case of three storage systems, as long as two are communicating, they can agree to detach a third storage system that is not communicating; however, the third storage system cannot detach the other two storage systems by its own choice. Confusion can arise when storage system communications are inconsistent. For example, storage system A may be communicating with storage system B instead of C, while storage system B may be communicating with both A and C. Therefore, A and B can detach C, or B and C can detach A, but more communication between pod members may be needed to resolve this issue.

[0196] When adding and removing storage systems, the group membership model needs to be considered. For example, if a fourth storage system is added, the "majority" in the storage system is now 3. A pod transitioning from three storage systems (two for a majority) to including a fourth (three for a majority) might require something similar to the previously described model to carefully add the storage system to the synchronization list. For example, the fourth storage system might start in an attached state but not yet attached, where it won't trigger a vote in the group. Once in this state, the original three pod members can each be updated to be aware of the fourth member and the new requirement for a majority of the four storage systems. Removing a storage system from a pod might similarly move that storage system to a "detached" state in local storage before updating other pod members. Variations of this would involve using a distributed consensus mechanism such as PAXOS or RAFT to handle any membership changes or detachment requests.

[0197] Another approach to managing membership transitions is to use an external system, separate from the storage system itself, to handle pod membership. For a storage system to become online for the pod, it must first contact the external pod membership system to verify its synchronization with the pod. Any storage system that is online for the pod should then maintain communication with the pod membership system and should wait or go offline in case of lost communication. Various clustering tools, such as Oracle RAC, Linux HA, Veritas Cluster Server, or IBM's HACMP, can be used to implement the external pod membership manager as a highly available cluster. The external pod membership manager can also utilize distributed configuration tools such as Etcd or Zookeeper, or reliable distributed databases such as Amazon's DynamoDB.

[0198] exist Figure 3DIn the illustrated examples, according to some embodiments of the invention, the illustrated storage system (311-402, 311-404, 311-406) can receive a request to read a portion of a dataset (311-426, 311-428) and process the request to read a portion of the dataset locally. The reader will understand that while requests to modify (e.g., write operations) the datasets (311-426, 311-428) require coordination among the storage systems (311-402, 311-404, 311-406) in the pod because the datasets (311-426, 311-428) should be kept consistent across all storage systems (311-402, 311-404, 311-406) in the pod, similar coordination is not required to respond to requests to read a portion of the datasets (311-426, 311-428). Therefore, the specific storage system receiving the read request can serve the read request locally by reading a portion of the datasets (311-426, 311-428) stored in the storage device of the storage system, without needing to synchronize with other storage systems in the pod. A read request received by a storage system for a replicated dataset in a replicated cluster is expected to avoid any communication in the vast majority of cases (at least when received by a storage system running within the cluster, which is also nominally running). Such a read is typically handled simply by reading from a local copy of the cluster dataset, without further interaction with other storage systems in the cluster.

[0199] Readers will understand that storage systems can take steps to ensure read consistency so that read requests return the same result regardless of which storage system processes them. For example, the content of a cluster dataset resulting from any set of updates received by any set of storage systems in the cluster should remain consistent across the cluster, at least during any time when updates are idle (all previous modification operations have been indicated as complete, and no new update requests have been received and processed in any way). More specifically, instances of a cluster dataset across a set of storage systems can only differ due to updates that have not yet been completed. This means, for example, that any two write requests that overlap within a volume block, or any combination of a write request with an overlapping snapshot, a comparison of a write, or a virtual block-range copy, must produce consistent results across all copies of the dataset. Two operations should not produce results as if they occurred in one order on one storage system in the replicated cluster and in a different order on another.

[0200] Furthermore, it ensures consistent temporal order for read requests. For example, if a read request is received and completed on the replicated cluster, and this read request is followed by another read request received by the replicated cluster for an overlapping address range, and these one or both reads overlap in any way with a modification request received by the replicated cluster in terms of time and volume address range (regardless of whether either the read or the modification is received by the same or different storage systems within the replicated cluster), then if the first read reflects the result of the update, the second read should also reflect the result of that update, rather than potentially returning data before the update. If the first read does not reflect the update, the second read may or may not reflect the update. This ensures that the "time" of a data segment cannot be rolled backward between two read requests.

[0201] exist Figure 3D In the illustrated example, the depicted storage systems (311-402, 311-404, 311-406) can also detect interruptions in data communication with one or more other storage systems and determine whether a particular storage system should remain in the pod. Interruptions in data communication with one or more other storage systems can occur for various reasons. For example, an interruption could be due to a failure of one of the storage systems, a network interconnect failure, or some other reason. A crucial aspect of synchronous replication clusters is ensuring that any failure handling does not lead to unrecoverable inconsistencies or any discrepancies in the response. For example, if a network failure occurs between two storage systems, at most one of the storage systems can continue to process new incoming I / O requests for the pod. And if one storage system continues to process, the other storage system cannot process any new requests (including read requests) to be completed.

[0202] exist Figure 3DIn the illustrated example, the depicted storage systems (311-402, 311-404, 311-406) can also determine whether a particular storage system should remain in the pod in response to the detection of an interruption in data communication with one or more other storage systems. As stated above, in order to be "online" as part of the pod, a storage system must consider itself synchronized with the pod and must communicate with all other storage systems that it considers synchronized with the pod. If a storage system cannot determine that it is synchronized and is communicating with all other synchronized storage systems, it can stop processing newly incoming requests to access the dataset (311-426, 311-428). Therefore, a storage system can determine whether it should remain online as part of the pod, for example, by determining whether it can communicate with all other storage systems that it considers to be synchronized with the pod (e.g., via one or more test messages), by determining that all other storage systems that it considers to be synchronized with the pod also consider the storage system attached to the pod, by a combination of two steps: the specific storage system must confirm that it can communicate with all other storage systems that it considers to be synchronized with the pod and all other storage systems that it considers to be synchronized with the pod also consider the storage system attached to the pod, or by some other mechanism.

[0203] exist Figure 3D In the illustrated examples, the depicted storage systems (311-402, 311-404, 311-406) may also make the datasets on the specific storage systems accessible for management and dataset operations in response to a determination that a particular storage system should remain in a pod. The storage system may make the datasets (311-426, 311-428) on the specific storage system accessible for management and dataset operations, for example, by accepting and processing requests for access to versions of the datasets (311-426, 311-428) stored on the storage system, by accepting and processing management operations associated with the datasets (311-426, 311-428) issued by a host or authorized administrator, by accepting and processing management operations associated with the datasets (311-426, 311-428) issued by another storage system, or in some other way.

[0204] However, in Figure 3DIn the illustrated example, the depicted storage systems (311-402, 311-404, 311-406) can, in response to a determination that a particular storage system should not remain in the pod, render the dataset on that particular storage system inaccessible for management and dataset operations. The storage system can render the dataset (311-426, 311-428) inaccessible for management and dataset operations, for example, by denying requests to access versions of the dataset (311-426, 311-428) stored on the storage system, by denying management operations associated with the dataset (311-426, 311-428) issued by the host or other authorized administrator, by denying management operations associated with the dataset (311-426, 311-428) issued by another storage system in the pod, or in some other way.

[0205] exist Figure 3D In the depicted example, the depicted storage systems (311-402, 311-404, 311-406) can also detect that an interruption in data communication with one or more other storage systems has been repaired, and make the dataset on the specific storage system accessible for management and dataset operations. The storage system can detect that an interruption in data communication with one or more other storage systems has been repaired, for example, by receiving a message from one or more other storage systems. In response to detecting that an interruption in data communication with one or more other storage systems has been repaired, once the previously separated storage systems are resynchronized with the storage systems that remain attached to the pod, the storage system can make the dataset on the specific storage system (311-426, 311-428) accessible for management and dataset operations.

[0206] exist Figure 3DIn the illustrated examples, the depicted storage systems (311-402, 311-404, 311-406) may also go offline from the pod, rendering the specific storage system unusable for management and dataset operations. The depicted storage systems (311-402, 311-404, 311-406) may go offline from the pod for various reasons, rendering the specific storage system unusable for management and dataset operations. For example, the depicted storage systems (311-402, 311-404, 311-406) may also go offline due to some failure of the storage system itself, due to updates or other maintenance occurring on the storage system, due to communication failures, or for many other reasons. In such an example, the depicted storage systems (311-402, 311-404, 311-406) can then update the dataset on the specific storage system to include all updates in the dataset, because the specific storage system was once offline relative to the pod and then came back online, allowing the specific storage system to be managed and the dataset to be manipulated, as described in more detail in the resynchronization section included below.

[0207] exist Figure 3D In the illustrated examples, the depicted storage systems (311-402, 311-404, 311-406) can also identify target storage systems used for asynchronously receiving datasets, where the target storage system is not one of multiple storage systems whose datasets are synchronously replicated. Such a target storage system can, for example, represent a backup storage system as well as some storage system that utilizes synchronous replication of the dataset. In fact, synchronous replication can be used to distribute copies of the dataset closer to certain server racks for better local read performance. One such scenario involves smaller top-of-rack storage systems symmetrically replicated to larger storage systems located centrally in a data center or campus, where these larger storage systems are more carefully managed for reliability or connected to external networks for asynchronous replication or backup services.

[0208] exist Figure 3D In the depicted examples, the described storage systems (311-402, 311-404, 311-406) can also identify portions of the dataset that are not currently being asynchronously copied to the target storage system by any other storage system, and asynchronously copy those portions to the target storage system, where two or more storage systems jointly copy the entire dataset to the target storage system. In this way, the work associated with asynchronously copying a specific dataset can be split among the members of the pod, such that each storage system in the pod is only responsible for asynchronously copying a subset of the dataset to the target storage system.

[0209] exist Figure 3DIn the illustrated example, the depicted storage systems (311-402, 311-404, 311-406) can also be detached from the pods, such that the specific storage systems detached from the pods are no longer included in the set of storage systems for synchronized replication of the dataset. For example, if Figure 3D The storage system (311-404) in the middle is from Figure 3D If the pods (311-430) are separated as shown, then pods (311-430) will only include storage systems (311-402, 311-406) as the storage systems that will synchronously replicate the dataset (311-426) included in pods (311-430). In such an example, separating a storage system from a pod could also include removing the dataset from the specific storage system separated from the pod. Continuing Figure 3D The storage system (311-404) in the middle is from Figure 4 The example shown is a pod (311-430) separation, and the dataset (311-426) included in the pod (311-430) can be deleted from the storage system (311-404) or otherwise removed.

[0210] Readers will understand that the pod model, with further support, can implement multiple unique management capabilities. Additionally, the pod model itself introduces several problems that can be addressed through implementation. For example, when (e.g., due to interconnect failure and another storage system of the pod winning in mediation) a storage system is offline to the pod but otherwise running, access to the offline pod's dataset on the offline storage system may still be expected or needed. One solution might simply be to enable the pod in a certain detached mode and allow access to the dataset. However, this solution can be risky and may lead to more difficult reconciliation of the pod's metadata and data when the storage system regains communication. Furthermore, separate paths may still exist for the host to access the offline storage system as well as the storage system that remains online. In this case, the host can issue I / O to both storage systems even though they are no longer synchronized because the host sees the target port reporting a volume with the same identifier and the host I / O driver assumes the host sees other paths to the same volume. This can lead to fairly serious data corruption because reads and writes to the two storage systems are no longer consistent, even if the host assumes they are. As a variation of this scenario, in clustered applications (such as shared storage clustered databases), a clustered application running on one host might be reading from or writing to one storage system, while the same clustered application running on another host might be reading from or writing to a "separate" storage system. However, the two instances of the clustered application are communicating with each other based on the assumption that the datasets they each see are completely consistent with the completed writes. Because they are inconsistent, this assumption is violated, and the application's dataset (e.g., the database) could eventually become corrupted very quickly.

[0211] One way to address these two issues is to allow offline pods, or rather, snapshots of offline pods, to be copied to a new pod with a new volume. This new volume has a sufficiently new identity that prevents host I / O drives and cluster applications from confusing the copied volume with the same one still online on another storage system. Since each pod maintains a complete copy of its dataset (which is crash-consistent but may differ slightly from a copy of the pod's dataset on another storage system), and since each pod has an independent copy of all the data and metadata required to operate on the pod's content, it's a straightforward problem to make a virtual copy of some or all of the volumes or snapshots in a pod a new volume in the new pod. For example, in a logical partition map implementation, all that's needed is to define a new volume in the new pod, where the new volume references the logical partition map in the copied pod associated with the pod's volume or snapshot, and that logical partition map is marked as a copy when written. Similar to how volume snapshots copied to a new volume can be implemented, the new volume should be treated as a new volume. The volume may have the same management name, but in a new pod namespace. However, they should have different base identifiers and logical unit identifiers that are different from the original volume.

[0212] In some cases, virtual network isolation technology can be used in ways such as creating a virtual LAN in the case of an IP network or a virtual SAN in the case of a Fibre Channel network, to ensure that the isolation of volumes presented to certain interfaces cannot be accessed from host network interfaces or host SCSI initiator ports that may also see the original volume. In this case, providing a copy of the volume with the same SCSI or other storage identifier as the original volume can be secure. This can be used, for example, when an application expects to see a specific set of storage identifiers so that it will not operate under excessive load upon reconfiguration.

[0213] Some of the techniques described here can also be used outside the active failure context to test the readiness to handle failures. Disaster recovery configurations typically require readiness testing (sometimes called "fire drills"), where frequent and repeated testing is considered necessary to ensure that most or all aspects of the disaster recovery plan are correct and take into account any recent changes to applications, datasets, or equipment. Readiness testing should not interfere with current production operations, including replication. In many cases, actual operations cannot be invoked on the active configuration, but a good approximation is to use storage operations to copy the production dataset, which can then be coupled with the use of a virtual network to create an isolated environment containing all the data required for critical applications that are considered essential to successfully establish themselves in the event of a disaster. Making such a copy of the dataset, which is being replicated synchronously (or even asynchronously), available within the site (or set of sites) where the disaster recovery readiness testing process is expected to take place, and then starting critical applications for that dataset to ensure they can be started and running, is a powerful tool because it helps ensure that no important part of the application dataset is omitted from the disaster recovery plan. If necessary and feasible, this can be coupled with a virtual isolation network (perhaps even with an isolated set of physical or virtual machines) to approximate a real-world disaster recovery takeover scenario as closely as possible. In effect, copying a pod (or set of pods) to another pod as a point-in-time image of the pod dataset immediately creates an isolated dataset containing all copied elements, which can then be operated on in essentially the same way as the original pods, allowing for individual separation from the original pods to single sites (or several sites). Furthermore, these are fast operations, and they can be dismantled and easily repeated, allowing tests to be repeated as frequently as desired.

[0214] Several enhancements can be made to further approach perfect disaster recovery testing. For example, in conjunction with isolated networks, SCSI logical unit identities or other types of identities can be copied into the target pod, allowing the test server, virtual machines, and applications to see the same identity. Furthermore, the server's management environment can be configured to respond to requests from a specific set of virtual networks in response to requests and actions on the original pod name, thus eliminating the need for scripts to use test variants with alternative "test" versions of object names. Further enhancements can be used where the host-side server infrastructure that will be taken over in the event of a disaster takeover can be used during testing. This includes situations where the disaster recovery data center has a fully stocked alternative server infrastructure, which will generally not be used until a disaster instructs it to do so. It also includes situations where the infrastructure can be used for non-critical operations (e.g., running analytics on production data, or simply supporting application development or other functions that may be important but can be stopped to obtain more critical functions when needed). Specifically, host definitions and configurations, along with the server infrastructure that will use them, can be set up as they will be used in an actual disaster recovery takeover event and tested as part of the disaster recovery takeover test, where the test volume connects to these host definitions from a copy of the virtual pod used to provide a snapshot of the dataset. Then, from the perspective of the storage system involved, these host definitions and configurations used for testing, as well as the volume-to-host connection configuration used during testing, can be reused when an actual disaster recovery takeover event is triggered, thereby greatly reducing the configuration difference between the test configuration and the actual configuration that will be used in the event of disaster recovery takeover.

[0215] In some cases, it may make sense to remove volumes from a first pod and move them into a new second pod that contains only those volumes. Pod memberships and high availability and recovery characteristics can then be tuned individually, allowing for the isolation of the management of the two resulting pod datasets. Operations that can be performed in one direction should also be possible in the other. To some extent, it makes sense to take two pods and merge them into one such that the volumes in each of the original two pods will track each other's storage system memberships, high availability and recovery characteristics, and events. Both operations can be performed safely and with reasonably minimal or no disruption to the running application by relying on the characteristics suggested for changing the mediation or group properties of the pods discussed in previous sections. For example, through mediation, the mediator of a pod can be changed using a sequence of steps comprising the following steps, wherein in steps, the individual storage systems in the pod are changed to depend on both the first and second mediators, and then each is changed to depend only on the second mediator. If a failure occurs in the middle of the sequence, some storage systems may depend on both the first and second mediators. However, in any case, recovery and fault handling will not result in some storage systems depending solely on the first mediator while others depend solely on the second. Recovery can be continued by temporarily depending on the victory of both the first and second group models. This may result in the availability of the failed pod depending on the attached resources for a very short period, thus reducing potential availability, but this period is short and the reduction in availability is often small. For mediators, if the change in mediator parameters is merely a change in the key used by the mediator and the mediator service used is the same, the potential reduction in availability is even less, because it now depends only on two calls to the same service and one call to that service, rather than on two separate calls to two different services.

[0216] Readers will notice that changing the group model can be quite complex. It may require additional steps where the storage system participates in a second group model, but does not depend on winning in that second group model, and then steps where it also depends on the second group model. This can be necessary to explain the fact that if only one system processes the change to depend on the group model, it will never win in the group because there will never be a majority. By appropriately modifying the high availability parameters (mediation, group model, takeover preference) using this model, safe procedures can be created for these operations to split a pod into two or combine two pods into one. This may require adding another capability: linking the second pod to the first pod for high availability, such that if both pods include compatible high availability parameters, the second pod linked to the first pod can depend on the first pod to determine and incentivize processes and operations related to separation, offline and synchronized states, and recovery and resynchronization actions.

[0217] To split a pod into two (an operation used to move some volumes into the newly created pod), a distributed operation can be formed, which can be described as: creating a second pod to move a set of volumes previously located in the first pod into it, copying high availability parameters from the first pod to the second pod to ensure they are link-compatible, and linking the second pod to the first pod for high availability. This operation can be encoded as a message and should be implemented by the storage systems in the pod in such a way that the storage system ensures the operation occurs entirely on its own storage system, or at least not at all in the event of a failure. Once all the synchronized storage systems of both pods have processed this operation, the storage systems can then process subsequent operations that change the second pod so that it is no longer linked to the first pod. As with other changes to the high availability characteristics of pods, this involves first changing the synchronized storage systems to depend on both the previous model (which was high availability linked to the first pod) and the new model (which is now high availability independent of itself). In the case of a mediator or group, this means that the storage system handling such a change will first depend on the mediator or group appropriately implemented for the first pod, and will additionally depend on a new, separate mediator (e.g., a new mediator key) or group that can be implemented for the second pod after a failure to request a mediator or test group. As described previously regarding the group change model, the intermediate step can be to set up the storage system to participate in the group of the second pod before the step where the storage system participates and depends on the group of the second pod. Once all synchronous storage systems have processed the change to depend on the new parameters of the mediator or group for both the first and second pods, the split is complete.

[0218] Merging the second pod into the first pod is essentially the reverse process. First, the second pod must be adapted to be compatible with the first pod by having the same list of storage systems and a compatible high-availability model. This may involve a set of steps, such as those described elsewhere in this document for adding or removing storage systems or for changing the mediator or swarm model. Depending on the implementation, it may only require achieving the same list of storage systems. Merging proceeds by processing operations on each of the synchronized storage systems to link the second pod to the first pod for high availability. The storage systems processing this operation will then depend on the first pod for high availability, and then on the second pod for high availability. Once all the synchronized storage systems for the second pod have processed the operation, the storage systems will then handle subsequent operations to remove the link between the second and first pods, migrate the volume from the second pod to the first pod, and delete the second pod. As long as host or application dataset modification or read operations are implemented to correctly point to the volume through the identity, and as long as the identity is maintained in a manner appropriate for the storage protocol or storage model (e.g., as long as the logical unit identifier of the volume and the use of the target port for accessing the volume are maintained in the case of SCSI), host or application dataset access can be maintained throughout these operations.

[0219] Migrating volumes between pods can be problematic. If the pods have the same set of synchronized membership storage systems, it can be straightforward: temporarily pause operations on the volume being migrated, transfer control of those volumes to the software and infrastructure controlling the new pod, and then resume operations. This allows for seamless migration, with applications continuing to function normally except for very brief operational pauses, provided the network and port migration between pods is correct. Depending on the implementation, pausing operations may not even be necessary, or it may be internal to the system so that the pause has no impact. Copying volumes between pods with different synchronized memberships is more problematic. If the target pod has a subset of synchronized memberships from the source pod, this is not a problem: the membership storage systems can be safely discarded without further work. However, if the target pod adds synchronized membership storage systems to a volume on the source pod, the added storage system must be synchronized to include the volume's contents before it can be used. Until synchronization, this makes the copied volume significantly different from the synchronized volume because fault handling is different, and request processing from unsynchronized membership storage systems either doesn't work, must be forwarded, or isn't as fast because reads must traverse the interconnect. In addition, the internal implementation will have to handle some volumes that are in sync and ready to handle failures, as well as other unsynchronized volumes.

[0220] In the face of failures, other issues related to operational reliability exist. Coordinating the migration of volumes between multiple storage system pods is a distributed operation. If the pod is the unit of failure handling and recovery, and if mediators, swarms, or any means are used to avoid split-brains, then when a volume is switched from one pod with a specific set of states, configurations, and relationships for failure handling, recovery, mediators, and swarms to another pod, the storage systems within the pod must carefully coordinate the changes associated with the handling of any volume. Operations cannot be distributed atomically between storage systems, but must be hierarchical in some way. Mediator and swarm models essentially provide pods with tools for achieving atomicity in distributed transactions, but this may not extend to operations between pods without being added to the implementation.

[0221] Even for two pods sharing the same first and second storage systems, consider a simple migration of a volume from the first pod to the second pod. To some extent, the storage systems will coordinate to define that the volume is now in the second pod and no longer in the first pod. If there is no inherent mechanism for transaction atomicity across the two pods' storage systems, the native implementation can leave the volume in the first pod on the first storage system and in the second pod on the second storage system in the event of a network failure that causes a failure to handle the detachment of the storage systems from the two pods. If each pod individually determines which storage system successfully detaches the other, the result might be a detachment of the other storage system for the same storage system for both pods. In this case, the volume migration recovery should be consistent; otherwise, it might result in detachment of different storage systems for the two pods. If the second storage system detaches for the first storage system for the first pod, and the first storage system detaches for the second storage system for the second pod, recovery might result in the volume being restored to the first pod on the first storage system and the second pod on the second storage system, where the volume runs and is exported to the host and storage applications. If, as an alternative, the first storage system is separated from the second storage system for the first pod, and the second storage system is separated from the first storage system for the second pod, recovery may result in the volume being dropped from the second pod due to the first storage system and from the first pod due to the second storage system, leading to the complete loss of the volume. Things become even more complicated if the pods in which the volume migrates are located on different sets of storage systems.

[0222] Solutions to these problems could involve using intermediate pods, along with the previously described techniques for splitting and combining pods. The intermediate pod may never be presented as a visible, managed object associated with a storage system. In this model, a volume to be moved from the first pod to the second pod is first split from the first pod into new intermediate pods using the previously described splitting operation. The storage system membership of the pod can then be adjusted to match the storage systems by adding or removing storage systems relative to it as needed. Subsequently, the intermediate pod can be combined with the second pod.

[0223] To further illustrate, Figure 3E A flowchart illustrating an example method for service I / O operations on a dataset (311-42) synchronized across multiple storage systems (311-38, 311-40) according to some embodiments of the present invention is provided. Although depicted in limited detail, Figure 3E The described storage systems (311-38, 311-40) are comparable to those in the above references. Figures 1A-1D , Figures 2A-2G , Figures 3A-3B Similar to the storage system described by it or any combination thereof. In fact, Figure 3E The described storage system may include the same, fewer, or additional components as the storage system described above.

[0224] Figure 3EThe dataset (311-42) described herein can be, for example, the content of a specific volume, specific shared content of a volume, or any other set of one or more data elements. The dataset (311-42) can be synchronized across multiple storage systems (311-38, 311-40), such that each storage system (311-38, 311-40) maintains a local copy of the dataset (311-42). In the example described herein, such a dataset (311-42) is synchronized and replicated across storage systems (311-38, 311-40) in such a way that any storage system (311-38, 311-40) can access the dataset (311-42) with the following performance characteristics: such that any storage system in the cluster operates substantially no better than any other storage system in the cluster, at least as long as the cluster and the specific storage system being accessed are nominally operational. In such a system, modifications to the dataset (311-42) should be made to copies of the dataset residing on each storage system (311-38, 311-40) in a manner that would produce consistent results when accessing the dataset (311-42) on any storage system (311-38, 311-40). For example, write requests to the dataset must be serviced on all storage systems (311-38, 311-40), or on a storage system (311-38, 311-40) that is not nominally running at the start of the write but remains nominally running during the write's completion. Similarly, some group of operations (e.g., two write operations to the same location within the dataset) must be performed in the same order on all storage systems (311-38, 311-40), or other steps, as described in more detail below, must be taken to make the dataset eventually identical on all storage systems (311-38, 311-40). Modifications to the datasets (311-42) do not need to be performed at exactly the same time, but some actions (e.g., issuing confirmations for write requests to the datasets, enabling read access to locations within the datasets targeted by write requests that have not yet been completed on the two storage systems) may be delayed until copies of the datasets on the respective storage systems (311-38, 311-40) have been modified.

[0225] exist Figure 3EIn the illustrated example method, designating one storage system (311-40) as the "leader" and another storage system (311-38) as the "follower" can indicate a relationship between various storage systems for the purpose of synchronously replicating a specific dataset across storage systems. In such an example, and as will be described in more detail below, the leader storage system (311-40) may be responsible for performing some processing of incoming I / O operations and passing such information to the follower storage systems (311-38), or performing other tasks not required by the follower storage systems (311-40). The leader storage system (311-40) may be responsible for performing tasks not required by the follower storage systems (311-38) for all incoming I / O operations, or alternatively, the leader-follower relationship may be specific only to a subset of I / O operations received by either storage system. For example, the leader-follower relationship may be specific to I / O operations for a first volume, a first set of volumes, a first set of logical addresses, a first set of physical addresses, or some other logical or physical descriptor. In this way, the first storage system can be used as the leader storage system for I / O operations on the first set of volumes (or other descriptors), while the second storage system can be used as the leader storage system for I / O operations on the second set of volumes (or other descriptors). Figure 3E The example method described depicts an embodiment in which multiple storage systems (311-38, 311-40) are synchronized in response to a request (311-04) received by the leader storage system (311-40) to modify the dataset (311-42). However, it is also possible to synchronize multiple storage systems (311-38, 311-40) in response to a request (311-04) received by the follower storage system (311-38) to modify the dataset (311-42), as will be described in more detail below.

[0226] Figure 3E The described example method includes a request (311-04) received (311-06) by the leader storage system (311-40) to modify a dataset (311-42). The request (311-04) to modify the dataset (311-42) can manifest as, for example, a request to write data to a location within the storage system (311-40) containing the data included in the dataset (311-42), a request to write data to a volume containing the data included in the dataset (311-42), a request to take a snapshot of the dataset (311-42), a virtual range copy, a UNMAP operation that essentially represents deleting a portion of the data in the dataset (311-42), a transformation of the dataset (311-42) (rather than changing a portion of the data within the dataset), or some other operation that results in a change to a portion of the data included in the dataset (311-42). Figure 3EIn the example method described, the request (311-04) for modifying the dataset (311-42) is issued by the host (311-02), which may be manifested as, for example, an application running on a virtual machine, an application running on a computing device connected to the storage system (311-40), or some other entity configured to access the storage system (311-40).

[0227] Figure 3E The illustrated example method also includes information (311-10) describing modifications to the dataset (311-42) generated (311-08) by the leader storage system (311-40). The leader storage system (311-40) may generate (311-08) the information (311-10) describing modifications to the dataset (311-42), for example, by determining the order and any other ongoing operations, by determining the appropriate result of overlapping modifications (e.g., the appropriate result of two requests modifying the same storage location), and by calculating any distributed state changes (such as common elements of metadata across all members of the pod (e.g., all storage systems across which the synchronized replication dataset spans)). The information (311-10) describing modifications to the dataset (311-42) may be embodied, for example, system-level information describing I / O operations to be performed by the storage system. The leader storage system (311-40) can generate (311-08) information (311-10) describing the modifications to the dataset (311-42) by processing the request (311-04) to modify the dataset (311-42) (just enough to figure out what should happen in order to serve the request (311-04) to modify the dataset (311-42). For example, the leader storage system (311-40) can determine whether the request (311-04) to modify the dataset (311-42) needs to be ordered relative to other requests to modify the dataset (311-42), or whether some other steps must be taken, as described in more detail below, to produce equivalent results on the various storage systems (311-38, 311-40).

[0228] Consider an example where a request (311-04) to modify dataset (311-42) is represented as a request to copy blocks from a first address range in dataset (311-42) to a second address range in dataset (311-42). In such an example, assume three other write operations (write A, write B, write C) target the first address range in dataset (311-42). In such an example, if the leader storage system (311-40) serves writes A and B (but not write C) before copying blocks from the first address range in dataset (311-42) to the second address range in dataset (311-42), then the follower storage system (311-38) must also serve writes A and B (but not write C) before copying blocks from the first address range in dataset (311-42) to the second address range in dataset (311-42) to produce a consistent result. Therefore, when the leader storage system (311-40) generates (311-08) information (311-10) describing the modification of the dataset (311-42), in this example, the leader storage system (311-40) can generate information identifying other operations that must be completed before the follower storage system (311-38) can process the request (311-04) to modify the dataset (311-42) (e.g., sequence numbers for writing A and writing B).

[0229] Consider an additional example of two requests (e.g., write A and write B) targeting an overlapping portion of dataset (311-42). In such an example, if the leader storage system (311-40) serves write A and subsequently write B, while the follower storage system (311-38) serves write B and subsequently write A, then dataset (311-42) will be inconsistent across these two storage systems (311-38, 311-40). Therefore, in this example, when the leader storage system (311-40) generates (311-08) information (311-10) describing the modifications to dataset (311-42), the leader storage system (311-40) can generate information identifying the order in which the requests should be executed (e.g., sequence numbers for write A and write B). Alternatively, the leader storage system (311-40) can generate (311-08) information (311-10) describing the modifications to the dataset (311-42) that includes information identifying the appropriate results of the two requests, instead of generating information (311-10) describing the modifications to the dataset (311-42) that requires intermediate behavior from the individual storage systems (311-38, 311-40). For example, if write B logically follows (and overlaps with) write A, the final result must be that the dataset (311-42) includes the portion of write B that overlaps with write A, not the portion of write A that overlaps with write B. Such a result can be facilitated by merging the results in memory and writing such a merged result to the dataset (311-42), rather than strictly requiring a particular storage system (311-38, 311-40) to perform write A and then subsequently write B. The reader will understand that more nuanced cases involve snapshots and virtual address range copies.

[0230] Readers will further understand that the correct results of any operation must be committed to a recoverable point before recovery can be confirmed. However, multiple operations can be committed together, or operations can be partially committed if recovery will ensure correctness. For example, a snapshot can be committed locally with a dependency on records expected to be written to A and B, but A or B itself may not be committed. If lost I / O cannot be recovered from other arrays, the snapshot cannot be confirmed, and recovery may end with a rollback snapshot. Furthermore, if a write to B overlaps with a write to A, the leader can "order" B after A, but A can actually be discarded, and then the write to A will only wait for B. Writes A, B, C, and D coupled with snapshots between A, B, and C, D can be committed and / or confirmed in some or all of them together, as long as recovery does not cause snapshot inconsistencies across arrays, and as long as it is confirmed that no later operations have been completed before the earlier operations have been persisted to a point where recovery is guaranteed.

[0231] Figure 3E The illustrated example method also includes sending (311-12) information (311-10) describing modifications to the dataset (311-42) from the leader storage system (311-40) to the follower storage system (311-38). Sending (311-12) information (311-10) describing modifications to the dataset (311-42) from the leader storage system (311-40) to the follower storage system (311-38) can be performed, for example, by the leader storage system (311-40) sending one or more messages to the follower storage system (311-38). The leader storage system (311-40) may also send an I / O payload (311-14) for a request (311-04) to modify the dataset (311-42) in the same message or in one or more different messages. For example, when a request (311-04) to modify dataset (311-42) is represented as a request to write data to dataset (311-42), the I / O payload (311-14) can be represented as data to be written to the storage within the follower storage system (311-38). In such an example, because the request (311-04) to modify dataset (311-42) is received (311-06) by the leader storage system (311-40), the follower storage system (311-38) has not yet received the I / O payload (311-14) associated with the request (311-04) to modify dataset (311-42). Figure 3E In the illustrated example method, information (311-10) describing modifications to the dataset (311-42) and an I / O payload (311-14) associated with a request (311-04) to modify the dataset (311-42) may be sent (311-12) from the leader storage system (311-40) to the follower storage system (311-38) via one or more data communication networks, one or more dedicated data communication links (e.g., a first link for sending the I / O payload and a second link for sending information describing modifications to the dataset), or via some other mechanism.

[0232] Figure 3EThe illustrated example method also includes information (311-10) describing modifications to the dataset (311-42) received (311-16) by the follower storage system (311-38). The follower storage system (311-38) may receive (311-16) information (311-10) describing modifications to the dataset (311-42) and I / O payload (311-14) from the leader storage system (311-40), for example, via one or more messages sent from the leader storage system (311-40) to the follower storage system (311-38). One or more messages can be sent from the leader storage system (311-40) to the follower storage system (311-38) via one or more dedicated data communication links between the two storage systems (311-38, 311-40) by writing messages to predetermined memory locations (e.g., queue locations) on the follower storage system (311-38) using RDMA or similar mechanisms or otherwise.

[0233] In one embodiment, the follower storage system (311-38) can receive (311-16) from the leader storage system (311-40) information (311-10) describing modifications to the dataset (311-42) and I / O payloads (311-14) using SCSI requests (writes from sender to receiver, or reads from receiver to sender) as a communication mechanism. In such an embodiment, SCSI write requests are used to encode information to be sent (which includes any data and metadata), and SCSI write requests can be delivered to a specific pseudo-device or via a specifically configured SCSI network, or via any other agreed addressing mechanism. Alternatively, the model can also issue a set of open SCSI read requests from the receiver to the sender using a specific device, a specifically configured SCSI network, or other agreed mechanisms. In response to one or more of these open SCSI requests, encoded information, including data and metadata, is delivered to the receiver. This model can be implemented on a Fibre Channel SCSI network, which is typically deployed as a “dark fiber” storage network infrastructure between data centers. This model also allows the use of the same network lines for host-to-remote array multipathing and bulk array-to-array communication.

[0234] Figure 3E The illustrated example method also includes a request (311-04) to modify the dataset (311-42) processed (311-18) by the follower storage system (311-38). Figure 3EIn the illustrated example method, the follower storage system (311-38) can process (311-18) a request (311-04) to modify the dataset (311-42) by means of information (311-10) describing the modification of the dataset (311-42) and an I / O payload (311-14) received from the leader storage system (311-40) to modify the contents of one or more storage devices (e.g., NVRAM devices, SSDs, HDDs) included in the follower storage system (311-38). Consider the following example where the request (311-04) to modify the dataset (311-42) is manifested as a write operation to a volume included in the dataset (311-42), and the information (311-10) describing the modification of the dataset (311-42) indicates that the write operation can only be performed after a previously issued write operation has been processed. In such an example, the request (311-04) to modify the dataset (311-42) can be performed by the follower storage system (311-38) first verifying that the previously issued write operation has been processed on the follower storage system (311-38) and then writing the I / O payload (311-14) associated with the write operation to one or more storage devices included in the follower storage system (311-38). In such an example, for example, when the I / O payload (311-14) has been submitted to persistent storage within the follower storage system (311-38), the request (311-04) to modify the dataset (311-42) can be considered complete and successfully processed.

[0235] Figure 3E The illustrated method also includes the completion of a request (311-04) from the follower storage system (311-38) to the leader storage system (311-40) to confirm (311-20) the completion of a request (311-04) to modify the dataset (311-42). Figure 3E In the illustrated example method, the completion of the request (311-04) by the follower storage system (311-20) to the leader storage system (311-40) to modify the dataset (311-42) can be performed by the follower storage system (311-38) sending an acknowledgment (311-22) message to the leader storage system (311-40). Such a message may include, for example, information identifying the completion of the specific request (311-04) to modify the dataset (311-42), as well as any additional information useful when the follower storage system (311-38) acknowledges the completion of the request (311-20) to modify the dataset (311-42). Figure 3EIn the example method described, the follower storage system (311-38) sends an acknowledgment (311-22) message to the leader storage system (311-40) to indicate the completion of the request (311-04) to the leader storage system (311-40) to confirm (311-20) the modification of the dataset (311-42).

[0236] Figure 3E The illustrated example method also includes a request (311-04) to modify the dataset (311-42) processed (311-24) by the leader storage system (311-40). Figure 3E In the illustrated example method, the leader storage system (311-40) can process (311-24) the request to modify the dataset (311-42) by modifying the contents of one or more storage devices (e.g., NVRAM devices, SSDs, HDDs) included in the leader storage system (311-40) according to information (311-10) describing the modification of the dataset (311-42) and an I / O payload (311-14) received as part of a request to modify the dataset (311-42) (311-04). Consider the following example where the request to modify the dataset (311-42) (311-04) is manifested as a write operation to a volume included in the dataset (311-42), and the information (311-10) describing the modification of the dataset (311-42) indicates that the write operation can only be performed after a previously issued write operation has been processed. In such an example, the request (311-04) to modify the dataset (311-42) can be performed by the leader storage system (311-40) first verifying that the previously issued write operation has been processed by the leader storage system (311-40) and then writing the I / O payload (311-14) associated with the write operation to one or more storage devices included in the leader storage system (311-40). In such an example, for example, when the I / O payload (311-14) has been submitted to persistent storage within the leader storage system (311-40), the request (311-04) to modify the dataset (311-42) can be considered complete and successfully processed.

[0237] Figure 3EThe illustrated example method also includes receiving (311-26) an indication (311-04) from the follower storage system (311-38) that the follower storage system (311-38) has processed the request (311-42) to modify the dataset (311-42). In this example, the indication that the follower storage system (311-38) has processed the request (311-04) to modify the dataset (311-42) is embodied in an acknowledgment (311-22) message sent from the follower storage system (311-38) to the leader storage system (311-40). The reader will understand that although many of the steps described above are depicted and presented in a specific order, this specific order is not actually required. In fact, since the follower storage system (311-38) and the leader storage system (311-40) are independent storage systems, some of the steps described above can be performed in parallel by the respective storage systems. For example, the follower storage system (311-38) can receive (311-16) information (311-10) describing the modification of the dataset (311-42), process (311-18) the request (311-04) to modify the dataset (311-42) before the leader storage system (311-40) has processed (311-24) the request (311-04) to modify the dataset (311-42), or confirm (311-20) the completion of the request (311-04) to modify the dataset (311-42). Alternatively, the leader storage system (311-40) may have processed (311-24) the request to modify the dataset (311-42) before the follower storage system (311-38) has received (311-16) information (311-10) describing the modification of the dataset (311-42), processed (311-18) the request to modify the dataset (311-42) (311-04), or confirmed (311-20) the completion of the request to modify the dataset (311-42) (311-04).

[0238] Figure 3E The illustrated method also includes the completion of a request (311-04) to modify the dataset (311-42) confirmed (311-34) by the leader storage system (311-40). Figure 3E In the example method described, the completion of the request (311-04) to modify the dataset (311-42) can be performed by using one or more acknowledgment (311-36) messages sent from the leader storage system (311-40) to the host (311-02) or via some other appropriate mechanism. Figure 3EIn the example method described, the leader storage system (311-40) can determine (311-28) whether the request (311-04) to modify the dataset (311-42) has been processed (311-18) by the follower storage system (311-38) before confirming (311-34) the completion of the request (311-04) to modify the dataset (311-42). The leader storage system (311-40) may determine, for example, whether it has received from the follower storage system (311-38) an acknowledgment message or other message indicating that the request (311-04) to modify the dataset (311-42) has been processed (311-18) by the follower storage system (311-38). In such an example, if the leader storage system (311-40) definitively (311-30) determines that the request (311-04) to modify the dataset (311-42) has been processed (311-18) by the follower storage system (311-38) and also by the leader storage system (311-24), then the leader storage system (311-40) can proceed by confirming (311-34) the completion of the request (311-04) to the host (311-02) that initiated the request (311-04) to modify the dataset (311-42). However, if the leader storage system (311-40) determines that the request (311-04) to modify the dataset (311-42) has not yet been processed (311-18) by the follower storage system (311-38) or has not yet been processed (311-24) by the leader storage system (311-38), then the leader storage system (311-40) may not have acknowledged (311-34) the host (311-02) that initiated the request (311-04) to modify the dataset (311-42). The completion of request (311-04) for modifying dataset (311-42) is only possible when the request (311-04) to modify dataset (311-42) has been successfully processed on all storage systems (311-38, 311-40) across which the synchronously replicated dataset (311-42) spans. Only then can the leader storage system (311-40) confirm (311-34) the completion of request (311-04) to modify dataset (311-42) to the host (311-02) that initiated the request (311-04) to modify dataset (311-42).

[0239] The reader will understand that, Figure 3EIn the illustrated example method, the sending (311-12) of information (311-10) from the leader storage system (311-40) to the follower storage system (311-38) describing modifications to the dataset (311-42), and the confirmation (311-20) from the follower storage system (311-38) to the leader storage system (311-40) of the completion of the request (311-04) to modify the dataset (311-42) can be performed using a single round-trip message. For example, a single round-trip message can be used by using Fibre Channel as the data interconnect. Typically, the SCSI protocol is used in conjunction with Fibre Channel. Such interconnects are often provided between data centers because some older replication technologies can be built to replicate data as SCSI transactions essentially over a Fibre Channel network. Furthermore, historically, Fibre Channel SCSI infrastructure has had less overhead and lower latency compared to Ethernet and TCP / IP-based networks. Furthermore, when using Fibre Channel to connect the interior of a data center to a block storage array, the Fibre Channel network can be extended to other data centers, allowing a host in one data center to switch to access a storage array in a remote data center when the local storage array fails.

[0240] SCSI can be used as a general communication mechanism, even though it is typically designed to be used with block storage protocols for storing and retrieving data in block-oriented volumes (or tapes). For example, SCSI reads or SCSI writes can be used to deliver or retrieve message data between storage controllers in a paired storage system. A typical implementation of a SCSI write requires two round trips: the SCSI initiator sends a SCSI CDB describing the SCSI write operation, the SCSI target receives the CDB, and the SCSI target sends a "ready to receive" message to the SCSI initiator. The SCSI initiator then sends the data to the SCSI target, and when the SCSI write is complete, the SCSI target responds to the SCSI initiator with a successful completion message. On the other hand, a SCSI read request requires only one round trip: the SCSI initiator sends a SCSI CDB describing the SCSI read operation, the SCSI target receives the CDB and responds with data, then responds with a successful completion message. As a result, in terms of distance, SCSI reads result in half the distance-dependent latency compared to SCSI writes. Therefore, a data communication receiver using a SCSI read request to receive a message may be faster than a message sender using a SCSI write request to send data. Using SCSI reads only requires the message sender to act as the SCSI target and the message receiver to act as the SCSI initiator. The message receiver can send a certain number of SCSI CDB read requests to any message sender, and the message sender will respond to one of the incomplete CDB read requests when message data is available. Since the SCSI subsystem may time out if a read request takes too long (e.g., 10 seconds), read requests should be responded to within a few seconds even if there is no message data to send.

[0241] As described in the SCSI Streaming Command standard from the T10 Technical Committee of the International Information Technology Standards Committee, SCSI tape requests support variable response data, which allows for greater flexibility in returning variable-sized message data. The SCSI standard also supports an immediate mode for SCSI write requests, which allows for a single round trip of the SCSI write command. The reader will understand that many of the embodiments described below also utilize single-round-trip message delivery.

[0242] To further illustrate, Figure 4 Examples of cloud-based storage systems (403) according to some embodiments of the present invention are illustrated. Figure 4In the examples depicted, for instance, the cloud-based storage system (403) is created entirely within a cloud computing environment (402) such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform, IBM Cloud, and Oracle Cloud. The cloud-based storage system (403) can be used to provide services similar to those offered by the aforementioned storage systems. For example, the cloud-based storage system (403) can be used to provide block storage services to users of the cloud-based storage system (403), or it can be used to provide storage services to users of the cloud-based storage system (403) by using solid-state storage, and so on.

[0243] Figure 4 The depicted cloud-based storage system (403) includes two cloud computing instances (404, 406), each used to support the execution of storage controller applications (408, 410). The cloud computing instances (404, 406) may, for example, be instances of cloud computing resources (e.g., virtual machines) that a cloud computing environment (402) may provide to support the execution of software applications such as storage controller applications (408, 410). In one embodiment, the cloud computing instances (404, 406) may be represented as Amazon Elastic Compute Cloud (EC2) instances. In such an example, an Amazon Machine Image (AMI) including the storage controller applications (408, 410) can be launched to create and configure virtual machines that can execute the storage controller applications (408, 410).

[0244] exist Figure 4 In the described example method, the storage controller application (408, 410) can be embodied as a computer program instruction module that, when executed, performs various storage tasks. For example, the storage controller application (408, 410) can be embodied as a computer program instruction module that, when executed, performs tasks related to the aforementioned controller ( Figure 1AThe tasks (110A, 110B) are the same as those in the cloud storage system (403), such as writing data received from users of the cloud storage system (403) to the cloud storage system (403), erasing data from the cloud storage system (403), retrieving data from the cloud storage system (403) and providing this data to users of the cloud storage system (403), monitoring and reporting disk utilization and performance, performing redundancy operations such as Independent Drive Redundant Array (RAID) or RAID-like data redundancy operations, compressing data, encrypting data, and deduplicating data. The reader will understand that, since there are two cloud computing instances (404, 406), each including a storage controller application (408, 410), in some embodiments, one cloud computing instance (404) can operate as the primary controller as described above, while the other cloud computing instance (406) can operate as the secondary controller as described above. In such an example, to save costs, the cloud instance operating as the primary controller (404) can be deployed on a relatively high-performance and relatively expensive cloud instance, while the cloud instance operating as the secondary controller (406) can be deployed on a relatively low-performance and relatively inexpensive cloud instance. The reader will understand that... Figure 4 The described storage controller applications (408, 410) may include the same source code executed in different cloud computing instances (404, 406).

[0245] Consider an example where the cloud computing environment (402) is represented as AWS and the cloud computing instance is represented as an EC2 instance. In such an example, AWS offers many types of EC2 instances. For example, AWS offers a general set of EC2 instances that include different levels of storage and processing power. In such an example, the cloud computing instance (404) operating as the primary controller can be deployed on an instance type with relatively large storage and processing power, while the cloud computing instance (406) operating as the secondary controller can be deployed on an instance type with relatively small storage and processing power. In such an example, in the event of a failover event that switches the roles of the primary and secondary controllers, a dual failover can actually be performed such that: 1) a first failover event occurs, in which the cloud computing instance (406) previously operating as the secondary controller begins to operate as the primary controller; and 2) a third cloud computing instance (not shown) of an instance type with relatively large storage and processing power accelerates rotation using a copy of the storage controller application, in which the third cloud computing instance begins to operate as the primary controller, while the cloud computing instance (406) initially operating as the secondary controller begins to operate as the secondary controller again. In such an example, the cloud computing instance (404) that previously operated as the primary controller can be terminated. The reader will understand that, in an alternative embodiment, the cloud computing instance (404) that operated as a secondary controller after a failover event can continue to operate as a secondary controller, while the cloud computing instance (406) that operated as the primary controller after a failover event can be terminated once a third cloud computing instance (not shown) assumes the primary role.

[0246] The reader will understand that while the above embodiments relate to an embodiment where one cloud instance (404) operates as the primary controller and a second cloud instance (406) operates as the secondary controller, other embodiments are also within the scope of the invention. For example, each cloud instance (404, 406) may operate as the primary controller for a portion of the address space supported by the cloud-based storage system (403), each cloud instance (404, 406) may operate as the primary controller for segmenting the service of I / O operations for the cloud-based storage system (403) in some other way, and so on. In fact, in other embodiments where cost savings may take precedence over performance requirements, there may only be a single cloud instance containing the storage controller application. In such an example, a controller failure may require more time to recover because a new cloud instance including the storage controller application will need to be spun up faster rather than having the already created cloud instance take over the role of servicing I / O operations that would otherwise be handled by the failed cloud instance.

[0247] Figure 4The cloud-based storage system (403) described includes cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422). Figure 4 The cloud computing instances (424a, 424b, 424n) depicted may, for example, be embodied as instances of cloud computing resources that a cloud computing environment (402) may provide to support the execution of software applications. Figure 4 The cloud computing instances (424a, 424b, 424n) may differ from the cloud computing instances (404, 406) mentioned above because... Figure 4 Cloud instances (424a, 424b, 424n) have local storage (414, 418, 422) resources, while cloud instances (404, 406) supporting the execution of storage controller applications (408, 410) do not require local storage resources. Cloud instances (424a, 424b, 424n) with local storage (414, 418, 422) can be, for example, embodied as EC2 M5 instances including one or more SSDs, EC2 R5 instances including one or more SSDs, and EC2 I3 instances including one or more SSDs, etc. In some embodiments, the local storage (414, 418, 422) must be embodied as solid-state storage (e.g., SSDs) rather than storage utilizing hard disk drives.

[0248] exist Figure 4 In the depicted example, each of the cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) may include a software daemon (412, 416, 420) that, when executed by the cloud computing instance (424a, 424b, 424n), can present itself to the storage controller application (408, 410) as if the cloud computing instance (424a, 424b, 424n) were a physical storage device (e.g., one or more SSDs). In such an example, the software daemon (412, 416, 420) may include computer program instructions similar to those typically included on the storage device to enable the storage controller application (408, 410) to send and receive the same commands that the storage controller would send to the storage device. In this way, the storage controller application (408, 410) may include code that is the same (or substantially the same) as the code that the controller in the aforementioned storage system would execute. In these and similar embodiments, communication between the storage controller application (408, 410) and the cloud computing instance (424a, 424b, 424n) with local storage (414, 418, 422) can utilize iSCSI, TCP-based NVMe, messaging, custom protocols, or some other mechanism.

[0249] exist Figure 4 In the illustrated example, cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) can each also be coupled to block storage (426, 428, 430) provided by the cloud computing environment (402). The block storage (426, 428, 430) provided by the cloud computing environment (402) can be, for example, represented as an Amazon Elastic Block Store (EBS) volume. For example, a first EBS volume (e.g., block storage (426)) can be coupled to a first cloud computing instance (424a), a second EBS volume (e.g., block storage (428)) can be coupled to a second cloud computing instance (424b), and a third EBS volume (e.g., block storage (430)) can be coupled to a third cloud computing instance (424n). In such an example, the block storage (426, 428, 430) provided by the cloud computing environment (402) can be utilized in a similar manner to how the NVRAM device described above is utilized, because the software daemons (412, 416, 420) (or some other module) executing in a specific cloud computing instance (424a, 424b, 424n) can initiate writes of data to their attached EBS volume and to their local storage (414, 418, 422) resources when they receive a request to write data. In some alternative embodiments, data can only be written to the local storage (414, 418, 422) resources within a specific cloud computing instance (424a, 424b, 424n). In an alternative embodiment, the actual RAM on each cloud instance (424a, 424b, 424n) with local storage (414, 418, 422) can be used as NVRAM instead of the block storage (426, 428, 430) provided by the cloud environment (402), thereby reducing the network utilization costs associated with using EBS volumes as NVRAM.

[0250] exist Figure 4In the depicted example, cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) can be utilized by cloud computing instances (404, 406) supporting the execution of storage controller applications (408, 410) to serve I / O operations against the cloud-based storage system (403). Consider an example where a first cloud computing instance (404) executing a storage controller application (408) is operating as the primary controller. In such an example, the first cloud computing instance (404) executing the storage controller application (408) can (directly or indirectly via a secondary controller) receive requests from users of the cloud-based storage system (403) to write data to the cloud-based storage system (403). In such an example, the first cloud instance (404) executing the storage controller application (408) can perform various tasks, such as deduplicating the data contained in the request, compressing the data contained in the request, and determining where to write the data contained in the request, before ultimately sending a request to one or more cloud instances (424a, 424b, 424n) with local storage (414, 418, 422). In some embodiments, any cloud instance (404, 406) can receive a request to read data from a cloud-based storage system (403) and can ultimately send the request to one or more cloud instances (424a, 424b, 424n) with local storage (414, 418, 422).

[0251] The reader will understand that when a cloud computing instance (424a, 424b, 424n) with local storage (414, 418, 422) receives a request to write data, the software daemon (412, 416, 420) or another computer program instruction module executing on the specific cloud computing instance (424a, 424b, 424n) can be configured not only to write data to its own local storage (414, 418, 422) resources and any appropriate block storage (426, 428, 430) provided by the cloud computing environment (402), but also the software daemon (412, 416, 420) or another computer program instruction module executing on the specific cloud computing instance (424a, 424b, 424n) can be configured to write data to the cloud-based object storage (432) attached to the specific cloud computing instance (424a, 424b, 424n). The cloud-based object storage (432) attached to a specific cloud computing instance (424a, 424b, 424n) can, for example, be embodied as Amazon Simple Storage Service (S3) storage accessible to the specific cloud computing instance (424a, 424b, 424n). In other embodiments, cloud computing instances (404, 406), each including a storage controller application (408, 410), can initiate the storage of data in both the local storage (414, 418, 422) and the cloud-based object storage (432) of the cloud computing instance (424a, 424b, 424n).

[0252] Readers will understand that software daemons (412, 416, 420) or other computer program instruction modules used to write data to block storage (e.g., local storage resources (414, 418, 422)) and also to cloud-based object storage (432) can execute on dissimilar types of processing units (e.g., different types of cloud computing instances, cloud computing instances containing different processing units). In fact, software daemons (412, 416, 420) or other computer program instruction modules used to write data to block storage (e.g., local storage resources (414, 418, 422)) and also to cloud-based object storage (432) can migrate between different types of cloud computing instances as needed.

[0253] As will be understood by the reader, as described above, the cloud-based storage system (403) can be used to provide block storage services to users of the cloud-based storage system (403). While the local storage (414, 418, 422) and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n) can support block-level access, the cloud-based object storage (432) attached to a specific cloud computing instance (424a, 424b, 424n) only supports object-based access. To address this issue, a software daemon (412, 416, 420) or another computer program instruction module executing on a specific cloud computing instance (424a, 424b, 424n) can be configured to retrieve data blocks, encapsulate the data blocks into objects, and write the objects to the cloud-based object storage (432) attached to the specific cloud computing instance (424a, 424b, 424n).

[0254] Consider an example where data is written in 1 MB blocks to cloud computing instances (424a, 424b, 424n) using local storage (414, 418, 422) and block storage (426, 428, 430) resources. In such an example, suppose a user of a cloud-based storage system (403) issues a request to write data, which, after being compressed and deduplicated by the storage controller (408, 410), results in 5 MB of data needing to be written. In such an example, writing data to the local storage (414, 418, 422) and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n) is relatively simple, because five blocks of 1 MB each are written to the local storage (414, 418, 422) and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). In such an example, a software daemon (412, 416, 420) or another computer program instruction module executing on a specific cloud computing instance (424a, 424b, 424n) may be configured to: 1) create a first object comprising the first 1 MB of data and write the first object to cloud-based object storage (432); 2) create a second object comprising the second 1 MB of data and write the second object to cloud-based object storage (432); 3) create a third object comprising the third 1 MB of data and write the third object to cloud-based object storage (432); and so on. Therefore, in some embodiments, the sizes of the objects written to cloud-based object storage (432) may be the same (or nearly the same). The reader will understand that in such an example, metadata associated with the data itself may be included in each object (e.g., the first 1 MB of the object is the data, and the remainder is metadata associated with that data).

[0255] The reader will understand that cloud-based object storage (432) can be incorporated into a cloud-based storage system (403) to improve the persistence of the cloud-based storage system (403). Continuing with the above examples of cloud computing instances (424a, 424b, 424n) as EC2 instances, the reader will understand that EC2 instances are only guaranteed to have 99.9% monthly uptime, and data stored in local instance storage persists only for the lifetime of the EC2 instance. Therefore, relying on cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) as the sole source of persistent data storage in the cloud-based storage system (403) can result in a relatively unreliable storage system. Similarly, EBS volumes are designed for 99.999% availability. Therefore, even relying on EBS as persistent data storage in the cloud-based storage system (403) can result in a storage system that is not persistent enough. However, Amazon S3 is designed to provide 99.999999999% durability, meaning that cloud-based storage systems that can incorporate S3 into their storage pools (403) are significantly more durable than various other options.

[0256] The reader will understand that while cloud-based storage systems (403) that incorporate S3 into their storage pools are significantly more persistent than other options, using S3 as the primary storage pool can result in storage systems with relatively slow response times and relatively long I / O latency. Therefore, Figure 4 The cloud-based storage system (403) described not only stores data in S3, but also stores data in the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). This allows read operations to be served from the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n), thereby reducing read latency when users of the cloud-based storage system (403) attempt to read data from the cloud-based storage system (403).

[0257] In some embodiments, all data stored in the cloud-based storage system (403) may be stored in either: 1) cloud-based object storage (432), and 2) at least one of the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). In these embodiments, the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n) can be effectively operated as a cache that generally includes all data also stored in S3, so that all data reads can be served by the cloud computing instances (424a, 424b, 424n) without requiring the cloud computing instances (424a, 424b, 424n) to access the cloud-based object storage (432). However, the reader will understand that in other embodiments, all data stored by the cloud-based storage system (403) may be stored in cloud-based object storage (432), but less data than all data stored by the cloud-based storage system (403) may be stored in at least one of the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n). In such an example, various strategies may be employed to determine which subset of the data stored by the cloud-based storage system (403) should reside in either: 1) cloud-based object storage (432), and 2) at least one of the local storage (414, 418, 422) resources and block storage (426, 428, 430) resources utilized by the cloud computing instances (424a, 424b, 424n).

[0258] As described above, when cloud computing instances (424a, 424b, 424n) with local storage are presented as EC2 instances, these instances can only guarantee 99.9% monthly uptime, and the data stored in the local instance storage persists only throughout the lifecycle of each cloud computing instance (424a, 424b, 424n) with local storage. Therefore, one or more computer program instruction modules executing within the cloud-based storage system (403) (e.g., a monitoring module executing on its own EC2 instance) can be designed to handle failures of one or more cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422). In such an example, the monitoring module can handle the failure of one or more cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) by creating one or more new cloud computing instances with local storage, retrieving data stored on the failed cloud computing instances (424a, 424b, 424n) from cloud-based object storage (432), and storing the data retrieved from cloud-based object storage (432) in local storage on the newly created cloud computing instances. The reader will understand that many variations of this process can be implemented.

[0259] Consider an example where all cloud computing instances (424a, 424b, 424n) with local storage (414, 418, 422) fail. In such an example, the monitoring module can create a new cloud computing instance with local storage, where a high-bandwidth instance type is selected that allows the maximum data transfer rate between the newly created high-bandwidth cloud computing instance with local storage and the cloud-based object storage (432). The reader will understand that selecting the instance type that allows the maximum data transfer rate between the new cloud computing instance and the cloud-based object storage (432) enables the new high-bandwidth cloud computing instance to be re-integrated with the data from the cloud-based object storage (432) as quickly as possible. Once the new high-bandwidth cloud computing instance is re-integrated with the data from the cloud-based object storage (432), a cheaper low-bandwidth cloud computing instance can be created, data can be migrated to the cheaper low-bandwidth cloud computing instance, and the high-bandwidth cloud computing instance can be terminated.

[0260] The reader will understand that in some embodiments, the number of new cloud computing instances created may significantly exceed the number of cloud computing instances required to store all the data in the local cloud-based storage system (403). This is to allow for faster data retrieval from the cloud-based object storage (432) to the new cloud computing instances, as each new cloud computing instance can retrieve a portion of the data stored in the cloud-based storage system (403) (in parallel). In these embodiments, once the data stored in the cloud-based storage system (403) has been retrieved to the newly created cloud computing instances, the data can be merged into a subset of the newly created cloud computing instances, and an excessive number of newly created cloud computing instances can be terminated.

[0261] Consider an example requiring 1000 cloud computing instances to locally store all valid data written by users to a cloud-based storage system (403). In such an example, assume all 1000 cloud computing instances fail. In such an example, the monitoring module might create 100,000 cloud computing instances, each responsible for retrieving different 1 / 100,000 chunks of valid data written by users to the cloud-based storage system (403) from the cloud-based object storage (432) and locally storing the different chunks in its retrieved dataset. In such an example, because each of the 100,000 cloud computing instances can retrieve data from the cloud-based object storage (432) in parallel, the caching layer can recover 100 times faster than an embodiment where the monitoring module creates only 1000 replacement cloud computing instances. In such an example, over time, data stored locally across 100,000 cloud computing instances can be consolidated into 1,000 cloud computing instances, and the remaining 99,000 cloud computing instances can be terminated.

[0262] The reader will understand that various performance aspects of the cloud-based storage system (403) can be monitored (e.g., by a monitoring module executing in an EC2 instance), allowing the cloud-based storage system (403) to scale up or down as needed. Consider examples of how the monitoring module monitors the performance of the cloud-based storage system (403): via communication with one or more cloud computing instances (404, 406), each used to support the execution of storage controller applications (408, 410); via monitoring communication between cloud computing instances (404, 406, 424a, 424b, 424n); via monitoring communication between cloud computing instances (404, 406, 424a, 424b, 424n) and cloud-based object storage (432); or in some other way. In such an example, suppose the monitoring module determines that the cloud computing instances (404, 406) used to support the execution of the storage controller applications (408, 410) are too small and insufficient to serve the I / O requests issued by users of the cloud-based storage system (403). In this example, the monitoring module can create a new, more powerful cloud computing instance that includes the storage controller applications (e.g., a cloud computing instance with more processing power, more storage, etc.), allowing the new, more powerful cloud computing instance to begin operating as the primary controller. Similarly, if the monitoring module determines that the cloud computing instances (404, 406) used to support the execution of the storage controller applications (408, 410) are too large and that cost savings can be achieved by switching to a smaller, less powerful cloud computing instance, the monitoring module can create a new, less powerful (and less expensive) cloud computing instance that includes the storage controller applications, allowing the new, less powerful cloud computing instance to begin operating as the primary controller.

[0263] Consider an example where the monitoring module determines that the utilization rate of the local storage jointly provided by the cloud computing instances (424a, 424b, 424n) has reached a predetermined utilization threshold (e.g., 95%) as an additional example for dynamically adjusting the size of the cloud-based storage system (403). In such an example, the monitoring module can create additional cloud computing instances with local storage to expand the local storage pool provided by the cloud computing instances. Alternatively, the monitoring module can create one or more new cloud computing instances with a larger amount of local storage compared to the existing cloud computing instances (424a, 424b, 424n), allowing data stored in the existing cloud computing instances (424a, 424b, 424n) to be migrated to one or more new cloud computing instances, and the existing cloud computing instances (424a, 424b, 424n) to be terminated, thereby expanding the local storage pool provided by the cloud computing instances. Similarly, if the local storage pool provided by the cloud computing instances is unnecessarily large, data can be merged and some cloud computing instances can be terminated.

[0264] The reader will understand that the cloud-based storage system (403) can automatically scale up and down by applying a set of predetermined rules that may be relatively simple or relatively complex through a monitoring module. In fact, the monitoring module may not only consider the current state of the cloud-based storage system (403), but may also apply predictive strategies such as those based on observed behavior (e.g., relatively low usage of the storage system from 10 PM to 6 AM each day) and predetermined fingerprints (e.g., an increase of X in IOPS for the storage system each time 100 virtual desktops are added to the virtual desktop infrastructure). In such an example, the dynamic scaling of the cloud-based storage system (403) can be based on current performance metrics, predicted workloads, and many other factors (including combinations thereof).

[0265] The reader will further understand that because the cloud-based storage system (403) can scale dynamically, it can operate even in a more dynamic way. Consider the example of garbage collection. In a traditional storage system, the amount of storage is fixed. Because of this, at some point, the storage system may be forced to perform garbage collection because the available storage has become so limited that the storage system is on the verge of running out of storage. In contrast, the cloud-based storage system (403) described here can always “add” additional storage (e.g., by adding more cloud computing instances with local storage). Because the cloud-based storage system (403) described here can always “add” additional storage, it can make more informed decisions about when to perform garbage collection. For example, the cloud-based storage system (403) can implement a strategy that garbage collection is only performed when the number of IOPS served by the cloud-based storage system (403) falls below a certain level. In some embodiments, given that the size of the cloud-based storage system (403) is not limited in the same way as that of a traditional storage system, other system-level functions (e.g., deduplication, compression) can also be turned on and off in response to system load.

[0266] Readers will understand that embodiments of the present invention address a problem with block storage services provided in some cloud computing environments, where only one cloud instance is allowed to connect to a block storage volume at a time. For example, in Amazon AWS, only a single EC2 instance can connect to an EBS volume. By using EC2 instances with local storage, embodiments of the present invention can provide multi-connectivity capabilities, where multiple EC2 instances can connect to another EC2 instance (drive instance) with local storage. In such embodiments, the drive instance may include software executing within the drive instance that allows the drive instance to support a specific amount of I / O from each connected EC2 instance. Therefore, some embodiments of the present invention can be embodied as a multi-connectivity block storage service, which may not include... Figure 4 All the components depicted.

[0267] In some embodiments, particularly where the cloud-based object storage (432) resource is embodied as Amazon S3, the cloud-based storage system (403) may include one or more modules (e.g., computer program instruction modules executing on EC2 instances) configured to ensure that, when local storage on a particular cloud instance is re-integrated with data from S3, the appropriate data is actually in S3. This problem largely arises because S3 implements an eventual consistency model, in which, when an existing object is overwritten, reads of that object will eventually (but not necessarily immediately) become consistent and will eventually (but not necessarily immediately) revert to a rewritten version of the object. To address this problem, in some embodiments of the invention, objects in S3 are never overwritten. Instead, a conventional “rewrite” would result in the creation of a new object (including an updated version of the data) and the eventual deletion of the old object (including a previous version of the data).

[0268] In some embodiments of the invention, as part of an attempt to never (or almost never) rewrite objects, the resulting objects can be tagged with serial numbers when data is written to S3. In some embodiments, these serial numbers can be maintained elsewhere (e.g., in a database) so that at any point in time, the serial number associated with the latest version of a particular data can be known. In this way, it can be determined whether S3 has the latest version of a particular data by simply reading the serial number associated with the object without actually reading the data from S3. The ability to make this determination can be particularly important in the event of a failure of a cloud computing instance with local storage, as it is not desirable to re-integrate the local storage of the replacement cloud computing instance with outdated data. In fact, since the cloud-based storage system (403) does not need to access the data to verify its validity, the data can remain encrypted, and access costs can be avoided.

[0269] exist Figure 4 In the illustrated example and as described above, the cloud computing instances (404, 406) used to support the execution of the storage controller applications (408, 410) can operate in a primary / secondary configuration, wherein one of the cloud computing instances (404, 406) used to support the execution of the storage controller applications (408, 410) is responsible for writing data to the local storage of the cloud computing instances (424a, 424b, 424n) that have local storage (414, 418, 422). However, in such an example, since each of the cloud computing instances (404, 406) used to support the execution of the storage controller applications (408, 410) can access the cloud computing instances (424a, 424b, 424n) that have local storage, both cloud computing instances (404, 406) used to support the execution of the storage controller applications (408, 410) can serve requests to read data from the cloud-based storage system (403).

[0270] To further illustrate, Figure 5 Examples of additional cloud-based storage systems (502) according to some embodiments of the present invention are illustrated. Figure 5 In the examples depicted, for instance, the cloud-based storage system (502) is created entirely within a cloud computing environment (402) such as AWS, Microsoft Azure, Google Cloud Platform, IBM Cloud, and Oracle Cloud. The cloud-based storage system (502) can be used to provide services similar to those offered by the aforementioned storage systems. For example, the cloud-based storage system (502) can be used to provide block storage services to users of the cloud-based storage system (502), or it can be used to provide storage services to users of the cloud-based storage system (502) by using solid-state storage, and so on.

[0271] Figure 5 The described cloud-based storage system (502) can be used in conjunction with... Figure 4 The cloud-based storage system (403) described operates in a slightly similar manner because Figure 5 The depicted cloud-based storage system (502) includes a storage controller application (506) executed in a cloud computing instance (504). However, in Figure 5In the depicted example, the cloud instance (504) executing the storage controller application (506) is a cloud instance (504) with local storage (508). In such an example, data written to the cloud-based storage system (502) can be stored in the local storage (508) of the cloud instance (504) and can also be stored in the cloud-based object storage (510) in the same manner as with the cloud-based object storage (432). For example, in some embodiments, the storage controller application (506) may be responsible for writing data to the local storage (508) of the cloud instance (504), while the software daemon (512) may be responsible for ensuring that the data is written to the cloud-based object storage (510) in the same manner as with the cloud-based object storage (432). In other embodiments, the same entity (e.g., the storage controller application) may be responsible for writing data to the local storage (508) of the cloud instance (504) and also for ensuring that the data is written to the cloud-based object storage (510) in the same manner as with the cloud-based object storage (432).

[0272] The reader will understand, Figure 5 The cloud-based storage system (502) described can represent more than Figure 4 The depicted cloud-based storage system is a cheaper and more robust version. In yet another alternative embodiment, Figure 5 The depicted cloud-based storage system (502) may include an additional cloud computing instance with local storage supporting the execution of a storage controller application (506), such that failover can occur if the cloud computing instance (504) executing the storage controller application (506) fails. Similarly, in other embodiments, Figure 5 The cloud-based storage system (502) described may include additional cloud computing instances with local storage to extend the amount of local storage provided by the cloud computing instances in the cloud-based storage system (502).

[0273] The reader will understand that the above references Figure 4 Many of the fault scenarios described will also apply to Figure 5 The cloud-based storage system described (502). Similarly, Figure 5 The cloud-based storage system (502) described can be dynamically scaled up and down in a similar manner to that described above. The performance of various system-level tasks can also be optimized. Figure 5 The cloud-based storage system (502) described herein operates in an intelligent manner as described above.

[0274] Readers will understand that, in an effort to improve the resilience of the aforementioned cloud-based storage system, various components may reside in different availability zones. For example, a first cloud instance supporting the execution of a storage controller application may reside in a first availability zone, while a second cloud instance, also supporting the execution of a storage controller application, may reside in a second availability zone. Similarly, cloud instances with local storage may be distributed across multiple availability zones. In fact, in some embodiments, an entire second cloud-based storage system can be created in different availability zones, where data in the original cloud-based storage system is replicated (synchronously or asynchronously) to the second cloud-based storage system, such that if the entire cloud-based storage system fails, a replacement cloud-based storage system (the second cloud-based storage system) can be introduced in a negligible amount of time.

[0275] Readers will understand that the cloud-based storage system described herein can be used as part of a storage system cluster. In fact, the cloud-based storage system described here can be paired with an on-premises storage system. In such an example, data stored in internal storage can be replicated (synchronously or asynchronously) to the cloud-based storage system, and vice versa.

[0276] To further illustrate, Figure 6 A flowchart illustrating an example method for servicing I / O operations in a cloud-based storage system (604) is provided. Although not depicted in great detail, Figure 6 The cloud-based storage system (604) described herein may be similar to the cloud-based storage system described above and may be supported by a cloud computing environment (602).

[0277] Figure 6 The described example method includes receiving (606) a request to write data to the cloud-based storage system (604). For example, a user communicatively coupled to the storage system in a cloud computing environment may receive the request to write data from an application running in the cloud computing environment or otherwise. In such an example, the request may include data to be written to the cloud-based storage system (604). In other embodiments, the request to write data to the cloud-based storage system (604) may occur during startup when the cloud-based storage system (604) is being started.

[0278] Figure 6The illustrated example method also includes data deduplication (608). Data deduplication is a data reduction technique used to eliminate duplicate copies of duplicate data. The cloud-based storage system (604) can deduplicatize data, for example, by comparing one or more portions of the data with data already stored in the cloud-based storage system (604), by comparing the fingerprint of one or more portions of the data with the fingerprint of data already stored in the cloud-based storage system (604), or otherwise (608). In such an example, duplicate data can be removed and replaced with references to existing copies of data already stored in the cloud-based storage system (604).

[0279] Figure 6 The illustrated example method also includes data compression (610). Data compression is a data reduction technique that encodes information using fewer bits than the original representation. The cloud-based storage system (604) can compress the data (610) by applying one or more data compression algorithms to the data (which may not include data already stored in the cloud-based storage system (604)).

[0280] Figure 6 The illustrated example method also includes encrypting the data (612). Data encryption is a technique involving converting data from a readable format to an encoded format that can only be read or processed after the data has been decrypted. A cloud-based storage system (604) can use an encryption key to encrypt data (which may have already been deduplicated and compressed) (612). The reader will understand that, although Figure 6 The described embodiments involve deduplication (608), compression (610), and encryption (612) of data, but there are other embodiments that perform fewer of these steps, and embodiments that perform the same or fewer steps in a different order.

[0281] Figure 6 The described example method also includes storing (614) data in the block storage of a cloud-based storage system (604). As described in more detail above, storing (614) data in the block storage of a cloud-based storage system (604) can be performed, for example, by storing (616) data in solid-state storage such as local storage (e.g., SSD) of one or more cloud computing instances. In such an example, data can be distributed across the local storage of many cloud computing instances along with parity data to achieve RAID or RAID-class data redundancy.

[0282] Figure 6The described example method also includes storing (618) data in an object storage of a cloud-based storage system (604). Storing data (618) in the object storage of the cloud-based storage system may include creating (620) one or more equal-sized objects, each equal-sized object comprising different chunks of data. In such an example, since each object includes both data and metadata, the data portion of each object may be equal-sized. In other embodiments, the data portion of each created object may not be equal-sized. For example, each object may include data from a predetermined number of blocks in a block storage used in the preceding paragraph or otherwise used.

[0283] Figure 6 The described example method also includes receiving (622) a request to read data from a cloud-based storage system (604) by the cloud-based storage system. For example, a user communicatively coupled to the storage system in a cloud computing environment may receive a request to read data from the cloud-based storage system (604) from an application running in the cloud computing environment or otherwise. The request may, for example, include the logical address of the data to be read from the cloud-based storage system (604).

[0284] Figure 6 The illustrated method also includes retrieving (624) data from the block storage of a cloud-based storage system (604). The reader will understand that the cloud-based storage system (604) can retrieve (624) data from its block storage, for example, by forwarding read requests to a cloud computing instance that includes the requested data in local storage, via a storage controller application. The reader will understand that retrieving (624) data from the block storage of the cloud-based storage system (604) can be faster than reading data from cloud-based object storage, although cloud-based object storage does include copies of the data.

[0285] The reader will understand that, Figure 6 In the illustrated method, the block storage of the cloud-based storage system (604) is characterized by low read latency relative to the object storage of the cloud-based storage system. Therefore, by serving read operations from block storage instead of object storage, the cloud-based storage system (604) can be able to serve read operations using low-latency block storage while still providing the resilience associated with object storage solutions offered by cloud service providers. Furthermore, the block storage of the cloud-based storage system (604) can provide relatively high bandwidth. As the reader of this invention will appreciate, the block storage of the cloud-based storage system (604) can be implemented in various ways.

[0286] To further illustrate, Figure 7A flowchart illustrating an example method for servicing I / O operations in a cloud-based storage system (604) is provided. Figure 7 The example methods described are Figure 6 The example methods described are similar because Figure 7 The illustrated example methods also include receiving (606) a request to write data to a cloud-based storage system (604), storing (614) the data in the block storage of the cloud-based storage system (604), and storing (618) the data in the object storage of the cloud-based storage system (604).

[0287] Figure 7 The described example method also includes detecting (702) that at least a portion of the block storage of a cloud-based storage system has become unavailable. As described in more detail below, detecting (702) that at least a portion of the block storage of a cloud-based storage system has become unavailable can, for example, be done by detecting that one or more cloud computing instances, including local storage, have become unavailable for execution.

[0288] Figure 7 The described example method also includes identifying (704) data stored in a block storage portion of a cloud-based storage system that has become unusable. Identifying data stored in a block storage portion of a cloud-based storage system that has become unusable (704) can be performed, for example, by using metadata used to map some identifier of the data (e.g., sequence number, address) to the location where the data is stored. This metadata, or separate metadata, can also map the data to one or more object identifiers used to identify the object containing the data stored in the object storage of the cloud-based storage system.

[0289] Figure 7 The described example method also includes retrieving (706) data stored in a portion of the block storage of a cloud-based storage system that has become unusable from the object storage of the cloud-based storage system. Retrieving (706) data stored in a portion of the block storage of a cloud-based storage system that has become unusable from the object storage of the cloud-based storage system can be performed, for example, by using the aforementioned metadata for mapping the data stored in the portion of the block storage of the cloud-based storage system that has become unusable to one or more objects containing that data stored in the object storage of the cloud-based storage system. In such an example, retrieving (706) the data can be performed by reading the object mapped to the data from the object storage of the cloud-based storage system.

[0290] Figure 7The described example method also includes storing the retrieved data (708) in the block storage of a cloud-based storage system. As described in more detail above, storing the retrieved data (708) in the block storage of a cloud-based storage system can be performed, for example, by creating an alternative cloud computing instance with local storage and storing the data in the local storage of one or more alternative cloud computing instances.

[0291] The reader will understand that while the above embodiments relate to instances where data stored in a portion of the block storage of a cloud-based storage system that has become unusable is substantially retrieved back to the block storage layer of the cloud-based storage system by retrieving the data from the object storage layer of the cloud-based storage system, other embodiments are also within the scope of the invention. For example, since data can be distributed across local storage of multiple cloud computing instances using data redundancy techniques such as RAID, in some embodiments, lost data can be retrieved back to the block storage layer of the cloud-based storage system through RAID reconstruction.

[0292] To further illustrate, Figure 8 A flowchart illustrating an additional example method for servicing I / O operations in a cloud-based storage system (804) is provided. Although not depicted in great detail, Figure 8 The cloud-based storage system (804) described herein may be similar to the cloud-based storage system described above and may be supported by a cloud computing environment (802).

[0293] Figure 8 The described example method includes receiving (806) a request to write data to the cloud-based storage system (804). For example, a user communicatively coupled to the storage system in a cloud computing environment may receive the request to write data from an application running in the cloud computing environment or otherwise. In such an example, the request may include data to be written to the cloud-based storage system (804). In other embodiments, the request to write data to the cloud-based storage system (804) may occur during startup when the cloud-based storage system (804) is being started.

[0294] Figure 8The described example method also includes data deduplication (808). Data deduplication is a data reduction technique used to eliminate duplicate copies of duplicate data. The cloud-based storage system (804) can deduplicatize data, for example, by comparing one or more portions of the data with data already stored in the cloud-based storage system (804), by comparing the fingerprint of one or more portions of the data with the fingerprint of data already stored in the cloud-based storage system (804), or otherwise (808). In such an example, duplicate data can be removed and replaced with references to existing copies of data already stored in the cloud-based storage system (804).

[0295] Figure 8 The illustrated example method also includes data compression (810). Data compression is a data reduction technique that encodes information using fewer bits than the original representation. The cloud-based storage system (804) can compress the data (810) by applying one or more data compression algorithms to the data (which may not include data already stored in the cloud-based storage system (804)).

[0296] Figure 8 The illustrated example method also includes encrypting the data (812). Data encryption is a technique involving converting data from a readable format to an encoded format that can only be read or processed after the data has been decrypted. A cloud-based storage system (804) can use an encryption key to encrypt data (which may have already been deduplicated and compressed) (812). The reader will understand that, although Figure 8 ...

Claims

1. A method comprising: Within the storage provided by the first storage layer of the virtual storage system, data loss within the dataset is detected, while the second storage layer stores data for the recovery of the dataset. The recovery point of the dataset is determined using recovery data stored in the second storage layer, wherein determining the recovery point includes selecting a recovery dataset from a plurality of recovery datasets stored in the second storage layer, the recovery dataset excluding one or more data portions associated with incomplete operations on the dataset at the time of data loss and operations logically dependent on the incomplete operations; as well as The data loss within the selected recovery dataset is recovered using the selected recovery dataset, wherein the recovery dataset represents a previous point in time up to a version of the dataset that is recoverable using the second storage layer.

2. The method according to claim 1, wherein, The first storage layer includes a hierarchical memory that provides transaction consistency and write confirmation, and the second storage layer includes a virtual drive provided by the virtual drive server of the virtual storage system.

3. The method according to claim 2, wherein, The first storage layer includes virtual drives provided by the virtual drive server of the virtual storage system, and the second storage layer includes object storage provided by a cloud service provider, which provides object storage independent of the virtual storage system.

4. The method according to claim 2, wherein, The recovery of the dataset is based on erase codes stored between virtual drives of the hierarchical memory, on data migrated from the hierarchical memory to object storage, or on mirrored data between virtual drives of the hierarchical memory.

5. The method according to claim 3, wherein, Recovering the dataset involves using a database of segment identifiers to determine whether segments in the object storage can be used to recover the dataset.

6. The method according to claim 1, wherein, The recovery point is determined at least in part based on a batch of backend updates that are defined as having been committed, wherein the recovery dataset corresponds to the boundary of the committed updates.

7. The method according to claim 2, wherein, The virtual drive server provides hierarchical storage, and incomplete operations include operations that have been confirmed at the first storage tier but have not yet been committed at the second storage tier.

8. The method according to claim 7, wherein, The virtual drive server includes corresponding local storage.

9. The method according to claim 7, wherein, The virtual drive server provides block-level data storage.

10. The method according to claim 1, wherein, Requests to write data to the virtual storage system are received by one or more virtual controllers running within a virtual machine, container, or bare metal server.

11. The method according to claim 2, wherein, The hierarchical storage is provided by multiple virtual drive servers, each including a virtual controller and local storage.

12. The method according to claim 2, wherein, At least a portion of the data stored in the hierarchical storage is deduplicated, encrypted, or compressed before being migrated from the hierarchical storage to object storage, which is a more persistent data storage than the hierarchical storage.

13. The method according to claim 2, wherein, The hierarchical memory of the virtual storage system is characterized by low read latency relative to the object storage provided by the virtual storage system.

14. A virtual storage system contained in a cloud computing environment, the virtual storage system comprising: One or more virtual drives are used to provide hierarchical memory for storage operations; as well as One or more virtual controllers, each executing within a cloud computing instance, the cloud computing instance including a computer processor and computer memory operatively coupled to the computer processor, the computer memory having computer program instructions set therein, the computer program instructions causing the virtual controllers to perform the following steps when executed by the computer processor: Data loss within the dataset is detected within the storage provided by the first storage layer of the virtual storage system, wherein the second storage layer stores data for the recovery of the dataset; Determining a recovery point for the dataset using recovery data stored in the second storage layer, wherein determining the recovery point includes selecting a recovery dataset from a plurality of recovery datasets stored in the second storage layer, the recovery dataset excluding one or more portions of data associated with incomplete operations on the dataset at the time of data loss and operations logically dependent on the incomplete operations; as well as The data loss within the selected recovery dataset is recovered using the selected recovery dataset, wherein the recovery dataset represents a previous point in time up to a version of the dataset that is recoverable using the second storage layer.

15. The virtual storage system according to claim 14, wherein, Recovery of the lost data is based on erase codes stored between virtual drives of the hierarchical memory, on data migrated from the hierarchical memory to object storage, or on mirrored data between virtual drives of the hierarchical memory.

16. The virtual storage system according to claim 14, wherein, Recovering the lost data includes: referencing a database of segment identifiers to determine whether segments in object storage can be used to recover a subset of the lost data.

17. The virtual storage system according to claim 14, wherein, The recovery point is determined at least in part based on the batch of backend updates that are defined as having been submitted.

18. The virtual storage system according to claim 14, wherein, The virtual drive server provides the hierarchical storage.

19. The virtual storage system according to claim 18, wherein, The virtual drive server includes corresponding local storage.

20. The virtual storage system according to claim 18, wherein, Virtual drive servers provide block-level data storage.

21. The virtual storage system according to claim 14, wherein, The hierarchical storage is provided by multiple virtual drive servers, each consisting of a virtual controller and local storage.

Images