Container recovery layer prioritization

Abstract

An illustrative method of container recovery using layer prioritization includes identifying a set of immutable layers including container images in a dataset used by a container system to run a containerized application on a first cluster; copying the set of immutable layers of the container image to a second cluster to prepare a recovery event; after copying the set of immutable layers of the container image to the second cluster, receiving a recovery request to recover the containerized application; and in response to the recovery request, replicating a set of variable layers included in the dataset to the second cluster, the second cluster configured to recover the containerized application on the second cluster using the replicated set of immutable layers and the replicated set of variable layers.

Description

[0001] Cross Reference

[0002] This application claims priority to U.S. patent application No. 17 / 873,988 filed on July 26, 2022, U.S. patent application No. 17 / 733,781 filed on April 29, 2022, U.S. patent application No. 17 / 733,748 filed on April 29, 2022, U.S. patent application No. 17 / 733,935 filed on April 29, 2022, and U.S. patent application No. 17 / 733,716 filed on April 29, 2022, the entire contents of each of which are incorporated herein by reference.

[0003] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are examples only and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals designate the same or similar elements.

[0004] Figure 1A A first example system for data storage according to some embodiments is described.

[0005] Figure 1B A second example system for data storage is described in accordance with some embodiments.

[0006] Figure 1C A third example system for data storage is described in accordance with some embodiments.

[0007] Figure 1D A fourth example system for data storage is described in accordance with some embodiments.

[0008] Figure 2A is a perspective diagram of a storage cluster having a plurality of storage nodes and internal storage coupled to each storage node to provide network attached storage according to some embodiments.

[0009] Figure 2B is a block diagram illustrating an interconnect switch coupling a plurality of storage nodes according to some embodiments.

[0010] Figure 2C is a multi-level block diagram showing the contents of a storage node and the contents of one of the non-volatile solid-state storage units according to some embodiments.

[0011] Figure 2D A storage server environment according to some embodiments is shown, which uses embodiments of the storage nodes and storage units of some previous figures.

[0012] Figure 2Eis a block diagram of blade hardware according to some embodiments, showing the control plane, compute and storage planes, and the authorities that interact with the underlying physical resources.

[0013] Figure 2F Depicted is a resiliency software layer in a blade of a storage cluster according to some embodiments.

[0014] Figure 2G Depicting authoritative and storage resources in a blade of a storage cluster according to some embodiments.

[0015] Figure 3A A diagram illustrating a storage system coupled for data communication with a cloud service provider according to some embodiments of the present disclosure.

[0016] Figure 3B A diagram illustrating a storage system according to some embodiments of the present disclosure.

[0017] Figure 3C An example of a cloud-based storage system according to some embodiments of the present disclosure is described.

[0018] Figure 3D An exemplary computing device is illustrated that may be specifically configured to perform one or more of the processes described herein.

[0019] Figure 3E An example of a storage system cluster for providing storage services according to some embodiments of the present disclosure is described.

[0020] Figure 4 An example container system according to some embodiments of the present disclosure is described.

[0021] Figure 5 An example configuration of a container system and a container-aware storage system according to some embodiments of the present disclosure is described.

[0022] Figure 6 An example of a container-aware storage system that receives a container image and stores the container image as a volume according to some embodiments of the present disclosure is described.

[0023] Figures 7A to 7B An example of a container-aware storage system that provides volumes to a container system according to some embodiments of the present disclosure is described.

[0024] Figures 8 to 11 A flowchart depicting an example method according to some embodiments of the present disclosure is described.

[0025] Figures 12A to 15 An example of using a data volume to store a container image according to some embodiments of the present disclosure is described.

[0026] Figures 15 to 18A flowchart depicting an example method according to some embodiments of the present disclosure is described.

[0027] Figures 19A to 19B The description depicts a computing environment for container recovery using a volume including a container image according to some embodiments of the present disclosure.

[0028] Figures 20 to 21 A flowchart depicting an example method according to some embodiments of the present disclosure is described.

[0029] Figures 22A to 22E An example configuration in which a recovery process is applied according to some embodiments of the present disclosure is described.

[0030] Figures 23 to 25 An example container recovery method according to some embodiments of the present disclosure is described.

[0031] Fig.26A 26D to 26D illustrate a computing environment for storage system-based monitoring and container remediation according to some embodiments of the present disclosure.

[0032] Fig. 27 A flowchart depicting an example method according to some embodiments of the present disclosure is described.

[0033] Fig.28 A flowchart depicting an example method according to some embodiments of the present disclosure is described.

[0034] Figures 29A to 29B A computing environment for storage system-based threat detection and container remediation according to some embodiments of the present disclosure is described.

[0035] Figures 30 to 31 A flowchart depicting an example method according to some embodiments of the present disclosure is described.

[0036] Figures 32A to 32B A computing environment depicting immutable nodes in a container system is described according to some embodiments of the present disclosure.

[0037] Fig.33 A flowchart depicting an example method according to some embodiments of the present disclosure is described. DETAILED DESCRIPTION

[0038] by Figure 1A Initially, example methods, systems, apparatuses, and products for container recovery layer prioritization according to embodiments of the present disclosure are described with reference to the accompanying drawings. Figure 1A An example system for data storage according to some implementations is described. For purposes of illustration and not limitation, system 100 (also referred to herein as a "storage system") includes a number of elements. It may be noted that system 100 may include the same, more, or fewer elements configured in the same or different manner in other embodiments.

[0039] The system 100 includes a number of computing devices 164A-B. The computing devices (also referred to herein as "client devices") may be embodied as servers, workstations, personal computers, laptop computers, or the like in a data center, for example. The computing devices 164A-B may be coupled for data communication with one or more storage arrays 102A-B via a storage area network ('SAN') 158 or a local area network ('LAN') 160.

[0040] SAN 158 may be implemented with a variety of data communication structures, devices, and protocols. For example, the structure of SAN 158 may include Fibre Channel, Ethernet, InfiniBand, Serial Attached Small Computer System Interface ('SAS'), or the like. Data communication protocols used with SAN 158 may include Advanced Technology Attachment ('ATA'), Fibre Channel Protocol, Small Computer System Interface ('SCSI'), Internet Small Computer System Interface ('iSCSI'), HyperSCSI, Non-Volatile Memory Express ('NVMe') over Fabric, or the like. It may be noted that SAN 158 is provided for illustration and not limitation. Other data communication couplings may be implemented between computing devices 164A-B and storage arrays 102A-B.

[0041] LAN 160 may also be implemented using a variety of structures, devices, and protocols. For example, the structure used for LAN 160 may include Ethernet (802.3), wireless (802.11), or the like. Data communication protocols used in LAN 160 may include Transmission Control Protocol ('TCP'), User Datagram Protocol ('UDP'), Internet Protocol ('IP'), Hypertext Transfer Protocol ('HTTP'), Wireless Access Protocol ('WAP'), Handheld Device Transport Protocol ('HDTP'), Session Initiation Protocol ('SIP'), Real Time Protocol ('RTP'), or the like.

[0042] Storage arrays 102A-B may provide persistent data storage for computing devices 164A-B. In an embodiment, storage array 102A may be housed in a chassis (not shown) and storage array 102B may be housed in another chassis (not shown). Storage arrays 102A and 102B may include one or more storage array controllers 110A-D (also referred to herein as "controllers"). Storage array controllers 110A-D may be embodied as modules of an automated computing machine including computer hardware, computer software, or a combination of computer hardware and software. In some embodiments, storage array controllers 110A-D may be configured to perform various storage tasks. Storage tasks may include writing data received from computing devices 164A-B to storage arrays 102A-B, erasing data from storage arrays 102A-B, retrieving data from storage arrays 102A-B and providing data to computing devices 164A-B, monitoring and reporting disk utilization and performance, performing redundancy operations such as redundant array of independent drives ('RAID') or RAID-like data redundancy operations, compressing data, encrypting data, and the like.

[0043] The storage array controllers 110A-D may be implemented in various ways, including as a field programmable gate array ('FPGA'), a programmable logic chip ('PLC'), an application specific integrated circuit ('ASIC'), a system on a chip ('SOC'), or any computing device including discrete components, such as a processing device, a central processing unit, a computer memory, or various adapters. The storage array controllers 110A-D may include, for example, a data communications adapter configured to support communications via the SAN 158 or the LAN 160. In some implementations, the storage array controllers 110A-D may be independently coupled to the LAN 160. In an implementation, the storage array controllers 110A-D may include an I / O controller or the like that couples the storage array controllers 110A-D for data communications to persistent storage resources 170A-B (also referred to herein as "storage resources") via a midplane (not shown). The persistent storage resources 170A-B primarily include any number of storage drives 171A-F (also referred to herein as "storage devices") and any number of non-volatile random access memory ('NVRAM') devices (not shown).

[0044] In some embodiments, the NVRAM device of the persistent storage resource 170A to B may be configured to receive data to be stored in the storage drive 171A to F from the storage array controller 110A to D. In some examples, the data may originate from the computing device 164A to B. In some examples, writing data to the NVRAM device may be faster than writing data directly to the storage drive 171A to F. In an embodiment, the storage array controller 110A to D may be configured to utilize the NVRAM device as a fast accessible buffer for data that is predetermined to be written to the storage drive 171A to F. Relative to a system in which the storage array controller 110A to D directly writes data to the storage drive 171A to F, the latency of a write request using the NVRAM device as a buffer may be improved. In some embodiments, the NVRAM device may be implemented with a computer memory in the form of high-bandwidth, low-latency RAM. The NVRAM device is referred to as "non-volatile" because the NVRAM device may receive or contain a unique power source that maintains the state of the RAM after the main power of the NVRAM device is lost. Such a power source may be a battery, one or more capacitors, or the like. In response to a loss of power, the NVRAM device may be configured to write the contents of the RAM to a persistent storage device, such as storage drives 171A-F.

[0045] In an embodiment, storage drives 171A-F may refer to any device configured to persistently record data, where "persistently" or "persistent" refers to the ability of the device to maintain the recorded data after power is lost. In some embodiments, storage drives 171A-F may correspond to non-disk storage media. For example, storage drives 171A-F may be one or more solid-state drives ('SSD'), flash-based storage devices, any type of solid-state non-volatile memory, or any other type of non-mechanical storage device. In other embodiments, storage drives 171A-F may include a mechanical or rotating hard disk, such as a hard disk drive ('HDD').

[0046] In some embodiments, the storage array controllers 110A-D may be configured to offload device management responsibilities from the storage drivers 171A-F in the storage arrays 102A-B. For example, the storage array controllers 110A-D may manage control information that may describe the status of one or more memory blocks in the storage drivers 171A-F. The control information may indicate, for example, that a particular memory block has failed and should no longer be written to, that a particular memory block contains boot code for the storage array controllers 110A-D, the number of program erase ('P / E') cycles that have been performed on a particular memory block, the age of the data stored in the particular memory block, the type of data stored in the particular memory block, and the like. In some embodiments, the control information may be stored as metadata with the associated memory blocks. In other embodiments, the control information for the storage drivers 171A-F may be stored in one or more particular memory blocks of the storage drivers 171A-F selected by the storage array controllers 110A-D. The selected memory blocks may be tagged with an identifier indicating that the selected memory blocks contain the control information. The identifier may be utilized by the storage array controllers 110A-D in conjunction with the storage drives 171A-F to quickly identify a memory block containing control information. For example, the storage controllers 110A-D may issue a command to locate a memory block containing control information. It may be noted that the control information may be so large that portions of the control information may be stored in multiple locations, for example, for redundancy purposes, or the control information may be otherwise distributed across multiple memory blocks in the storage drives 171A-F.

[0047] In an embodiment, the storage array controller 110A-D may offload device management responsibilities from the storage drives 171A-F of the storage array 102A-B by retrieving control information describing the state of one or more memory blocks in the storage drives 171A-F from the storage drives 171A-F. Retrieving the control information from the storage drives 171A-F may be performed, for example, by the storage array controller 110A-D querying the storage drives 171A-F for the location of the control information for a particular storage drive 171A-F. The storage drives 171A-F may be configured to execute instructions that enable the storage drives 171A-F to identify the location of the control information. The instructions may be executed by a controller (not shown) associated with or otherwise located on the storage drives 171A-F, and may cause the storage drives 171A-F to scan a portion of each memory block to identify a memory block storing control information for the storage drives 171A-F. Storage drives 171A-F may respond by sending a response message to storage array controller 110A-D that includes the location of control information for storage drives 171A-F. In response to receiving the response message, storage array controller 110A-D may issue a request to read data stored at an address associated with the location of control information for storage drives 171A-F.

[0048] In other embodiments, the storage array controllers 110A-D may further offload device management responsibilities from the storage drives 171A-F by performing storage drive management operations in response to receiving the control information. The storage drive management operations may include, for example, operations that are typically performed by the storage drives 171A-F (e.g., a controller (not shown) associated with the particular storage drives 171A-F). The storage drive management operations may include, for example, ensuring that data is not written to failed memory blocks within the storage drives 171A-F, ensuring that data is written to memory blocks within the storage drives 171A-F in such a manner that adequate wear leveling is achieved, and the like.

[0049] In an embodiment, the storage arrays 102A to B may implement two or more storage array controllers 110A to D. For example, the storage array 102A may include a storage array controller 110A and a storage array controller 110B. In a given example, a single storage array controller 110A to D (e.g., storage array controller 110A) of the storage system 100 may be designated as having a primary status (also referred to herein as a "primary controller"), and other storage array controllers 110A to D (e.g., storage array controller 110A) may be designated as having a secondary status (also referred to herein as a "secondary controller"). The primary controller may have specific rights, such as permission to change data in persistent storage resources 170A to B (e.g., write data to persistent storage resources 170A to B). At least some of the rights of the primary controller may supersede the rights of the secondary controller. For example, when the primary controller has the right to change data in persistent storage resources 170A to B, the secondary controller may not have permission to change the data in persistent storage resources 170A to B. The status of the storage array controllers 110A through D may change. For example, the storage array controller 110A may be designated as having a secondary status, and the storage array controller 110B may be designated as having a primary status.

[0050] In some embodiments, a primary controller, such as storage array controller 110A, may be used as a primary controller for one or more storage arrays 102A-B, and a secondary controller, such as storage array controller 110B, may be used as a secondary controller for one or more storage arrays 102A-B. For example, storage array controller 110A may be the primary controller for storage array 102A and storage array 102B, and storage array controller 110B may be the secondary controller for storage arrays 102A and 102B. In some embodiments, storage array controllers 110C and 110D (also referred to as "storage processing modules") may have neither primary nor secondary status. Storage array controllers 110C and 110D implemented as storage processing modules may be used as a communication interface between primary and secondary controllers (e.g., storage array controllers 110A and 110B, respectively) and storage array 102B. For example, storage array controller 110A of storage array 102A may send a write request to storage array 102B via SAN 158. The write request may be received by both storage array controllers 110C and 110D of storage array 102B. Storage array controllers 110C and 110D facilitate communication, for example, sending the write request to the appropriate storage drives 171A through F. It may be noted that in some embodiments, storage processing modules may be used to increase the number of storage drives controlled by the primary and secondary controllers.

[0051] In an implementation, storage array controllers 110A-D are communicatively coupled to one or more storage drives 171A-F and one or more NVRAM devices (not shown) included as part of storage arrays 102A-B via a midplane (not shown). Storage array controllers 110A-D may be coupled to the midplane via one or more data communications links, and the midplane may be coupled to storage drives 171A-F and NVRAM devices via one or more data communications links. The data communications links described herein are collectively illustrated by data communications links 108A-D, and may include, for example, a Peripheral Component Interconnect Express ('PCIe') bus.

[0052] Figure 1B An example system for data storage is described according to some embodiments. Figure 1B The storage array controller 101 described in Figure 1A 1. The storage array controllers 110A to D described above are shown in FIG. 1. In one example, the storage array controller 101 may be similar to the storage array controller 110A or the storage array controller 110B. For purposes of illustration and not limitation, the storage array controller 101 includes a number of elements. It may be noted that the storage array controller 101 may include the same, more, or fewer elements configured in the same or different manner in other embodiments. It may be noted that Figure 1A Elements of may be included below to help illustrate features of storage array controller 101 .

[0053] The storage array controller 101 may include one or more processing devices 104 and a random access memory ('RAM') 111. The processing device 104 (or controller 101) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device 104 (or controller 101) may be a complex instruction set computing ('CISC') microprocessor, a reduced instruction set computing ('RISC') microprocessor, a very long instruction word ('VLIW') microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. The processing device 104 (or controller 101) may also be one or more special-purpose processing devices, such as an ASIC, an FPGA, a digital signal processor ('DSP'), a network processor, or the like.

[0054] Processing device 104 may be connected to RAM 111 via data communication link 106, which may be embodied as a high-speed memory bus, such as a double data rate 4 ('DDR4') bus. Stored in RAM 111 is an operating system 112. In some embodiments, instructions 113 are stored in RAM 111. Instructions 113 may include computer program instructions for performing operations in a direct-mapped flash memory system. In one embodiment, a direct-mapped flash memory system is a system that addresses data blocks within a flash drive directly and without requiring address translation performed by a storage controller of the flash drive.

[0055] In an embodiment, storage array controller 101 includes one or more host bus adapters 103A-C that are coupled to processing device 104 via data communication links 105A-C. In an embodiment, host bus adapters 103A-C may be computer hardware that connects a host system (e.g., a storage array controller) to other networks and storage arrays. In some examples, host bus adapters 103A-C may be Fibre Channel adapters that enable storage array controller 101 to connect to a SAN, Ethernet adapters that enable storage array controller 101 to connect to a LAN, or the like. Host bus adapters 103A-C may be coupled to processing device 104 via data communication links 105A-C such as, for example, a PCIe bus.

[0056] In an embodiment, storage array controller 101 may include a host bus adapter 114 coupled to an expander 115. Expander 115 may be used to attach a host system to a greater number of storage drives. Expander 115 may be, for example, a SAS expander that is used to enable host bus adapter 114 to attach to storage drives in embodiments where host bus adapter 114 is embodied as a SAS controller.

[0057] In an embodiment, the storage array controller 101 may include a switch 116 coupled to the processing device 104 via a data communication link 109. The switch 116 may be a computer hardware device that can create multiple endpoints from a single endpoint, thereby enabling multiple devices to share a single endpoint. For example, the switch 116 may be a PCIe switch that couples to a PCIe bus (e.g., data communication link 109) and presents multiple PCIe connection points to the midplane.

[0058] In an embodiment, storage array controller 101 includes a data communication link 107 for coupling storage array controller 101 to other storage array controllers. In some examples, data communication link 107 may be a Quick Path Interconnect (QPI) interconnect.

[0059] A conventional storage system using a conventional flash drive may implement processes across the flash drive that is part of the conventional storage system. For example, a higher level process of the storage system may span the flash drive startup and control process. However, the flash drive of the conventional storage system may include its own storage controller that also executes the processes. Thus, for the conventional storage system, both a higher level process (e.g., initiated by the storage system) and a lower level process (e.g., initiated by the storage controller of the storage system) may be executed.

[0060] To address various deficiencies of conventional storage systems, operations may be performed by higher level processes rather than lower level processes. For example, a flash memory system may include a flash drive that does not include a storage controller that provides the process. Thus, the operating system of the flash memory system itself may initiate and control the process. This may be achieved with a direct mapped flash memory system that addresses data blocks within the flash drive directly and without the need for address translation performed by the flash drive's storage controller.

[0061] In an embodiment, storage drive 171A to F may be one or more partitioned storage devices. In some embodiments, one or more partitioned storage devices may be shingled HDDs. In an embodiment, one or more storage devices may be flash-based SSDs. In a partitioned storage device, a partition namespace on the partitioned storage device may be addressed by groups of blocks grouped and aligned by natural size, thereby forming a number of addressable regions. In an embodiment utilizing an SSD, the natural size may be based on the erase block size of the SSD. In some embodiments, regions of the partitioned storage device may be defined during initialization of the partitioned storage device. In an embodiment, regions may be dynamically defined as data is written to the partitioned storage device.

[0062] In some embodiments, regions may be heterogeneous, with some regions each being a page group and other regions being multiple page groups. In an embodiment, some regions may correspond to erase blocks and other regions may correspond to multiple erase blocks. In an embodiment, for a heterogeneous mix of programming modes, manufacturers, product types, and / or product generations of storage devices applied to heterogeneous assemblies, upgrades, distributed storage, etc., regions may be any combination of different numbers of pages in page groups and / or erase blocks. In some embodiments, regions may be defined as having usage characteristics, such as properties of supporting data with a particular kind of life (e.g., very short life or very long life). These properties may be used by the partitioned storage device to determine how the region will be managed over the expected life of the region.

[0063] It should be understood that a region is a virtual structure. Any particular region may not have a fixed position at a storage device. Before allocation, a region may not have any position at a storage device. A region may correspond to a number representing a block of virtual allocatable space, which is the size of an erase block or other block size in various embodiments. When a system allocates or opens a region, the region is allocated to a flash memory or other solid-state memory, and when the system writes to the region, a page is written to the mapped flash memory or other solid-state memory of the partition storage device. When the system closes a region, an associated erase block or other block of size is completed. At a certain point in time in the future, the system may delete a region, which will free the allocated space of the region. During its lifespan, a region may be moved around to different locations of a partition storage device, for example, when the partition storage device performs internal maintenance.

[0064] In an embodiment, the region of a partitioned storage device may be in different states. A region may be in an empty state, where data has not yet been stored in the region. An empty region may be explicitly opened, or may be implicitly opened by writing data to the region. This is the initial state of a region on a new partitioned storage device, but it may also be the result of a region reset. In some embodiments, an empty region may have a specified location in the flash memory of a partitioned storage device. In an embodiment, the location of an empty region may be selected when the region is first opened or first written to (or if the write is buffered in a memory, then later). A region may be implicitly or explicitly in an open state, where the region in the open state may be written to store data with a write or append command. In an embodiment, a copy command to copy data from different regions may also be used to write to a region in an open state. In some embodiments, a partitioned storage device may have a limit on the number of open regions at a specific time.

[0065] A region in a closed state is a region that has been partially written to but has entered a closed state after an explicit close operation was issued. A region in a closed state may be left for future writes, but some runtime overhead consumed by keeping the region in an open state may be reduced. In an embodiment, a partitioned storage device may have a limit on the number of closed regions at a particular time. A region in a complete state is a region that is storing data and can no longer be written to. A region may be in a complete state after a write has written data to the entire region, or as a result of a region complete operation. Prior to a complete operation, a region may have been fully written or may not have been fully written. However, after a complete operation, it may not be possible to open the region for further writing without first performing a region reset operation.

[0066] The mapping from regions to erase blocks (or to shingled tracks in an HDD) can be arbitrary, dynamic, and hidden from view. The process of opening a region can be an operation that allows a new region to be dynamically mapped to the underlying storage of a partitioned storage device, and then allows data to be written by appending writes to the region until the region reaches capacity. The region can end at any point in time, after which further data cannot be written to the region. When the data stored in the region is no longer needed, the region can be reset, which effectively deletes the contents of the region from the partitioned storage device so that the physical storage maintained by the region can be used for subsequent storage of data. Once a region has been written and completed, the partitioned storage device ensures that the data stored in the region is not lost until the region is reset. In the time between writing data to the region and resetting the region, the region can be moved around between shingled tracks or erase blocks as part of maintenance operations within the partitioned storage device, such as by copying data to keep the data refreshed or to deal with memory cell aging in an SSD.

[0067] In embodiments utilizing HDDs, resetting a zone may allow shingled tracks to be assigned to new, open zones that may be opened at some point in the future. In embodiments utilizing SSDs, resetting a zone may cause the zone's associated physical erase blocks to be erased and subsequently reused for storage of data. In some embodiments, a zoned storage device may have a limit on the number of open zones at a point in time to reduce the amount of overhead dedicated to keeping zones open.

[0068] The operating system of the flash memory system can identify and maintain a list of allocation units across multiple flash drives of the flash memory system. An allocation unit can be an entire erase block or multiple erase blocks. The operating system can maintain a mapping or address range that directly maps addresses to erase blocks of the flash drives of the flash memory system.

[0069] The erase blocks mapped directly to the flash drive can be used to rewrite data and erase data. For example, an operation can be performed on one or more allocation units containing first data and second data, where the first data is to be retained and the second data is no longer used by the flash memory system. The operating system can initiate a process to write the first data to a new location within other allocation units and erase the second data and mark the allocation unit as available for subsequent data. Therefore, the process can be performed only by the higher-level operating system of the flash memory system without the need for additional lower-level processes to be performed by the controller of the flash memory drive.

[0070] Advantages of having the process performed only by the operating system of the flash memory system include increasing the reliability of the flash memory system's flash drives because no unnecessary or redundant write operations are performed during the process. One possible novel feature of this document is the concept of starting and controlling the process on the operating system of the flash memory system. Additionally, the process can be controlled by the operating system across multiple flash drives. This is in contrast to having the process performed by the storage controller of the flash memory drive.

[0071] The storage system may consist of two storage array controllers that share a set of drives for failover purposes, or it may consist of a single storage array controller that provides storage services that utilize multiple drives, or it may consist of a distributed network of storage array controllers, each storage array controller having a certain number of drives or a certain amount of flash devices, wherein the storage array controllers in the network cooperate to provide a complete storage service and cooperate in various aspects of the storage service, including storage allocation and garbage collection.

[0072] Figure 1C A third example system 117 for data storage according to some embodiments is described. For purposes of illustration and not limitation, the system 117 (also referred to herein as a "storage system") includes a number of elements. It may be noted that the system 117 may include the same, more, or fewer elements configured in the same or different manner in other embodiments.

[0073] In one embodiment, system 117 includes dual peripheral component interconnect ('PCI') flash memory devices 118 with individually addressable fast write storage devices. System 117 may include storage device controller 119. In one embodiment, storage device controller 119A-D may be a CPU, ASIC, FPGA, or any other circuit system that can implement the control structure required according to the present disclosure. In one embodiment, system 117 includes flash memory devices (e.g., including flash memory devices 120a-n) that are operably coupled to various channels of storage device controller 119. Flash memory devices 120a-n can be presented to controller 119A-D as an addressable collection of flash memory pages, erase blocks, and / or control elements, which are sufficient to allow storage device controller 119A-D to program and retrieve various aspects of the flash memory. In one embodiment, the storage device controllers 119A to D may perform operations on the flash memory devices 120a to n, including storing and retrieving the data contents of a page, placing and erasing any block, tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells, tracking and predicting error codes and failures within the flash memory, controlling voltage levels associated with programming and retrieving the contents of flash memory cells, and the like.

[0074] In one embodiment, the system 117 may include a RAM 121 to store addressable fast write data separately. In one embodiment, the RAM 121 may be one or more separate discrete devices. In another embodiment, the RAM 121 may be integrated into the storage device controller 119A to D or multiple storage device controllers. The RAM 121 may also be used for other purposes, such as temporary program storage for a processing device (e.g., a CPU) in the storage device controller 119.

[0075] In one embodiment, system 117 may include energy storage device 122, such as a rechargeable battery or capacitor. Energy storage device 122 may store enough energy to power storage device controller 119, an amount of RAM (e.g., RAM 121), and an amount of flash memory (e.g., flash memory 120a to 120n) for enough time to write the contents of the RAM to the flash memory. In one embodiment, if the storage device controller detects a loss of external power, the storage device controller 119A to D may write the contents of the RAM to the flash memory.

[0076] In one embodiment, the system 117 includes two data communication links 123a, 123b. In one embodiment, the data communication links 123a, 123b may be PCI interfaces. In another embodiment, the data communication links 123a, 123b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). The data communication links 123a, 123b may be based on the Non-Volatile Memory Express ('NVMe') or NVMe over Fabric ('NVMf') specifications, which allow external connections to the storage device controllers 119A to D from other components in the storage system 117. It should be noted that for convenience, the data communication links are interchangeably referred to herein as PCI buses.

[0077] The system 117 may also include an external power source (not shown) which may be provided through one or both of the data communication links 123a, 123b, or may be provided separately. An alternative embodiment includes a separate flash memory (not shown) dedicated to storing the contents of the RAM 121. The storage device controllers 119A-D may present a logical device on the PCI bus, which may include an addressable fast write logical device, or a different portion of the logical address space of the storage device 118, which may be presented as PCI memory or a persistent storage device. In one embodiment, operations to store to the device are directed to the RAM 121. In the event of a power failure, the storage device controllers 119A-D may write the storage contents associated with the addressable fast write logical memory to a flash memory (e.g., flash memory 120a-n) for long-term persistent storage.

[0078] In one embodiment, the logic device may include some representation of some or all of the contents of flash memory devices 120a-n, wherein the representation allows a storage system (e.g., storage system 117) including storage device 118 to directly address flash memory pages and directly reprogram erase blocks from storage system components external to the storage device over a PCI bus. The representation may also allow one or more external components to control and retrieve other aspects of the flash memory, including some or all of the following: tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells across all flash memory devices; tracking and predicting error codes and failures within and across flash memory devices; controlling voltage levels associated with programming and retrieving the contents of flash memory cells; and the like.

[0079] In one embodiment, the energy storage device 122 may be sufficient to ensure that ongoing operations on the flash memory devices 120a to 120n are completed. The energy storage device 122 may power the storage device controllers 119A to D and the associated flash memory devices (e.g., 120a to n) for the operations described, as well as for storing fast write RAM to flash memory. The energy storage device 122 may be used to store cumulative statistics and other parameters saved and tracked by the flash memory devices 120a to n and / or the storage device controller 119. A separate capacitor or energy storage device (e.g., a smaller capacitor close to or embedded in the flash memory device itself) may be used for some or all of the operations described herein.

[0080] Various schemes may be used to track and optimize the life of the energy storage components, such as adjusting voltage levels over time, partially discharging the energy storage device 122 to measure corresponding discharge characteristics, etc. If the available energy decreases over time, the effective available capacity of the addressable fast write storage device may be reduced to ensure that it can be safely written based on the currently available energy storage.

[0081] Figure 1D A third example storage system 124 for data storage according to some embodiments is illustrated. In one embodiment, the storage system 124 includes storage controllers 125a, 125b. In one embodiment, the storage controllers 125a, 125b are operably coupled to dual PCI storage devices. The storage controllers 125a, 125b are operably coupled (e.g., via a storage network 130) to some number of host computers 127a to n.

[0082] In one embodiment, two storage controllers (e.g., 125a and 125b) provide storage services, such as SCSI block storage arrays, file servers, object servers, database or data analysis services, etc. The storage controllers 125a, 125b may provide services to host computers 127a to n external to the storage system 124 through some number of network interfaces (e.g., 126a to d). The storage controllers 125a, 125b may provide integrated services or applications entirely within the storage system 124, thereby forming a converged storage and computing system. The storage controllers 125a, 125b may utilize fast write memory within or across the storage devices 119a to d to log ongoing operations to ensure that operations are not lost in the event of a power failure, storage controller removal, storage controller or storage system shutdown, or some failure of one or more software or hardware components within the storage system 124.

[0083] In one embodiment, the storage controllers 125a, 125b operate as PCI masters for one or the other PCI bus 128a, 128b. In another embodiment, 128a and 128b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). Other storage system embodiments may operate the storage controllers 125a, 125b as multi-masters for two PCI buses 128a, 128b. Alternatively, a PCI / NVMe / NVMf switching infrastructure or structure may connect multiple storage controllers. Some storage system embodiments may allow storage devices to communicate directly with each other, rather than just with the storage controllers. In one embodiment, the storage device controller 119a may operate under guidance from the storage controller 125a to retrieve data from a computer that has been stored in RAM (e.g., Figure 1C The recalculation may be performed by combining data in RAM 121 of the storage controller and transmitting the data to be stored in the flash memory device. For example, after the storage controller has determined that the operation has been fully committed across the storage system, or when the fast write memory on the device has reached a certain used capacity, or after a certain amount of time, a recalculated version of the RAM contents may be transmitted to ensure improved data security or to free up addressable fast write capacity for reuse. For example, this mechanism may be used to avoid a second transmission from the storage controller 125a, 125b over a bus (e.g., 128a, 128b). In one embodiment, the recalculation may include compressing the data, attaching indexes or other metadata, combining multiple data segments together, performing erasure code calculations, etc.

[0084] In one embodiment, under direction from the storage controller 125a, 125b, the storage device controller 119a, 119b is operable to retrieve data from the memory stored in RAM (e.g., Figure 1CThe storage device 128a, 128b may be used to calculate data from the data in the RAM 121 of the storage device 125a and transfer the data to the other storage device without involving the storage controller 125a, 125b. This operation may be used to mirror data stored in one storage controller 125a to another storage controller 125b, or it may be used to offload compression, data aggregation and / or erasure coding calculations and transfers to the storage device to reduce the load on the storage controller or the storage controller interface 129a, 129b to the PCI bus 128a, 128b.

[0085] The storage device controllers 119A-D may include mechanisms for implementing high availability primitives for use by other parts of the storage system external to the dual PCI storage device 118. For example, a reservation or exclusion primitive may be provided so that in a storage system with two storage controllers providing highly available storage services, one storage controller may prevent the other storage controller from accessing or continuing to access the storage device. This may be used, for example, in situations where one controller detects that the other controller is not functioning properly or the interconnect between the two storage controllers itself may not be functioning properly.

[0086] In one embodiment, a storage system for use with dual PCI direct mapped storage devices having individually addressable fast write storage devices includes a system for managing erase blocks or groups of erase blocks as allocation units for storing data on behalf of a storage service, or for storing metadata associated with the storage service (e.g., indexes, logs, etc.), or for properly managing the storage system itself. Flash pages, which may be several kilobytes in size, may be written as data arrives or when the storage system is about to keep data for a long interval (e.g., exceeding a defined time threshold). To commit data more quickly, or to reduce the number of writes to the flash memory devices, the storage controller may first write the data to an individually addressable fast write storage device on yet another storage device.

[0087] In one embodiment, the storage controllers 125a, 125b may enable the use of erase blocks within and across storage devices (e.g., 118) based on the age and expected remaining life of the storage devices or based on other statistical data. The storage controllers 125a, 125b may enable obsolete collection and data migration between storage devices based on pages that are no longer needed, as well as manage flash page and erase block lifespans, and manage overall system performance.

[0088] In one embodiment, the storage system 124 may utilize mirroring and / or erasure coding schemes as part of storing data in addressable fast write storage devices and / or as part of writing data to allocation units associated with erase blocks. Erasure codes may be used across storage devices, as well as within erase blocks or allocation units, or within and across flash memory devices on a single storage device to provide redundancy for single or multiple storage device failures, or to prevent internal corruption of flash memory pages caused by flash operations or flash cell degradation. Various levels of mirroring and erasure coding may be used to recover from multiple types of failures occurring alone or in combination.

[0089] refer to Figure 2A The embodiments depicted in Figure 1 to 1 illustrate a storage cluster that stores user data (e.g., user data originating from one or more users or client systems or other sources external to the storage cluster). The storage cluster distributes user data across storage nodes housed within a chassis or across multiple chassis using erasure coding and redundant copies of metadata. Erasure coding refers to a data protection or reconstruction method in which data is stored across a set of different locations (e.g., disks, storage nodes, or geographic locations). Flash memory is a type of solid-state memory that can be integrated with embodiments, but embodiments can be extended to other types of solid-state memory or other storage media, including non-solid-state memory. Control of storage locations and workloads is distributed across storage locations in a clustered peer-to-peer system. Tasks such as mediating communications between various storage nodes, detecting when a storage node becomes unavailable, and balancing I / O (input and output) across various storage nodes are all handled on a distributed basis. In some embodiments, data is arranged or distributed across multiple storage nodes in the form of data fragments or stripes that support data recovery. Independent of the input and output mode, ownership of data can be reassigned within the cluster. This architecture, described in more detail below, allows a storage node in a cluster to fail while the system remains operational because data can be rebuilt from other storage nodes and thus remain available for input and output operations.In various embodiments, a storage node may be referred to as a cluster node, a blade, or a server.

[0090] The storage cluster may be housed in a chassis (i.e., a housing that houses one or more storage nodes). The chassis contains a mechanism for providing power to each storage node (e.g., a power distribution bus) and a communication mechanism (e.g., a communication bus) that enables communication between storage nodes. According to some embodiments, the storage cluster may operate as a standalone system in one location. In one embodiment, the chassis contains at least two instances of both the power distribution and communication buses that can be independently enabled or disabled. The internal communication bus may be an Ethernet bus, however, other technologies (e.g., PCIe, InfiniBand, and others) are also applicable. The chassis provides ports for external communication buses to enable communication between multiple chassis and with client systems directly or through switches. External communications may use technologies such as Ethernet, InfiniBand, Fibre Channel, etc. In some embodiments, the external communication bus uses different communication bus technologies for inter-chassis and client communications. If a switch is deployed within a chassis or between chassis, the switch may be used as a conversion between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, the storage cluster may be accessed by clients using a proprietary interface or a standard interface, such as a Network File System ('NFS'), Common Internet File System ('CIFS'), Small Computer System Interface ('SCSI'), or Hypertext Transfer Protocol ('HTTP'). Conversion from the client protocol may occur at a switch, a chassis external communication bus, or within each storage node. In some embodiments, multiple chassis may be coupled or connected to each other via an aggregator switch. A portion and / or all of the coupled or connected chassis may be designated as a storage cluster. As discussed above, each chassis may have multiple blades, each blade having a media access control ('MAC') address, but in some embodiments, the storage cluster appears to the external network as having a single cluster IP address and a single MAC address.

[0091] Each storage node may be one or more storage servers, and each storage server is connected to one or more non-volatile solid-state storage units, which may be referred to as storage units or storage devices. One embodiment includes a single storage server in each storage node and non-volatile solid-state storage units between 1 and 8, but this one example is not meant to be restrictive. The storage server may include a processor, DRAM, and an interface for power distribution of an internal communication bus and each power bus. In some embodiments, inside the storage node, the interface and the storage unit share a communication bus, such as PCI Express. The non-volatile solid-state storage unit may directly access the internal communication bus interface through the storage node communication bus, or request the storage node to access the bus interface. The non-volatile solid-state storage unit contains an embedded CPU, a solid-state storage controller, and a certain amount of solid-state mass storage devices, such as between 2 and 32 terabytes ('TB') in some embodiments. An embedded volatile storage medium such as DRAM and an energy storage device are included in the non-volatile solid-state storage unit. In some embodiments, the energy storage device is a capacitor, a supercapacitor, or a battery, which transmits a subset of DRAM content to a stable storage medium in the event of power loss. In some embodiments, the non-volatile solid-state memory cells are comprised of storage class memory such as phase change or magnetoresistive random access memory ('MRAM') that replaces DRAM and implements a device with reduced power hold-up.

[0092] One of the many features of storage nodes and non-volatile solid-state storage devices is the ability to actively rebuild data in a storage cluster. Storage nodes and non-volatile solid-state storage devices can determine when a storage node or non-volatile solid-state storage device in a storage cluster is unreachable, regardless of whether an attempt is made to read data involving the storage node or non-volatile solid-state storage device. The storage nodes and non-volatile solid-state storage devices then collaborate to recover and rebuild the data in at least a portion of the new location. This constitutes active reconstruction because the system does not need to wait until the data is needed for a read access initiated from a client system employing the storage cluster before rebuilding the data. These and further details of the memory and its operation are discussed below.

[0093] Figure 2A1 is a perspective view of a storage cluster 161 according to some embodiments, the storage cluster 161 having a plurality of storage nodes 150 and internal solid-state storage coupled to each storage node to provide a network attached storage or storage area network. A network attached storage device, a storage area network or a storage cluster or other storage may include one or more storage clusters 161, each storage cluster 161 having one or more storage nodes 150, which are flexible and reconfigurable arrangements of both physical components and the amount of storage provided thereby. The storage cluster 161 is designed to be assembled in a rack, and one or more racks may be set up and populated according to the needs of the storage. The storage cluster 161 has a chassis 138 with a plurality of slots 142. It should be understood that the chassis 138 may be referred to as a housing, a shell, or a rack unit. In one embodiment, the chassis 138 has 14 slots 142, but other numbers of slots are easily designed. For example, some embodiments have four slots, eight slots, sixteen slots, thirty-two slots, or other suitable number of slots. In some embodiments, each slot 142 can accommodate one storage node 150. The chassis 138 includes a baffle 148 that can be used to mount the chassis 138 on a rack. The fan 144 provides air circulation for cooling the storage node 150 and its components, but other cooling components can be used, or an embodiment without a cooling component can be designed. The switch structure 146 couples the storage nodes 150 in the chassis 138 together and couples to the network to communicate with the storage. In the embodiment described herein, the slot 142 on the left side of the switch structure 146 and the fan 144 is shown as being occupied by the storage node 150, while the slot 142 on the right side of the switch structure 146 and the fan 144 is empty and can be used to insert the storage node 150 for illustration purposes. This configuration is an example, and one or more storage nodes 150 can occupy the slot 142 in various other arrangements. In some embodiments, the storage node arrangement does not have to be sequential or adjacent. The storage node 150 is hot-swappable, which means that the storage node 150 can be inserted into the slot 142 in the chassis 138, or removed from the slot 142 without stopping or shutting down the system. After a storage node 150 is inserted or removed from a slot 142, the system automatically reconfigures to recognize and adapt to the change. In some embodiments, reconfiguration includes restoring redundancy and / or rebalancing data or load.

[0094] Each storage node 150 may have multiple components. In the embodiment shown herein, the storage node 150 includes a printed circuit board 159 populated by a CPU 156 (i.e., a processor), a memory 154 coupled to the CPU 156, and a non-volatile solid-state storage device 152 coupled to the CPU 156, but other installations and / or components may be used in other embodiments. The memory 154 has instructions executed by the CPU 156 and / or data operated by the CPU 156. As further explained below, the non-volatile solid-state storage device 152 includes flash memory, or in other embodiments, includes other types of solid-state memory.

[0095] refer to Figure 2A , the storage cluster 161 is scalable, which means that it is easy to add storage capacity with uneven storage sizes, as described above. In some embodiments, one or more storage nodes 150 can be inserted into or removed from each chassis, and the storage cluster self-configures. The plug-in storage nodes 150, whether installed in the chassis at the time of delivery or added later, can have different sizes. For example, in one embodiment, the storage node 150 can have any multiple of 4TB, such as 8TB, 12TB, 16TB, 32TB, etc. In other embodiments, the storage node 150 can have any multiple of other storage amounts or capacities. The storage capacity of each storage node 150 is broadcasted and affects the decision on how to stripe data. For maximum storage efficiency, the embodiment can be self-configured as wide as possible in the stripe, subject to the predetermined requirement of continuing operation in the case of losing up to one or up to two non-volatile solid-state storage devices 152 units or storage nodes 150 in the chassis.

[0096] Figure 2B 1 is a block diagram showing a communication interconnect 173 and a power distribution bus 172 coupling a plurality of storage nodes 150. Return to Reference Figure 2A In some embodiments, the communication interconnect 173 may be included in or implemented with the switch fabric 146. In some embodiments, where multiple storage clusters 161 occupy a rack, the communication interconnect 173 may be included in or implemented with a top-of-rack switch. Figure 2B 1, storage cluster 161 is enclosed within a single chassis 138. External port 176 is coupled to storage node 150 via communication interconnect 173, while external port 174 is coupled directly to storage node. External power port 178 is coupled to power distribution bus 172. Storage node 150 may include non-volatile solid-state storage devices 152 of varying amounts and capacities, as described in reference to FIG. Figure 2A Description. In addition, Figure 2BAs described in the specification, one or more storage nodes 150 may be computing-only storage nodes. The authority 168 is implemented on the non-volatile solid-state storage device 152, for example as a list or other data structure stored in a memory. In some embodiments, the authority is stored within the non-volatile solid-state storage device 152 and is supported by software executed on a controller or other processor of the non-volatile solid-state storage device 152. In other embodiments, the authority 168 is implemented on the storage node 150, for example as a list or other data structure stored in a memory 154 and is supported by software executed on the CPU 156 of the storage node 150. In some embodiments, the authority 168 controls how and where data is stored in the non-volatile solid-state storage device 152. This control helps determine which type of erasure coding scheme is applied to the data, and which storage nodes 150 have which parts of the data. Each authority 168 can be assigned to a non-volatile solid-state storage device 152. In various embodiments, each authority may control a range of inode numbers, segment numbers, or other data identifiers that are assigned to data by the file system, by the storage nodes 150, or by the non-volatile solid-state storage devices 152.

[0097] In some embodiments, each piece of data and each piece of metadata has redundancy in the system. In addition, each piece of data and each piece of metadata has an owner, which can be called an authority. If the authority is unreachable, for example, due to a failure of a storage node, there is a succession plan for how to find the data or the metadata. In various embodiments, there are redundant copies of authority 168. In some embodiments, authority 168 is related to storage node 150 and non-volatile solid-state storage device 152. Each authority 168 covering a range of data segment numbers or other identifiers of data can be assigned to a specific non-volatile solid-state storage device 152. In some embodiments, all these ranges of authority 168 are distributed on the non-volatile solid-state storage device 152 of the storage cluster. Each storage node 150 has a network port that provides access to the non-volatile solid-state storage device 152 of the storage node 150. In some embodiments, data can be stored in segments associated with segment numbers, and the segment number is the indirectness of the configuration of a RAID (Redundant Array of Independent Disks) stripe. Therefore, the assignment and use of authority 168 establishes indirectness to data. According to some embodiments, indirectness may be referred to as the ability to reference data indirectly (in this case via authority 168). A segment identifies a set of non-volatile solid-state storage devices 152 and a local identifier into the set of non-volatile solid-state storage devices 152 that may contain data. In some embodiments, the local identifier is an offset into the device and may be reused sequentially by multiple segments. In other embodiments, the local identifier is unique to a particular segment and is never reused. The offset in the non-volatile solid-state storage device 152 is applied to locate data (in the form of a RAID stripe) for writing to or reading from the non-volatile solid-state storage device 152. The data is striped across multiple units of the non-volatile solid-state storage device 152, which may include or be different from the non-volatile solid-state storage device 152 having the authority 168 for a particular data segment.

[0098] If the location where a particular data segment is located changes, such as during data movement or data reconstruction, the authority 168 of the data segment should be consulted at the non-volatile solid-state storage device 152 or storage node 150 with the authority 168. In order to locate a specific piece of data, an embodiment calculates a hash value of the data segment or applies an index node number or a data segment number. The output of this operation points to a non-volatile solid-state storage device 152 with an authority 168 for the specific piece of data. In some embodiments, this operation has two stages. The first stage maps an entity identifier (ID) (such as a segment number, an index node number, or a directory number) to an authoritative identifier. This mapping may include, for example, the calculation of a hash or a bit mask. The second stage is to map the authoritative identifier to a specific non-volatile solid-state storage device 152, which can be accomplished by explicit mapping. The operation is repeatable so that when the calculation is performed, the result of the calculation can be repeatedly and reliably pointed to a specific non-volatile solid-state storage device 152 with the authority 168. The operation may include a reachable storage node group as input. If the set of reachable non-volatile solid-state storage units changes, the best set also changes. In some embodiments, the saved value is the current assignment (always true), and the calculated value is the target assignment toward which the cluster will attempt to reconfigure. This calculation can be used to determine the best non-volatile solid-state storage device 152 for authority in the presence of a set of non-volatile solid-state storage devices 152 that are reachable and constitute the same cluster. The calculation also determines an ordered set of peer non-volatile solid-state storage devices 152, which will also record the authority to non-volatile solid-state memory mapping so that the authority can be determined even if the assigned non-volatile solid-state storage device is unreachable. If a particular authority 168 is not available in some embodiments, a copy or alternative authority 168 can be consulted.

[0099] refer to Figure 2A and 2B, two of the many tasks of the CPU 156 on the storage node 150 are to decompose write data and reassemble read data. When the system has determined that data is to be written, the authority 168 of the data is located as described above. When the segment ID of the data has been determined, the write request is forwarded to the non-volatile solid-state storage device 152 of the host that is currently determined to be the authority 168 determined from the segment. The host CPU 156 of the storage node 150 on which the non-volatile solid-state storage device 152 and the corresponding authority 168 reside then decomposes or fragments the data and transmits the data to various non-volatile solid-state storage devices 152. According to the erasure coding scheme, the transmitted data is written as data stripes. In some embodiments, the request extracts the data, and in other embodiments, the data is pushed. In contrast, when reading data, the authority 168 containing the segment ID of the data is located as described above. The host CPU 156 of the storage node 150 on which the non-volatile solid-state storage device 152 and the corresponding authority 168 reside requests data from the non-volatile solid-state storage device and the corresponding storage node pointed to by the authority. In some embodiments, the data is read from the flash memory as a data stripe. The host CPU 156 of the storage node 150 then reassembles the read data, corrects any errors (if any) according to the appropriate erasure coding scheme, and forwards the reassembled data to the network. In other embodiments, some or all of these tasks can be handled in the non-volatile solid-state storage device 152. In some embodiments, the segment host requests that data be sent to the storage node 150 by requesting a page from the storage device and then sending the data to the storage node that made the original request.

[0100] In an embodiment, authority 168 operates to determine how operations will be performed for specific logical elements. Each logical element can be operated by a specific authority across multiple storage controllers of the storage system. Authority 168 can communicate with multiple storage controllers so that multiple storage controllers jointly perform operations on those specific logical elements.

[0101] In embodiments, the logical element may be, for example, a file, a directory, an object bucket, an individual object, a delineated portion of a file or object, other forms of key-value databases or tables. In embodiments, performing an operation may involve, for example, ensuring consistency, structural integrity, and / or recoverability of other operations on the same logical element, reading metadata and data associated with the logical element, determining which data should be persistently written to the storage system to preserve any changes to the operation, or metadata and data may be determined where to store across modular storage devices attached to multiple storage controllers in the storage system.

[0102] In some embodiments, the operations are token-based transactions to efficiently communicate within a distributed system. Each transaction may be accompanied or associated with a token that gives permission to perform the transaction. In some embodiments, the authority 168 can maintain the pre-transaction state of the system until the operation is completed. Token-based communication can be completed without a global lock across the system, and can also restart operations in the event of an interruption or other failure.

[0103] In some systems, such as in UNIX-style file systems, data is handled with index nodes (index nodes or inodes), which specify data structures that represent objects in the file system. For example, an object may be a file or a directory. Metadata may accompany the object as attributes (such as permission data and creation timestamps, among other attributes). Segment numbers may be assigned to all or part of such objects in the file system. In other systems, data segments are handled with segment numbers assigned elsewhere. For the purposes of discussion, an allocation unit is an entity, and an entity may be a file, a directory, or a segment. That is, an entity is a unit of data or metadata stored by a storage system. Entities are grouped into groups called authorities. Each authority has an authority owner, which is a storage node that has the exclusive right to update entities in the authority. In other words, a storage node contains an authority, and the authority contains entities.

[0104] According to some embodiments, a segment is a logical container for data. A segment is an address space between the media address space, and the physical flash locations (i.e., data segment numbers) are in this address space. A segment may also contain metadata that enables data redundancy to be restored (rewritten to a different flash location or device) without involving higher level software. In one embodiment, the internal format of a segment contains client data and a media map to determine the location of the data. Where applicable, each data segment is protected, for example, from memory and other failures by breaking the segment into several data and parity slices. According to an erasure coding scheme, data and parity slices are coupled to the host CPU 156 (see Figure 2E and 2G ) of the non-volatile solid-state storage device 152, i.e., striped. In some embodiments, the use of the term segment refers to the container and its location in the address space of the segment. According to some embodiments, the use of the term stripe refers to the same group of shards as a segment and includes how the shards are distributed along with redundancy or parity information.

[0105] A series of address space conversions occur across the entire storage system. At the top are directory entries (file names) linked to index nodes. Index nodes point to media address spaces that logically store data. Media addresses can be mapped through a series of indirect media to distribute the load of large files, or implement data services such as deduplication or snapshots. Media addresses can be mapped through a series of indirect media to distribute the load of large files, or implement data services such as deduplication or snapshots. The segment address is then converted into a physical flash location. According to some embodiments, the physical flash location has an address range bounded by the amount of flash in the system. Media addresses and segment addresses are logical containers, and in some embodiments use 128-bit or larger identifiers so that they are actually infinite, where the possibility of reuse is calculated to be longer than the expected life of the system. In some embodiments, addresses from logical containers are allocated in a hierarchical manner. Initially, each non-volatile solid-state storage device 152 unit can be assigned a range of address space. Within this assigned range, the non-volatile solid-state storage device 152 can allocate addresses without synchronizing with other non-volatile solid-state storage devices 152.

[0106] Data and metadata are stored by a set of underlying storage layouts that are optimized for different workload patterns and storage devices. These layouts incorporate a variety of redundancy schemes, compression formats, and indexing algorithms. Some of these layouts store information about the authority and the authoritative master, while others store file metadata and file data. Redundancy schemes include error correction codes that allow for damaged bits within a single storage device (e.g., a NAND flash chip), erasure codes that allow for failure of multiple storage nodes, and replication schemes that allow for data center or regional failures. In some embodiments, low-density parity check ('LDPC') codes are used within a single storage unit. In some embodiments, Reed-Solomon encoding is used within a storage cluster, and mirroring is used within a storage grid. An ordered log-structured index (e.g., a log-structured merge tree) can be used to store metadata, and large data may not be stored in a log-structured layout.

[0107] In order to maintain consistency across multiple copies of an entity, storage nodes implicitly agree on two things through computation: (1) the authority containing the entity, and (2) the storage node containing the authority. Assigning entities to authorities can be accomplished by pseudo-randomly assigning entities to authorities, by partitioning entities into several ranges based on externally generated keys, or by placing a single entity into each authority. Examples of pseudo-random schemes are linear hashing and hashing of the replication under scalable hashing ('RUSH') family, including controlled replication under scalable hashing ('CRUSH'). In some embodiments, pseudo-random assignment is only used to assign authorities to nodes because the node group can change. The authority group cannot change, so any subjective function can be applied in these embodiments. Some placement schemes automatically place authorities on storage nodes, while other placement schemes rely on an explicit mapping of authorities to storage nodes. In some embodiments, a pseudo-random scheme is used to map from each authority to a set of candidate authority owners. The pseudo-random data allocation function associated with CRUSH can assign authorities to storage nodes and create a list of where the authorities are assigned. Each storage node has a copy of the pseudo-random data allocation function and can derive the same calculations for allocation and subsequent lookup or location of the authority. In some embodiments, each pseudo-random scheme requires a set of reachable storage nodes as input in order to infer the same target node. Once an entity has been placed in an authority, the entity can be stored on a physical device so that expected failures do not result in unexpected data loss. In some embodiments, the rebalancing algorithm attempts to store copies of all entities within an authority in the same layout and on the same set of machines.

[0108] Examples of expected failures include device failures, stolen machines, data center fires, and regional disasters, such as nuclear or geological events. Different failures result in different levels of acceptable data loss. In some embodiments, the theft of a storage node affects neither the security nor the reliability of the system, while regional events can result in no data loss, a few seconds or minutes of lost updates, or even complete data loss, depending on the system configuration.

[0109] In an embodiment, the placement of data for storing redundancy is independent of the authoritative placement for data consistency. In some embodiments, the storage node containing authority does not contain any persistent storage. On the contrary, the storage node is connected to a non-volatile solid-state storage unit that does not contain authority. The communication interconnection between the storage node and the non-volatile solid-state storage unit consists of a variety of communication technologies and has uneven performance and fault tolerance. In some embodiments, as mentioned above, the non-volatile solid-state storage unit is connected to the storage node via PCI express, and the storage node is connected together in a single chassis using an Ethernet backplane, and the chassis are connected together to form a storage cluster. In some embodiments, the storage cluster is connected to the client using Ethernet or Fibre Channel. If multiple storage clusters are configured into a storage grid, then the Internet or other long-distance network links (such as "metropolitan scale" links or dedicated links that do not cross the Internet) are used to connect multiple storage clusters.

[0110] The authoritative owner has the exclusive right to modify the entity, migrate the entity from one non-volatile solid-state storage unit to another non-volatile solid-state storage unit, and add and delete copies of the entity. This allows the redundancy of the underlying data to be maintained. When the authoritative owner fails, is about to retire, or is overloaded, the authority is transferred to the new storage node. Transient failures make it very important to ensure that all non-faulty machines agree on the new authoritative position. The ambiguity caused by transient failures can be automatically implemented through consensus protocols (such as Paxos, hot and cold failover schemes), through manual intervention by remote system administrators or local hardware administrators (for example, by physically removing the failed machine from the cluster, or pressing a button on the failed machine). In some embodiments, a consensus protocol is used and failover is automatic. According to some embodiments, if too many failures or replication events occur in too short a period of time, the system enters a self-preservation mode and stops replication and data movement activities until the administrator intervenes.

[0111] The system transmits messages between storage nodes and non-volatile solid-state storage units as authority is transferred between storage nodes and authority owners update entities in their authority. Regarding persistent messages, messages with different purposes are of different types. Depending on the type of message, the system maintains different sorting and persistence guarantees. When persistent messages are processed, the messages are temporarily stored in multiple persistent and non-persistent storage hardware technologies. In some embodiments, messages are stored in RAM, NVRAM, and on NAND flash memory devices, and various protocols are used to efficiently utilize each storage medium. Delay-sensitive client requests can be saved in replicated NVRAM and then subsequently saved in NAND, while background rebalancing operations are saved directly to NAND.

[0112] Persistent messages are persistently stored before transmission. This allows the system to continue to service client requests even in the face of failures and component replacements. Although many hardware components contain unique identifiers visible to system administrators, manufacturers, hardware supply chains, and persistent monitoring quality control infrastructure, applications running on top of infrastructure addresses virtualize the addresses. These virtualized addresses do not change over the life of the storage system, regardless of component failures and replacements. This allows each component of the storage system to be replaced over time without reconfiguration or interruption of client request processing, i.e., the system supports non-disruptive upgrades.

[0113] In some embodiments, virtualized addresses are stored with sufficient redundancy. The continuous monitoring system correlates hardware and software states with hardware identifiers. This allows for detection and prediction of failures due to faulty components and manufacturing details. In some embodiments, the monitoring system also enables proactive transfer of authority and entities away from affected devices before a failure occurs by removing components from critical paths.

[0114] Figure 2C 1 is a multi-level block diagram showing the contents of a storage node 150 and the contents of a non-volatile solid-state storage device 152 of the storage node 150. In some embodiments, data is passed to and from the storage node 150 by a network interface controller ('NIC') 202. Each storage node 150 has a CPU 156 and one or more non-volatile solid-state storage devices 152, as discussed above. Figure 2C Moving down one level in FIG. 1 , each non-volatile solid-state storage device 152 has relatively fast non-volatile solid-state memory, such as non-volatile random access memory ('NVRAM') 204 and flash memory 206. In some embodiments, NVRAM 204 may be a component that does not require program / erase cycles (DRAM, MRAM, PCM), and may be a memory that can support being written to more frequently than the memory is read. Figure 2CMoving down another level in the NVRAM 204, in one embodiment, the NVRAM 204 is implemented as a high-speed volatile memory, such as a dynamic random access memory (DRAM) 216, backed by an energy reserve 218. The energy reserve 218 provides sufficient power to keep the DRAM 216 powered long enough to transfer the contents to the flash memory 206 in the event of a power failure. In some embodiments, the energy reserve 218 is a capacitor, supercapacitor, battery, or other device that supplies a suitable energy supply sufficient to enable the transfer of the contents of the DRAM 216 to a stable storage medium in the event of a power loss. The flash memory 206 is implemented as a plurality of flash memory dies 222, which may be referred to as a package of flash memory dies 222 or an array of flash memory dies 222. It should be appreciated that the flash memory dies 222 may be packaged in any number of ways, one die per package, multiple dies per package (i.e., a multi-chip package), a hybrid package, as a die on a printed circuit board or other substrate, as an encapsulated die, and the like. In the embodiment shown, the non-volatile solid-state storage device 152 has a controller 212 or other processor and an input-output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or the network interface controller 202 of the flash node 150. A flash input-output (I / O) port 220 is coupled to a flash die 222, and a direct memory access unit (DMA) 214 is coupled to the controller 212, the DRAM 216, and the flash die 222. In the embodiment shown, the I / O port 210, the controller 212, the DMA unit 214, and the flash I / O port 220 are implemented on a programmable logic device ('PLD') 208, such as an FPGA. In this embodiment, each flash die 222 has pages organized as 16kB (kilobyte) pages 224 and registers 226 through which data can be written to or read from the flash die 222. In other embodiments, other types of solid-state memory are used instead of or in addition to the flash memory illustrated within flash die 222 .

[0115] In various embodiments disclosed herein, a storage cluster 161 can be compared to a general storage array. Storage nodes 150 are part of a collection that creates a storage cluster 161. Each storage node 150 has a data slice and the computation required to provide the data. Multiple storage nodes 150 collaborate to store and retrieve data. The memory or storage device typically used in a storage array is less involved in processing and manipulating data. The memory or storage device in the storage array receives commands to read, write or erase data. The memory or storage device in the storage array does not know the larger system in which they are embedded, or what the data means. The memory or storage device in the storage array may include various types of memory, such as RAM, solid-state drives, hard disk drives, etc. The non-volatile solid-state storage device 152 unit described herein has multiple interfaces that are active at the same time and serve multiple purposes. In some embodiments, some functions of the storage node 150 are moved to the storage unit 152, thereby converting the storage unit 152 into a combination of the storage unit 152 and the storage node 150. Placing the computation (relative to storing data) in the storage unit 152 makes this computation closer to the data itself. Various system embodiments have a hierarchy of storage node tiers with different capabilities. In contrast, in a storage array, a controller owns and knows everything about all the data that the controller manages in a rack or storage device. In a storage cluster 161, as described herein, multiple controllers in multiple non-volatile sold-state storage devices 152 units and / or storage nodes 150 collaborate in various ways (e.g., for erasure coding, data sharding, metadata communication and redundancy, storage capacity expansion or contraction, data recovery, etc.).

[0116] Figure 2D Demonstrate storage server environment and its use Figure 2A 150 and storage device 152 units. In this version, each non-volatile solid-state storage device 152 unit is in the chassis 138 (see Figure 2A ) on a PCIe (Peripheral Component Interconnect Express) board having a processor (eg, controller 212 (see Figure 2C )), FPGA, flash memory 206 and NVRAM 204 (which is a supercapacitor-backed DRAM 216, see Figure 2B and 2C ). The non-volatile solid-state storage device 152 unit may be implemented as a single board containing the storage device and may be the maximum allowable failure domain within the chassis. In some embodiments, up to two non-volatile solid-state storage device 152 units may fail and the device will continue without data loss.

[0117] In some embodiments, physical storage is divided into named areas based on application usage. NVRAM 204 is a continuous block of reserved memory in non-volatile solid-state storage device 152 DRAM 216, and is supported by NAND flash memory. NVRAM 204 is logically divided into multiple memory areas (e.g., spool_region) for two writes as spools. The space within the NVRAM 204 spool is managed independently by each authority 168. Each device provides a certain amount of storage space to each authority 168. The authority 168 further manages the life and allocation within the space. Examples of spools include distributed transactions or concepts. When the main power of the non-volatile solid-state storage device 152 unit fails, the on-board super capacitor provides a short lasting power retention. During this retention interval, the content of NVRAM 204 is refreshed to flash memory 206. At the next power-on, the content of NVRAM 204 is restored from flash memory 206.

[0118] As with the storage unit controllers, the responsibilities of the logical "controller" are distributed across each blade containing the authority 168. This distribution of logical control is Figure 2D 1 and 2. Shown in FIG. 1 are a host controller 242, a mid-tier controller 244, and a storage unit controller 246. Management of the control plane and storage plane are handled independently, but the components can be physically co-located on the same blade. Each authority 168 effectively acts as an independent controller. Each authority 168 provides its own data and metadata structures, its own background workers, and maintains its own life cycle.

[0119] Figure 2E is Figure 2D Used in a storage server environment Figure 2A A hardware block diagram of a blade 252 of an embodiment of a storage node 150 and a storage unit 152 of FIG. 1 shows a control plane 254, a compute and storage plane 256, 258, and an authority 168 interacting with underlying physical resources. The control plane 254 is partitioned into several authorities 168, which can run on any blade 252 using the compute resources in the compute plane 256. The storage plane 258 is partitioned into a set of devices, each of which provides access to the flash memory 206 and NVRAM 204 resources. In one embodiment, the compute plane 256 can perform the operations of a storage array controller on one or more devices (e.g., a storage array) of the storage plane 258, as described herein.

[0120] exist Figure 2EIn the compute and storage planes 256, 258 of the system, authorities 168 interact with the underlying physical resources (i.e., devices). From the perspective of the authorities 168, their resources are striped across all physical devices. From the perspective of the devices, they provide resources to all authorities 168, regardless of where the authority happens to be running. Each authority 168 has or has been allocated one or more partitions 260 of memory in the storage unit 152, such as partitions 260 in flash memory 206 and NVRAM 204. Each authority 168 uses those allocated partitions 260 belonging to it for writing or reading user data. Authorities can be associated with different amounts of physical storage for the system. For example, one authority 168 may have a greater number of partitions 260 or larger-sized partitions 260 in one or more storage units 152 than one or more other authorities 168.

[0121] Figure 2F Depicts the elasticity software layer in the blade 252 of a storage cluster according to some embodiments. In the elastic architecture, the elasticity software is symmetrical, i.e., the computing module 270 of each blade runs Figure 2F The storage manager 274 executes read and write requests from other blades 252 for data and metadata stored in the local storage unit 152 NVRAM 204 and flash memory 206. The authority 168 satisfies client requests by issuing necessary reads and writes to the blade 252, and the corresponding data or metadata resides on the storage unit 152 of the blade 252. The endpoint 272 parses the client connection request received from the switch fabric 146 supervisory software, relays the client connection request to the authority 168 responsible for implementation, and relays the response of the authority 168 to the client. The symmetric three-tier structure enables a high degree of concurrency in the storage system. In these embodiments, elasticity scales horizontally efficiently and reliably. In addition, elasticity implements a unique horizontal expansion technology that can evenly balance work across all resources regardless of client access patterns and maximize concurrency by eliminating many of the requirements for inter-blade coordination that typically occurs with conventional distributed locking.

[0122] Still reference Figure 2F, the authorities 168 running in the compute modules 270 of the blades 252 perform the internal operations needed to satisfy client requests. One feature of resiliency is that the authorities 168 are stateless, i.e., they cache active data and metadata in their own blade 252 DRAM for fast access, but the authorities store each update in their NVRAM 204 partitions on three separate blades 252 until the update has been written to flash memory 206. In some embodiments, all storage system writes to NVRAM 204 are written in triplicate to partitions on three separate blades 252. With the triply mirrored NVRAM 204 and persistent storage protected by parity and Reed-Solomon RAID checksums, the storage system can survive the simultaneous failure of two blades 252 without losing data, metadata, or access to either.

[0123] Because authorities 168 are stateless, they can migrate between blades 252. Each authority 168 has a unique identifier. In some embodiments, NVRAM 204 and flash memory 206 partitions are associated with the identifier of the authority 168, rather than the blade 252 on which they are running. Therefore, when an authority 168 migrates, the authority 168 continues to manage the same storage partitions from its new location. When a new blade 252 is installed in an embodiment of a storage cluster, the system automatically rebalances the load by partitioning the storage of the new blade 252 for use by the system's authorities 168, migrating selected authorities 168 to the new blade 252, starting the endpoints 272 on the new blade 252, and including them in the client connection allocation algorithm of the switch fabric 146.

[0124] From their new locations, the migrated authorities 168 save the contents of their NVRAM 204 partitions on flash memory 206, process read and write requests from other authorities 168, and satisfy client requests directed to them by endpoints 272. Similarly, if a blade 252 fails or is removed, the system reallocates its authorities 168 among the remaining blades 252 in the system. The reallocated authorities 168 continue to perform their original functions from their new locations.

[0125] Figure 2GDepicts the authorities 168 and storage resources in the blades 252 of a storage cluster according to some embodiments. Each authority 168 is specifically responsible for a partition of flash memory 206 and NVRAM 204 on each blade 252. The authorities 168 manage the content and integrity of their partitions independently of other authorities 168. The authorities 168 compress incoming data and temporarily store it in their NVRAM 204 partitions, and then merge, RAID protect, and save the data in the storage segments in their flash memory 206 partitions. When the authority 168 writes data to the flash memory 206, the storage manager 274 performs the necessary flash conversions to optimize write performance and maximize media life. In the background, the authority 168 "junks up," or reclaims space occupied by data that has been discarded by clients by rewriting the data. It should be appreciated that because the partitions of the authorities 168 are disjoint, no distributed locking is required to execute clients and writes or to perform background functions.

[0126] The embodiments described herein may utilize various software, communication and / or network protocols. In addition, the configuration of hardware and / or software may be adjusted to accommodate various protocols. For example, embodiments may utilize Active Directory, which is a file system in Windows. TM A database-based system that provides authentication, directory, policy and other services in an environment. In these embodiments, LDAP (Lightweight Directory Access Protocol) is an example application protocol for querying and modifying items in a directory service provider such as Active Directory. In some embodiments, a network lock manager ('NLM') is used as a facility to work in conjunction with a network file system ('NFS') to provide System V-style advisory file and record locking on a network. The Server Message Block ('SMB') protocol (one version of which is also called the Common Internet File System ('CIFS')) can be integrated with the storage system discussed in this article. SMP operates as an application layer network protocol that is typically used to provide shared access to files, printers, and serial ports and a variety of communications between nodes on a network. SMB also provides an authenticated inter-process communication mechanism. AMAZON TMS3 (Simple Storage Service) is a web service provided by Amazon Web Services, and the system described herein can interface with Amazon S3 through web service interfaces (REST (Representational State Transfer), SOAP (Simple Object Access Protocol), and BitTorrent). The RESTful API (Application Programming Interface) breaks down transactions to create a series of small modules. Each module handles a specific underlying part of the transaction. The control or permissions provided by these embodiments (especially for object data) can include the use of access control lists ('ACLs'). An ACL is a list of permissions attached to an object, and the ACL specifies which users or system processes are granted access to an object, and which operations are allowed on a given object. The system can utilize Internet Protocol version 6 ('IPv6') as well as IPv4 as communication protocols, which provide identification and location systems for computers on a network and route traffic across the Internet. Packet routing between network systems can include equal cost multipath routing ('ECMP'), which is a routing strategy in which next-hop packet forwarding to a single destination can occur on multiple "best paths" that are tied for first place in the routing metric calculation. Multipath routing can be used in conjunction with most routing protocols because it is a hop-by-hop decision limited to a single router. The software can support multi-tenancy, which is an architecture in which a single instance of a software application serves multiple customers. Each customer may be referred to as a tenant. In some embodiments, tenants may be given the ability to customize parts of an application, but not the code of the application. An embodiment may maintain an audit log. An audit log is a document that records events in a computing system. In addition to recording which resources are accessed, an audit log entry typically includes destination and source addresses, timestamps, and user login information to comply with various regulations. An embodiment may support various key management strategies, such as encryption key rotation. In addition, the system may support some variations of dynamic root passwords or dynamically changing passwords.

[0127] Figure 3A A diagram illustrating a storage system 306 coupled for data communication with a cloud service provider 302 according to some embodiments of the present disclosure. Although not described in detail, Figure 3A The storage system 306 depicted in FIG. 3 may be similar to the storage system 306 described above with reference to FIG. Figures 1A to 1D and Figures 2A to 2G In some embodiments, Figure 3AThe storage system 306 depicted in the diagram may be embodied as a storage system including unbalanced active / active controllers, a storage system including balanced active / active controllers, a storage system including active / active controllers (in which less than all of the resources of each controller are utilized so that each controller has reserve resources available to support failover), a storage system including fully active / active controllers, a storage system including data set isolation controllers, a storage system including a two-tier architecture with a front-end controller and a back-end integrated storage controller, a storage system including a scale-out cluster of dual-controller arrays, and combinations of these embodiments.

[0128] exist Figure 3A In the example depicted in , the storage system 306 is coupled to the cloud service provider 302 via a data communication link 304. The data communication link 304 may be embodied as a dedicated data communication link, a data communication path provided by using one or a data communication network such as a wide area network ('WAN') or a LAN, or some other mechanism capable of conveying digital information between the storage system 306 and the cloud service provider 302. Such a data communication link 304 may be completely wired, completely wireless, or some aggregation of wired and wireless data communication paths. In such an example, one or more data communication protocols may be used to exchange digital information between the storage system 306 and the cloud service provider 302 via the data communication link 304. For example, the digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using the Handheld Device Transfer Protocol ('HDTP'), the Hypertext Transfer Protocol ('HTTP'), the Internet Protocol ('IP'), the Real-time Transport Protocol ('RTP'), the Transmission Control Protocol ('TCP'), the User Datagram Protocol ('UDP'), the Wireless Application Protocol ('WAP'), or other protocols.

[0129] Figure 3A The cloud service provider 302 depicted in FIG. 304 may embody a system and computing environment that provides a wide range of services to users of the cloud service provider 302, for example, by sharing computing resources via a data communication link 304. The cloud service provider 302 may provide on-demand access to a shared pool of configurable computing resources, such as computer networks, servers, storage devices, applications, and services. The shared pool of configurable resources may be quickly provisioned and released to users of the cloud service provider 302 with minimal management effort. Generally, users of the cloud service provider 302 are unaware of the exact computing resources utilized by the cloud service provider 302 to provide services. Although in many cases such a cloud service provider 302 may be accessible via the Internet, readers skilled in the art will recognize that any system that abstracts the use of shared resources to provide services to users via any data communication link may be considered a cloud service provider 302.

[0130] exist Figure 3A , cloud service provider 302 may be configured to provide various services to storage system 306 and users of storage system 306 by implementing various service models. For example, cloud service provider 302 may be configured to provide services by implementing an infrastructure as a service ('IaaS') service model, by implementing a platform as a service ('PaaS') service model, by implementing a software as a service ('SaaS') service model, by implementing an authentication as a service ('AaaS') service model, by implementing a storage as a service model (in which cloud service provider 302 provides access to its storage infrastructure for use by storage system 306 and users of storage system 306), etc. The reader will appreciate that cloud service provider 302 may be configured to provide additional services to storage system 306 and users of storage system 306 by implementing additional service models, as the service models described above are included for explanation purposes only and in no way represent a limitation on the services that may be provided by cloud service provider 302 or a limitation on the service models that may be implemented by cloud service provider 302.

[0131] exist Figure 3A In the example depicted in , cloud service provider 302 may be embodied as, for example, a private cloud, a public cloud, or a combination of private and public clouds. In embodiments where cloud service provider 302 is embodied as a private cloud, cloud service provider 302 may be dedicated to providing services to a single organization, rather than providing services to multiple organizations. In embodiments where cloud service provider 302 is embodied as a public cloud, cloud service provider 302 may provide services to multiple organizations. In other alternative embodiments, cloud service provider 302 may be embodied as a mix of private and public cloud services with a hybrid cloud deployment.

[0132] Although not in Figure 3A Although not explicitly depicted in , the reader will appreciate that a number of additional hardware components and additional software components may be necessary to facilitate the presentation of cloud services to the storage system 306 and users of the storage system 306. For example, the storage system 306 may be coupled to (or even include) a cloud storage gateway. Such a cloud storage gateway may be embodied as, for example, a hardware-based or software-based appliance located inside the storage system 306. Such a cloud storage gateway may operate as a bridge between local applications executing on the storage system 306 and remote cloud-based storage utilized by the storage system 306. By using a cloud storage gateway, an organization may move a primary iSCSI or NAS to the cloud service provider 302, thereby enabling the organization to save space on its internal storage system. Such a cloud storage gateway may be configured to emulate a disk array, a block-based device, a file server, or other storage system that may convert SCSI commands, file server commands, or other appropriate commands into a REST space protocol that facilitates communication with the cloud service provider 302.

[0133] In order to enable storage system 306 and users of storage system 306 to take advantage of services provided by cloud service provider 302, a cloud migration process may occur during which data, applications, or other elements from an organization's local systems (or even from another cloud environment) are moved to cloud service provider 302. In order to successfully migrate data, applications, or other elements to the environment of cloud service provider 302, middleware such as cloud migration tools may be utilized to bridge the gap between the environment of cloud service provider 302 and the environment of the organization. Such cloud migration tools may also be configured to address the potential high network costs and long transfer times associated with migrating large amounts of data to cloud service provider 302, as well as security issues associated with sensitive data passing through data communications networks to cloud service provider 302. In order to further enable storage system 306 and users of storage system 306 to take advantage of services provided by cloud service provider 302, cloud orchestrators may also be used to arrange and coordinate automated tasks in pursuit of creating a unified process or workflow. Such cloud orchestrators may perform tasks such as configuring various components (whether the components are cloud components or internal components) and managing the interconnections between these components. Cloud orchestrators may simplify inter-component communications and connections to ensure that links are properly configured and maintained.

[0134] exist Figure 3A In the example depicted in , and as briefly described above, the cloud service provider 302 may be configured to provide services to the storage system 306 and users of the storage system 306 by using a SaaS service model, thereby eliminating the need to install and run applications on local computers, which can simplify the maintenance and support of the applications. According to various embodiments of the present disclosure, such applications may take many forms. For example, the cloud service provider 302 may be configured to provide access to a data analysis application to the storage system 306 and users of the storage system 306. Such a data analysis application may be configured to, for example, receive a large amount of telemetry data called back by the storage system 306. Such telemetry data may describe various operating characteristics of the storage system 306, and may be analyzed for various purposes, including, for example, determining the health of the storage system 306, identifying workloads executed on the storage system 306, predicting when the storage system 306 will exhaust various resources, recommending configuration changes, hardware or software upgrades, workflow migrations, or other actions that may improve the operation of the storage system 306.

[0135] The cloud service provider 302 may also be configured to provide access to a virtualized computing environment to the storage system 306 and users of the storage system 306. Such a virtualized computing environment may be embodied, for example, as a virtual machine or other virtualized computer hardware platform, a virtual storage device, virtualized computer network resources, etc. Examples of such a virtualized environment may include a virtual machine created to emulate an actual computer, a virtualized desktop environment that separates a logical desktop from a physical machine, a virtualized file system that allows unified access to different types of specific file systems, and many others.

[0136] although Figure 3A The example depicted in FIG. 3 illustrates storage system 306 coupled for data communication with cloud service provider 302, but in other embodiments, storage system 306 may be part of a hybrid cloud deployment in which private cloud elements (e.g., private cloud services, on-premises infrastructure, etc.) and public cloud elements (e.g., public cloud services, infrastructure, etc. that may be provided by one or more cloud service providers) are combined to form a single solution, orchestrated among various platforms. Such a hybrid cloud deployment may utilize hybrid cloud management software, such as, for example, Microsoft Azure. TM Azure TM Arc, which centralizes the management of hybrid cloud deployments to any infrastructure and enables services to be deployed anywhere. In such an instance, the hybrid cloud management software can be configured to create, update, and delete resources (both physical and virtual) that form a hybrid cloud deployment, allocate compute and storage to specific workloads, monitor workloads and resources for performance, policy compliance, updates and patches, security status, or perform a variety of other tasks.

[0137] The reader will appreciate that by pairing the storage system described herein with one or more cloud service providers, a variety of offerings may be realized. For example, a Disaster Recovery as a Service ('DRaaS') may be provided, wherein cloud resources are utilized to protect applications and data from disruptions caused by disasters, including in embodiments where the storage system may be used as a primary data repository. In such embodiments, full system backups may be performed, which allows for business continuity in the event of a system failure. In such embodiments, cloud data backup technology (either by itself or as part of a larger DRaaS solution) may also be integrated into an overall solution including the storage system described herein and a cloud service provider.

[0138] The storage systems and cloud service providers described herein can be used to provide a wide range of security features. For example, the storage system can encrypt data at rest (and data can be sent to or from an encrypted storage system), and key management as a service ('KMaaS') can be utilized to manage encryption keys, keys for locking and unlocking storage devices, and the like. Likewise, a cloud data security gateway or similar mechanism can be utilized to ensure that data stored within the storage system does not inappropriately end up being stored in the cloud as part of a cloud data backup operation. In addition, micro-segmentation or identity-based segmentation can be used in the data center containing the storage system or within the cloud service provider to create secure zones in the data center and cloud deployments that isolate workloads from each other.

[0139] To explain further, Figure 3B A diagram illustrating a storage system 306 according to some embodiments of the present disclosure. Although not described in detail, Figure 3B The storage system 306 depicted in FIG. 3 may be similar to the storage system 306 described above with reference to FIG. Figures 1A to 1D and Figures 2A to 2G The storage system described above is described because the storage system may include many of the components described above.

[0140] Figure 3B The storage system 306 depicted in can include a number of storage resources 308, which can be embodied in a variety of forms. For example, the storage resources 308 can include nano-RAM or another form of non-volatile random access memory utilizing carbon nanotubes deposited on a substrate, 3D cross-point non-volatile memory, flash memory (including single-level cell ('SLC') NAND flash memory, multi-level cell ('MLC') NAND flash memory, triple-level cell ('TLC') NAND flash memory, quad-level cell ('QLC') NAND flash memory), or other. Likewise, the storage resources 308 can include non-volatile magnetoresistive random access memory ('MRAM'), including spin transfer torque ('STT') MRAM. The example storage resources 308 can alternatively include non-volatile phase change memory ('PCM'), quantum memory that allows storage and retrieval of photon quantum information, resistive random access memory ('ReRAM'), storage class memory ('SCM'), or other forms of storage resources, including any combination of the resources described herein. The reader will appreciate that other forms of computer memory and storage devices may be utilized by the memory system described above, including DRAM, SRAM, EEPROM, general purpose memory, and many others. Figure 3A The storage resources 308 depicted in may be embodied in a variety of form factors, including, but not limited to, dual in-line memory modules ('DIMMs'), non-volatile dual in-line memory modules ('NVDIMMs'), M.2, U.2, and others.

[0141] Figure 3BThe storage resources 308 depicted in FIG. 304 may include various forms of SCM. The SCM may effectively treat fast, non-volatile memory (e.g., NAND flash) as an extension of DRAM, so that the entire data set may be treated as an in-memory data set that resides entirely in DRAM. The SCM may include non-volatile media, such as, for example, NAND flash. Such NAND flash may be accessed using NVMe, which may use the PCIe bus as its transport, thereby providing relatively low access latency compared to older protocols. In fact, network protocols for SSDs in all-flash arrays may include NVMe (ROCE, NVME TCP) using Ethernet, Fibre Channel (NVMe FC), InfiniBand (iWARP), and others that may treat fast, non-volatile memory as an extension of DRAM. In view of the fact that DRAM is typically byte-addressable and fast, non-volatile memory (e.g., NAND flash) is block-addressable, a controller software / hardware stack may be required to convert block data into bytes stored in the media. Examples of media and software that may be used as SCM may include, for example, 3D XPoint, Intel Memory Drive Technology, Samsung's Z-SSD, and others.

[0142] Figure 3B The storage resource 308 depicted in FIG. 3 may also include racetrack memory (also known as magnetic domain wall memory). Such racetrack memory may be embodied in a form of non-volatile solid-state memory that relies on the inherent strength and orientation of the magnetic field generated by electrons in a solid-state device as they spin, in addition to their charge. By using a spin-coherent current to move magnetic domains along a nanoscale Permalloy wire, when current is passed through the wire, the magnetic domains may pass through a magnetic read / write head positioned near the wire, which alters the magnetic domains to record a pattern of bits. To fabricate a racetrack memory device, many such wires and read / write elements may be packaged together.

[0143] Figure 3B The example storage system 306 depicted in the example storage system 306 may implement various storage architectures. For example, a storage system according to some embodiments of the present disclosure may utilize block storage, where data is stored in blocks, and each block essentially acts as an individual hard drive. A storage system according to some embodiments of the present disclosure may utilize object storage, where data is managed as objects. Each object may include the data itself, a variable amount of metadata, and a globally unique identifier, where object storage may be implemented at multiple levels (e.g., device level, system level, interface level). A storage system according to some embodiments of the present disclosure utilizes file storage, where data is stored in a hierarchical structure. Such data may be saved in files and folders and presented in the same format to both the system storing it and the system retrieving it.

[0144] Figure 3BThe example storage system 306 depicted in FIG. 300 may be embodied as a storage system in which additional storage resources may be added by using a scale-up model, additional storage resources may be added by using a scale-out model, or by some combination thereof. In a scale-up model, additional storage may be added by adding additional storage devices. However, in a scale-out model, additional storage nodes may be added to a cluster of storage nodes, wherein such storage nodes may include additional processing resources, additional network resources, and the like.

[0145] Figure 3B The example storage system 306 depicted in FIG. 300 can utilize the storage resources described above in a variety of different ways. For example, some portions of the storage resources can be used to act as a write cache, storage resources within the storage system can be used as a read cache, or tiering can be implemented within the storage system by placing data within the storage system according to one or more tiering policies.

[0146] Figure 3B The storage system 306 depicted in also includes communication resources 310, which can be used to facilitate data communication between components within the storage system 306, and between the storage system 306 and computing devices external to the storage system 306, including embodiments where those resources are separated by relatively wide spaces. The communication resources 310 can be configured to facilitate data communication between components within the storage system and computing devices external to the storage system using a variety of different protocols and data communication structures. For example, the communication resources 310 can include Fibre Channel ('FC') technology, such as FC fabric and FC protocol that can convey SCSI commands over a FC network, FC over Ethernet ('FCoE') technology that encapsulates and transmits FC frames over Ethernet, InfiniBand ('IB') technology in which a switch fabric topology is utilized to facilitate transmission between channel adapters, NVM Express ('NVMe') technology, and NVMe over fabric ('NVMeoF') technology that can access non-volatile storage media attached via a PCI express ('PCIe') bus, and others. In fact, the storage systems described above may directly or indirectly utilize neutrino communication techniques and devices by which information (including binary information) is transmitted using neutrino beams.

[0147] The communication resources 310 may also include mechanisms for accessing storage resources 308 within the storage system 306 utilizing Serial Attached SCSI ('SAS'), a Serial ATA ('SATA') bus interface for connecting storage resources 308 within the storage system 306 to a host bus adapter within the storage system 306, Internet Small Computer System Interface ('iSCSI') technology for providing block-level access to storage resources 308 within the storage system 306, and other communication resources that may be used to facilitate data communications between components within the storage system 306 and data communications between the storage system 306 and computing devices external to the storage system 306.

[0148] Figure 3B The storage system 306 depicted in FIG. 3 also includes processing resources 312, which can be used to execute computer program instructions and perform other computing tasks within the storage system 306. The processing resources 312 can include one or more ASICs customized for some specific purpose and one or more CPUs. The processing resources 312 can also include one or more DSPs, one or more FPGAs, one or more systems on a chip ('SoC'), or other forms of processing resources 312. The storage system 306 can utilize the storage resources 312 to perform a variety of tasks, including but not limited to supporting the execution of software resources 314, which will be described in more detail below.

[0149] Figure 3B The storage system 306 depicted in FIG. 3 also includes software resources 314 that, when executed by processing resources 312 within the storage system 306, can perform a number of tasks. The software resources 314 can include, for example, one or more computer program instruction modules that, when executed by processing resources 312 within the storage system 306, can be used to implement various data protection techniques. Such data protection techniques can be implemented, for example, by system software executed on computer hardware within the storage system, by a cloud service provider, or in other ways. Such data protection techniques can include data archiving, data backup, data replication, data snapshots, data and database cloning, and other data protection techniques.

[0150] The software resources 314 may also include software that can be used to implement software defined storage ('SDS'). In such an example, the software resources 314 may include one or more computer program instruction modules that, when executed, can be used for policy-based data storage provisioning and management independent of the underlying hardware. Such software resources 314 can be used to implement storage virtualization to separate storage hardware from the software that manages the storage hardware.

[0151] The software resources 314 may also include software that can be used to facilitate and optimize I / O operations directed to the storage system 306. For example, the software resources 314 may include software modules that perform various data reduction techniques, such as, for example, data compression, data deduplication, and others. The software resources 314 may include software modules that intelligently group I / O operations together to facilitate better use of the underlying storage resources 308, software modules that perform data migration operations to migrate from within the storage system, and software modules that perform other functions. Such software resources 314 may be embodied as one or more software containers or in many other ways.

[0152] To explain further, Figure 3C An example of a cloud-based storage system 318 is described in accordance with some embodiments of the present disclosure. Figure 3C In the example depicted in FIG. 1 , the cloud-based storage system 318 is entirely within a cloud computing environment 316, such as, for example, Amazon Web Services ('AWS'). TM 、Microsoft Azure TM 、Google Cloud Platform TM 、IBM Cloud TM , Oracle Cloud TM The cloud-based storage system 318 may be used to provide services similar to those that may be provided by the storage systems described above.

[0153] Figure 3C 326. The cloud-based storage system 318 depicted in FIG. 31 includes two cloud computing instances 320, 322, each for supporting the execution of a storage controller application 324, 326. The cloud computing instances 320, 322 may be embodied as instances of cloud computing resources (e.g., virtual machines) that may be provided by the cloud computing environment 316 to support the execution of software applications such as the storage controller applications 324, 326. For example, each of the cloud computing instances 320, 322 may be executed on an Azure VM, wherein each Azure VM may include high-speed temporary storage that may be used as a cache (e.g., as a read cache). In one embodiment, the cloud computing instances 320, 322 may be embodied as Amazon Elastic Compute Cloud ('EC2') instances. In such an example, an Amazon Machine Image ('AMI') that includes the storage controller application 324, 326 may be launched to create and configure a virtual machine that may execute the storage controller application 324, 326.

[0154] exist Figure 3CIn the example method depicted in FIG. 1 , the storage controller applications 324, 326 may be embodied as computer program instruction modules that, when executed, perform various storage tasks. For example, the storage controller applications 324, 326 may be embodied as computer program instruction modules that, when executed, perform the same tasks as described above. Figure 1A The controllers 110A, 110B in the cloud-based storage system 318 perform the same tasks, such as writing data to the cloud-based storage system 318, erasing data from the cloud-based storage system 318, retrieving data from the cloud-based storage system 318, monitoring and reporting disk utilization and performance, performing redundancy operations (such as RAID or RAID-like data redundancy operations), compressing data, encrypting data, deduplicating data, etc. The reader will understand that because there are two cloud computing instances 320, 322, each of which includes a storage controller application 324, 326, in some embodiments, one cloud computing instance 320 can operate as the primary controller described above, while the other cloud computing instance 322 can operate as the secondary controller described above. The reader will understand that Figure 3C The storage controller applications 324, 326 depicted in FIG. 3 may include the same source code that executes within different cloud computing instances 320, 322 (eg, different EC2 instances).

[0155] The reader should appreciate that other embodiments that do not include primary and secondary controllers are also within the scope of the present disclosure. For example, each cloud computing instance 320, 322 may operate as a primary controller for some portion of the address space supported by the cloud-based storage system 318, each cloud computing instance 320, 322 may operate as a primary controller where the service of I / O operations directed to the cloud-based storage system 318 is divided in some other manner, etc. In fact, in other embodiments where cost savings may take precedence over performance requirements, there may be only a single cloud computing instance containing a storage controller application.

[0156] Figure 3C The cloud-based storage system 318 depicted in FIG. 1 includes cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338. The cloud computing instances 340a, 340b, 340n may embody, for example, instances of cloud computing resources that may be provided by the cloud computing environment 316 to support the execution of software applications. Figure 3C The cloud computing instances 340a, 340b, 340n may be different from the cloud computing instances 320, 322 described above because Figure 3CThe cloud computing instances 340a, 340b, 340n of the storage controller applications 324, 326 have local storage 330, 334, 338 resources, while the cloud computing instances 320, 322 supporting the execution of the storage controller applications 324, 326 do not need to have local storage resources. The cloud computing instances 340a, 340b, 340n with local storage 330, 334, 338 can be embodied as, for example, EC2 M5 instances including one or more SSDs, EC2 R5 instances including one or more SSDs, EC2 I3 instances including one or more SSDs, etc. In some embodiments, the local storage 330, 334, 338 must be embodied as a solid-state storage device (e.g., an SSD) rather than a storage device utilizing a hard disk drive.

[0157] exist Figure 3C In the example depicted in FIG. 1 , each of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338 may include a software daemon 328, 332, 336 that, when executed by the cloud computing instances 340a, 340b, 340n, may present itself to the storage controller application 324, 326 as if the cloud computing instance 340a, 340b, 340n were a physical storage device (e.g., one or more SSDs). In such an example, the software daemon 328, 332, 336 may include computer program instructions similar to those that would typically be contained on a storage device, such that the storage controller application 324, 326 may send and receive the same commands that a storage controller would send to the storage device. In this manner, the storage controller application 324, 326 may include code that is the same (or substantially the same) as the code that would be executed by the controller in the storage system described above. In these and similar embodiments, communication between the storage controller applications 324, 326 and cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338 may utilize iSCSI, NVMe over TCP, messaging, a custom protocol, or some other mechanism.

[0158] exist Figure 3C, each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 may also be coupled to a block storage 342, 344, 346 provided by the cloud computing environment 316, such as, for example, an Amazon Elastic Block Store ('EBS') volume. In such an example, the block storage 342, 344, 346 provided by the cloud computing environment 316 may be utilized in a manner similar to how the NVRAM devices described above are utilized, in that a software daemon 328, 332, 336 (or some other module) executing within a particular cloud computing instance 340a, 340b, 340n may initiate writing data to its attached EBS volume as well as writing data to its local storage 330, 334, 338 resource upon receiving a request to write data. In some alternative embodiments, data may be written only to local storage 330, 334, 338 resources within a particular cloud composite instance 340a, 340b, 340n. In an alternative embodiment, instead of using block storage 342, 344, 346 provided by the cloud computing environment 316 as NVRAM, actual RAM on each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 is used as NVRAM, thereby reducing the network utilization costs that would be associated with using EBS volumes as NVRAM. In yet another embodiment, high-performance block storage resources such as one or more Azure Ultra disks may be used as NVRAM.

[0159] The storage controller applications 324, 326 may be used to perform various tasks, such as deduplicating data contained in the request, compressing data contained in the request, determining where to write the data contained in the request, etc., before ultimately sending a deduplicated, encrypted, or otherwise potentially updated version of a request to write data to one or more of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338. In some embodiments, any of the cloud computing instances 320, 322 may receive a request to read data from the cloud-based storage system 318, and may ultimately send the request to read data to one or more of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338.

[0160] When a request to write data is received by a particular cloud computing instance 340a, 340b, 340n having a local storage 330, 334, 338, the software daemon 328, 332, 336 may be configured to write the data not only to its own local storage 330, 334, 338 resources and any appropriate block storage 342, 344, 346 resources, but the software daemon 328, 332, 336 may also be configured to write the data to a cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n. The cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n may be embodied as, for example, Amazon Simple Storage Service ('S3'). In other embodiments, cloud computing instances 320, 322, each including a storage controller application 324, 326, may initiate storage of data in local storage 330, 334, 338 and cloud-based object storage 348 of the cloud computing instances 340a, 340b, 340n. In other embodiments, instead of using both cloud computing instances 340a, 340b, 340n and cloud-based object storage 348 with local storage 330, 334, 338 (also referred to herein as "virtual drives") to store data, a persistent storage layer may be implemented in other ways. For example, one or more Azure Ultra disks may be used to persistently store data (e.g., after the data has been written to the NVRAM layer).

[0161] While the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n can support block-level access, the cloud-based object storage 348 attached to a particular cloud computing instance 340a, 340b, 340n only supports object-based access. Thus, the software daemons 328, 332, 336 can be configured to retrieve data blocks, package those data blocks into objects, and write the objects to the cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n.

[0162] Consider an example in which data is written to local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by cloud computing instances 340a, 340b, 340n in 1MB blocks. In such an example, assume that a user of the cloud-based storage system 318 issues a request to write data that results in the need to write 5MB of data after the data is compressed and deduplicated by the storage controller applications 324, 326. In such an example, writing data to the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n is relatively simple because 5 blocks of 1MB in size are written to the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n. In such an example, the software daemons 328, 332, 336 may also be configured to create five objects containing different 1MB chunks of data. Thus, in some embodiments, each object written to the cloud-based object store 348 may be identical (or nearly identical) in size. The reader will appreciate that in such an example, metadata associated with the data itself may be included in each object (e.g., the first 1MB of the object is the data, and the remainder is the metadata associated with the data). The reader will appreciate that the cloud-based object store 348 may be incorporated into the cloud-based storage system 318 to increase the persistence of the cloud-based storage system 318.

[0163] In some embodiments, all data stored by the cloud-based storage system 318 may be stored in both: 1) a cloud-based object store 348, and 2) at least one of a local storage 330, 334, 338 resource or a block storage 342, 344, 346 resource utilized by the cloud computing instances 340a, 340b, 340n. In such embodiments, the local storage 330, 334, 338 resource and the block storage 342, 344, 346 resource utilized by the cloud computing instances 340a, 340b, 340n may effectively operate as a cache that typically includes all data also stored in S3, so that all reads of data may be serviced by the cloud computing instances 340a, 340b, 340n without requiring the cloud computing instances 340a, 340b, 340n to access the cloud-based object store 348. However, the reader will appreciate that in other embodiments, all data stored by the cloud-based storage system 318 may be stored in the cloud-based object store 348, but less than all data stored by the cloud-based storage system 318 may be stored in at least one of the local storage 330, 334, 338 resources or block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n. In such instances, various policies may be utilized to determine which subset of data stored by the cloud-based storage system 318 should reside in both: 1) the cloud-based object store 348, and 2) at least one of the local storage 330, 334, 338 resources or block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n.

[0164] One or more computer program instruction modules executing within the cloud-based storage system 318 (e.g., a monitoring module executing on its own EC2 instance) can be designed to handle the failure of one or more of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338. In such an example, the monitoring module can handle the failure of one or more of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 by creating one or more new cloud computing instances having local storage, retrieving data stored on the failed cloud computing instances 340a, 340b, 340n from the cloud-based object storage 348, and storing the data retrieved from the cloud-based object storage 348 in the local storage on the newly created cloud computing instances. The reader will appreciate that many variations of this process can be implemented.

[0165] The reader should appreciate that various performance aspects of the cloud-based storage system 318 may be monitored (e.g., by a monitoring module executing in an EC2 instance) so that the cloud-based storage system 318 may be scaled up or scaled out as needed. For example, if the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 are undersized and are not adequately servicing I / O requests issued by users of the cloud-based storage system 318, the monitoring module may create a new, more powerful cloud computing instance (e.g., a type of cloud computing instance that includes more processing power, more memory, etc.) that includes the storage controller application so that the new, more powerful cloud computing instance may begin operating as the primary controller. Similarly, if the monitoring module determines that the cloud computing instances 320, 322 used to support the execution of the storage controller applications 324, 326 are oversized and cost savings may be obtained by switching to a smaller, weaker cloud computing instance, the monitoring module may create a new, weaker (and less expensive) cloud computing instance that includes the storage controller application so that the new, weaker cloud computing instance may begin operating as the primary controller.

[0166] The storage system described above may implement intelligent data backup techniques by which data stored in the storage system may be replicated and stored in different locations to avoid data loss in the event of an equipment failure or some other form of disaster. For example, the storage system described above may be configured to check each backup to avoid restoring the storage system to an undesirable state. Consider an example in which malware infects the storage system. In such an example, the storage system may include software resources 314 that may scan each backup to identify backups captured before the malware infected the storage system and those backups captured after the malware infected the storage system. In such an example, the storage system may restore itself from a backup that does not contain the malware, or at least not restore the portion of the backup that contains the malware. In such an instance, the storage system may include software resources 314 that may scan each backup to identify the presence of malware (or a virus, or something else undesirable), for example, by identifying write operations served by the storage system and originating from a network subnet suspected of having presented malware, by identifying write operations served by the storage system and originating from users suspected of having presented malware, by identifying write operations served by the storage system and checking the contents of the write operations against a fingerprint of the malware, and in many other ways.

[0167] The reader should further appreciate that backups (typically in the form of one or more snapshots) may also be used to perform rapid recovery of the storage system. Consider an example in which a storage system is infected with ransomware that locks users out of the storage system. In such an example, software resources 314 within the storage system may be configured to detect the presence of ransomware, and may be further configured to use retained backups to recover the storage system to a point in time prior to the point in time when the ransomware infected the storage system. In such an example, the presence of ransomware may be explicitly detected through the use of a software tool utilized by the system, through the use of a key (e.g., a USB drive) inserted into the storage system, or in a similar manner. Likewise, the presence of ransomware may be inferred in response to system activity satisfying a predetermined fingerprint, such as, for example, no reads or writes into the system for a predetermined period of time.

[0168] The reader will appreciate that the various components described above can be grouped into one or more optimized computing packages as a converged infrastructure. Such a converged infrastructure can include pools of computer, storage, and network resources that can be shared by multiple applications and managed in a collective manner using a policy-driven process. Such a converged infrastructure can be implemented using a converged infrastructure reference architecture, a standalone appliance, a software-driven hyper-convergence approach (e.g., a hyper-convergence infrastructure), or in other ways.

[0169] The reader will appreciate that the storage systems described in this disclosure can be used to support various types of software applications. In fact, the storage system can be "application-aware" in the sense that the storage system can obtain, maintain, or otherwise access information describing connected applications (e.g., applications that utilize the storage system) to optimize the operation of the storage system based on intelligence about the applications and their utilization patterns. For example, the storage system can optimize data layout, optimize cache behavior, optimize 'QoS' levels, or perform some other optimization designed to improve the storage performance experienced by the application.

[0170] As an example of one type of application that may be supported by the storage system described herein, storage system 306 may be used to support artificial intelligence ('AI') applications, database applications, XOps initiatives (e.g., DevOps initiatives, DataOps initiatives, MLOps initiatives, ModelOps initiatives, PlatformOps initiatives), electronic design automation tools, event-driven software applications, high-performance computing applications, simulation applications, high-speed data capture and analysis applications, machine learning applications, media production applications, media serving applications, picture archiving and communication system ('PACS') applications, software development applications, virtual reality applications, augmented reality applications, and many other types of applications by providing storage resources to such applications.

[0171] Given the fact that storage systems include computing resources, storage resources, and various other resources, storage systems may be well suited to support resource-intensive applications, such as, for example, AI applications. AI applications may be deployed in a variety of areas, including: predictive maintenance in manufacturing and related areas, healthcare applications (e.g., patient data and risk analysis), retail and marketing deployments (e.g., search advertising, social media advertising), supply chain solutions, fintech solutions (e.g., business analytics and reporting tools), operational deployments (e.g., real-time analytics tools), application performance management tools, IT infrastructure management tools, and many others.

[0172] Such AI applications can enable devices to perceive their environment and take actions that maximize their chances of success in achieving a certain goal. Examples of such AI applications can include IBM Watson TM 、Microsoft Oxford TM 、Google DeepMind TM 、Baidu Minwa TM and others.

[0173] The storage system described above may also be well suited for supporting other types of resource-intensive applications, such as, for example, machine learning applications. Machine learning applications can perform various types of data analysis to automate analytical model building. Using algorithms that iteratively learn from data, machine learning applications can enable computers to learn without being explicitly programmed. A specific area of ​​machine learning is called reinforcement learning, which involves taking appropriate actions in specific situations to maximize rewards.

[0174] In addition to the resources already described, the storage system described above may also include a graphics processing unit ('GPU'), occasionally referred to as a visual processing unit ('VPU'). Such a GPU may be embodied as a specialized electronic circuit that rapidly manipulates and alters memory to accelerate the creation of images in a frame buffer for output to a display device. Such a GPU may be included within any computing device that is part of the storage system described above, including as one of many individually scalable components of the storage system, where other examples of individually scalable components of such a storage system may include storage components, memory components, computing components (e.g., CPUs, FPGAs, ASICs), network components, software components, and others. In addition to the GPU, the storage system described above may also include a neural network processor ('NNP') for various aspects of neural network processing. Such an NNP may be used in place of (or in addition to) a GPU, and may also be independently scalable.

[0175] As described above, the storage system described herein can be configured to support artificial intelligence applications, machine learning applications, big data analysis applications, and many other types of applications. The rapid growth of such applications is driven by three technologies: deep learning (DL), GPU processors, and big data. Deep learning is a computing model that uses massively parallel neural networks inspired by the human brain. Deep learning models write their own software by learning from a large number of examples, rather than experts handcrafting software. Such GPUs can include thousands of cores that are well suited to running algorithms that loosely represent the parallel nature of the human brain.

[0176] Advances in deep neural networks, including the development of multi-layer neural networks, have sparked a new wave of algorithms and tools for data scientists to exploit their data with artificial intelligence (AI). With improved algorithms, larger datasets, and a variety of frameworks, including open source software libraries for machine learning across a range of tasks, data scientists are tackling new use cases such as self-driving cars, natural language processing and understanding, computer vision, machine reasoning, strong AI, and many others. Applications of this technology can include: machine and vehicle object detection, recognition, and avoidance; visual identification, classification, and tagging; algorithmic financial trading strategy performance management; simultaneous localization and mapping; predictive maintenance of high-value equipment; protection against cybersecurity threats, automation of expertise; image recognition and classification; question answering; robotics; text analysis (extraction, classification) and text generation and translation; and many others. Applications of AI technology have been implemented in a wide range of products, including, for example: Amazon Echo's voice recognition technology, which allows users to talk to their machines; Google Translate TM , which allows machine-based language translation; Spotify’s Discover Weekly, which provides recommendations for new songs and artists that users may like based on their usage and traffic analytics; Quill’s text generation product, which takes structured data and turns it into narrative stories; chatbots, which provide real-time, context-specific answers to questions in a conversational format; and many others.

[0177] Data is at the heart of modern AI and deep learning algorithms. Before training can begin, one problem that must be solved centers around collecting labeled data, which is critical to training accurate AI models. A full AI deployment may require the continuous collection, cleaning, transformation, labeling, and storage of large amounts of data. Adding additional high-quality data points directly translates into more accurate models and better insights. Data samples may go through a series of processing steps, including, but not limited to: 1) ingesting data from external sources into the training system and storing the data in raw form, 2) cleaning and transforming the data in a format that is convenient for training, including linking data samples to appropriate labels, 3) exploring parameters and models, testing quickly with smaller datasets, and iterating to converge on the most promising models to push into production clusters, 4) performing a training phase to select random batches of input data, including both new and old samples, and feeding those into production GPU servers for computation to update model parameters, and 5) evaluation, including using a held-out portion of the data not used in training, in order to evaluate the model accuracy on the held-out data. This life cycle can apply to any type of parallelized machine learning, not just neural networks or deep learning. For example, a standard machine learning framework may rely on CPUs instead of GPUs, but the data ingestion and training workflows may be the same. Readers will appreciate that a single shared storage data center creates a coordination point throughout the lifecycle without the need for additional copies of data during the ingestion, preprocessing, and training phases. Ingested data is rarely used for just one purpose, and shared storage gives the flexibility to train multiple different models or apply traditional analytics to the data.

[0178] The reader will appreciate that each stage in the AI ​​data pipeline may have different requirements for a data center (e.g., a storage system or a collection of storage systems). A scale-out storage system must deliver uncompromising performance for a variety of access types and patterns, from small, metadata-heavy files to large files, from random access patterns to sequential access patterns, and from low concurrency to high concurrency. The storage system described above can serve as an ideal AI data center because the system can serve unstructured workloads. In the first stage, the data is ideally ingested and stored on the same data center that will be used in subsequent stages to avoid excessive data replication. The next two steps can be completed on a standard computing server that optionally includes a GPU, and then in the fourth and final stage, the full training production job is run on a powerful GPU-accelerated server. Typically, there is a production pipeline next to the experimental pipeline operating on the same dataset. In addition, GPU-accelerated servers can be used independently for different models, or can be combined to train on a larger model, or even distributed training across multiple systems. If the shared storage layer is slow, each stage must copy the data to the local storage device, resulting in wasted time temporarily storing the data on different servers. The ideal data center for an AI training pipeline delivers performance similar to data stored locally on server nodes, while also having the simplicity and performance to enable all pipeline stages to operate simultaneously.

[0179] In order for the storage system described above to be used as a data center or as part of an AI deployment, in some embodiments, the storage system may be configured to provide DMA between storage devices contained in the storage system and one or more GPUs used in an AI or big data analytics pipeline. One or more GPUs may be coupled to the storage system, for example, via NVMe over Fabric ('NVMe-oF'), so that bottlenecks such as the host CPU can be bypassed and the storage system (or a component contained therein) can directly access the GPU memory. In such an example, the storage system may utilize an API hook to the GPU to transfer data directly to the GPU. For example, the GPU may be embodied as an Nvidia TM The GPU and the storage system may support GPUDirect Storage ('GDS') software, or have similar proprietary software, which enables the storage system to transfer data to the GPU via RDMA or similar mechanisms.

[0180] Although the previous paragraphs discuss deep learning applications, the reader will appreciate that the storage system described herein may also be part of a distributed deep learning ('DDL') platform to support the execution of DDL algorithms. The storage system described above may also be paired with other technologies such as TensorFlow, an open source software library for dataflow programming across a range of tasks that may be used in machine learning applications such as neural networks to facilitate the development of such machine learning models, applications, and the like.

[0181] The storage system described above can also be used in a neuromorphic computing environment. Neuromorphic computing is a form of computing that mimics brain cells. To support neuromorphic computing, the architecture of interconnected "neurons" replaces the traditional computing model with low-power signals that pass directly between neurons to achieve more efficient computing. Neuromorphic computing can utilize very large scale integration (VLSI) systems containing electronic analog circuits to mimic the neurobiological architecture present in the nervous system, as well as analog, digital, mixed-mode analog / digital VLSI and software systems that implement models of nervous systems for perception, motor control or multi-sensory integration.

[0182] The reader will appreciate that the storage system described above may be configured to support the storage or use of (among other types of data) blockchain and derivative projects, such as, for example, open source blockchains and IBM TM Related tools that are part of the Hyperledger project, permissioned blockchains that allow a certain number of trusted parties to access the blockchain, blockchain products that enable developers to build their own distributed ledger projects, and others. The blockchain and storage system described in this article can be used to support on-chain storage of data as well as off-chain storage of data.

[0183] Off-chain storage of data can be implemented in various ways, and can occur when the data itself is not stored in the blockchain. For example, in one embodiment, a hash function can be utilized, and the data itself can be fed into the hash function to generate a hash value. In such an instance, a hash of a large amount of data can be embedded in a transaction, rather than the data itself. The reader will appreciate that in other embodiments, alternatives to blockchain can be used to facilitate decentralized storage of information. For example, one alternative to blockchain that can be used is blockweave. While conventional blockchains store each transaction for verification, blockweaves allow for secure decentralization without using the entire chain, thereby enabling low-cost on-chain storage of data. Such blockweaves can utilize a consensus mechanism based on proof of access (PoA) and proof of work (PoW).

[0184] The storage systems described above may be used alone or in combination with other computing devices to support in-memory computing applications. In-memory computing involves storing information in RAM distributed across a cluster of computers. The reader will appreciate that the storage systems described above, particularly those that are configurable with customizable amounts of processing resources, storage resources, and memory resources (e.g., those systems in which blades containing configurable amounts of each type of resource) may be configured in a manner that can provide an infrastructure that supports in-memory computing. Likewise, the storage systems described above may include component portions (e.g., NVDIMMs, 3D crosspoint storage devices that provide persistent, fast random access memory) that may actually provide an improved in-memory computing environment compared to in-memory computing environments that rely on RAM distributed across dedicated servers.

[0185] In some embodiments, the storage system described above may be configured to operate as a hybrid in-memory computing environment that includes a common interface to all storage media (e.g., RAM, flash memory, 3D cross-point storage devices). In such an embodiment, users may not know the details about where their data is stored, but they can still use the same complete, unified API to address the data. In such an embodiment, the storage system may move the data (in the background) to the fastest tier available, including intelligently placing the data depending on various characteristics of the data or depending on some other heuristic method. In such an instance, the storage system may even utilize existing products such as Apache Ignite and GridGain to move data between various storage tiers, or the storage system may utilize custom software to move data between various storage tiers. The storage system described herein may implement various optimizations to improve the performance of in-memory computing, such as, for example, making the computing occur as close to the data as possible.

[0186] The reader should further understand that in some embodiments, the storage system described above can be paired with other resources to support the applications described above. For example, an infrastructure may include primary computing in the form of servers and workstations that specifically use general-purpose computing on graphics processing units ('GPGPU') to accelerate deep learning applications that are interconnected into computing engines to train parameters of deep neural networks. Each system may have an Ethernet external connection, an infinite bandwidth external connection, some other form of external connection, or some combination thereof. In such an instance, GPUs can be grouped for a single large training, or used independently to train multiple models. The infrastructure may also include a storage system such as the one described above to provide, for example, horizontally scaled all-flash file or object storage, through which data can be accessed via high-performance protocols such as NFS, S3, etc. The infrastructure may also include redundant top-of-rack Ethernet switches connected to storage devices and computing, such as via ports in an MLAG port channel, to achieve redundancy. The infrastructure may also include additional computing in the form of white box servers, optionally with GPUs, for data ingestion, preprocessing, and model debugging. The reader will understand that additional infrastructure is also possible.

[0187] The reader will appreciate that the storage system described above (whether alone or in coordination with other computing devices) can be configured to support other AI-related tools. For example, the storage system can utilize tools such as ONXX or other open neural network exchange formats, which make it easier to transfer models written in different AI frameworks. Likewise, the storage system can be configured to support tools such as Amazon's Gluon, which allows developers to prototype, build, and train deep learning models. In fact, the storage system described above can be part of a larger platform, such as IBM's TM A private data cloud that includes integrated data science, data engineering, and application building services.

[0188] The reader should further appreciate that the storage system described above may also be deployed as an edge solution. Such an edge solution may be in place to optimize cloud computing systems by performing data processing at the edge of the network close to the data source. Edge computing can push applications, data, and computing power (i.e., services) away from a centralized point to the logical extremes of the network. By using an edge solution such as the storage system described above, computing tasks may be performed using computing resources provided by such a storage system, data may be stored using the storage resources of the storage system, and cloud-based services may be accessed by using various resources of the storage system (including network resources). By performing computing tasks on an edge solution, storing data on an edge solution, and generally utilizing an edge solution, the consumption of expensive cloud-based resources may be avoided, and in fact, performance improvements may be experienced relative to a greater reliance on cloud-based resources.

[0189] While many tasks can benefit from the utilization of edge solutions, certain specific uses may be particularly well suited for deployment in such environments. For example, devices such as drones, self-driving cars, robots, and others may require extremely fast processing, so fast, in fact, that sending data up to a cloud environment and back to receive data processing support may simply be too slow. As an additional example, some IoT devices (such as connected cameras) may not be well suited to utilizing cloud-based resources because sending data to the cloud may not be practical (not just from a privacy perspective, a security perspective, or a financial perspective) simply because of the sheer amount of data involved. Therefore, many tasks that are truly about data processing, storage, or communication may be better suited for platforms that include edge solutions (such as the storage systems described above).

[0190] The storage system described above can be used alone or in combination with other computing resources as a network edge platform that combines computing resources, storage resources, network resources, cloud technologies, network virtualization technologies, etc. As part of the network, the edge can present characteristics similar to other network facilities, from customer premises and backhaul aggregation facilities to points of presence (PoP) and regional data centers. The reader will understand that network workloads (such as virtual network functions (VNFs) and others) will reside on the network edge platform. Through the combination of containers and virtual machines, the network edge platform can rely on controllers and schedulers that are no longer geographically co-located with data processing resources. As microservices, the functions can be split into control planes, user and data planes, or even state machines, allowing independent optimization and expansion techniques to be applied. Such user and data planes can be implemented by adding accelerators (both of which reside in server platforms, such as FPGAs and smart NICs) and commercial chips and programmable ASICs enabled by SDN.

[0191] The storage systems described above can also be optimized for use in big data analytics, including utilization as part of a composable data analytics pipeline, where a containerized analytics architecture, for example, makes analytics capabilities more composable. Big data analytics can generally be described as the process of examining large and diverse data sets to uncover hidden patterns, unknown correlations, market trends, customer preferences, and other useful information that can help organizations make smarter business decisions. As part of the process, semi-structured and unstructured data (such as, for example, Internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone call detail records, IoT sensor data, and other data) can be converted into a structured form.

[0192] The storage system described above may also support (including being implemented as a system interface) applications that perform tasks in response to human voice. For example, the storage system may support the execution of an intelligent personal assistant application, such as, for example, Amazon's Alexa TM , Apple Siri TM , Google Voice TM 、Samsung Bixby TM , Microsoft Cortana TM and others. Although the example described in the previous sentence utilizes voice as input, the storage system described above may also support chatbots, conversational robots, conversational robots, or artificial dialogue entities or other applications configured to conduct conversations via auditory or textual methods. Likewise, the storage system may actually execute such applications to enable users, such as system administrators, to interact with the storage system via voice. Such applications are typically capable of voice interaction, music playback, making to-do lists, setting alarms, streaming podcasts, playing audiobooks, and providing weather, traffic, and other real-time information (e.g., news), but in embodiments according to the present disclosure, such applications may be used as an interface for various system management operations.

[0193] The storage system described above may also implement an AI platform for fulfilling the vision of self-driving storage devices. Such an AI platform may be configured to present global predictive intelligence by collecting and analyzing a large number of storage system telemetry data points for easy management, analysis, and support. In fact, such a storage system may be able to predict both capacity and performance, as well as generate intelligent recommendations on workload deployment, interaction, and optimization. Such an AI platform may be configured to scan all incoming storage system telemetry data against an outgoing fingerprint library to predict and resolve events in real time before they impact the customer environment, and capture hundreds of performance-related variables for predicting performance loads.

[0194] The storage system described above may support serial or concurrent execution of artificial intelligence applications, machine learning applications, data analytics applications, data transformations, and other tasks that may collectively form an AI ladder. Such an AI ladder may be effectively formed by combining these elements to form a complete data science pipeline, where there are dependencies between the elements of the AI ​​ladder. For example, AI may require some form of machine learning to have occurred, machine learning may require some form of analysis to have occurred, analysis may require some form of data and information architecture to have occurred, and so on. Thus, each element may be viewed as a rung in the AI ​​ladder, which may collectively form a complete and complex AI solution.

[0195] The storage system described above can also be used alone or in combination with other computing environments to deliver an AI-everywhere experience, where AI permeates a wide range of aspects of business and life. For example, AI can play an important role in delivering deep learning solutions, deep reinforcement learning solutions, artificial general intelligence solutions, self-driving cars, cognitive computing solutions, commercial UAVs or drones, conversational user interfaces, enterprise taxonomies, ontology management solutions, machine learning solutions, smart dust, smart robots, smart workplaces, and many other aspects.

[0196] The storage system described above can also be used alone or in combination with other computing environments to present a wide range of transparent immersive experiences (including those using digital twins of various "things" (such as people, places, processes, systems, etc.), where the technology can introduce transparency between people, enterprises, and things. Such transparent immersive experiences can be presented as augmented reality technology, connected home, virtual reality technology, brain-computer interface, human augmentation technology, nanotube electronics, volumetric displays, 4D printing technology, or other.

[0197] The storage system described above can also be used alone or in combination with other computing environments to support various digital platforms. Such digital platforms may include, for example, 5G wireless systems and platforms, digital twin platforms, edge computing platforms, IoT platforms, quantum computing platforms, serverless PaaS, software-defined security, neuromorphic computing platforms, etc.

[0198] The storage system described above may also be part of a multi-cloud environment, where multiple cloud computing and storage services are deployed in a single heterogeneous architecture. To facilitate the operation of such a multi-cloud environment, DevOps tools may be deployed to enable orchestration across clouds. Similarly, continuous development and continuous integration tools may be deployed to standardize processes around continuous integration and delivery, new feature rollouts, and provisioning of cloud workloads. By standardizing these processes, a multi-cloud strategy may be implemented that enables the utilization of the best provider for each workload.

[0199] The storage system described above can be used as part of a platform to enable the use of cryptographic anchors that can be used to authenticate the origin and content of a product to ensure that it matches the blockchain record associated with the product. Similarly, as part of a toolkit to protect data stored on the storage system, the storage system described above can implement various cryptographic techniques and schemes, including lattice cryptography. Lattice cryptography can involve cryptographic primitive constructions that involve lattices in the construction itself or in security proofs. Unlike public key schemes such as RSA, Diffie-Hellman, or elliptic curve cryptography, which are vulnerable to quantum computer attacks, some lattice-based constructions appear to be resistant to attacks by both classical and quantum computers.

[0200] A quantum computer is a device that performs quantum computing. Quantum computing is computing that uses quantum mechanical phenomena, such as superposition and entanglement. Quantum computers are different from traditional computers based on transistors because such traditional computers need to encode data as binary digits (bits), each of which is always in one of two finite states (0 or 1). Compared to traditional computers, quantum computers use qubits, which can be in a superposition of states. A quantum computer maintains a series of qubits, where a single qubit can represent 1, 0, or any quantum superposition of the states of those two qubits. A pair of qubits can be in any quantum superposition of 4 states, and three qubits can be in any superposition of 8 states. A quantum computer with n qubits can typically be in any superposition of up to 2^n different states at the same time, while a traditional computer can only be in one of these states at any time. A quantum Turing machine is a theoretical model of such a computer.

[0201] The storage system described above may also be paired with an FPGA-accelerated server as part of a larger AI or ML infrastructure. Such an FPGA-accelerated server may reside near the storage system described above (e.g., in the same data center), or even incorporated into an appliance comprising one or more storage systems, one or more FPGA-accelerated servers, a network infrastructure supporting communication between one or more storage systems and one or more FPGA-accelerated servers, and other hardware and software components. Alternatively, the FPGA-accelerated server may reside in a cloud computing environment that can be used to perform computationally related tasks for AI and ML jobs. Any of the embodiments described above may be used together as an FPGA-based AI or ML platform. The reader will appreciate that in some embodiments of an FPGA-based AI or ML platform, the FPGA contained in the FPGA-accelerated server may be reconfigured for different types of ML models (e.g., LSTM, CNN, GRU). The ability to reconfigure the FPGA contained in the FPGA-accelerated server can achieve acceleration of ML or AI applications based on optimal numerical precision and the memory model used. The reader will appreciate that by treating a collection of FPGA-accelerated servers as a pool of FPGAs, any CPU in a data center can use the pool of FPGAs as a shared hardware microservice, rather than limiting servers to the dedicated accelerators plugged into them.

[0202] The FPGA-accelerated servers and GPU-accelerated servers described above can implement a computing model where, instead of keeping a small amount of data in the CPU and running a long instruction stream on it as in a more traditional computing model, the machine learning model and parameters are fixed to a high-bandwidth on-chip memory through which a large amount of data flows. For such a computing model, an FPGA may even be more efficient than a GPU because the FPGA can be programmed with only the instructions required to run such a computing model.

[0203] The storage system described above can be configured to provide parallel storage, for example, by using a parallel file system such as BeeGFS. Such a parallel file system can include a distributed metadata architecture. For example, a parallel file system can include multiple metadata servers across which metadata is distributed, and components including services for clients and storage servers.

[0204] The system described above can support the execution of a wide range of software applications. Such software applications can be deployed in various ways, including container-based deployment models. Various tools can be used to manage containerized applications. For example, Docker Swarm, Kubernetes, and others can be used to manage containerized applications. Containerized applications can be used to facilitate serverless, cloud-native computing deployment and management models for software applications. In order to support serverless, cloud-native computing deployment and management models for software applications, containers can be used as part of an event handling mechanism (e.g., AWS Lambdas) so that various events cause the containerized application to be launched to operate as an event handler.

[0205] The systems described above may be deployed in a variety of ways, including in a manner to support fifth generation ('5G') networks. 5G networks may support substantially faster data communications than previous generations of mobile communications networks, and therefore may lead to the disaggregation of data and computing resources, as modern large-scale data centers may become less prominent and may be replaced by more local micro data centers, such as near mobile network towers. The systems described above may be included in such local micro data centers, and may be part of or paired with a multi-access edge computing ('MEC') system. Such a MEC system may enable cloud computing capabilities and IT service environments at the edge of a cellular network. By running applications and performing related processing tasks closer to cellular clients, network congestion may be reduced, and applications may perform better.

[0206] The storage system described above may also be configured to implement NVMe partitioned namespaces. By using NVMe partitioned namespaces, the logical address space of the namespace is divided into multiple regions. Each region provides a logical block address range that must be written sequentially and explicitly reset before being rewritten, thereby creating a namespace that exposes the natural boundaries of the device and offloading the management of internal mapping tables to the host. To implement NVMe partitioned namespaces ('ZNS'), a ZNS SSD or some other form of partitioned block device that uses zones to expose the namespace logical address space may be utilized. With the zones aligned with the internal physical properties of the device, several inefficiencies in data placement can be eliminated. In such an embodiment, each region may be mapped to, for example, a separate application so that functions such as wear leveling and garbage collection can be performed on a region-by-region or application-by-application basis rather than across the entire device. To support ZNS, the storage controller described herein may be configured to use, for example, Linux TM Kernel partition block device interface or other tools to interact with partition block devices.

[0207] The storage system described above may also be configured to implement zoned storage in other ways, such as, for example, by using shingled magnetic recording (SMR) storage devices. In instances where zoned storage is used, a device-managed embodiment may be deployed, where the storage device hides this complexity by managing it in firmware, presenting an interface like any other storage device. Alternatively, zoned storage may be implemented via a host-managed embodiment, which relies on the operating system to know how to handle the drive, and only writes to certain areas of the drive in sequence. Zoned storage may similarly be implemented using a host-aware embodiment, where a combination of drive-managed and host-managed implementations are deployed.

[0208] The storage system described herein can be used to form a data lake. The data lake can operate as the first location to which an organization's data flows, where the data can be in a raw format. Metadata tags can be implemented to facilitate searches of data elements in the data lake, particularly in embodiments where the data lake contains multiple data stores (e.g., unstructured data, semi-structured data, structured data) in formats that are not easily accessed or read. From the data lake, data can flow downstream into a data warehouse, where the data can be stored in a format that is deeply processed, packaged, and consumed. The storage system described above can also be used to implement such a data warehouse. In addition, a data mart or data center can allow for more easily consumed data, where the storage system described above can also be used to provide the underlying storage resources required by the data mart or data center. In an embodiment, queries to the data lake may require a read-time schema approach, where the data is applied to a plan or schema when it is pulled from a storage location rather than when it enters the storage location.

[0209] The storage systems described herein may also be configured to implement a recovery point objective ('RPO'), which may be established by a user, established by an administrator, established as a system default, established as part of a storage class or service that the storage system is participating in rendering, or established in some other manner. A "recovery point objective" is a target for the maximum time difference between the last update of a source data set and the last recoverable update of a replicated data set that will correctly recover from a continuously or frequently updated copy of the source data set if there is reason to do so. Updates can be correctly recovered if all updates processed on the source data set prior to the last recoverable update of the replicated data set are properly accounted for.

[0210] In synchronous replication, the RPO will be zero, meaning that under normal operation, all updates done on the source data set should be present and recoverable correctly on the replicated data set. In replication that strives to be as close to synchronous as possible, the RPO can be as low as a few seconds. In snapshot-based replication, the RPO can be roughly calculated as the time interval between snapshots plus the time to transfer modifications between the previously transferred snapshot and the most recent snapshot to be replicated.

[0211] If updates accumulate faster than they can be replicated, then the RPO can be missed. For snapshot-based replication, if the amount of data to be replicated accumulates between two snapshots is greater than can be replicated between taking a snapshot and replicating the accumulated updates to that snapshot to the replica, then the RPO can be missed. Again, in snapshot-based replication, if the data to be replicated accumulates faster than can be transferred in the time between subsequent snapshots, then replication can begin to fall further behind, which can extend the missed time between the expected recovery point objective and the actual recovery point represented by the last correctly replicated update.

[0212] The storage systems described above may also be part of a shared-nothing storage cluster. In a shared-nothing storage cluster, each node of the cluster has local storage and communicates with other nodes in the cluster over a network, where the storage used by the cluster is (generally speaking) provided only by the storage connected to each individual node. A collection of nodes that synchronously replicate data sets may be an example of a shared-nothing storage cluster because each storage system has local storage and communicates with other storage systems over a network, where those storage systems (generally speaking) do not use storage from elsewhere that they have shared access to through some interconnect. In contrast, some of the storage systems described above are themselves built as shared storage clusters because there are drive racks that are shared by paired controllers. However, other storage systems described above are built as shared-nothing storage clusters because all storage is local to a particular node (e.g., a blade) and all communications are through a network that links the compute nodes together.

[0213] In other embodiments, other forms of shared-nothing storage clusters may include embodiments where any node in the cluster has a local copy of all storage devices they need, and where data is mirrored to other nodes in the cluster via synchronous replication to ensure that data is not lost, or because other nodes are also using the storage devices. In such embodiments, if a new cluster node needs some data, the data can be copied to the new node from other nodes that have copies of the data.

[0214] In some embodiments, a mirror copy based shared storage cluster may store multiple copies of all storage data of the cluster, wherein each subset of data is copied to a specific set of nodes, and different subsets of data are copied to different sets of nodes. In some variations, embodiments may store all storage data of the cluster in all nodes, while in other variations, the nodes may be partitioned so that a first set of nodes will all store the same data set, and a second, different set of nodes will all store different data sets.

[0215] The reader will appreciate that a RAFT-based database (such as etcd) can operate like a shared-nothing storage cluster, where all RAFT nodes store all data. However, the amount of data stored in a RAFT cluster can be limited so that additional copies do not consume too much storage. Assuming that containers do not tend to be too large, and their bulk data (data manipulated by applications running in containers) is stored elsewhere, such as in an S3 cluster or an external file server, then a cluster of container servers may also be able to replicate all data to all cluster nodes. In such an instance, container storage can be provided directly by the cluster through its shared-nothing storage model, where those containers provide images of the execution environments that form part of an application or service.

[0216] To explain further, Figure 3D An exemplary computing device 350 is illustrated that may be specifically configured to perform one or more of the processes described herein. Figure 3D As shown in FIG. 3 , computing device 350 may include a communication interface 352, a processor 354, a storage device 356, and an input / output ('I / O') module 358 communicatively coupled to one another via a communication infrastructure 360. Figure 3D An exemplary computing device 350 is shown in FIG. Figure 3D The components described in the embodiment are not intended to be limiting. Additional or alternative components may be used in other embodiments. Figure 3D Components of computing device 350 are shown in FIG.

[0217] Communication interface 352 may be configured to communicate with one or more computing devices. Examples of communication interface 352 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0218] The processor 354 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the performance of one or more of the instructions, processes, and / or operations described herein. The processor 354 may perform operations by executing computer-executable instructions 362 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 356.

[0219] Storage device 356 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, storage device 356 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including data described herein) may be temporarily and / or permanently stored in storage device 356. For example, data representing computer executable instructions 362 configured to direct processor 354 to perform any of the operations described herein may be stored within storage device 356. In some examples, the data may be arranged in one or more databases residing within storage device 356.

[0220] The I / O module 358 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 358 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 358 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0221] The I / O module 358 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 358 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation. In some examples, any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 350.

[0222] To explain further, Figure 3EAn example of a storage system cluster 376 for providing storage services (also referred to herein as 'data services') is illustrated. The storage system cluster 376 depicted in FIG. 3 includes a plurality of storage systems 374a, 374b, 374c, 374d, 374n, each of which may be similar to the storage systems described herein. The storage systems 374a, 374b, 374c, 374d, 374n in the storage system cluster 376 may be embodied as the same storage system or different types of storage systems. For example, Figure 3E The two storage systems 374a, 374n depicted in FIG. 3 are depicted as cloud-based storage systems because the resources that together form each of the storage systems 374a, 374n are provided by different cloud service providers 370, 372. For example, the first cloud service provider 370 may be Amazon AWS. TM , and the second cloud service provider 372 is Microsoft Azure TM , but in other embodiments, one or more public clouds, private clouds, or a combination thereof may be used to provide the underlying resources used to form a particular storage system in the storage system cluster 376.

[0223] According to some embodiments of the present disclosure, Figure 3E The example depicted in 380 includes an edge management service 382 for delivering storage services. The delivered storage services (also referred to herein as 'data services') may include, for example, services that provide a certain amount of storage to consumers, services that provide storage to consumers according to a predetermined service level agreement, services that provide storage to consumers according to predetermined regulatory requirements, and many others.

[0224] Figure 3E The edge management service 382 depicted in FIG. 3 may be embodied as one or more computer program instruction modules executed, for example, on computer hardware, such as one or more computer processors. Alternatively, the edge management service 382 may be embodied as one or more computer program instruction modules executed on a virtualized execution environment, such as one or more virtual machines, in one or more containers, or in some other manner. In other embodiments, the edge management service 382 may be embodied as a combination of the embodiments described above, including embodiments in which one or more computer program instruction modules contained in the edge management service 382 are distributed across multiple physical or virtual execution environments.

[0225] The edge management service 382 may operate as a gateway for providing storage services to storage consumers, where the storage services utilize storage devices provided by one or more storage systems 374a, 374b, 374c, 374d, 374n. For example, the edge management service 382 may be configured to provide storage services to host devices 378a, 378b, 378c, 378d, 378n that are executing one or more applications that consume the storage services. In such an example, the edge management service 382 may operate as a gateway between the host devices 378a, 378b, 378c, 378d, 378n and the storage systems 374a, 374b, 374c, 374d, 374n, rather than requiring the host devices 378a, 378b, 378c, 378d, 378n to directly access the storage systems 374a, 374b, 374c, 374d, 374n.

[0226] Figure 3E Edge Management Service 382 Figure 3E The host devices 378a, 378b, 378c, 378d, 378n of the edge management service 382 may expose the storage service module 380, but in other embodiments, the edge management service 382 may expose the storage service module 380 to other consumers of various storage services. The various storage services may be presented to consumers via one or more user interfaces, via one or more APIs, or through some other mechanism provided by the storage service module 380. Thus, Figure 3E The storage service module 380 depicted in may be embodied as one or more computer program instruction modules executed on physical hardware, a virtualized execution environment, or a combination thereof, wherein execution of such modules enables consumers of storage services to be provided, select, and access various storage services.

[0227] Figure 3E The edge management service 382 also includes a system management service module 384 . Figure 3EThe system management service module 384 includes one or more computer program instruction modules, which, when executed, coordinate with the storage systems 374a, 374b, 374c, 374d, 374n to perform various operations to provide storage services to the host devices 378a, 378b, 378c, 378d, 378n. The system management service module 384 may be configured to, for example, perform tasks such as provisioning storage resources from the storage systems 374a, 374b, 374c, 374d, 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c, 374d, 374n, migrating data sets or workloads among the storage systems 374a, 374b, 374c, 374d, 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c, 374d, 374n, setting one or more adjustable parameters (i.e., one or more configurable settings) on the storage systems 374a, 374b, 374c, 374d, 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c, 374d, 374n, and the like. For example, many of the services described below relate to embodiments in which the storage systems 374a, 374b, 374c, 374d, 374n are configured to operate in a certain manner. In such instances, the system management service module 384 may be responsible for using an API (or some other mechanism) provided by the storage systems 374a, 374b, 374c, 374d, 374n to configure the storage systems 374a, 374b, 374c, 374d, 374n to operate in the manner described below.

[0228] In addition to configuring the storage systems 374a, 374b, 374c, 374d, 374n, the edge management service 382 itself may be configured to perform various tasks required to provide various storage services. Consider an instance where the storage service includes a service that, when selected and applied, causes personally identifiable information ('PII') contained in a data set to be obfuscated when the data set is accessed. In such an instance, the storage systems 374a, 374b, 374c, 374d, 374n may be configured to obfuscate the PII when serving read requests directed to the data set. Alternatively, the storage systems 374a, 374b, 374c, 374d, 374n may service reads by returning data containing the PII, but the edge management service 382 may itself obfuscate the PII as the data passes through the edge management service 382 on its way from the storage systems 374a, 374b, 374c, 374d, 374n to the host devices 378a, 378b, 378c, 378d, 378n.

[0229] Figure 3EThe storage systems 374a, 374b, 374c, 374d, 374n depicted in the drawings may be embodied as described above with reference to Figures 1A to 3D One or more of the storage systems described, including variations thereof. In fact, the storage systems 374a, 374b, 374c, 374d, 374n can be used as a storage resource pool, where the individual components in the pool have different performance characteristics, different storage characteristics, etc. For example, one of the storage systems 374a can be a cloud-based storage system, another storage system 374b can be a storage system that provides block storage, another storage system 374c can be a storage system that provides file storage, another storage system 374d can be a relatively high-performance storage system, and another storage system 374n can be a relatively low-performance storage system, etc. In alternative embodiments, there can be only a single storage system.

[0230] Figure 3E The storage systems 374a, 374b, 374c, 374d, 374n depicted in the diagram may also be organized into different fault domains such that a failure of one storage system 374a should be completely independent of a failure of another storage system 374b. For example, each storage system may receive power from an independent power system, each storage system may be coupled for data communications over an independent data communications network, and so on. In addition, storage systems in a first fault domain may be accessed via a first gateway, while storage systems in a second fault domain may be accessed via a second gateway. For example, the first gateway may be a first instance of the edge management service 382, ​​and the second gateway may be a second instance of the edge management service 382, ​​including embodiments in which each instance is different or each instance is part of a distributed edge management service 382.

[0231] As an illustrative example of available storage services, storage services may be presented to a user associated with different levels of data protection. For example, a user may be presented with storage services that, when selected and implemented, assure the user that data associated with the user will be protected such that various recovery point objectives ('RPOs') may be guaranteed. A first available storage service may ensure, for example, that some data set associated with a user will be protected such that any data older than 5 seconds in the past may be recovered in the event of a failure of a primary data store, while a second available storage service may ensure that a data set associated with a user will be protected such that any data older than 5 minutes in the past may be recovered in the event of a failure of a primary data store.

[0232] Additional instances of storage services that may be presented to a user, selected by the user, and ultimately applied to a data set associated with the user may include one or more data compliance services. Such data compliance services may be embodied, for example, as services that may provide data compliance services to consumers (i.e., users) to ensure that the user's data set is managed in a manner that complies with various regulatory requirements. For example, one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner that complies with the General Data Protection Regulation ('GDPR'), one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner that complies with the Sarbanes-Oxley Act of 2002 ('SOX'), or one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner that complies with certain other regulations. In addition, one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner that complies with certain non-governmental guidance (e.g., compliance with best practices for auditing purposes), one or more data compliance services may be provided to a user to ensure that the user's data set is managed in a manner that complies with specific customer or organizational requirements, and so on.

[0233] Consider an example where a specific data compliance service is designed to ensure that a user's data set is managed in a manner that complies with the requirements set out in the GDPR. While a list of all of the requirements of the GDPR can be found in the regulation itself, for illustrative purposes, the example requirement set out in the GDPR requires that a pseudonymization process must be applied to stored data in order to transform the personal data in such a way that the resulting data cannot be attributed to a specific data subject without the use of additional information. For example, data encryption techniques may be applied to make the original data unintelligible, and such data encryption techniques cannot be reversed without access to the correct decryption key. Therefore, the GDPR may require that the decryption key be stored separately from the pseudonymized data. A specific data compliance service may be provided to ensure compliance with the requirements set out in this paragraph.

[0234] In order to provide such a specific data compliance service, the data compliance service may be presented to the user (e.g., via a GUI) and selected by the user. In response to receiving a selection of a specific data compliance service, one or more storage service policies may be applied to a data set associated with the user to implement the specific data compliance service. For example, a storage service policy may be applied that requires that a data set be encrypted before being stored in a storage system, before being stored in a cloud environment, or before being stored elsewhere. In order to implement such a policy, not only may a requirement that a data set be encrypted when stored be implemented, but also a requirement that a data set be encrypted before transmitting the data set (e.g., sending the data set to another party). In such an instance, a storage service policy may also be implemented that requires that any encryption keys used to encrypt the data set are not stored on the same system that stores the data set itself. The reader will appreciate that many other forms of data compliance services may be provided and implemented in accordance with embodiments of the present disclosure.

[0235] The storage systems 374a, 374b, 374c, 374d, and 374n in the storage system group 376 may be jointly managed by one or more group management modules. The group management module may be Figure 3E The cluster management module may perform tasks such as monitoring the health of each storage system in the cluster, initiating updates or upgrades on one or more storage systems in the cluster, migrating workloads for load balancing or other performance purposes, and many other tasks. Therefore, and for many other reasons, the storage systems 374a, 374b, 374c, 374d, 374n may be coupled to each other via one or more data communication links to exchange data between the storage systems 374a, 374b, 374c, 374d, 374n.

[0236] The storage system described herein may support various forms of data replication. For example, two or more storage systems may synchronously replicate data sets between each other. In synchronous replication, different copies of a particular data set may be maintained by multiple storage systems, but all accesses (e.g., reads) to the data set should produce consistent results, regardless of which storage system the access is directed to. For example, reads from any storage system directed to a synchronously replicated data set should return the same result. Therefore, although updates to versions of a data set do not need to occur completely simultaneously, precautions must be taken to ensure consistent access to the data set. For example, if an update (e.g., write) directed to a data set is received by a first storage system, the update may be confirmed as complete only if all storage systems that synchronously replicate the data set have applied the update to their copies of the data set. In such an instance, synchronous replication may be implemented using I / O forwarding (e.g., writes received at a first storage system are forwarded to a second storage system), communication between storage systems (e.g., each storage system indicates that it has completed the update), or in other ways.

[0237] In other embodiments, a data set may be replicated using checkpoints. In checkpoint-based replication (also known as 'near synchronous replication'), a set of updates to a data set (e.g., one or more write operations directed to the data set) may occur between different checkpoints, such that the data set has been updated to a particular checkpoint only if all updates to the data set prior to the particular checkpoint have been completed. Consider an example in which a first storage system stores a real-time copy of a data set that is being accessed by users of the data set. In such an example, assume that the data set is replicated from the first storage system to a second storage system using checkpoint-based replication. For example, the first storage system may send a first checkpoint (at time t=0) to the second storage system, followed by a first set of updates to the data set, followed by a second checkpoint (at time t=1) , followed by a second set of updates to the data set, followed by a third checkpoint (at time t=2). In such an example, if the second storage system has performed all updates in the first set of updates, but has not yet performed all updates in the second set of updates, then the copy of the data set stored on the second storage system may be up-to-date until the second checkpoint. Alternatively, if the second storage system has performed all updates in both the first set of updates and the second set of updates, then the copy of the data set stored on the second storage system may be up-to-date until the third checkpoint. The reader will appreciate that various types of checkpoints may be used (e.g., metadata-only checkpoints), that checkpoints may be dispersed based on various factors (e.g., time, number of operations, RPO settings), and so on.

[0238] In other embodiments, a data set may be replicated via snapshot-based replication (also referred to as "asynchronous replication"). In snapshot-based replication, a snapshot of a data set may be sent from a replication source, such as a first storage system, to a replication target, such as a second storage system. In such embodiments, each snapshot may include the entire data set or a subset of the data set, such as, for example, only the portion of the data set that has changed since the last snapshot was sent from the replication source to the replication target. The reader will appreciate that snapshots may be sent on demand, based on a policy that takes into account various factors (e.g., time, number of operations, RPO settings), or in some other manner.

[0239] The storage systems described above may be configured, either individually or in combination, to function as continuous data protection storage. Continuous data protection storage is a feature of a storage system that records updates to a data set so that a consistent image of the previous contents of the data set can be accessed at a low temporal granularity (typically on the order of a few seconds or even less), and backtracked to a reasonable period of time (typically hours or days). These allow access to the most recent consistent point in time of the data set, and also allow access to a point in time of the data set that may have immediately preceded an event (e.g., causing part of the data set to be corrupted or otherwise lost), while maintaining a maximum number of updates close to that event. Conceptually, they are like a series of snapshots of a data set that are taken very frequently and saved for a long time, but continuous data protection storage is typically implemented very differently from snapshots. A storage system that implements data continuous data protection storage may further provide a means of accessing these points in time, accessing one or more of these points in time as a snapshot or clone copy, or restoring the data set to one of those recorded points in time.

[0240] Over time, to reduce overhead, some of the time points saved in the continuous data protection storage can be merged with other nearby time points, essentially removing some of these time points from the storage. This can reduce the capacity required for storage updates. A limited number of these time points can also be converted into snapshots of longer persistence. For example, such a storage can maintain a low-granularity sequence of time points going back a few hours from now, with some time points merged or deleted to reduce overhead by up to an additional day. Going back further in the past than that, some of these time points can be converted into snapshots that represent a consistent point-in-time image only every few hours.

[0241] Although some embodiments are primarily described in the context of storage systems, a skilled reader in the art will recognize that embodiments of the present disclosure may also take the form of a computer program product disposed on a computer-readable storage medium for use with any suitable processing system. Such a computer-readable storage medium may be any storage medium for machine-readable information, including magnetic media, optical media, solid-state media, or other suitable media. Examples of such media include magnetic or floppy disks in hard drives, optical disks for optical drives, magnetic tapes, and other media that will occur to those skilled in the art. Those skilled in the art will immediately recognize that any computer system with suitable programming components will be able to perform the steps described herein, as embodied in a computer program product. Those skilled in the art will also recognize that although some embodiments described in this specification are directed to software installed and executed on computer hardware, alternative embodiments implemented as firmware or hardware are fully within the scope of the present disclosure.

[0242] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided according to the principles described herein. When executed by a processor of a computing device, the instructions may direct the processor and / or the computing device to perform one or more operations, including one or more operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0243] The non-transitory computer-readable media referred to herein may include any non-transitory storage media that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0244] The advantages and features of the present disclosure can be further described by the following statements:

[0245] 1. A method comprising: identifying, by a recovery process, a set of immutable layers of a container image contained in a data set used by a container system to run a containerized application on a first cluster; replicating, by the recovery process, the set of immutable layers of the container image to a second cluster; after replicating the set of immutable layers of the container image to the second cluster, receiving, by the recovery process, a recovery request to recover the containerized application; and replicating, by the recovery process, a set of variable layers contained in the data set to the second cluster in response to the recovery request, the second cluster being configured to use the replicated set of immutable layers and the replicated set of variable layers to recover the containerized application on the second cluster.

[0246] 2. The method of any of the preceding statements, wherein the set of immutable layers of the container image comprises container image layers accessed and used by one or more computing nodes of the first cluster to run the containerized application on the first cluster.

[0247] 3. A method according to any of the preceding statements, wherein the set of mutable layers is provided by a storage system as persistent storage for the containerized application running on the first cluster.

[0248] 4. A method according to any of the preceding statements, wherein replicating the set of immutable layers contained in the data set to the second cluster comprises prioritizing more frequently used immutable layers over less frequently used immutable layers.

[0249] 5. A method according to any of the preceding statements, wherein replicating the set of variable layers contained in the data set to the second cluster comprises prioritizing more frequently used variable layers over less frequently used variable layers.

[0250] 6. A method according to any of the preceding statements, wherein in response to the recovery request, copying the set of variable layers contained in the data set to the second cluster includes identifying and copying only the variable layers in the set of variable layers that have changed since the set of variable layers was last copied to the second cluster.

[0251] 7. A method according to any of the preceding statements, wherein the set of immutable layers comprises only the unique immutable layer in the data set.

[0252] 8. A method according to any of the preceding statements, wherein the data set is managed by a container-aware storage system, which provides the immutable layer and the mutable layer for use by the container system to run the containerized application on the first cluster.

[0253] 9. A method as in any of the preceding statements, wherein said container-aware storage system manages and provides said immutable layer and said mutable layer as volumes.

[0254] 10. The method of any of the preceding statements, wherein the container-aware storage system comprises a containerized instance of the container-aware storage system running on a compute node of the first cluster.

[0255] 11. A method according to any of the preceding statements, wherein the recovery process is included in the container-aware storage system.

[0256] 12. A method according to any of the preceding statements, further comprising: determining, by the container-aware storage system, an interruption of one or more storage services provided by the container-aware storage system; and providing an alert from the container-aware storage system to a container orchestrator for the container system based on the interruption.

[0257] 13. A method according to any of the preceding statements, further comprising: detecting a new immutable layer or an updated immutable layer in the set of immutable layers included in the data set; and replicating the new immutable layer or the updated immutable layer to the second cluster.

[0258] 14. A method according to any of the preceding statements, further comprising: receiving a restore request to restore the containerized application back to the first cluster; and identifying any of the variable layers that have changed on the second cluster and copying it from the second cluster to the first cluster.

[0259] 15. A system comprising: one or more memories storing computer-executable instructions; and one or more processors for executing the computer-executable instructions to: identify a set of immutable layers of a container image contained in a data set used by a container system to run a containerized application on a first cluster; copy the set of immutable layers of the container image to a second cluster; after copying the set of immutable layers of the container image to the second cluster, receive a restore request to restore the containerized application; and in response to the restore request, copy a set of variable layers contained in the data set to the second cluster, the second cluster being configured to use the copied set of immutable layers and the copied set of variable layers to restore the containerized application on the second cluster.

[0260] 16. A system according to any of the preceding statements, wherein: the set of immutable layers of the container image includes container image layers that are accessed and used by one or more computing nodes of the first cluster to run the containerized application on the first cluster; and the set of mutable layers is provided by a storage system as persistent storage for the containerized application running on the first cluster.

[0261] 17. A system according to any of the preceding statements, wherein the data set is managed by a container-aware storage system, which provides the immutable layer and the mutable layer for use by the container system to run the containerized application on the first cluster.

[0262] 18. The system of any of the preceding statements, wherein the container-aware storage system manages and provides the immutable layer and the mutable layer as volumes.

[0263] 19. A system according to any of the preceding statements, wherein the one or more processors are further configured to execute the computer executable instructions to: receive a restore request to restore the containerized application back to the first cluster; and identify any of the variable layers that have changed on the second cluster and copy it from the second cluster to the first cluster.

[0264] 20. A non-transitory computer-readable medium storing computer instructions that, when executed, direct one or more processors of one or more computing devices to: identify a set of immutable layers of a container image included in a data set used by a container system to run a containerized application on a first cluster; copy the set of immutable layers of the container image to a second cluster; after copying the set of immutable layers of the container image to the second cluster, receive a restore request to restore the containerized application; and in response to the restore request, copy a set of variable layers included in the data set to the second cluster, the second cluster being configured to use the copied set of immutable layers and the copied set of variable layers to restore the containerized application on the second cluster.

[0265] 1. A method comprising: providing, by a storage system, one or more storage services to a container operating within a container system; determining, by the storage system, an interruption to the one or more storage services; and providing, based on the interruption and from the storage system, an alert associated with the container to a container orchestrator for the container system.

[0266] 2. The method of any of the preceding statements, wherein said determining said interruption comprises determining a port error based on a connection between a first storage area network port associated with said storage system and a second storage area network port associated with a node hosting said container.

[0267] 3. A method according to any of the preceding statements, wherein said determining said interruption comprises: determining a loss of connectivity between a first storage area network port associated with said storage system and a second storage area network port associated with a node hosting said container.

[0268] 4. A method according to any of the preceding statements, wherein: the connection comprises a storage area network connection; and the connection implements at least one of: Fibre Channel, Internet Small Computer System Interface, Fibre Channel over Ethernet, or Non-Volatile Flash Memory over Fibre Channel.

[0269] 5. A method according to any of the preceding statements, wherein said providing said one or more storage services comprises: providing a volume comprising a container image.

[0270] 6. A method according to any of the preceding statements, wherein said providing said one or more storage services comprises: providing data storage to said container.

[0271] 7. A method according to any of the preceding statements, wherein the alarm indicates an error associated with the operation of the container.

[0272] 8. A system comprising: one or more memories storing computer executable instructions; and one or more processors for executing the computer executable instructions to: provide one or more storage services to a container operating within a container system by a storage system; determine, by the storage system, an interruption to the one or more storage services; and provide, based on the interruption, an alert associated with the container from the storage system to a container orchestrator for the container system.

[0273] 9. A system according to any of the preceding statements, wherein the one or more processors further execute the computer executable instructions to: determine a port error based on a connection between a first storage area network port associated with the storage system and a second storage area network port associated with a node hosting the container.

[0274] 10. A system according to any of the preceding statements, wherein the one or more processors further execute the computer executable instructions to: determine a loss of connectivity between a first storage area network port associated with the storage system and a second storage area network port associated with a node hosting the container.

[0275] 11. A system according to any of the preceding statements, wherein: the connection comprises a storage area network connection; and the connection implements at least one of: Fibre Channel, Internet Small Computer System Interface, Fibre Channel over Ethernet, or Non-Volatile Flash Memory over Fibre Channel.

[0276] 12. The system of any of the preceding statements, wherein, to provide the one or more storage services, the one or more processors further execute the computer executable instructions to: provide a volume comprising a container image.

[0277] 13. The system of any of the preceding statements, wherein, in order to provide the one or more storage services, the one or more processors further execute the computer executable instructions to: provide data storage to the container.

[0278] 14. A system according to any of the preceding statements, wherein said alarm indicates an error associated with the operation of said container.

[0279] 15. A non-transitory computer-readable medium storing computer instructions that, when executed, direct one or more processors of one or more computing devices to: provide, by a storage system, one or more storage services to a container operating within a container system; determine, by the storage system, an interruption to the one or more storage services; and provide, based on the interruption and from the storage system, an alert associated with the container to a container orchestrator for the container system.

[0280] 16. A non-transitory computer readable medium according to any of the preceding statements, wherein the one or more processors further execute the computer instructions to: determine a port error based on a connection between a first storage area network port associated with the storage system and a second storage area network port associated with a node hosting the container.

[0281] 17. A non-transitory computer readable medium according to any of the preceding statements, wherein the one or more processors further execute the computer instructions to: determine a loss of connectivity between a first storage area network port associated with the storage system and a second storage area network port associated with a node hosting the container.

[0282] 18. A non-transitory computer readable medium according to any of the preceding statements, wherein: the connection comprises a storage area network connection; and the connection implements at least one of: Fibre Channel, Internet Small Computer System Interface, Fibre Channel over Ethernet, or Non-Volatile Flash Memory over Fibre Channel.

[0283] 19. The non-transitory computer readable medium of any of the preceding statements, wherein, to provide the one or more storage services, the one or more processors further execute the computer instructions to: provide a volume comprising a container image.

[0284] 20. The non-transitory computer readable medium according to any one of the preceding statements, wherein the alarm indicates an error associated with the operation of the container.

[0285] 1. A method comprising: providing storage services to a container system by a storage system; determining, by the storage system and based on providing the storage services to the container system, a potential security threat; and performing a remediation action by the storage system based on the potential security threat.

[0286] 2. A method according to any of the preceding statements, wherein said providing said storage service comprises: providing a volume comprising a container image.

[0287] 3. A method according to any of the preceding statements, wherein said providing said storage service comprises: providing storage to a container, wherein said volume comprises said storage.

[0288] 4. A method according to any of the preceding statements, wherein the determining the potential security threat comprises: determining that the storage data associated with providing the storage service to the container system comprises the potential security threat based on one or more threat characteristics, wherein the potential security threat comprises ransomware.

[0289] 5. The method of any of the preceding statements, wherein said performing said remediation action comprises: initiating one or more remediation operations for stored data including said ransomware.

[0290] 6. A method according to any of the preceding statements, wherein the determining the potential security threat comprises: using a machine learning model to determine that one or more storage operations indicate the potential security threat, wherein the one or more storage operations are associated with the storage service.

[0291] 7. A method according to any of the preceding statements, wherein said performing said remedial action comprises: providing a notification to a container orchestrator, wherein said notification indicates said potential security threat, and wherein said container orchestrator operates within said container system.

[0292] 8. A system comprising: one or more memories storing computer executable instructions; and one or more processors for executing the computer executable instructions to: provide storage services to a container system by a storage system; determine, by the storage system and based on providing the storage services to the container system, a potential security threat; and perform remedial actions by the storage system based on the potential security threat.

[0293] 9. The system of any of the preceding statements, wherein, to provide the storage service, the one or more processors further execute the computer executable instructions to: provide a volume comprising a container image.

[0294] 10. The system of any of the preceding statements, wherein, to provide the storage service, the one or more processors further execute the computer executable instructions to: provide storage to a container, wherein the volume comprises the storage.

[0295] 11. A system according to any of the preceding statements, wherein, in order to determine the potential security threat, the one or more processors further execute the computer executable instructions to: determine that the storage data associated with providing the storage service to the container system includes the potential security threat based on one or more threat characteristics, wherein the potential security threat includes ransomware.

[0296] 12. The system of any of the preceding statements, wherein the one or more processors further execute the computer executable instructions to: initiate one or more remediation operations for stored data including the ransomware.

[0297] 13. A system according to any of the preceding statements, wherein, in order to determine the potential security threat, the one or more processors further execute the computer executable instructions to: use a machine learning model to determine that one or more storage operations indicate the potential security threat, wherein the one or more storage operations are associated with the storage service.

[0298] 14. The system of any of the preceding statements, wherein, to perform the remedial action, the one or more processors further execute the computer-executable instructions to: provide a notification to a container orchestrator, wherein the notification indicates the potential security threat, and wherein the container orchestrator operates within the container system.

[0299] 15. A non-transitory computer-readable medium storing computer instructions that, when executed, direct one or more processors of one or more computing devices to: provide storage services to a container system by a storage system; determine, by the storage system and based on providing the storage services to the container system, a potential security threat; and perform remedial actions by the storage system based on the potential security threat.

[0300] 16. The non-transitory computer readable medium of any of the preceding statements, wherein, to provide the storage service, the one or more processors further execute the computer instructions to: provide a volume comprising a container image.

[0301] 17. The non-transitory computer readable medium of any of the preceding statements, wherein, to provide the storage service, the one or more processors further execute the computer instructions to: provide storage to a container, wherein the volume comprises the storage.

[0302] 18. A non-transitory computer-readable medium according to any of the preceding statements, wherein, in order to determine the potential security threat, the one or more processors further execute the computer-executable instructions to: determine that the storage data associated with providing the storage service to the container system includes the potential security threat based on one or more threat characteristics, wherein the potential security threat includes ransomware.

[0303] 19. The non-transitory computer readable medium according to any of the preceding statements, wherein the one or more processors further execute the computer executable instructions to: initiate one or more remediation operations for stored data including the ransomware.

[0304] 20. A non-transitory computer-readable medium according to any of the preceding statements, wherein, in order to determine the potential security threat, the one or more processors further execute the computer-executable instructions to: use a machine learning model to determine that one or more storage operations indicate the potential security threat, wherein the one or more storage operations are associated with the storage service.

[0305] 1. A method comprising: mounting a volume comprising a container image by a container runtime operating on a node within a container system, wherein the node comprises a storage resource; creating an instance of a container associated with the container image on the node and based on access to the volume; and mapping access to the storage resource to one or more volumes stored away from the node for the container, wherein based on mapping the access to the storage resource to the one or more volumes, the storage resource on the node is immutable for the container.

[0306] 2. A method according to any of the preceding statements, wherein an application operating in the container has write access to the storage resource of the node.

[0307] 3. A method according to any of the preceding statements, further comprising: mapping the one or more volumes to one or more directories within a user space of the node.

[0308] 4. A method according to any of the preceding statements, wherein the volume comprises the one or more volumes, and wherein the one or more volumes comprise data associated with the operation of the application within the container.

[0309] 5. A method according to any of the preceding statements, further comprising: providing, by the container runtime, an identifier associated with the container image to a storage system; and receiving volume data associated with the volume; wherein the mounting of the volume is based on the volume data.

[0310] 6. A method according to any of the preceding statements, wherein the one or more volumes are mapped to one or more directories used by applications operating within the container.

[0311] 7. A method according to any of the preceding statements, wherein the one or more volumes are mapped to one or more directories within a kernel namespace, wherein the one or more directories are associated with operation of the kernel, and wherein the container operates within user space.

[0312] 8. A system comprising: one or more memories storing computer executable instructions; and one or more processors for executing the computer executable instructions to: mount a volume comprising a container image by a container runtime operating on a node within a container system, wherein the node comprises a storage resource; create an instance of a container associated with the container image on the node and based on access to the volume; and map access to the storage resource to one or more volumes stored away from the node for the container, wherein based on mapping the access to the storage resource to the one or more volumes, the storage resource on the node is immutable for the container.

[0313] 9. A system according to any of the preceding statements, wherein an application operating in the container has write access to the storage resources of the node.

[0314] 10. The system of any of the preceding statements, wherein the one or more processors further execute the computer executable instructions to: map the one or more volumes to one or more directories within a user space of the node.

[0315] 11. A system as in any of the preceding statements, wherein the volume comprises the one or more volumes, and wherein the one or more volumes comprise data associated with operation of an application within the container.

[0316] 12. A system according to any of the preceding statements, wherein the one or more processors further execute the computer executable instructions to: provide, by the container runtime, an identifier associated with the container image to a storage system; and receive volume data associated with the volume; wherein mounting the volume is based on the volume data.

[0317] 13. A system according to any of the preceding statements, wherein the one or more volumes are mapped to one or more directories used by applications operating within the container.

[0318] 14. A system according to any of the preceding statements, wherein the one or more volumes are mapped to one or more directories within a kernel namespace, wherein the one or more directories are associated with operation of the kernel, and wherein the container operates within user space.

[0319] 15. A non-transitory computer-readable medium storing computer instructions that, when executed, direct one or more processors of one or more computing devices to: mount a volume comprising a container image via a container runtime operating on a node within a container system, wherein the node comprises a storage resource; create an instance of a container associated with the container image on the node and based on access to the volume; and map, for the container, access to the storage resource to one or more volumes stored away from the node, wherein based on mapping the access to the storage resource to the one or more volumes, the storage resource on the node is immutable with respect to the container.

[0320] 16. The non-transitory computer readable medium according to any of the preceding statements, wherein an application operating in said container has write access to said storage resources of said node.

[0321] 17. A non-transitory computer readable medium according to any of the preceding statements, wherein said one or more processors further execute said computer instructions to: map said one or more volumes to one or more directories within a user space of said node.

[0322] 18. The non-transitory computer readable medium of any of the preceding statements, wherein the volume comprises the one or more volumes, and wherein the one or more volumes comprise data associated with operation of an application within the container.

[0323] 19. A non-transitory computer readable medium according to any of the preceding statements, wherein the one or more processors further execute the computer instructions to: provide, by the container runtime, an identifier associated with the container image to a storage system; and receive volume data associated with the volume; wherein mounting the volume is based on the volume data.

[0324] 20. A non-transitory computer readable medium according to any of the preceding statements, wherein the one or more volumes are mapped to one or more directories used by an application operating within the container.

[0325] One or more embodiments may be described herein with the help of method steps that illustrate the performance of specific functions and their relationships. For ease of description, the boundaries and sequences of these functional building blocks and method steps are arbitrarily defined herein. Alternative boundaries and sequences may be defined as long as the specified functions and relationships are properly performed. Therefore, any such alternative boundaries or sequences are within the scope and spirit of the claims. In addition, for ease of description, the boundaries of these functional building blocks have been arbitrarily defined. Alternative boundaries may be defined as long as certain important functions are properly performed. Similarly, flow chart blocks may also be arbitrarily defined herein to illustrate certain important functionality.

[0326] To the extent used, the flow chart block boundaries and sequences could have been defined in other ways and still perform certain significant functionality. Therefore, such alternative definitions of functional building blocks and flow chart blocks and sequences are within the scope and spirit of the claims. Those skilled in the art will also recognize that the functional building blocks and other illustrative boxes, modules, and components herein may be implemented as described or by discrete components, application specific integrated circuits, processors executing appropriate software, and the like, or any combination thereof.

[0327] Although specific combinations of various functions and features of one or more embodiments are explicitly described herein, other combinations of these features and functions are also possible. The present disclosure is not limited to the specific examples disclosed herein, and explicitly incorporates these other combinations.

[0328] In some embodiments, one or more storage systems or one or more elements of a storage system (e.g., features, services, operations, components, etc. of a storage system), such as any of the illustrative storage systems or storage system elements described herein, may be implemented in and / or provide storage services to one or more container systems. A container system may include any system that supports the execution of one or more containerized applications or services. Such services may be software deployed as infrastructure for building applications, for operating a runtime environment, and / or as infrastructure for other services. In the discussion that follows, descriptions of containerized applications generally also apply to containerized services.

[0329] A container can combine one or more elements of a containerized software application with a runtime environment for operating those elements of the software application bundled in a single image. For example, each container of a containerized application may include executable code and various dependencies, libraries, and / or other components of the software application, as well as network configurations and configuration access to additional resources, which are used by the elements of the software application within a specific container so that these elements can be operated. A containerized application can be represented as a collection of such containers, which together represent all elements of the application and are combined with various runtime environments required for all these elements to run. Therefore, containerized applications can be abstracted from the host operating system as a combined collection of lightweight and portable packaging and configurations, where containerized applications can be uniformly deployed and consistently executed in different computing environments using different container-compatible operating systems or different infrastructures. In some embodiments, the containerized application shares a kernel with the host computer system and is executed as an isolated environment (an isolated collection of files and directories, processes, system and network resources, and configuration access to additional resources and capabilities), which is isolated by the host system's operating system in conjunction with a container management framework. When executed, a containerized application may provide one or more containerized workloads and / or services.

[0330] A container system may include and / or utilize a cluster of nodes. For example, a container system may be configured to manage the deployment and execution of containerized applications on one or more nodes in a cluster. Containerized applications may utilize resources of a node, such as memory, processing, and / or storage resources provided and / or accessed by the node. Storage resources may include any of the illustrative storage resources described herein, and may include on-node resources such as local files and directory trees, off-node resources such as external network file systems, databases, or object repositories, or both on-node resources and off-node resources. Access to additional resources and capabilities of containers that may be configured for containerized applications may include specialized computing capabilities such as GPUs and AI / ML engines, or specialized hardware such as sensors and cameras.

[0331] In some embodiments, the container system may include a container orchestration system (also referred to as a container orchestrator, container orchestration platform, etc.) that is designed to be reasonably simple and automated for many use cases to deploy, scale, and manage containerized applications. In some embodiments, the container system may include and / or communicate with a storage management system that is configured to provide and manage storage resources (e.g., virtual volumes) for private or shared use by cluster nodes and / or containers of containerized applications.

[0332] Figure 4An example container system 400 is illustrated. In this example, the container system 400 includes a container storage system 402 that can be configured to perform one or more storage management operations to organize, provide, and manage storage resources for use by one or more containerized applications 404-1 through 404-L of the container system 400. In particular, the container storage system 402 can organize storage resources into one or more storage pools 406 of storage resources for use by the containerized applications 404-1 through 404-L. The container storage system itself can be implemented as a containerized service.

[0333] Container system 400 may include or be implemented by one or more container orchestration systems, including Kubernetes TM 、Mesos TM 、Docker Swarm TM Etc. The container orchestration system may manage the container system 400 running on the cluster 408 through services implemented by a control node depicted as 410, and may further manage the relationship between the container storage system or individual containers and their storage, memory and CPU limits, networking, and their access to additional resources or services.

[0334] The control plane of the container system 400 can implement services including: deploying applications via the controller 412, monitoring applications via the controller 412, providing interfaces via the API server 414, and scheduling deployments via the scheduler 416. In this example, the controller 412, the scheduler 416, the API server 414, and the container storage system 402 are implemented on a single node (node ​​410). In other examples, for resiliency, the control plane can be implemented by multiple redundant nodes, where if a node providing management services for the container system 400 fails, another redundant node can provide management services for the cluster 408.

[0335] The data plane of container system 400 may include a set of nodes that provide a container runtime for executing containerized applications. Individual nodes within cluster 408 may execute a container runtime, such as Docker. TM , and executes a container manager or node agent, such as kubelet in Kubernetes (not depicted), which communicates with the control plane via a local network-connected agent (sometimes referred to as a proxy), such as proxy 418. Proxy 418 can route network traffic to and from containers using, for example, Internet Protocol (IP) port numbers. For example, a containerized application can request a storage class from the control plane, where the request is handled by the container manager, and the container manager communicates the request to the control plane using proxy 418.

[0336] Cluster 408 may include a set of nodes that run containers of managed containerized applications. A node may be a virtual machine or a physical machine. A node may be a host system.

[0337] Container storage system 402 can orchestrate storage resources to provide storage to container system 400. For example, container storage system 402 can provide persistent storage for containerized applications 404-1 through 404-L using storage pool 406. Container storage system 402 itself can be deployed as a containerized application by a container orchestration system.

[0338] For example, container storage system 402 applications may be deployed within cluster 408 and perform management functions to provide storage for containerized applications 404. Management functions may include determining one or more storage pools from available storage resources, provisioning virtual volumes on one or more nodes, replicating data, responding to and recovering from host and network failures, or handling storage operations. Storage pool 406 may include storage resources from one or more local or remote sources, where the storage resources may be different types of storage, including block storage, file storage, and object storage as examples.

[0339] Container storage system 402 may also be deployed on a set of nodes, and the container orchestration system may provide persistent storage for these nodes. In some instances, container storage system 402 may be deployed on all nodes in cluster 408 using, for example, Kubernetes DaemonSet. In this instance, nodes 420-1 to 420-N provide container runtimes at which container storage system 402 executes. In other instances, some but not all nodes in the cluster may execute container storage system 402.

[0340] The container storage system 402 can handle storage on the node and communicate with the control plane of the container system 400 to provide dynamic volumes, including persistent volumes. Persistent volumes can be mounted on the node as virtual volumes, such as virtual volumes 422-1 and 422-P. After the virtual volume 422 is mounted, the containerized application can request and use or otherwise be configured to use the storage provided by the virtual volume 422. In this example, the container storage system 402 can install a driver on the kernel of the node, wherein the driver handles storage operations directed to the virtual volume. In this example, the driver can receive storage operations directed to the virtual volume, and in response, the driver can perform storage operations on one or more storage resources within the storage pool 406 under the guidance of the container or using additional logic within the container, which implements the container storage system 402 as a containerized service.

[0341] The container storage system 402 may determine available storage resources in response to being deployed as a containerized service. For example, the storage resources 424-1 to 424-M may include local storage, remote storage (storage on a separate node in the cluster), or both local storage and remote storage. The storage resources may also include storage from external sources, such as various combinations of block storage systems, file storage systems, and object storage systems. The storage resources 424-1 to 424-M may include storage resources of any type and / or configuration (e.g., any of the illustrative storage resources described above), and the container storage system 402 may be configured to determine available storage resources in any suitable manner, including based on a configuration file. For example, the configuration file may specify account and authentication information for a cloud-based object storage device 348 or a cloud-based storage system 318. The container storage system 402 may also determine the availability of one or more storage devices 356 or one or more storage systems. The total storage from one or more of the storage device 356, the storage system, the cloud-based storage system 318, the edge management service 382, ​​the cloud-based object storage device 348, or any other storage resource, or any combination or sub-combination of these storage resources, can be used to provide the storage pool 406. The storage pool 406 is used to provision storage for one or more virtual volumes mounted on one or more of the nodes 420 within the cluster 408.

[0342] In some embodiments, the container storage system 402 may create multiple storage pools. For example, the container storage system 402 may aggregate storage resources of the same type into individual storage pools. In this example, the storage type may be one of the following: storage device 356, storage array 102, cloud-based storage system 318, storage via edge management service 382, ​​or cloud-based object storage device 348. Alternatively, it may be storage configured with a certain level or type of redundancy or distribution, such as a specific combination of striping, mirroring, or erasure coding.

[0343] The container storage system 402 may be executed within a cluster 408 as a containerized container storage system service, wherein container instances implementing elements of the containerized container storage system service may operate on different nodes within the cluster 408. In this example, the containerized container storage system service may operate in conjunction with the container orchestration system of the container system 400 to handle storage operations, mount virtual volumes to provide storage to nodes, aggregate available storage into a storage pool 406, provision storage for virtual volumes from a storage pool 406, generate backup data, replicate data between nodes, clusters, environments, and other storage system operations. In some examples, the containerized container storage system service may provide storage services across multiple clusters operating in different computing environments. For example, other storage system operations may include the storage system operations described above with respect to FIGS. 1 to 3. The persistent storage provided by the containerized container storage system service may be used to implement stateful and / or resilient containerized applications.

[0344] The container storage system 402 may be configured to perform any suitable storage operations of a storage system. For example, the container storage system 402 may be configured to perform one or more illustrative storage management operations described herein to manage storage resources used by the container system.

[0345] In some embodiments, one or more storage operations, including one or more illustrative storage management operations described herein, may be containerized. For example, one or more storage operations may be implemented as one or more containerized applications configured to be executed to perform the storage operations. Such containerized storage operations may be executed in any suitable runtime environment to manage any storage system, including any illustrative storage system described herein.

[0346] In some embodiments, a storage system, such as the container storage system 402 described above, may be configured to store, manage, and provide immutable container images (e.g., images of containerized applications) to a container system for use by the container system to run container instances of the container images (e.g., container instances of containerized applications). The storage system may be further configured to provide storage services, such as persistent data storage, to the container system (e.g., to container instances running in the container system), and may utilize elements of the storage services to store, manage, and provide container images to the container system. Such a storage system may be referred to as a container-aware storage system.

[0347] In some embodiments, a container-aware storage system may use volumes to store, manage, and provide an immutable container image to a container system for use by the container system to run a container instance of the container image. For example, a storage system may receive an immutable container image and store the immutable container image as a volume. This may include the storage system converting the immutable container image to a volume, examples of which are described herein, and storing the volume to one or more storage resources. The storage system may then detect a request from the container system, such as a request to run a container instance of the immutable container image in the container system, and provide the volume to the container system in response to the request. The provision of the volume may include mapping the volume to the container instance, such as by attaching and mounting a virtual volume mapped to the volume to a host node for use by the container instance. The container system may then use the immutable container image from the volume to run a container instance of the immutable container image in the container system.

[0348] In some embodiments, in addition to providing an immutable container image to a container system, a storage system may also provide persistent data storage for use by the container system to run a container instance of the immutable container image in the container system, such as when the container instance is a stateful container instance or is otherwise configured to write data to persistent storage. The storage system may provide persistent data storage in any suitable manner. As an example, the storage system may provide an additional volume to the container system that is configured to provide persistent storage for the container instance of the immutable container image in the container system. As another example, a volume that includes an immutable container image and is provided by the storage system to the container system may include both an immutable layer that includes the immutable container image (e.g., an immutable snapshot of the container image) and a writable layer that is configured to provide persistent storage for the container instance of the immutable container image in the container system.

[0349] Various illustrative embodiments of a container-aware storage system and the manner in which the container-aware storage system can store, manage, and provide immutable container images and optional persistent data storage to a container system, and the manner in which the container system can use the immutable container images and persistent data storage to run container instances of the container images, will now be described in more detail with reference to several figures.

[0350] Figure 5An example configuration 500 of a container system 502 and a container-aware storage system 504 (also referred to as storage system 504) is illustrated, which is configured to store, manage, and provide immutable container images and optional persistent data storage to the container system 502. The container system 502 and the storage system 504 may be communicatively coupled and configured to communicate with each other in any suitable manner. For example, the container system 502 and the storage system 504 may communicate through an interface 506, which may be any suitable interface (e.g., one or more APIs). The storage system 504 may provide interfaces that allow the storage system 504 to function as a container image storage layer as well as a persistent storage layer. For example, the storage system may provide interfaces that are configured to receive and interpret requests from the container system 502, such as requests for container images, persistent storage, and / or volumes.

[0351] Container system 502 may include orchestration system 508 and container runtime system 510. Orchestration system 508 may deploy, scale, and manage containerized applications within container system 502, such as selecting nodes of a cluster by managing the deployment of containerized applications. Container runtime system 510 may provide a container runtime environment on the nodes of the cluster. When a containerized application is deployed to a node for execution on the node, container runtime system 510 may perform operations to execute the containerized application within the container runtime environment on the node.

[0352] Execution of the containerized application may include the container runtime system 510 running a container instance 512 representing a container image of the containerized application in a container runtime environment on the node. To do this, the container runtime system 510 may receive a request to run a container instance 512 of a container image, and in response to the request, access and use a container image associated with the request to run the container instance 512. The container system 510 may access the container image from the storage system 504, rather than accessing the container image from a local storage of a node or accessing the container image from a container image repository via a network connection and loading the container image into the local storage of the node as in conventional container systems.

[0353] In response to one or more requests from the container system 502, the storage system 504 can provide the container system 502 with a container image 514 and optional persistent storage 516. The container runtime system 510 can use the container image 514 to run a container instance 512 of the container image 514 in the container runtime environment of the node. In addition, when persistent storage 516 is provided, the container runtime system 510 can use the persistent storage 516 to persistently store data associated with running the container instance 512 of the container image 514 in the container runtime environment of the node.

[0354] The storage system 504 can be configured to serve as a repository for container images that can receive, store, manage, and provide container images. For example, the storage system 504 can be configured to receive and respond to push and pull requests to receive and provide container images. In certain embodiments, the storage system 504 is configured to receive container images, store the container images as volumes, manage the volumes, and provide the volumes to the container system 502 for running container instances of the container images.

[0355] Figure 6 An example is illustrated in which storage system 504 receives container image 602 and stores container image 602 as volume 604. Container image 602 can be immutable and can include one or more image layers. In the illustrated example, container image 602 includes three image layers—image layer 1, image layer 2, and image layer 3. The image layers can have a hierarchical relationship. For example, image layer 3 can be a top-level image layer and can depend on and reference the next image layer, image layer 2, which can depend on and reference the next image layer, image layer 1. In some embodiments, each image layer can be considered an individual immutable container image.

[0356] Storage system 504 can receive container image 602 from any suitable source in any suitable manner. In some embodiments, container image 602 can be pushed from a container runtime system to storage system 504 via a push operation, for example.

[0357] Storage system 504 may receive container image 602 and store it as volume 604. This may include converting container image 602 to volume 604. The conversion may include opening each image layer in container image 602 and writing data from the image layers to an immutable volume snapshot. Specifically, in the illustrated example, image layer 3 may be converted to snapshot 3 of volume 604, image layer 2 may be converted to snapshot 2 of volume 604, and image layer 1 may be converted to snapshot 1 of volume 604. Snapshots may be defined to have relationships that correspond to relationships between image layers of container image 602. For example, snapshot 3 may depend on and reference snapshot 2, and snapshot 2 may depend on and reference snapshot 1. Relationships between snapshots may be defined in any suitable manner, including by snapshots referencing other snapshots, metadata indicating relationships between snapshots, and / or using indexes or naming conventions indicating relationships. In the illustrated example, all snapshots are associated with volume 604 (e.g., are considered snapshots of volume 604). In some examples, volume 604 may be considered to be volume snapshot 3, which references volume snapshot 2, and volume snapshot 2 references volume snapshot 1. Storage system 504 can store any metadata that can be used to identify volume 604 as being associated with container image 602 (eg, identifying that volume 604 should be provided in response to a request for container image 602 ).

[0358] As used herein, a volume (e.g., volume 604) may be an atomic storage unit that can be mapped to physical storage, such as a logical drive or disk. A volume is an identifiable data storage unit that is configured to be managed by the storage system 504 (e.g., by applying storage operations to the volume) and accessed by clients.

[0359] Where container image 602 is stored as volume 604, storage system 604 may apply one or more storage services of storage system 604 to volume 604, including one or more of the storage services described herein. For example, volume 604 may be the subject of storage service operations associated with logical mapping (e.g., attaching, mounting, etc.), deduplication, compression, replication, snapshots, cloning, obsolete collection, migration, disaster recovery, etc.

[0360] Where container image 602 is stored as volume 604, storage system 604 can prepare to provide volume 604 to container system 502 for use by container system 502 to run a container instance of container image 602. In some examples, storage system 504 can provide volume 604 to container system 502 in response to a request for container image 602. The request can come from container system 502, such as from orchestration system 508 or container runtime system 510. Storage system 504 can detect the request for container image 602 in any suitable manner, such as by detecting a fetch request or a run request associated with container image 602 (e.g., a request to run a container instance of container image 602).

[0361] Storage system 604 may provide volume 604 to container system 502 in any manner that allows container system 502 to run a container instance of container image 602. In some examples, the manner in which storage system 604 provides volume 604 to container system 502 may depend on the configuration of container system 502, such as the configuration of orchestration system 508 or container runtime system 510. In other examples, storage system 604 may provide volume 604 to container system 502 in any suitable manner, and how container system 502 uses volume 604 may depend on the configuration of container system 502, such as the configuration of orchestration system 508 or container runtime system 510.

[0362] In some embodiments, storage system 504 may provide volume 604 to container system 502 by attaching and mounting volume 604 on a node on which the container instance will run. For example, storage system 504 may detect a request for container image 602 on a node, identify volume 604 as being associated with container image 602, and provide volume 604 by attaching and mounting volume 604 on the node, such as by attaching and mounting a virtual volume that maps to volume 604.

[0363] In some instances where container image 602 includes multiple image layers represented as snapshots of volume 604 in storage system 504, mounting of volume 604 may include storage system 504 mounting the snapshots as individual volumes in an order representing the hierarchical relationship of the image layers to each other in container image 602. In other instances where container image 602 includes multiple image layers represented as snapshots of volume 604 in storage system 504, mounting of volume 604 may include storage system 504 mounting a virtual volume mapped to volume 604 in any suitable manner, such as by mapping to the highest level snapshot (snapshot 3) of volume 604.

[0364] With volume 604 mounted on the node, the container system 502 may access volume 604 and read container image data from volume 604. For example, the container runtime system 510 may read container image data from volume 604 and use the container image data to run a container instance of container image 602. In some embodiments, the container runtime system 510 may do this by reading container image 602 from volume 604 and loading container image 602 into the node's local storage (e.g., using a container extraction operation). The container runtime system 510 may then use the container image 602 in the node's local storage to run the container instance of container image 602 on the node. In other embodiments, the container runtime system 510 may not load the container image 602 into the node's local storage, but may access and use the container image 602 in the mounted volume 604 to run the container instance of container image 602 on the node. In such other embodiments, resources of the node may be conserved, such as by not having to use the node's local storage to store the container image 602. This may allow a node to be provisioned with fewer local storage resources than would be required to support a conventional container runtime environment and / or to use local storage resources for other purposes. By not loading container image data into the node's local storage, container startup time may be reduced compared to a conventional container runtime environment. Additionally, the amount of data fetched onto the node may be reduced, which may conserve network resources and further reduce container startup time. This savings in resource usage and startup time may be significant when performing a restart operation (e.g., when a node is restarted) and / or when a container instance is moved from a source node to one or more other nodes (e.g., due to an outage on the source node).

[0365] As described above, in some embodiments, the storage system 504 may be configured to provide persistent data storage for container instances that are running or to be run in the container system 502. In some examples, whether the storage system 504 provides persistent data storage for a container instance may depend on the needs of the container instance. If the container instance will not use persistent storage, the container system 502 may not provide a request for persistent storage for the container instance to the storage system 504, and the storage system 504 may not provide persistent storage for the container instance. On the other hand, if the container instance is to use persistent storage, the container system 502 may provide a request for persistent storage for the container instance to the storage system 504, and the storage system 504 may provide persistent storage for the container instance.

[0366] The storage system 504 can be configured to provide persistent storage for container instances in any suitable manner. In some embodiments, the storage system 504 provides volumes to which container instances can write persistent data. The storage system 504 can attach and mount volumes on nodes for access by container instances running on the nodes.

[0367] In some embodiments, volumes provided for persistent storage may be provided in addition to volumes 604 containing data representing container images 602. Thus, volumes 604 may provide container images 602 (in the form of volume snapshots) for use by container system 502 to run container instances, and additional volumes may provide persistent storage to which running container instances may write persistent data and from which running container instances may read persistent data. Fig. 7A The storage system 504 is illustrated as providing to the container system 502 1) a volume 604 containing data representing the immutable container image 602 and 2) an instance of an additional separate volume 702 configured to provide persistent storage.

[0368] In some other embodiments, the volume provided for persistent storage may be included in and / or provided with a volume containing an immutable container image. In such an embodiment, for example, the storage system 504 may provide a single volume to the container system 502, the volume including an immutable layer including data representing the container image (this immutable layer may include multiple sublayers representing multiple snapshots) and a writable layer configured to provide persistent storage for use by the container instance. Figure 7B The storage system 504 is illustrated as providing an instance of a volume 704 to the container system 502, and the volume 704 includes an immutable layer 706 and a writable layer 708. The immutable layer 706 includes data representing the immutable container image 602 (e.g., volume 604, which may include multiple sub-layers representing multiple volume snapshots). The writable layer 708 may include one or more writable volumes (e.g., writable volume 710) configured for storing persistent data for the container instance.

[0369] The storage system 504 may be configured to create a volume 704 including a writable layer 708 and an immutable layer 706 at any suitable time. For example, the storage system 504 may create the volume 704 in response to receiving one or more requests for a container image 602 and a persistent data store from the container system 502. In other examples, the storage system 504 may create the volume 704 before receiving one or more requests for a container image 602 and a persistent data store from the container system 502.

[0370] The storage system 504 may be configured to create a volume 704 including a writable layer 708 and an immutable layer 706 in any suitable manner. For example, the volume 704 may be created to represent a hierarchical chain of volumes that are dependent on each other in a manner corresponding to the dependencies that the container system 502 will use to run a container instance of the container image 602. In some embodiments, for example, the storage system 504 may provide a writable volume 710 that is defined as dependent on the volume 604 (e.g., the writable volume 710 is a writable layer on top of the volume 604). The volume 704 may be the writable volume 710 or an additional volume that is defined as dependent on the writable volume 710. The writable volume 710 may include one or more individual writable volumes located on top of the volume 604. Thus, the volume 704 may define a stack of dependent volumes that the container system 502 may use to run a container instance of the container image 602 and read and write persistent data while running the container instance. The stacking path of volume 704 can mirror the stacking path of the container instance. Volume 704 can be attached and mounted as a single point from which the entire container instance can be started and run.

[0371] The container system 502 (e.g., the container runtime system 510) can map any area in the container instance to a volume in the storage system 504. For example, the container system 502 can attach the namespace of the volume to a location in the container instance. Thus, the container system 502 can access and use the mounted volumes, such as volume 704 or volumes 604 and 702, to run the container instance from the container image 602 and read and write persistent data. The container system 502 can provide a temporary layer, which can be a layer to which temporary data can be read and written while the container instance is running. The temporary layer can be a layer that is located on top of the volume provided by the storage system 504.

[0372] In embodiments where a single volume, such as volume 704, contains writable and immutable container data in one place, the container system 502 (e.g., container runtime system 510) may not have to request immutable container data and persistent storage separately from different sources. For example, the container system 502 may request a container instance running a container image, such as by requesting a volume associated with the container image. In response, the storage system 504 may provide a single volume containing writable and immutable container data in one place.

[0373] Figures 8 to 11 A flowchart depicting an example method is described. Although the flowchart depicts illustrative operations according to some embodiments, other embodiments may omit, add, reorder, combine, and / or modify any of the operations shown in the flowchart. In some implementations, one or more operations shown in the flowchart may be performed by a storage system (e.g., storage system 504), a container system (e.g., container system 400 or 502), any component of a storage system and / or container system, and / or any implementation of a storage system and / or container system.

[0374] like Figure 8 , example method 800 includes: receiving an immutable container image at 802; storing the container image as a volume at 804; detecting a request for the container image from a container system at 806; providing the volume to the container system at 808; and using the volume to run a container instance in the container system at 810. The operations of method 800 may be performed in any manner described herein.

[0375] In some embodiments, the storing at 804 of method 800 may be performed as follows: Fig. 9 Execute as described in Fig. 9 , example method 900 includes: at 902, opening an image layer in an immutable container image (e.g., the immutable container image received at 802 in method 800); at 904, converting the image layer to a volume snapshot; at 906, storing the volume snapshot; and at 908, storing metadata for the volume snapshot. The operations of method 900 may be performed in any manner described herein. The metadata stored at 908 may include any data associated with the volume snapshot, including metadata associating the volume snapshot with a container image (e.g., metadata indicating an identifier of a container image and / or container image layer associated with the volume snapshot), any tags associated with the container image and / or container image layer associated with the volume snapshot; an image hash of the container image and / or container image layer associated with the volume snapshot, a time associated with receiving the container image and / or container image layer associated with the volume snapshot, an identifier of the volume snapshot, one or more relationships to one or more other volume snapshots (e.g., a reference from one snapshot to another snapshot), and the like.

[0376] After 908, method 900 further includes determining whether another image layer exists in the container image at 910. If another layer exists, method 900 is repeated again starting with 902. If another layer does not exist, method 900 ends.

[0377] like Fig.10, an example method 1000 includes: at 1002, receiving a request for a container image from a container system; at 1004, identifying a volume corresponding to the container image; at 1006, providing the volume to the container system; at 1008, receiving a request for persistent storage from the container system; and at 1010, providing the persistent storage volume to the container system 1010. The operations of method 1000 may be performed in any manner described herein, including as described in reference to Fig. 7A The requests received at 1002 and 1008 may be received in parallel or sequentially in any suitable order.

[0378] like Fig.11 , an example method 1100 includes: at 1102, receiving a request for a volume that contains a container image and provides persistent storage from a container system; at 1104, providing the volume to the container system; at 1106, running a container instance in the container system using the volume and writing persistent data to the container instance. The operations of method 1100 may be performed in any manner described herein, including as described in reference to Figure 7B describe.

[0379] As described herein, a volume may be specified to include one or more container images. Such volumes may be used by a container system, a container runtime, a container orchestrator, and / or a storage system to efficiently deploy or redeploy containers among nodes of a cluster. Deploying containers using volumes may benefit from the advantages of using volumes, such as accessing data through volume mount points and accessing container images associated with the volume via a storage area network.

[0380] In some embodiments, the disclosed volumes may include data storage in addition to one or more container images. The data storage may be data generated during operation of one or more containers associated with the one or more container images within the data volume. In this way, the same volume may include both container images and persistent and / or temporary storage of data generated during operation of container instances based on the container images. In addition to or in lieu of data generated during operation of the one or more containers, the data storage may also include configuration data available to one or more containers associated with the container images within the volume.

[0381] As described in more detail below, advantages of using volumes that include container images include the container runtime mounting the volumes on the nodes and accessing the container images. The container runtime, based on accessing the container images via the mounted volumes, can create instances of containers associated with the container images without having to copy the entire container image to the nodes. Other advantages can include faster spin-up times when creating container instances on a given node of the cluster, and in some embodiments, the ability to use the same volume that includes the container image as a storage device available to the container. One of more of these features can allow containers to be efficiently deployed and / or redeployed (e.g., spun up / down) among the nodes of the container system cluster.

[0382] In a container system, a container orchestrator may deploy one or more containers among one or more nodes of a cluster. In some embodiments, the container orchestrator may be operable to deploy and / or redeploy containers within a cluster without knowledge of volumes associated with container images. In other embodiments, the container orchestrator may be operable to deploy and / or redeploy containers within a cluster based on knowledge of one or more volumes storing container images.

[0383] In some instances, after initial deployment, containers may fail and / or be redeployed for various reasons. As one example of a container orchestrator determining container redeployment based on a metric regarding load balancing among cluster nodes, the container orchestrator may determine to move a container operating on a first node to a second node. In other instances, the container orchestrator may redeploy containers using reasons such as performance of containerized applications, security threats, system upgrades, increase or decrease in computing resources, failure of nodes, etc., such as by spinning down containers on one node and spinning up containers on one or more other nodes.

[0384] Failure or redeployment of containers may occur frequently enough that faster movement of containers among nodes of a container system may result in significant improvements in container system performance or resource utilization. The disclosed volumes including container images may be the basis for faster deployment of containers. Deployment or redeployment may be faster based on local access to a storage system storing volumes associated with container images. In contrast, conventional systems may access container images via an Internet access repository.

[0385] Furthermore, because the disclosed volumes can use logical structures and application programming interfaces that are compatible with existing components of the container system that use the storage system, the amount of modification of components of the container system that use the disclosed volumes can be reduced. For example, the container runtime can have an API to communicate with the storage system to identify, mount, read, write, or otherwise modify the characteristics of data associated with the volume.

[0386] As another example, accessing a container image associated with a volume based on a container runtime can avoid bursts of network traffic within a container system similar to accessing storage content within a volume. For example, network traffic can be managed by accessing portions of a container image based on a container runtime during scheduled time periods, rather than copying the entire container image from a remote network location to local storage of a node that creates a container instance.

[0387] Fig. 12A An example storage system 1202 is illustrated for storing a volume 1204 including a container image according to some embodiments. The storage system 1202 may be similar to the example storage system 1202 described above. Figures 1A to 11 The invention may be implemented in one or more of the storage systems described.

[0388] Volume 1204 may include volume metadata 1206 and container image 1210. In this example, volume metadata 1206 includes container metadata 1208 that describes one or more layers 1212 of container image 1210. Metadata 1206 may describe a structured collection of data objects that represent logical relationships between layers of container image 1210. In certain embodiments, different types of logical relationships may be implemented. In this example, the logical relationships between image layers 1212 may be structured as a directed acyclic graph.

[0389] In certain implementations, dependencies between image layers 1212 may be represented by directed links between data objects associated with image layers 1212. For example, image layer 1212-1 may be a base layer that does not depend on another image layer, image layer 1212-2 may depend on image layer 1212-1, and image layer 1212-3 may depend on image layer 1212-2. Dependencies between image layers may include one or more references from a first dependent image layer to a second image layer.

[0390] The one or more layers 1212 may be as described above. Figure 6 In this example, container image 1210 includes image layers 1212-1 to 1212-3. In other examples, such as Fig. 12B As described, volume metadata 1206 may include container metadata 1208 and storage metadata.

[0391] Fig. 12B An instance storage system 1202 is illustrated storing a volume 1204 including a container image, according to some embodiments.

[0392] As about Fig. 12A1204. As depicted, volume 1204 may include volume metadata 1206 and container image 1210. In this example, volume 1204 additionally includes storage metadata 1220 and container storage 1222-1 to 1222-M. In certain embodiments, the container runtime may mount a first volume that provides access to container image 1210, mount a second volume (e.g., volume 1224) that provides access to container storage 1221-1, and mount an Mth volume (e.g., volume 1228) that provides access to container storage 1222-M. In this example, the mounted volumes associated with container storage 1221 may be used to store data during container operation. In the event that a container is redeployed, fails, or otherwise becomes unavailable, data stored in volumes associated with container storage 1221 may be persisted.

[0393] The storage metadata 1220 may describe a data set associated with the volume 1224 and, in an example, one or more snapshots 1226 of the data set. The storage metadata 1220 may represent the volume 1224 based on a directed acyclic graph or in any other suitable manner.

[0394] In certain embodiments, volume 1224 may be specified by storage metadata 1220 as a directed acyclic graph, such as a B-tree, where the leaves of the storage metadata 1220 representation may contain pointers to stored data objects. Volume addresses or volume references and offsets may be used to navigate the storage metadata 1220 representation of volume 1224. For example, the storage metadata 1220 representation of volume 1224 may include data objects as leaf nodes, and intermediate nodes may be references to leaf nodes. Volume 1224 may include an aggregation of data objects at the leaf nodes of storage metadata 1220, where the organization of the data objects is specified by the intermediate nodes. Data objects may be logical extents, where the logical extents may be any specified size, such as 1MB, 4MB, or some other size.

[0395] Snapshot 1226 may be specified by a metadata representation similar to the metadata representation of a volume. For example, snapshot 1226 may be a directed acyclic graph in which snapshot 1226 includes references and data objects associated with one or more changes to a data set of volume 1224. In addition to one or more references to the metadata representation of volume 1224 associated with data objects that have not been changed by the one or more changes associated with a given snapshot, the given snapshot may also include data objects associated with one or more changes to the volume 1224 data set. Based on the given snapshot's reference to the volume 1224 data set, the given snapshot may be considered to be dependent on volume 1224.

[0396] As depicted in this example, storage metadata 1220 describes container storage 1222-1 through 1222-M. Container storage 1222-1 represents volume 1224 and snapshots 1226-1 and 1226-2 of volume 1224. Snapshot 1226-1 depends on volume 1224, and snapshot 1226-2 depends on snapshot 1226-1. Container storage 1222-M represents volume 1228 and snapshot 1230-1 of volume 1228. Snapshot 1230-1 depends on volume 1228.

[0397] As about Figures 12A to 12B Description, describing various characteristics of a volume including one or more container images, and in some examples, describing one or more storage volumes. Figures 13 to 15 As described, volumes including one or more container images may be implemented in various computing environment architectures. In various architectures, container orchestrators, container runtimes, and / or storage systems may have different roles and characteristics in deploying or redeploying containers among nodes of a container system. Fig.13 The first example described describes a storage system that is implemented independently of a container system. In this example, the storage system can provide volumes including container images to a container runtime or container orchestrator operating within the container system. Fig.14 The second example described describes a cluster including a storage system, such as Figures 5 to 11 In this example, the cluster and the storage system operate within a cloud environment provided by the same cloud service provider and utilize storage resources that are external to the cluster and provided by the cloud service provider. In other embodiments, the storage resources may be implemented in the cluster. Fig.15 The third example described describes a container system in which a container runtime uses volumes to access container images, and in which the volumes are obtained from a cloud-based object storage that operates as a repository for volumes including container images. In the following figures, components with reference numbers described with respect to previous figures are similarly implemented.

[0398] Fig.13 An example computing environment 1300 is illustrated using a volume 1204 including a container image 1210 according to some embodiments.

[0399] In computing environment 1300, a container orchestrator, such as Figure 5The described orchestration system 508 can deploy or redeploy containers among the nodes 420 of the cluster 408. The container orchestrator can communicate information about the container deployment to the container runtime 1302 running on the node 420, wherein the container runtime can include a data volume module 1304 configured to interface with (e.g., interpret) a volume including a container image. Based on the received information, the container runtime 1302 can communicate with the storage system 1306 to request a container image 1210 associated with the container. The request can be made in any suitable manner, such as by sending a request for the container image 1210 and / or the volume 1204 to the storage system 1306, wherein the volume 1204 can be used to access the container image 1210. The storage system 1306 can generate, store and / or provide a volume including one or more container images. In this example, as described above, the storage system 1306 is external to the cluster 408 and is implemented external to and / or independently of the container system in which the cluster 408 operates.

[0400] In this example, the container runtime 1302 accesses the volume 1204 within the storage system 1306. The container runtime 1302 can access the container image 1210 associated with the volume 1204 to create an instance of the container image 1210 on the node 420-1 within the cluster 408. To create the instance of the container image 1210, the container runtime 1302 can request the volume 1204 from the storage system 1306. The container runtime 1302 can be implemented similarly to the container runtime 402, wherein the container runtime 1302 can include fewer, additional, or all of the features described above with respect to the container runtime 402. In this example, the container runtime 1302 includes a data volume module 1304 configured to interpret the volume including the container image and / or otherwise interface with the volume.

[0401] The data volume module 1304 can be configured to access one or more image layers of a container image from a volume 1204 including one or more container images 1210. The data volume module 1304 can implement the above-mentioned Figures 5 to 11 Any operation described, including creating a container image within a volume, accessing a container image within a volume, modifying a container image within a volume, moving a volume from one node to another within a container system, etc.

[0402] Similar to About Figures 5 to 11 1204, the storage system 1306 can use the identifier of the container image 1210 to select the volume 1204 associated with the identifier. The storage system 1306 can be implemented similarly to the storage systems 402 and 504, wherein the storage system 1306 can include fewer, additional, or all of the features described above with respect to the storage systems 402 and 504.

[0403] Continuing with this example, to access volume 1204, container runtime 1302 may send a request 1308-1 for volume 1204 to storage system 1306. Request 1308-1 may indicate an identifier of container image 1210. In response to request 1308-1, storage system 1306 may provide volume data 1310-1, where volume data 1310-1 is associated with volume 1204. Container runtime 1302 may use volume data 1310-1 to mount volume 1204 to access container image 1210. In some examples, volume data may be a network path or network location and credentials, such as a user name and / or password for an account associated with one or more storage resources. Based on mounting and accessing volume 1204, container runtime 1302 may create an instance of container image 1210 in response to deployment of an application associated with container image 1210.

[0404] In contrast, in conventional systems, a container image repository is used to store and retrieve container images that are not included as part of a volume. TM In some embodiments, the node 420 on which the container image 1210 is to be instantiated can be connected to the storage system 1306 via a storage area network. The storage area network can be Fibre Channel, iSCSI, NVMe, or some other storage area network implementation, such as Figures 1A to 1D Storage area network 158 is depicted. Accessing container images via a storage area network can avoid network activity that affects other container system processes that may use other types of network connections. In addition, accessing container images via a storage area network using volume data 1310-1 contrasts with traditional container runtimes that may access container images via a wide area network (e.g., the Internet) to reach a repository.

[0405] In some embodiments, the container orchestrator may deploy the same application on multiple nodes 420 of the cluster 408. For example, the deployment configuration file may specify a replication factor Z. Based on the replication factor Z, the container orchestrator may deploy Z instances of the container image 1210 associated with the application among the Z nodes 420. Figures 5 to 11 The description of the container runtime 402 mounts a volume based on the container runtime 1302 to access the container image, avoiding the network load generated by transmitting Z copies of the container image 1210.

[0406] In this example, to deploy the first instance of container image 1210, container runtime 1302 on node 420-1 issues request 1308-1 and receives volume data 1310-1. Figures 5 to 111, the container runtime 1302 on the node 420-1 may mount the volume 1204 using the volume data 1310-1 to access and use the container image 1210 to run the container instance on the node 420-1, thereby spinning up the container associated with the container image 1210 on the node 420-1. Similarly, to deploy the instance Z of the container image 1210, the container runtime 1302 on the node 420-N issues a request 1308-Z and receives the volume data 1310-Z. The container runtime 1302 on the node 420-N may mount the same volume 1204 using the volume data 1310-Z to access and use the container image 1210 to run the container instance on the node 420-N, thereby spinning up another container associated with the container image 1210 on the node 420-N.

[0407] Fig.14 An example computing environment 1400 is illustrated using a volume 1204 including a container image 1210 in accordance with some embodiments.

[0408] In computing environment 1400, similar to computing environment 1300, a container orchestrator may deploy or redeploy containers among nodes 420 of cluster 408. The container orchestrator may communicate information about the container deployment to a container runtime 1302 running on node 420. Based on the received information, container runtime 1302 may communicate with storage system 1406 to request a container image 1210 associated with the container. The request may be made in any suitable manner, such as by sending a request for container image 1210 and / or volume 1204 to storage system 1306, where volume 1204 may be used to access container image 1210. Storage system 1402 may generate, store, and / or provide a volume including one or more container images. However, in computing environment 1300, compared to computing environment 1300, in computing environment 1400, storage system 1402 may be implemented among one or more nodes 410 and 420 of cluster 408.

[0409] Storage system 1402 may be implemented similarly to storage systems 402 and 504 , wherein storage system 1402 may include fewer, additional, or all of the features described above with respect to storage systems 402 and 504 .

[0410] The deployment and redeployment of containers can be performed by a container orchestrator, container runtime 1302, and / or storage system 1402 in computing environment 1400, similar to the Fig.13 The container orchestrator, container runtime 1302, and / or storage system 1306 are described.

[0411] Fig.15An example computing environment 1500 is illustrated using a volume 1204 including a container image 1210 according to some embodiments.

[0412] In computing environment 1500, similar to computing environment 1300, a container orchestrator may deploy or redeploy containers among nodes 420 of cluster 408. The container orchestrator may communicate information about the container deployment to container runtime 1302 running on node 420. As described with respect to computing environment 1300, based on the received information, container runtime 1302 may communicate with storage system 1306 to request container image 1210. In contrast to the above example, in this example, container runtime 1502 may use volume data to directly or indirectly access cloud-based object repository 348 storing volume 1204. In some examples, cloud-based object repository 348 may be provided by a third-party cloud service provider, wherein cloud-based object repository 348 may be used as a repository for volumes that include container images but do not include features described with respect to container-aware storage system 504. In this example, volume data may be used to mount volumes, or may include volume metadata, including container metadata and / or storage metadata. Based on access to volume 1204, container runtime 1502 can access container image 1210 to create a container instance, as described in detail in relation to Fig.13 describe.

[0413] Continuing with this example, the container orchestrator can interpret the configuration file to determine that the container image is associated with a volume stored within a data repository, such as cloud-based object storage 348. For example, the configuration file can include identifiers associated with one or more container images within volume 1204. The configuration file can include information indicating the deployment of the container within the container system, including the number of rep...

Claims

1. A method comprising: identifying, by a recovery process, a set of immutable layers of a container image contained in a data set used by a container system to run a containerized application on a first cluster; copying the set of immutable layers of the container image to a second cluster via the recovery process; receiving, by the restore process, a restore request to restore the containerized application after copying the set of immutable layers of the container image to the second cluster; and A set of variable layers included in the data set is replicated to the second cluster in response to the restore request by the restore process, and the second cluster is configured to use the replicated set of immutable layers and the replicated set of variable layers to restore the containerized application on the second cluster.

2. The method of claim 1, wherein the set of immutable layers of the container image comprises container image layers accessed and used by one or more compute nodes of the first cluster to run the containerized application on the first cluster. 3 . The method of claim 1 , wherein the set of mutable layers is provided by a storage system as persistent storage for the containerized application running on the first cluster. 4 . The method of claim 1 , wherein replicating the set of immutable layers contained in the data set to the second cluster comprises prioritizing more frequently used immutable layers over less frequently used immutable layers. 5 . The method of claim 1 , wherein replicating the set of variable layers contained in the data set to the second cluster comprises prioritizing more frequently used variable layers over less frequently used variable layers.

6. The method of claim 1 , wherein replicating the set of variable layers contained in the data set to the second cluster in response to the restore request comprises identifying and replicating only variable layers in the set of variable layers that have changed since the set of variable layers was last replicated to the second cluster.

7. The method of claim 1, wherein the set of immutable layers comprises only a unique immutable layer in the data set.

8. The method of claim 1, wherein the data set is managed by a container-aware storage system that provides the immutable layer and the mutable layer for use by the container system to run the containerized application on the first cluster.

9. The method of claim 8, wherein the container-aware storage system manages and provides the immutable layer and the mutable layer as volumes.

10. The method of claim 8, wherein the container-aware storage system comprises a containerized instance of the container-aware storage system running on a compute node of the first cluster.

11. The method of claim 8, wherein the recovery process is included in the container-aware storage system.

12. The method according to claim 11, further comprising: determining, by the container-aware storage system, an interruption of one or more storage services provided by the container-aware storage system; and An alert is provided from the container-aware storage system to a container orchestrator for the container system based on the interruption.

13. The method according to claim 1, further comprising: detecting a new or updated immutable layer in the set of immutable layers included in the data set; and The new immutable layer or the updated immutable layer is copied to the second cluster.

14. The method of claim 1, further comprising: receiving a restore request to restore the containerized application back to the first cluster; and Any of the variable layers that have changed on the second cluster are identified and copied from the second cluster to the first cluster.

15. A system comprising: one or more memories storing computer-executable instructions; and one or more processors configured to execute the computer executable instructions to: identifying a set of immutable layers of a container image contained in a data set used by a container system to run a containerized application on a first cluster; copying the set of immutable layers of the container image to a second cluster; receiving a restore request to restore the containerized application after copying the set of immutable layers of the container image to the second cluster; and In response to the restore request, a set of variable layers included in the data set is replicated to the second cluster, the second cluster being configured to use the replicated set of immutable layers and the replicated set of variable layers to restore the containerized application on the second cluster.

16. The system of claim 15, wherein: the set of immutable layers of the container image comprising container image layers accessed and used by one or more compute nodes of the first cluster to run the containerized application on the first cluster; and The set of mutable layers is provided by a storage system as persistent storage for the containerized application running on the first cluster.

17. The system of claim 15, wherein the data set is managed by a container-aware storage system that provides the immutable layer and the mutable layer for use by the container system to run the containerized application on the first cluster.

18. The system of claim 17, wherein the container-aware storage system manages and provides the immutable layer and the mutable layer as volumes.

19. The system of claim 15, wherein the one or more processors are further configured to execute the computer executable instructions to: receiving a restore request to restore the containerized application back to the first cluster; and Any of the variable layers that have changed on the second cluster are identified and copied from the second cluster to the first cluster.

20. A non-transitory computer-readable medium storing computer instructions that, when executed, direct one or more processors of one or more computing devices to: identifying a set of immutable layers of a container image contained in a data set used by a container system to run a containerized application on a first cluster; copying the set of immutable layers of the container image to a second cluster; receiving a restore request to restore the containerized application after copying the set of immutable layers of the container image to the second cluster; and In response to the restore request, a set of variable layers included in the data set is replicated to the second cluster, the second cluster being configured to use the replicated set of immutable layers and the replicated set of variable layers to restore the containerized application on the second cluster.

Images