Dynamic plane selection in data storage system

Abstract

The invention relates to dynamic plane selection in a data storage system. A storage system is provided. The storage system includes a plurality of non-volatile memory modules and a storage system controller. The one or more non-volatile memory modules include a multi-planar die. A processing device of the storage system controller is configured to determine a number of planes for accessing data while the multi-plane die should be changed. In response to determining that a number of the planes of the multi-plane die used to access data at the same time should be changed, the processing device is configured to move one or more portions from an existing erase block to a new erase block, the size of the existing erase block being different from the size of the new erase block.

Description

Technical Field

[0001] The present application relates to dynamic plane selection in data storage systems. Background Art

[0002] Solid-state memory, such as flash, is currently used in solid-state drives (SSDs) to enhance or replace traditional hard disk drives (HDDs), writable CD (compact disk) or writable DVD (digital versatile disk) drives (collectively referred to as rotating media), and tape drives for storing large amounts of data. Flash and other solid-state memories have different characteristics than rotating media. However, for compatibility reasons, many solid-state drives are designed to conform to hard disk drive standards, which makes it difficult to provide enhanced features or take advantage of the unique aspects of flash and other solid-state memories.

[0003] It is against this background that the embodiments emerge. Summary of the invention

[0004] An embodiment of the present disclosure provides a storage system, comprising: a plurality of non-volatile memory modules, one non-volatile memory module comprising a multi-plane die; a storage system controller operably coupled to the plurality of storage devices, the storage system controller comprising a processing device, the processing device being configured to: determine that the number of planes for accessing data while the multi-plane die should be changed; in response to determining that the number of planes for accessing data while the multi-plane die should be changed, allocate blocks using a new block size by combining erase block sets at the same address in separate planes based on the newly determined number of planes.

[0005] Another embodiment of the present disclosure provides a method, comprising: determining that the number of planes used to access data simultaneously in a multi-plane die should be changed, wherein the multi-plane die is part of a non-volatile memory module among multiple non-volatile memory modules of a storage system; in response to determining that the number of planes used to access data simultaneously in the multi-plane die should be changed, allocating blocks using a new block size by combining erase block sets at the same address in separate planes based on the newly determined number of planes.

[0006] Yet another embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing instructions which, when executed, cause a processing device of a storage system controller to perform the following operations: determine that the number of planes used to access data simultaneously of a multi-plane die should be changed, the multi-plane die being part of one of a plurality of non-volatile memory modules of a storage system; in response to determining that the number of planes used to access data simultaneously of the multi-plane die should be changed, allocate blocks using a new block size by combining erase block sets at the same address in separate planes based on the newly determined number of planes. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The described embodiments and their advantages may be best understood by referring to the following description in conjunction with the accompanying drawings, which in no way limit any changes in form and detail that may be made to the described embodiments by those skilled in the art without departing from the spirit and scope of the described embodiments.

[0008] The present disclosure is illustrated by way of example and not limitation and may be more fully understood with reference to the following detailed description when considered in conjunction with the accompanying drawings as described below.

[0009] Figure 1A A first example system for data storage is shown in accordance with some embodiments.

[0010] Figure 1B A second example system for data storage is shown in accordance with some embodiments.

[0011] Figure 1C A third example system for data storage is shown in accordance with some embodiments.

[0012] Figure 1D A fourth example system for data storage is shown in accordance with some embodiments.

[0013] Figure 2A is a perspective diagram of a storage cluster having a plurality of storage nodes and an internal storage device coupled to each storage node to provide network attached storage according to some embodiments.

[0014] Figure 2B is a block diagram illustrating an interconnect switch coupling a plurality of storage nodes, according to some embodiments.

[0015] Figure 2C is a multi-level block diagram illustrating the contents of a storage node and the contents of one of the non-volatile solid-state storage units, according to some embodiments.

[0016] Figure 2D A storage server environment according to some embodiments is shown, which uses embodiments of the storage nodes and storage units of some previous figures.

[0017] Figure 2E is a block diagram of blade hardware according to some embodiments, showing the control plane, compute and storage planes, and authorities interacting with underlying physical resources.

[0018] Figure 2F Depicted is a resiliency software layer in a blade of a storage cluster according to some embodiments.

[0019] Figure 2GAuthorization and storage resources in blades of a storage cluster are depicted according to some embodiments.

[0020] Figure 3A A diagram of a storage system coupled for data communication with a cloud service provider is set forth in accordance with some embodiments of the present disclosure.

[0021] Figure 3B A diagram of a storage system according to some embodiments of the present disclosure is illustrated.

[0022] Figure 3C An example of a cloud-based storage system according to some embodiments of the present disclosure is set forth.

[0023] Figure 3D An exemplary computing device 350 is shown that may be specifically configured to perform one or more of the processes described herein.

[0024] Figure 3E An example of a series of storage systems 376 for providing storage services is shown.

[0025] Figure 3F An example container system is shown.

[0026] Figure 4 is a block diagram illustrating an example storage system according to some embodiments of the present disclosure.

[0027] Figure 5 is a block diagram illustrating an example non-volatile memory module according to some embodiments of the present disclosure.

[0028] Fig. 6A is a block diagram illustrating an example erase block according to some embodiments of the present disclosure.

[0029] Figure 6B is a block diagram illustrating an example erase block according to some embodiments of the present disclosure.

[0030] Figure 7 is a flow chart illustrating a method for performing a storage operation according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0031] The following embodiments describe a storage cluster that stores user data, such as user data from one or more users or client systems or other sources outside the storage cluster. The storage cluster distributes user data between storage nodes contained in a chassis using erasure coding and redundant copies of metadata. Erasure coding is a data protection method in which data is broken down into fragments, extended and encoded with redundant data blocks, and stored in a set of different locations, such as disks, storage nodes, or geographic locations. Flash memory is a solid-state memory that can be integrated with an embodiment, but the embodiment can be extended to other types of solid-state memory or other storage media, including non-solid-state memory. The control of storage locations and workloads is distributed on storage locations in a cluster peer system. Tasks such as regulating communication between storage nodes, detecting when a storage node is unavailable, and balancing I / O (input and output) on each storage node are all processed on a distributed basis. In some embodiments, data is arranged or distributed on multiple storage nodes in the form of data fragments or stripes that support data recovery. The ownership of data can be reassigned within the cluster, regardless of the input and output mode. This architecture, described in more detail below, allows a storage node in a cluster to fail, but the system remains operational because data can be rebuilt from other storage nodes, thereby remaining available for input and output operations.In various embodiments, a storage node may be referred to as a cluster node, a blade, or a server.

[0032] The storage cluster is contained in a chassis, that is, a housing that contains one or more storage nodes. The chassis contains a mechanism for supplying power to each storage node, such as a power distribution bus, and a communication mechanism, such as a communication bus that enables communication between storage nodes. According to some embodiments, the storage cluster can be operated as a standalone system in one location. In one embodiment, the chassis accommodates at least two instances of power distribution and internal and external communication buses that can be independently enabled or disabled. The internal communication bus can be an Ethernet bus, but other technologies, such as peripheral component interconnect (PCI) high speed, InfiniBand, etc., are also applicable. The chassis provides a port for an external communication bus for communicating between multiple chassis and with client systems directly or through a switch. External communication can use technologies such as Ethernet, InfiniBand, Fibre Channel, etc. In some embodiments, the external communication bus uses different communication bus technologies for inter-chassis and client communication. If a switch is deployed in a chassis or between chassis, the switch can act as a conversion between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, clients can access the storage cluster using proprietary interfaces or standard interfaces such as Network File System (NFS), Common Internet File System (CIFS), Small Computer System Interface (SCSI), or Hypertext Transfer Protocol (HTTP). Conversion from the client protocol can occur at the switch, at the chassis external communication bus, or within each storage node.

[0033] Each storage node may be one or more storage servers, and each storage server is connected to one or more non-volatile solid-state memory units, which may be referred to as storage units. One embodiment includes a single storage server in each storage node, and one to eight non-volatile solid-state memory units, but this example is not intended to be limiting. The storage server may include a processor, a dynamic random access memory (DRAM), and an interface for an internal communication bus and power distribution for each power bus. In some embodiments, inside the storage node, the interface and the storage unit share a communication bus, such as PCI Express. The non-volatile solid-state memory unit may directly access the internal communication bus interface through the storage node communication bus, or request the storage node to access the bus interface. The non-volatile solid-state memory unit accommodates an embedded central processing unit (CPU), a solid-state storage controller, and a certain number of solid-state mass storage devices, such as between 2-32 terabytes (TB) in some embodiments. The non-volatile solid-state memory unit includes an embedded volatile storage medium such as DRAM and an energy storage device. In some embodiments, the energy storage device is a capacitor, a supercapacitor, or a battery, which can transfer a subset of the DRAM content to a stable storage medium in the event of a power outage. In some embodiments, the non-volatile solid-state memory cells are comprised of storage class memories such as phase change or other resistive random access memory (RRAM) or magnetoresistive random access memory (MRAM), which can replace DRAM and achieve device retention power reduction.

[0034] In some embodiments, the storage node has one or more non-volatile solid-state storage units, each of which has a non-volatile random access memory (NVRAM) and a flash memory. The NVRAM and the flash memory can be independently addressed by the storage node, or more specifically, by the processor of the storage node. The storage node writes user data to the NVRAM for storage in the flash memory. The storage node, such as the processor of the storage node, also writes metadata to the NVRAM, which can be used as a work area for the processor. Direct memory access (DMA) is used to transfer user data from the NVRAM to the flash memory to store the user data. DMA is also used to transfer the contents of the NVRAM to the flash memory in the event of a power outage. In some embodiments, a controller in the non-volatile solid-state storage device manages a mapping or conversion table and performs various duties to manage user data in the flash memory.

[0035] Example methods, devices, and products for a configurable storage system according to embodiments of the present disclosure are described with reference to the drawings and the following disclosure.

[0036] Figure 1AAn example system for data storage according to some embodiments is shown. For purposes of illustration and not limitation, system 100 (also referred to herein as a "storage system") includes a number of elements. It may be noted that system 100 may include the same, more, or fewer elements configured in the same or different manner in other embodiments.

[0037] The system 100 includes a plurality of computing devices 164A-B. The computing devices (also referred to herein as "client devices") may be embodied as, for example, servers in a data center, workstations, personal computers, notebooks, etc. The computing devices 164A-B may be coupled to communicate data with one or more storage arrays 102A-B via a storage area network ('SAN') 158 or a local area network ('LAN') 160.

[0038] SAN 158 may be implemented with a variety of data communication structures, devices, and protocols. For example, the structure of SAN 158 may include Fibre Channel, Ethernet, InfiniBand, Serial Attached Small Computer System Interface ('SAS'), etc. Data communication protocols for SAN 158 may include Advanced Technology Attachment ('ATA'), Fibre Channel Protocol, Small Computer System Interface ('SCSI'), Internet Small Computer System Interface ('iSCSI'), HyperSCSI, Fabric-based Non-Volatile Memory Express ('NVMe'), etc. It may be noted that SAN 158 is provided for illustration and not limitation. Other data communication couplings may be implemented between computing devices 164A-B and storage arrays 102A-B.

[0039] LAN 160 may also be implemented with various structures, devices and protocols. For example, the structure of LAN 160 may include Ethernet (802.3), wireless (802.11), etc. The data communication protocols used for LAN 160 may include transmission control protocol ('TCP'), user datagram protocol ('UDP'), Internet protocol ('IP'), hypertext transfer protocol ('HTTP'), wireless access protocol ('WAP'), handheld device transfer protocol ('HDTP'), session initiation protocol ('SIP'), real-time protocol ('RTP'), etc.

[0040] The storage arrays 102A-B may provide persistent data storage for the computing devices 164A-B. In some embodiments, the storage array 102A may be housed in a chassis (not shown) and the storage array 102B may be housed in another chassis (not shown). The storage arrays 102A and 102B may include one or more storage array controllers 110A-D (also referred to herein as "controllers"). The storage array controllers 110A-D may be embodied as modules of automated computing machinery, including computer hardware, computer software, or a combination of computer hardware and software. In some embodiments, the storage array controllers 110A-D may be configured to perform various storage tasks. Storage tasks may include writing data received from computing devices 164A-B to storage arrays 102A-B, erasing data from storage arrays 102A-B, retrieving data from storage arrays 102A-B, and providing data to computing devices 164A-B, monitoring and reporting storage device utilization and performance, performing redundancy operations (e.g., redundant array of independent drives ('RAID') or RAID-like data redundancy operations), compressing data, encrypting data, and the like.

[0041] The storage array controllers 110A-D may be implemented in various ways, including as a field programmable gate array ('FPGA'), a programmable logic chip ('PLC'), an application specific integrated circuit ('ASIC'), a system on a chip ('SOC'), or any computing device including discrete components such as a processing device, a central processing unit, a computer memory, or various adapters. The storage array controllers 110A-D may include, for example, a data communication adapter configured to support communication via a SAN 158 or a LAN 160. In some embodiments, the storage array controllers 110A-D may be independently coupled to a LAN 160. In some embodiments, the storage array controllers 110A-D may include an I / O controller, etc., which couples the storage array controllers 110A-D to communicate data with permanent storage resources 170A-B (also referred to herein as "storage resources") via a midplane (not shown). The permanent storage resources 170A-B may include any number of storage drives 171A-F (also referred to herein as "storage devices") and any number of non-volatile random access memory ('NVRAM') devices (not shown).

[0042] In some embodiments, the NVRAM device of the permanent storage resource 170A-B can be configured to receive data from the storage array controller 110A-D to be stored in the storage drive 171A-F. In some examples, the data can be derived from the computing device 164A-B. In some examples, writing data to the NVRAM device can be implemented faster than writing data directly to the storage drive 171A-F. In some embodiments, the storage array controller 110A-D can be configured to use the NVRAM device as a fast access buffer for data that is predetermined to be written to the storage drive 171A-F. Relative to the system in which the storage array controller 110A-D directly writes data to the storage drive 171A-F, the latency of the write request using the NVRAM device as a buffer can be improved. In some embodiments, the NVRAM device can be implemented with a computer memory in the form of a high-bandwidth, low-latency RAM. The NVRAM device is called "non-volatile" because the NVRAM device can receive or contain a unique power supply that maintains the state of the RAM after the main power supply of the NVRAM device is powered off. Such a power supply can be a battery, one or more capacitors, and the like. In response to a power loss, the NVRAM device may be configured to write the contents of the RAM to a permanent storage device, such as storage drives 171A-F.

[0043] In some embodiments, storage drives 171A-F may refer to any device configured to permanently record data, where "permanently" or "permanent" refers to the device's ability to maintain recorded data after power is removed. In some embodiments, storage drives 171A-F may correspond to non-disk storage media. For example, storage drives 171A-F may be one or more solid-state drives ('SSD'), flash memory-based storage devices, any type of solid-state non-volatile memory, or any other type of non-mechanical storage device. In other embodiments, storage drives 171A-F may include a mechanical or rotating hard disk, such as a hard disk drive ('HDD').

[0044] In some embodiments, the storage array controllers 110A-D may be configured to offload device management responsibilities from the storage drivers 171A-F in the storage arrays 102A-B. For example, the storage array controllers 110A-D may manage control information that may describe the status of one or more memory blocks in the storage drivers 171A-F. For example, the control information may indicate that a particular memory block has failed and should no longer be written to, that a particular memory block contains boot code for the storage array controllers 110A-D, the number of program-erase ('P / E') cycles that have been performed on a particular memory block, the age of the data stored in a particular memory block, the type of data stored in a particular memory block, and the like. In some embodiments, the control information may be stored as metadata with the associated memory blocks. In other embodiments, the control information for the storage drivers 171A-F may be stored in a particular memory block of the storage drivers 171A-F selected by the storage array controllers 110A-D. The selected memory block may be marked with an identifier indicating that the selected memory block contains the control information. The identifier may be utilized by the storage array controller 110A-D and the storage drives 171A-F to quickly identify a memory block containing control information. For example, the storage controller 110A-D may issue a command to locate a memory block containing control information. It may be noted that the control information may be so large that portions of the control information may be stored in multiple locations, for example, the control information may be stored in multiple locations for redundancy purposes, or the control information may be otherwise distributed across multiple memory blocks in the storage drives 171A-F.

[0045] In some embodiments, the storage array controller 110A-D may offload device management responsibilities from the storage drives 171A-F of the storage array 102A-B by retrieving control information from the storage drives 171A-F that describes the state of one or more memory blocks in the storage drives 171A-F. Retrieving control information from the storage drives 171A-F may be performed, for example, by the storage array controller 110A-D querying the storage drives 171A-F for the location of the control information for a particular storage drive 171A-F. The storage drives 171A-F may be configured to execute instructions that cause the storage drives 171A-F to identify the location of the control information. The instructions may be executed by a controller (not shown) associated with or otherwise located on the storage drives 171A-F, and may cause the storage drives 171A-F to scan a portion of each memory block to identify the memory blocks that store the control information for the storage drives 171A-F. The storage drives 171A-F may respond by sending a response message to the storage array controller 110A-D containing the location of the control information for the storage drives 171A-F. In response to receiving the response message, the storage array controller 110A-D may issue a request to read the data stored at the address associated with the location of the control information for the storage drives 171A-F.

[0046] In other embodiments, the storage array controllers 110A-D may further offload device management responsibilities from the storage drives 171A-F by performing storage drive management operations in response to receiving control information. The storage drive management operations may include, for example, operations that are typically performed by the storage drives 171A-F (e.g., a controller (not shown) associated with a particular storage drive 171A-F). The storage drive management operations may include, for example, ensuring that data is not written to a failed memory block within the storage drive 171A-F, ensuring that data is written to a memory block within the storage drive 171A-F so that adequate wear leveling is achieved, and the like.

[0047] In some embodiments, the storage array 102A-B may implement two or more storage array controllers 110A-D. For example, the storage array 102A may include a storage array controller 110A and a storage array controller 110B. At a given moment, a single storage array controller 110A-D (e.g., storage array controller 110A) of the storage system 100 may be assigned a primary state (also referred to herein as a "primary controller"), and other storage array controllers 110A-D (e.g., storage array controller 110A) may be assigned a secondary state (also referred to herein as a "secondary controller"). The primary controller may have specific permissions, such as permission to change data in the permanent storage resources 170A-B (e.g., write data to the permanent storage resources 170A-B). At least some permissions of the primary controller may replace permissions of the secondary controller. For example, when the primary controller has permission to change data in the permanent storage resources 170A-B, the secondary controller may not have this permission. The state of the storage array controllers 110A-D may change. For example, storage array controller 110A may be assigned a secondary status and storage array controller 110B may be assigned a primary status.

[0048] In some embodiments, a primary controller, such as storage array controller 110A, may act as a primary controller for one or more storage arrays 102A-B, and a secondary controller, such as storage array controller 110B, may act as a secondary controller for the one or more storage arrays 102A-B. For example, storage array controller 110A may be a primary controller for storage array 102A and storage array 102B, and storage array controller 110B may be a secondary controller for storage arrays 102A and 102B. In some embodiments, storage array controllers 110C and 110D (also referred to as "storage processing modules") may not have both primary and secondary status. Storage array controllers 110C and 110D implemented as storage processing modules may act as a communication interface between primary and secondary controllers (e.g., storage array controllers 110A and 110B, respectively) and storage array 102B. For example, storage array controller 110A of storage array 102A may send a write request to storage array 102B via SAN 158. The write request may be received by both storage array controllers 110C and 110D of storage array 102B. Storage array controllers 110C and 110D facilitate communications, such as sending write requests to appropriate storage drives 171A-F. It may be noted that in some embodiments, storage processing modules may be used to increase the number of storage drives controlled by primary and secondary controllers.

[0049] In some embodiments, the storage array controllers 110A-D are communicatively coupled to one or more storage drives 171A-F and one or more NVRAM devices (not shown) included as part of the storage arrays 102A-B via a midplane (not shown). The storage array controllers 110A-D may be coupled to the midplane via one or more data communications links, and the midplane may be coupled to the storage drives 171A-F and the NVRAM devices via one or more data communications links. The data communications links described herein are collectively shown as data communications links 108A-D, and may include, for example, a Peripheral Component Interconnect Express ('PCIe') bus.

[0050] Figure 1B An example system for data storage is shown according to some embodiments. Figure 1B The storage array controller 101 shown in FIG. 1 may be similar to that of Figure 1A In one example, the storage array controller 101 may be similar to the storage array controller 110A or the storage array controller 110B. For purposes of illustration and not limitation, the storage array controller 101 includes a number of elements. It is noted that the storage array controller 101 may include the same, more, or fewer elements configured in the same or different manner in other embodiments. It is noted that the following may include Figure 1A 1 to help illustrate the features of storage array controller 101.

[0051] The storage array controller 101 may include one or more processing devices 104 and a random access memory ('RAM') 111. The processing device 104 (or controller 101) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, and the like. More specifically, the processing device 104 (or controller 101) may be a complex instruction set computing ('CISC') microprocessor, a reduced instruction set computing ('RISC') microprocessor, a very long instruction word ('VLIW') microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. The processing device 104 (or controller 101) may also be one or more special-purpose processing devices, such as an ASIC, an FPGA, a digital signal processor ('DSP'), a network processor, and the like.

[0052] Processing device 104 may be connected to RAM 111 via data communication link 106, which may be embodied as a high-speed memory bus, such as a double data rate 4 ('DDR4') bus. Stored in RAM 111 is an operating system 112. In some embodiments, instructions 113 are stored in RAM 111. Instructions 113 may include computer program instructions for performing operations in a direct-mapped flash memory system. In one embodiment, a direct-mapped flash memory system is a system that directly addresses data blocks within a flash drive and does not require address translation performed by a storage controller of the flash drive.

[0053] In some embodiments, the storage array controller 101 includes one or more host bus adapters 103A-C coupled to the processing device 104 via data communication links 105A-C. In some embodiments, the host bus adapters 103A-C may be computer hardware that connects a host system (e.g., a storage array controller) to other networks and storage arrays. In some examples, the host bus adapters 103A-C may be Fibre Channel adapters that enable the storage array controller 101 to connect to a SAN, Ethernet adapters that enable the storage array controller 101 to connect to a LAN, and so on. The host bus adapters 103A-C may be coupled to the processing device 104 via data communication links 105A-C, such as a PCIe bus.

[0054] In some embodiments, storage array controller 101 may include a host bus adapter 114 coupled to an expander 115. Expander 115 may be used to attach a host system to a greater number of storage drives. For example, in embodiments where host bus adapter 114 is embodied as a SAS controller, expander 115 may be a SAS expander for attaching host bus adapter 114 to storage drives.

[0055] In some embodiments, the storage array controller 101 may include a switch 116 coupled to the processing device 104 via the data communication link 109. The switch 116 may be a computer hardware device that can create multiple endpoints from a single endpoint, thereby allowing multiple devices to share a single endpoint. For example, the switch 116 may be a PCIe switch that is coupled to a PCIe bus (e.g., the data communication link 109) and presents multiple PCIe connection points to the midplane.

[0056] In some embodiments, storage array controller 101 includes a data communication link 107 for coupling storage array controller 101 to other storage array controllers. In some examples, data communication link 107 may be a Quick Path Interconnect (QPI) interconnect.

[0057] A conventional storage system using conventional flash drives may implement a process on a flash drive that is part of the conventional storage system. For example, a higher level process of the storage system may initiate and control a process on the flash drive. However, the flash drive of the conventional storage system may include its own storage controller that also executes the process. Thus, for the conventional storage system, both a higher level process (e.g., initiated by the storage system) and a lower level process (e.g., initiated by the storage controller of the storage system) may be executed.

[0058] To address various deficiencies of conventional storage systems, operations may be performed by higher level processes rather than lower level processes. For example, a flash storage system may include a flash drive that does not include a storage controller that provides the process. Thus, the operating system of the flash storage system itself may initiate and control the process. This may be accomplished by a direct mapped flash storage system that directly addresses data blocks within the flash drive and does not require address translation performed by the flash drive's storage controller.

[0059] In some embodiments, the storage drive 171A-F may be one or more partitioned storage devices. In some embodiments, the one or more partitioned storage devices may be shingled HDDs. In some embodiments, the one or more storage devices may be flash-based SSDs. In a partitioned storage device, the partition namespace on the partitioned storage device may be addressed by groups of blocks, which are grouped and arranged according to native sizes to form several addressable areas. In some embodiments utilizing SSDs, the native size may be based on the erase block size of the SSD. In some embodiments, the zones of the partitioned storage device may be defined during the initialization of the partitioned storage device. In some embodiments, the zones may be dynamically defined when data is written to the partitioned storage device.

[0060] In some embodiments, the zones may be heterogeneous, with some zones being page groups and other zones being multiple page groups. In some embodiments, some zones may correspond to erase blocks and other zones may correspond to multiple erase blocks. In an embodiment, a zone may be any combination of different numbers of pages in a page group and / or erase block, for a heterogeneous mix of programming modes, manufacturers, product types, and / or product generations of a storage device, such as for heterogeneous assemblies, upgrades, distributed storage, and the like. In some embodiments, a zone may be defined as having usage characteristics, such as attributes that support data with a specific lifespan (e.g., ultra-short lifespan or ultra-long lifespan). A partitioned storage device may use these attributes to determine how to manage a zone within its expected lifespan.

[0061] It should be understood that a zone is a virtual construct. Any particular zone may not have a fixed position on a storage device. Before allocation, a zone may not have any position on a storage device. A zone may correspond to a number representing a virtual allocatable space block, which is the size of an erase block or other block size in various embodiments. When the system allocates or opens a zone, the zone is allocated to a flash or other solid-state storage memory, and when the system writes to a zone, a page is written to the mapped flash or other solid-state storage memory of the partition storage device. When the system closes the zone, the associated erase block or other size block is completed. At some point in the future, the system may delete a zone, which will free up the allocated space of this zone. During its lifespan, a zone may be moved to a different location of a partition storage device, such as when the partition storage device performs internal maintenance.

[0062] In some embodiments, the zones of the partitioned storage device may be in different states. The zone may be in a clear state, where data has not yet been stored at the zone. The cleared area may be opened explicitly, or implicitly opened by writing data to the zone. This is the initial state of the zone on the fresh partitioned storage device, but it may also be the result of a zone reset. In some embodiments, the cleared zone may have a specified location in the flash memory of the partitioned storage device. In an embodiment, the location of the cleared zone may be selected when the zone is first opened or first written (or subsequently selected when the write is buffered in the memory). A zone may be implicitly or explicitly in an open state, where the zone in the open state may be written to store data using a write or append command. In an embodiment, the zone in the open state may also be written using a copy command to copy data from different zones. In some embodiments, the partitioned storage device may have a limit on the number of open zones at a specific time.

[0063] A zone in a closed state is one that has been partially written to but has been put into a closed state after an explicit close operation was issued. A zone in a closed state may be retained for future writes, but some runtime overhead consumed may be reduced by keeping the zone in an open state. In some embodiments, a partitioned storage device may have a limit on the number of closed zones at a particular time. A zone in a full state is one that is storing data and can no longer be written to. A zone may be in a full state after a write has written data to the entire zone, or due to a zone complete operation. Prior to a complete operation, a zone may or may not have been fully written. However, after a complete operation, the zone may not be opened or written to further without first performing a zone reset operation.

[0064] The mapping of zones to erase blocks (or to shingled tracks in an HDD) can be arbitrary, dynamic, and hidden from view. The process of opening a zone can be an operation that allows a new zone to be dynamically mapped to the underlying storage of a partitioned storage device, and then allows data to be written by appending writes to the zone until the zone reaches capacity. The zone can be completed at any time, after which further data may not be written to the zone. When the data stored at the zone is no longer needed, the zone can be reset, which effectively deletes the zone contents from the partitioned storage device, making the physical storage maintained by the zone available for subsequent data storage. Once a zone is written and completed, the partitioned storage device ensures that the data stored in the zone is not lost until the zone is reset. In the time between writing data to the zone and resetting the zone, zones can be moved between shingled tracks or erase blocks as part of maintenance operations within the partitioned storage device, such as by copying data to keep data refreshed or to handle memory cell aging in an SSD.

[0065] In some embodiments utilizing HDDs, a zone reset may allow shingled tracks to be assigned to new open zones that may be opened at some time in the future. In some embodiments utilizing SSDs, a zone reset may cause the zone's associated physical erase blocks to be erased and subsequently reused to store data. In some embodiments, a zoned storage device may have a limit on the number of open zones at a point in time to reduce the amount of overhead dedicated to keeping zones open.

[0066] The operating system of the flash memory system can identify and maintain a list of allocation units across multiple flash drives of the flash memory system. An allocation unit can be an entire erase block or multiple erase blocks. The operating system can maintain a mapping or address range that directly maps addresses to erase blocks of the flash drives of the flash memory system.

[0067] Direct mapping to the erase block of the flash drive can be used to rewrite data and erase data. For example, operations can be performed on one or more allocation units containing first data and second data, where the first data is to be retained and the second data is no longer used by the flash memory system. The operating system can initiate this process to write the first data to a new location within other allocation units, erase the second data and mark the allocation unit as available for subsequent data. Therefore, this process can be performed only by the higher-level operating system of the flash memory system without the need for additional lower-level processes performed by the controller of the flash drive.

[0068] Advantages of this process being performed only by the operating system of the flash memory system include increased reliability of the flash drives of the flash memory system, since no unnecessary or redundant write operations are performed during this process. A possible novelty here is the concept of initiating and controlling the process on the operating system of the flash memory system. In addition, this process can be controlled by the operating system across multiple flash drives. This is in contrast to the storage controller of the flash drive executing the process.

[0069] A storage system may consist of two storage array controllers that share a set of drives for failover purposes, or it may consist of a single storage array controller that provides storage services that utilize multiple drives, or it may consist of a distributed network of storage array controllers, each with a certain number of drives or a certain amount of flash storage, where the storage array controllers in the network collaborate to provide complete storage services and collaborate on various aspects of the storage services, including storage allocation and garbage collection.

[0070] Figure 1C A third example system 117 for data storage according to some embodiments is shown. For purposes of illustration and not limitation, the system 117 (also referred to herein as a "storage system") includes a number of elements. It may be noted that the system 117 may include the same, more, or fewer elements configured in the same or different manner in other embodiments.

[0071] In one embodiment, system 117 includes dual peripheral component interconnect ('PCI') flash memory devices 118 with individually addressable fast write storage. System 117 may include a memory device controller 119. In one embodiment, memory device controllers 119A-D may be a CPU, ASIC, FPGA, or any other circuitry that may implement the control structures required according to the present disclosure. In one embodiment, system 117 includes flash memory devices (e.g., including flash memory devices 120a-n) operatively coupled to various channels of memory device controller 119. Flash memory devices 120a-n may be presented to controllers 119A-D as addressable collections of flash pages, erase blocks, and / or control elements sufficient to allow memory device controllers 119A-D to program and retrieve various aspects of the flash. In one embodiment, the storage device controllers 119A-D may perform operations on the flash memory devices 120a-n including storing and retrieving the data contents of a page, arranging and erasing any block, tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells, tracking and predicting error codes and failures within the flash memory, controlling voltage levels associated with programming and retrieving the contents of flash cells, and the like.

[0072] In one embodiment, the system 117 may include a RAM 121 for storing individually addressable fast-write data. In one embodiment, the RAM 121 may be one or more separate discrete devices. In another embodiment, the RAM 121 may be integrated into the storage device controller 119A-D or multiple storage device controllers. The RAM 121 may also be used for other purposes, such as temporary program storage for a processing device (e.g., a CPU) in the storage device controller 119.

[0073] In one embodiment, system 117 may include a stored energy device 122, such as a rechargeable battery or capacitor. The stored energy device 122 may store enough energy to power storage device controller 119, a certain amount of RAM (e.g., RAM 121), and a certain amount of flash memory (e.g., flash memory 120a-120n) for enough time to write the contents of the RAM to the flash memory. In one embodiment, if the storage device controller detects that external power is lost, the storage device controller 119A-D may write the contents of the RAM to the flash memory.

[0074] In one embodiment, the system 117 includes two data communication links 123a, 123b. In one embodiment, the data communication links 123a, 123b may be PCI interfaces. In another embodiment, the data communication links 123a, 123b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). The data communication links 123a, 123b may be based on the Non-Volatile Memory Express ('NVMe') or Fabric-based NVMe ('NVMf') specifications, which allow external connections to the storage device controllers 119A-D from other components in the storage system 117. It should be noted that for convenience, the data communication links may be interchangeably referred to herein as PCI buses.

[0075] The system 117 may also include an external power source (not shown) which may be provided via one or both of the data communication links 123a, 123b, or may be provided separately. An alternative embodiment includes a separate flash memory (not shown) dedicated to storing the contents of the RAM 121. The storage device controllers 119A-D may present a logical device on the PCI bus, which may include an addressable fast write logical device, or a different portion of the logical address space of the storage device 118, which may be presented as PCI memory or permanent storage. In one embodiment, store operations to the device are directed to the RAM 121. In the event of a power failure, the storage device controllers 119A-D may write the storage contents associated with the addressable fast write logical storage to a flash memory (e.g., flash memory 120a-n) for long-term permanent storage.

[0076] In one embodiment, the logic device may include some representation of some or all of the contents of flash memory devices 120a-n, wherein the representation allows a storage system (e.g., storage system 117) including storage device 118 to directly address flash memory pages and directly reprogram erase blocks of storage system components external to the storage device over a PCI bus. The representation may also allow one or more external components to control and retrieve other aspects of the flash memory, including some or all of the following: tracking statistics related to the use and reuse of flash memory pages, erase blocks, and cells across all flash memory devices; tracking and predicting error codes and failures within and across flash memory devices; controlling voltage levels associated with programming and retrieving the contents of flash cells; and the like.

[0077] In one embodiment, the storage energy device 122 may be sufficient to ensure completion of ongoing operations on the flash memory devices 120a-120n. The storage energy device 122 may power the storage device controllers 119A-D and associated flash memory devices (e.g., 120a-n) for these operations, as well as for storing fast write RAM to flash memory. The storage energy device 122 may be used to store cumulative statistics and other parameters maintained and tracked by the flash memory devices 120a-n and / or the storage device controller 119. A separate capacitor or storage energy device (e.g., a smaller capacitor close to or embedded in the flash memory device itself) may be used for some or all of the operations described herein.

[0078] Various schemes may be used to track and optimize the life of the stored energy components, such as adjusting voltage levels over time, partially discharging the stored energy device 122 to measure corresponding discharge characteristics, etc. If the available energy decreases over time, the effectively available capacity of the addressable fast write storage may be reduced to ensure that it can be safely written based on the currently available storage energy.

[0079] Figure 1D A third example storage system 124 for data storage is shown in accordance with some embodiments. In one embodiment, the storage system 124 includes storage controllers 125a, 125b. In one embodiment, the storage controllers 125a, 125b are operatively coupled to dual PCI storage devices. The storage controllers 125a, 125b may be operatively coupled (e.g., via a storage network 130) to a number of host computers 127a-n.

[0080] In one embodiment, two storage controllers (e.g., 125a and 125b) provide storage services, such as SCS block storage arrays, file servers, object servers, database or data analysis services, etc. The storage controllers 125a, 125b can provide services to host computers 127a-n external to the storage system 124 through a number of network interfaces (e.g., 126a-d). The storage controllers 125a, 125b can provide integrated services or applications entirely within the storage system 124, thereby forming a converged storage and computing system. The storage controllers 125a, 125b can utilize fast write memory within or across the storage devices 119a-d to log ongoing operations to ensure that these operations are not lost in the event of a power failure, storage controller removal, storage controller or storage system shutdown, or certain failures of one or more software or hardware components within the storage system 124.

[0081] In one embodiment, the storage controllers 125a, 125b act as PCI masters for one or other PCI buses 128a, 128b. In another embodiment, 128a and 128b may be based on other communication standards (e.g., HyperTransport, InfiniBand, etc.). Other storage system embodiments may operate the storage controllers 125a, 125b as multi-masters for both PCI buses 128a, 128b. Alternatively, a PCI / NVMe / NVMf switching infrastructure or structure may connect multiple storage controllers. Some storage system embodiments may allow storage devices to communicate directly with each other, rather than just with storage controllers. In one embodiment, the storage device controller 119a may operate under the direction of the storage controller 125a to retrieve data from a computer that has been stored in RAM (e.g., Figure 1C The recalculated version of the RAM contents may be delivered to ensure improved security of the data or to free up addressable fast write capacity for reuse. For example, this mechanism may be used to avoid a second pass from the storage controller 125a, 125b over the bus (e.g., 128a, 128b). In one embodiment, the recalculation may include compressing the data, appending indexes or other metadata, combining multiple data segments together, performing erasure code calculations, and the like.

[0082] In one embodiment, under the direction of the storage controller 125a, 125b, the storage device controller 119a, 119b may be used to store the data stored in the RAM (e.g., Figure 1CThe data in the RAM 121 of the storage device 125a) is calculated and passed to other storage devices without the participation of the storage controllers 125a, 125b. This operation can be used to mirror data stored in one storage controller 125a to another storage controller 125b, or it can be used to share compression, data aggregation and / or erasure coding calculations and pass them to storage devices to reduce the load on the storage controller or the storage controller interface 129a, 129b to the PCI bus 128a, 128b.

[0083] The storage device controllers 119A-D may include mechanisms for implementing high availability primitives for use by other parts of the storage system external to the dual PCI storage device 118. For example, a reservation or exclusion primitive may be provided so that in a storage system having two storage controllers providing highly available storage services, one storage controller may prevent another storage controller from accessing or continuing to access the storage device. This may be used, for example, in situations where one controller detects that the other controller is not functioning properly or the interconnect between the two storage controllers may itself be not functioning properly.

[0084] In one embodiment, a storage system for use with dual PCI direct mapped storage devices having individually addressable fast write storage includes a system for managing erase blocks or groups of erase blocks as allocation units for storing data on behalf of a storage service, or for storing metadata associated with the storage service (e.g., indexes, logs, etc.), or for properly managing the storage system itself. Flash pages may be only a few kilobytes in size and may be written as data arrives or when the storage system persists data for a long period of time (e.g., exceeding a defined time threshold). To commit data more quickly, or to reduce the number of writes to the flash memory devices, the storage controller may first write the data to individually addressable fast write storage on one or more storage devices.

[0085] In one embodiment, the storage controllers 125a, 125b may initiate the use of erase blocks within and across storage devices (e.g., 118) based on the age and expected remaining life of the storage devices, or based on other statistical data. The storage controllers 125a, 125b may initiate garbage collection and data migration between storage devices based on pages that are no longer needed, as well as manage the life of flash pages and erase blocks, and manage overall system performance.

[0086] In one embodiment, the storage system 124 may utilize mirroring and / or erasure coding schemes as part of storing data to addressable fast write storage and / or as part of writing data to allocation units associated with erase blocks. Erasure codes may be used across storage devices, within erase blocks or allocation units, or within and across flash memory devices on a single storage device to provide redundancy against single or multiple storage device failures, or to prevent internal corruption of flash memory pages due to flash memory operation or flash memory cell degradation. Different levels of mirroring and erasure coding may be used to recover from a variety of failures occurring alone or in combination.

[0087] refer to Figure 2A -G depicts an embodiment that shows a storage cluster that stores user data, such as user data from one or more users or client systems or other sources outside the storage cluster. The storage cluster uses erasure coding and redundant copies of metadata to distribute user data between storage nodes housed in a chassis or across multiple chassis. Erasure coding is a data protection or reconstruction method in which data is stored across a set of different locations, such as disks, storage nodes, or geographic locations. Flash memory is a solid-state memory that can be integrated with an embodiment, but the embodiment can be extended to other types of solid-state memory or other storage media, including non-solid-state memory. The control of storage locations and workloads is distributed on storage locations in a cluster peer system. Tasks such as regulating communication between individual storage nodes, detecting when a storage node is unavailable, and balancing I / O (input and output) on individual storage nodes are all processed on a distributed basis. In some embodiments, data is arranged or distributed on multiple storage nodes in the form of data fragments or stripes that support data recovery. The ownership of data can be reassigned within the cluster, regardless of the input and output mode. This architecture, described in more detail below, allows a storage node in a cluster to fail, but the system remains operational because data can be rebuilt from other storage nodes, thereby remaining available for input and output operations.In various embodiments, a storage node may be referred to as a cluster node, a blade, or a server.

[0088] The storage cluster can be housed in a chassis, that is, a housing that houses one or more storage nodes. The chassis contains a mechanism for supplying power to each storage node, such as a power distribution bus, and a communication mechanism, such as a communication bus that enables communication between storage nodes. According to some embodiments, the storage cluster can operate as a standalone system in one location. In one embodiment, the chassis accommodates at least two instances of power distribution and communication buses that can be enabled or disabled independently. The internal communication bus can be an Ethernet bus, but other technologies, such as PCIe, Infiniband, etc., are also applicable. The chassis provides a port for an external communication bus for communicating between multiple chassis and with client systems directly or through a switch. External communications can use technologies such as Ethernet, Infiniband, Fibre Channel, etc. In some embodiments, the external communication bus uses different communication bus technologies for inter-chassis and client communications. If a switch is deployed in a chassis or between chassis, the switch can act as a conversion between multiple protocols or technologies. When multiple chassis are connected to define a storage cluster, clients may access the storage cluster using proprietary interfaces or standard interfaces, such as Network File System ('NFS'), Common Internet File System ('CIFS'), Small Computer System Interface ('SCSI'), or Hypertext Transfer Protocol ('HTTP'). Conversion from the client protocol may occur at a switch, at a chassis external communication bus, or within each storage node. In some embodiments, multiple chassis may be coupled or connected to each other via an aggregator switch. Some and / or all of the coupled or connected chassis may be represented as a storage cluster. As discussed above, each chassis may have multiple blades, each blade having a media access control ('MAC') address, but in some embodiments, the storage cluster is presented to the external network as having a single cluster IP address and a single MAC address.

[0089] Each storage node can be one or more storage servers, and each storage server is connected to one or more non-volatile solid-state memory units, which can be referred to as storage units or storage devices. One embodiment includes a single storage server in each storage node, and one to eight non-volatile solid-state memory units, but this example is not intended to be restrictive. The storage server may include a processor, DRAM, and an interface for an internal communication bus and power distribution for each power bus. In some embodiments, inside the storage node, the interface and the storage unit share a communication bus, such as PCI Express. The non-volatile solid-state memory unit can directly access the internal communication bus interface through the storage node communication bus, or request the storage node to access the bus interface. The non-volatile solid-state memory unit accommodates an embedded CPU, a solid-state storage controller, and a certain number of solid-state mass storage devices, such as between 2-32 terabytes ('TB') in some embodiments. Embedded volatile storage media such as DRAM and energy storage devices are included in the non-volatile solid-state memory unit. In some embodiments, the energy storage device is a capacitor, a supercapacitor, or a battery, which can transfer a subset of DRAM content to a stable storage medium in the case of power failure. In some embodiments, the non-volatile solid-state memory cells are comprised of storage class memories such as phase change or magnetoresistive random access memory ('MRAM'), which can replace DRAM and achieve device retention power reduction.

[0090] One of the many features of storage nodes and non-volatile solid-state storage devices is the ability to actively rebuild data in a storage cluster. Storage nodes and non-volatile solid-state storage devices can determine when a storage node or non-volatile solid-state storage device in a storage cluster is inaccessible, regardless of whether an attempt is made to read data involving the storage node or non-volatile solid-state storage device. The storage node and non-volatile solid-state storage device then collaborate to recover and rebuild the data at at least a portion of the new location. This constitutes an active reconstruction because the system does not wait until the data is needed for a read access initiated from a client system that employs the storage cluster when rebuilding the data. These and further details of the storage memory and its operation are discussed below.

[0091] Figure 2A1 is a perspective view of a storage cluster 161 according to some embodiments, which has multiple storage nodes 150 and internal solid-state storage coupled to each storage node to provide a network attached storage or storage area network. The network attached storage, storage area network or storage cluster or other storage storage can include one or more storage clusters 161, each storage cluster having one or more storage nodes 150, arranged in a flexible and reconfigurable manner in terms of physical components and the amount of storage storage provided thereby. The storage cluster 161 is designed to be assembled in a rack, and one or more racks can be set and filled for storage storage as needed. The storage cluster 161 has a chassis 138 with multiple slots 142. It should be understood that the chassis 138 can be referred to as a housing, shell or rack unit. In one embodiment, the chassis 138 has fourteen slots 142, but other numbers of slots are easy to design. For example, some embodiments have four slots, eight slots, sixteen slots, thirty-two slots or other suitable numbers of slots. In some embodiments, each slot 142 can accommodate a storage node 150. The chassis 138 includes fins 148 that can be used to mount the chassis 138 on a rack. The fan 144 provides air circulation to cool the storage node 150 and its components, but other cooling components can also be used, or an embodiment without a cooling component can be designed. The switch structure 146 couples the storage nodes 150 in the chassis 138 together and couples to the network to communicate with the storage. In the embodiment depicted herein, for illustrative purposes, the slot 142 on the left side of the switch structure 146 and the fan 144 is shown as being occupied by the storage node 150, while the slot 142 on the right side of the switch structure 146 and the fan 144 is empty and can be used to insert the storage node 150. This configuration is an example, and one or more storage nodes 150 can occupy the slot 142 in various other arrangements. In some embodiments, the storage node arrangement does not need to be sequential or adjacent. The storage node 150 is hot-swappable, which means that the storage node 150 can be inserted into the slot 142 in the chassis 138, or removed from the slot 142 without stopping the system or turning off the system power. When a storage node 150 is inserted into or removed from a slot 142, the system automatically reconfigures to recognize and adapt to the changes. In some embodiments, reconfiguring includes restoring redundancy and / or rebalancing data or load.

[0092] Each storage node 150 may have multiple components. In the embodiment shown here, the storage node 150 includes a printed circuit board 159 populated by a CPU 156 (i.e., a processor), a memory 154 coupled to the CPU 156, and a non-volatile solid-state storage device 152 coupled to the CPU 156, but other mountings and / or components may be used in other embodiments. The memory 154 has instructions executed by the CPU 156 and / or data operated by the CPU 156. As further explained below, the non-volatile solid-state storage device 152 includes flash, or in other embodiments, includes other types of solid-state memory.

[0093] refer to Figure 2A , the storage cluster 161 is scalable, which means that storage capacity with non-uniform storage sizes can be easily added as described above. In some embodiments, one or more storage nodes 150 can be inserted into or removed from each chassis, and the storage cluster can self-configure. Plug-in storage nodes 150, whether installed in the chassis at delivery or added later, can have different sizes. For example, in one embodiment, the storage node 150 may have any multiple of 4 TB, for example, 8TB, 12TB, 16TB, 32TB, etc. In other embodiments, the storage node 150 may have any multiple of other storage amounts or capacities. The storage capacity of each storage node 150 is broadcast and affects the decision on how to stripe the data. In order to obtain maximum storage efficiency, the embodiment can self-configure as wide as possible in the stripe, but meet the predetermined requirements for continuous operation, in which at most one or two units or storage nodes 150 of non-volatile solid-state storage devices 152 are lost in the chassis.

[0094] Figure 2B 1 is a block diagram showing a communication interconnect 173 and a power distribution bus 172 coupling a plurality of storage nodes 150. Return to Reference Figure 2A In some embodiments, the communication interconnect 173 may be included in or implemented with the switch fabric 146. In some embodiments, where multiple storage clusters 161 occupy one rack, the communication interconnect 173 may be included in or implemented with a top-of-rack switch. Figure 2B As shown, storage cluster 161 is enclosed within a single chassis 138. External port 176 is coupled to storage node 150 via communication interconnect 173, while external port 174 is directly coupled to storage node. External power port 178 is coupled to power distribution bus 172. Storage node 150 may include non-volatile solid-state storage devices 152 of varying numbers and capacities, as described in reference to FIG. Figure 2A In addition, one or more storage nodes 150 may be Figure 2BOnly computing storage nodes are shown. Authorization 168 is implemented on the non-volatile solid-state storage device 152, for example as a list or other data structure stored in memory. In some embodiments, the authorization is stored in the non-volatile solid-state storage device 152 and supported by software executed on a controller or other processor of the non-volatile solid-state storage device 152. In another embodiment, the authorization 168 is implemented on the storage node 150, for example as a list or other data structure stored in the memory 154 and supported by software executed on the CPU 156 of the storage node 150. In some embodiments, the authorization 168 controls the manner and location of data storage in the non-volatile solid-state storage device 152. This control helps determine which type of erasure coding scheme is applied to the data, and which storage nodes 150 have which parts of the data. Each authorization 168 can be assigned to the non-volatile solid-state storage device 152. In various embodiments, each authorization can control the range of index node numbers, segment numbers, or other data identifiers assigned to data by the file system, storage node 150, or non-volatile solid-state storage device 152.

[0095] In some embodiments, each piece of data and each piece of metadata has redundancy in the system. In addition, each piece of data and each piece of metadata has an owner, which can be called an authorization. If the authorization cannot be accessed, for example due to a storage node failure, there is a series of plans for how to find the data or the metadata. In various embodiments, there are redundant copies of authorization 168. In some embodiments, authorization 168 is related to storage node 150 and non-volatile solid-state storage device 152. Each authorization 168 covering the range of data segment numbers or other identifiers of data can be assigned to a specific non-volatile solid-state storage device 152. In some embodiments, all these ranges of authorization 168 are distributed on the non-volatile solid-state storage device 152 of the storage cluster. Each storage node 150 has a network port for providing access to the non-volatile solid-state storage device 152 of the storage node 150. Data can be stored in segments associated with segment numbers, which in some embodiments are indirect ways of RAID (Redundant Array of Independent Disks) stripe configurations. Therefore, the assignment and use of authorization 168 establishes indirect access to data. According to some embodiments, indirect access may refer to the ability to indirectly reference data, in this case, via authorization 168. A segment identifies a set of non-volatile solid-state storage devices 152 and a local identifier in the set of non-volatile solid-state storage devices 152, which may contain data. In some embodiments, the local identifier is an offset in the device and may be reused sequentially by multiple segments. In other embodiments, the local identifier is unique to a particular segment and is never reused. The offset in the non-volatile solid-state storage device 152 is used to locate data to write to the non-volatile solid-state storage device 152 or read from the non-volatile solid-state storage device 152 (in the form of a RAID stripe). The data is striped across multiple units of the non-volatile solid-state storage device 152, which may contain or be different from the non-volatile solid-state storage device 152 with the authorization 168 for a particular data segment.

[0096] If the location of a particular data segment has changed, such as during data movement or data reconstruction, the authorization 168 of the data segment should be queried at the non-volatile solid-state storage device 152 or storage node 150 with the authorization 168. In order to locate a specific piece of data, the embodiment calculates the hash value of the data segment or applies the index node number or data segment number. The output of this operation points to the non-volatile solid-state storage device 152 with the authorization 168 of the specific piece of data. In some embodiments, this operation is divided into two stages. The first stage maps the entity identifier (ID) (such as a segment number, an index node number, or a directory number) to the authorization identifier. This mapping may include calculations such as hashing or bit masks. The second stage is to map the authorization identifier to a specific non-volatile solid-state storage device 152, which can be done by explicit mapping. This operation is repeatable so that when the calculation is performed, the calculation result can be repeatedly and reliably pointed to the specific non-volatile solid-state storage device 152 with the authorization 168. This operation may include a set of accessible storage nodes as input. If the set of accessible non-volatile solid-state storage units changes, the optimal set will also change. In some embodiments, the persistent value is the current assignment (always true) and the calculated value is the target assignment that the cluster will attempt to reconfigure. In the presence of a set of non-volatile solid-state storage devices 152 that are accessible and constitute the same cluster, this calculation can be used to determine the best non-volatile solid-state storage device 152 for authorization. The calculation also determines an ordered set of peer non-volatile solid-state storage devices 152, which will also record the mapping of authorizations to non-volatile solid-state storage devices so that authorizations can be determined even if the assigned non-volatile solid-state storage device is inaccessible. In some embodiments, if a specific authorization 168 is not available, a duplicate or alternative authorization 168 can be queried.

[0097] refer to Figure 2A and 2B, two of the many tasks of the CPU 156 on the storage node 150 are to split the write data and reassemble the read data. When the system determines that data is to be written, the authorization 168 of the data is located as described above. When the segment ID of the data has been determined, the write request is forwarded to the non-volatile solid-state storage device 152 of the host currently determined as the authorization 168 determined according to the segment. Then, the host CPU 156 of the storage node 150 (the non-volatile solid-state storage device 152 and the corresponding authorization 168 reside thereon) splits or slices the data and transmits the data out to various non-volatile solid-state storage devices 152. According to the erasure coding scheme, the transmitted data is written as a data stripe. In some embodiments, the request pulls the data, while in other embodiments, the data is pushed. In contrast, when reading data, as described above, the authorization 168 containing the segment ID of the data is located. The host CPU 156 of the storage node 150 on which the non-volatile solid-state storage device 152 and the corresponding authorization 168 reside requests data from the non-volatile solid-state storage device and the corresponding storage node to which the authorization points. In some embodiments, data is read from the flash storage device as a stripe of data. The host CPU 156 of the storage node 150 then reassembles the read data, corrects any errors (if any) according to an appropriate erasure coding scheme, and forwards the reassembled data to the network. In other embodiments, some or all of these tasks may be handled in the non-volatile solid-state storage device 152. In some embodiments, the segment host requests that data be sent to the storage node 150 by requesting the page from the storage device and then sending the data to the storage node that issued the original request.

[0098] In an embodiment, authorization 168 operates to determine how operations will be performed for a specific logical element. Each logical element can be operated by a specific authorization on multiple storage controllers of the storage system. Authorization 168 can communicate with multiple storage controllers so that multiple storage controllers jointly perform operations for these specific logical elements.

[0099] In embodiments, the logical element may be, for example, a file, a directory, an object bucket, an individual object, a delineated portion of a file or object, other forms of key-value databases, or a table. In embodiments, performing an operation may involve, for example, ensuring consistency, structural integrity, and / or recoverability with other operations on the same logical element, reading metadata and data associated with the logical element, determining which data should be persistently written to the storage system to preserve any changes to the operation, or determining where metadata and data are stored on modular storage devices attached to multiple storage controllers in the storage system.

[0100] In some embodiments, operations are token-based transactions to efficiently communicate within a distributed system. Each transaction may be accompanied or associated with a token that provides authority to perform the transaction. In some embodiments, authorization 168 can maintain the pre-transaction state of the system until the operation is completed. Token-based communication can be completed without a global lock on the system, and operations can also be restarted in the event of an interruption or other failure.

[0101] In some systems, such as UNIX-type file systems, data is handled by index nodes (or inodes), which specify data structures that represent objects in the file system. For example, an object can be a file or a directory. Metadata may accompany the object, such as attributes such as permission data and creation timestamps, as well as other attributes. Segment numbers can be assigned to all or part of such objects in the file system. In other systems, data segments are handled by segment numbers assigned elsewhere. For the purposes of discussion, a distribution unit is an entity, which can be a file, directory, or segment. That is, an entity is a unit of data or metadata stored by a storage system. Entities are grouped into collections called authorizations. Each authorization has an authorization owner, which is a storage node that has exclusive rights to update entities in the authorization. In other words, storage nodes contain authorizations, which in turn contain entities.

[0102] According to some embodiments, a segment is a logical container for data. A segment is an address space between the media address space and the physical flash location, i.e., the data segment number is located in this address space. A segment may also contain metadata, which enables data redundancy to be restored (rewritten to a different flash location or device) without involving higher-level software. In one embodiment, the internal format of a segment contains client data and a media map to determine the location of the data. Where applicable, the segments are protected from memory and other failures, for example, by breaking each data segment into several data and parity slices. The data and parity slices are distributed, i.e., striped, across the non-volatile solid-state storage device 152 coupled to the host CPU 156 according to an erasure coding scheme (see Figure 2E and 2G ). In some embodiments, the use of the term segment refers to a container and its location in the address space of the segment. According to some embodiments, the use of the term stripe refers to a group of shards that is the same as a segment, including how the shards are distributed along with redundancy or parity information.

[0103] A series of address space transformations occur throughout the storage system. At the top are directory entries (file names) linked to index nodes. Index nodes point to media address space where data is logically stored. Media addresses can be mapped through a series of indirect media to distribute the load of large files, or implement data services such as deduplication or snapshots. The segment address is then converted to a physical flash location. According to some embodiments, the physical flash location has an address range limited by the amount of flash in the system. Media addresses and segment addresses are logical containers, and in some embodiments, 128-bit or larger identifiers are used so that they are virtually infinite, and the possibility of reuse is calculated to be longer than the expected life of the system. In some embodiments, addresses from logical containers are allocated in a hierarchical manner. Initially, each unit of the non-volatile solid-state storage device 152 can be assigned an address space range. Within this assigned range, the non-volatile solid-state storage device 152 is able to assign addresses without synchronizing with other non-volatile solid-state storage devices 152.

[0104] Data and metadata are stored by a set of underlying storage layouts optimized for different workload patterns and storage devices. These layouts include a variety of redundancy schemes, compression formats, and indexing algorithms. Some of these layouts store information about authorization and authorization masters, while other layouts store file metadata and file data. Redundancy schemes include error correction codes that can tolerate damaged bits within a single storage device (such as a NAND flash chip), erasure codes that can tolerate failures of multiple storage nodes, and replication schemes that can tolerate data center or regional failures. In some embodiments, low-density parity check ('LDPC') codes are used within a single storage unit. In some embodiments, Reed-Solomon coding is used within a storage cluster and mirroring is used within a storage grid. Metadata can be stored using an ordered log-structured index (such as a log-structured merge tree), and large data may not be stored in a log-structured layout.

[0105] To maintain consistency across multiple copies of an entity, storage nodes implicitly agree on two things through computation: (1) the grants that contain the entity, and (2) the storage nodes that contain the grants. Entities can be assigned to grants by pseudo-randomly assigning them to grants, splitting the entities into ranges based on an externally generated key, or placing a single entity into each grant. Examples of pseudo-random schemes are linear hashing and the Replication Under Scalable Hashing ('RUSH') hash family, including Controlled Replication Under Scalable Hashing ('CRUSH'). In some embodiments, pseudo-random assignment is used only to assign grants to nodes because the set of nodes can change. The set of grants cannot change, so any subjective function can be applied in these embodiments. Some placement schemes automatically place grants on storage nodes, while other placement schemes rely on an explicit mapping of grants to storage nodes. In some embodiments, a pseudo-random scheme is used to map each grant to a set of candidate grant owners. A pseudo-random data distribution function associated with CRUSH can assign grants to storage nodes and create a list of grant assignment locations. Each storage node has a copy of the pseudo-random data distribution function and can reach the same calculation for distribution and subsequently find or locate the authorization. In some embodiments, each pseudo-random scheme requires a set of accessible storage nodes as input in order to derive the same target nodes. Once an entity is placed in an authorization, the entity can be stored on a physical device so that expected failures do not occur and cause unexpected data loss. In some embodiments, the rebalancing algorithm attempts to store copies of all entities within an authorization in the same layout and on the same set of machines.

[0106] Examples of expected failures include device failure, stolen machines, data center fires, and regional disasters such as nuclear accidents or geological events. Different failures result in different levels of acceptable data loss. In some embodiments, the theft of a storage node does not affect the security and reliability of the system, while regional events may result in no data loss, lost updates for a few seconds or minutes, or even complete data loss, depending on the system configuration.

[0107] In an embodiment, placing data for storage redundancy is independent of placing authorization for data consistency. In some embodiments, a storage node containing authorization does not contain any permanent storage device. Instead, the storage node is connected to a non-volatile solid-state storage unit that does not contain authorization. The communication interconnection between the storage node and the non-volatile solid-state storage unit consists of a variety of communication technologies and has non-uniform performance and fault tolerance. In some embodiments, as mentioned above, the non-volatile solid-state storage unit is connected to the storage node via PCI high-speed, the storage nodes are connected together using an Ethernet backplane in a single chassis, and the chassis are connected together to form a storage cluster. In some embodiments, the storage cluster is connected to the client using Ethernet or Fibre Channel. If multiple storage clusters are configured as a storage grid, the multiple storage clusters are connected using the Internet or other long-distance networking links, such as "metropolitan scale" links or dedicated links that do not traverse the Internet.

[0108] The authorized owner has exclusive rights to modify the entity, migrate the entity from one non-volatile solid-state storage unit to another non-volatile solid-state storage unit, and add and delete copies of the entity. This allows redundancy of the underlying data to be maintained. When the authorized owner fails, is about to be decommissioned, or is overloaded, the authorization is transferred to the new storage node. Transient failures make it very important to ensure that all non-faulty machines agree on the new authorized location. Ambiguities caused by transient failures can be automatically implemented through consensus protocols (such as Paxos, hot-warm failover schemes), manual intervention by remote system administrators, or local hardware administrators (such as physically removing the failed machine from the cluster, or pressing a button on the failed machine). In some embodiments, failover is automatic using a consensus protocol. According to some embodiments, if too many failures or replication events occur in too short a time, the system will enter a self-protection mode and interrupt replication and data movement activities until the administrator intervenes.

[0109] As grants are passed between storage nodes and grant owners update entities in their grants, the system passes messages between storage nodes and non-volatile solid-state storage units. Regarding permanent messages, messages with different purposes have different types. Depending on the type of message, the system maintains different ordering and persistence guarantees. When processing permanent messages, messages are temporarily stored in a variety of persistent and non-persistent storage hardware technologies. In some embodiments, messages are stored on RAM, NVRAM, and NAND flash devices, and various protocols are used to efficiently utilize each storage medium. Latency-sensitive client requests can be persisted in replicated NVRAM and then in NAND, while background rebalancing operations persist directly to the NAND.

[0110] Persistent messages are stored durably before transmission. This allows the system to continue to service client requests in the face of failures and component replacements. Although many hardware components contain unique identifiers visible to system administrators, manufacturers, hardware supply chains, and continuous monitoring quality control infrastructure, applications running on the infrastructure virtualize addresses. These virtualized addresses do not change over the life of the storage system, regardless of component failures or replacements. This allows each component of the storage system to be replaced over time without reconfiguration or interruption of client request processing, i.e., the system supports non-disruptive upgrades.

[0111] In some embodiments, virtualized addresses are stored with sufficient redundancy. A continuous monitoring system correlates hardware and software states with hardware identifiers. This allows for detection and prediction of failures due to faulty components and manufacturing details. In some embodiments, by removing components from the critical path, the monitoring system can also proactively transfer authorizations and entities away from affected devices before a failure occurs.

[0112] Figure 2C 1 is a multi-level block diagram showing the contents of a storage node 150 and the contents of a non-volatile solid-state storage device 152 of the storage node 150. In some embodiments, data is transferred to and from the storage node 150 via a network interface controller ('NIC') 202. Each storage node 150 has a CPU 156, and one or more non-volatile solid-state storage devices 152, as discussed above. Figure 2C Moving down one level in FIG. 1 , each non-volatile solid-state storage device 152 has relatively fast non-volatile solid-state memory, such as non-volatile random access memory ('NVRAM') 204, and flash memory 206. In some embodiments, NVRAM 204 may be a component that does not require program / erase cycles (DRAM, MRAM, PCM), and may be a memory that can support a write frequency that is much higher than the frequency at which the memory is read. Figure 2CMoving down one level in the NVRAM 204, in one embodiment, NVRAM 204 is implemented as a high-speed volatile memory, such as dynamic random access memory (DRAM) 216, backed up by an energy reserve 218. The energy reserve 218 provides sufficient power to keep the DRAM 216 powered long enough to transfer the contents to the flash memory 206 in the event of a power failure. In some embodiments, the energy reserve 218 is a capacitor, a supercapacitor, a battery, or another device that provides an appropriate energy supply sufficient to transfer the contents of the DRAM 216 to a stable storage medium in the event of a power outage. The flash memory 206 is implemented as a plurality of flash dies 222, which may be referred to as a package of flash dies 222 or an array of flash dies 222. It should be understood that the flash dies 222 may be packaged in any number of ways, including a single die per package, multiple dies per package (i.e., a multi-chip package), a hybrid package, as a bare die on a printed circuit board or other substrate, as an encapsulated die, and the like. In the illustrated embodiment, the non-volatile solid-state storage device 152 has a controller 212 or other processor, and an input-output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or the network interface controller 202 of the flash storage node 150. A flash input-output (I / O) port 220 is coupled to a flash die 222, and a direct memory access unit (DMA) 214 is coupled to the controller 212, the DRAM 216, and the flash die 222. In the illustrated embodiment, the I / O port 210, the controller 212, the DMA unit 214, and the flash I / O port 220 are implemented on a programmable logic device ('PLD') 208, such as an FPGA. In this embodiment, each flash die 222 has pages organized into sixteen kB (kilobyte) pages 224, and registers 226 that can be used to write data to or read data from the flash die 222. In other embodiments, other types of solid-state memory are used instead of or in addition to the flash memory shown within flash die 222 .

[0113] In various embodiments disclosed herein, a storage cluster 161 can be contrasted with a general storage array. A storage node 150 is part of a collection that creates a storage cluster 161. Each storage node 150 has a data slice and the computation required to provide the data. Multiple storage nodes 150 collaborate to store and retrieve data. Storage memories or storage devices generally used in storage arrays are less involved in the processing and manipulation of data. The storage memories or storage devices in the storage array receive commands to read, write, or erase data. The storage memories or storage devices in the storage array do not know the larger system in which they are embedded, nor do they know what the data means. The storage memories or storage devices in the storage array may include various types of storage memories, such as RAM, solid-state hard disks, hard disk drives, etc. The units of the non-volatile solid-state storage devices 152 described herein have multiple interfaces that are active at the same time and serve multiple purposes. In some embodiments, some functions of the storage node 150 are transferred to the storage unit 152, thereby transforming the storage unit 152 into a combination of the storage unit 152 and the storage node 150. Putting computation (relative to storing data) into the storage unit 152 will bring this computation closer to the data itself. Various system embodiments have a hierarchy of storage node layers with different functions. In contrast, in a storage array, a controller owns and knows everything about all the data that the controller manages in a rack or storage device. In a storage cluster 161, as described herein, multiple units of non-volatile solid-state storage devices 152 and / or multiple controllers in a storage node 150 collaborate in various ways (e.g., for erasure coding, data sharding, metadata communication and redundancy, storage capacity expansion or contraction, data recovery, etc.).

[0114] Figure 2D shows a storage server environment that uses Figure 2A -C storage node 150 and storage device 152 unit embodiment. In this version, each non-volatile solid-state storage device 152 unit is in the chassis 138 (see Figure 2A ) in a PCIe (Peripheral Component Interconnect Express) board having, for example, a controller 212 (see Figure 2C ) of the processor, FPGA, flash memory 206 and NVRAM 204 (which is DRAM 216 supporting supercapacitors, see Figure 2B and 2C ). The non-volatile solid-state storage device 152 unit can be implemented as a single board containing the storage device and can be the largest tolerable failure domain within the chassis. In some embodiments, up to two non-volatile solid-state storage device 152 units can fail, but the device can still continue without losing any data.

[0115] In some embodiments, the physical storage device is divided into named areas based on application use. NVRAM 204 is a continuous reserved memory block in the DRAM 216 of the non-volatile solid-state storage device 152 and is supported by NAND flash. NVRAM 204 is logically divided into multiple memory areas, two of which are written as temporary storage areas (spool) (e.g., spool_region). The space within the temporary storage area of ​​NVRAM 204 is managed independently by each authorization 168. Each device provides a certain amount of storage space to each authorization 168. The authorization 168 further manages the life and allocation within the space. Examples of temporary storage areas include distributed transactions or concepts. When the main power supply of the non-volatile solid-state storage device 152 unit fails, the on-board supercapacitor provides a short period of power retention. During this retention interval, the content of NVRAM 204 is refreshed to the flash memory 206. At the next power-on, the content of NVRAM 204 is restored from the flash memory 206.

[0116] With respect to the storage unit controller, the responsibility of the logical "controller" is distributed across each blade that contains authorization 168. This distribution of logical control is Figure 2D 1 and 2. The control plane and storage plane management are handled independently, but the parts may be physically co-located on the same blade. Each authority 168 effectively acts as an independent controller. Each authority 168 provides its own data and metadata structures, its own background workers, and maintains its own life cycle.

[0117] Figure 2E is used Figure 2D Storage server environment Figure 2A -C, showing a blade 252 hardware block diagram of an embodiment of a storage node 150 and a storage unit 152, showing a control plane 254, a compute and storage plane 256, 258, and an authorization 168 interacting with underlying physical resources. The control plane 254 is segmented into a number of authorizations 168, which can run on any blade 252 using the compute resources in the compute plane 256. The storage plane 258 is segmented into a set of devices, each of which provides access to the Flash 206 and NVRAM 204 resources. In one embodiment, the compute plane 256 can perform the operations of a storage array controller on one or more devices of the storage plane 258 (e.g., a storage array), as described herein.

[0118] exist Figure 2EIn the computing and storage planes 256, 258 of the system, the authorization 168 interacts with the underlying physical resources (i.e., devices). From the perspective of the authorization 168, its resources are striped across all physical devices. From the perspective of the device, it provides resources to all authorizations 168, regardless of where the authorization happens to be running. Each authorization 168 has or has been assigned one or more partitions 260 of storage memory in the storage unit 152, such as partitions 260 in flash memory 206 and NVRAM 204. Each authorization 168 uses those allocated partitions 260 belonging to it to write or read user data. Authorizations can be associated with different physical storage amounts of the system. For example, the number of partitions 260 or the size of the partitions 260 of one authorization 168 in one or more storage units 152 may be higher than one or more other authorizations 168.

[0119] Figure 2F Depicts the elasticity software layer in the blade 252 of the storage cluster according to some embodiments. In the elastic architecture, the elasticity software is symmetrical, that is, the computing module 270 of each blade runs Figure 2F The storage manager 274 executes read and write requests from other blades 252 for data and metadata stored in the local storage unit 152 NVRAM 204 and flash 206. The authorization 168 implements the client request by issuing the necessary reads and writes to the blade 252 on whose storage unit 152 the corresponding data or metadata resides. The endpoint 272 parses the client connection request received from the switch fabric 146 supervisory software, relays the client connection request to the authorization 168 responsible for implementation, and relays the response of the authorization 168 to the client. The symmetrical three-tier structure enables the storage system to have a high degree of parallelism. In these embodiments, elasticity scales out efficiently and reliably. In addition, elasticity implements a unique scale-out technology that evenly balances work across all resources regardless of client access patterns and maximizes parallelism by eliminating much of the need for inter-blade coordination that typically occurs in traditional distributed locking.

[0120] Still reference Figure 2F, the authorities 168 running in the compute module 270 of the blade 252 perform the internal operations required to fulfill the client request. One feature of resiliency is that the authorities 168 are stateless, that is, they cache active data and metadata in the DRAM of their own blade 252 for fast access, but the authorities store each update in their partitions of NVRAM 204 on three separate blades 252 until the update is written to flash 206. In some embodiments, all storage system writes to NVRAM 204 are three copies, written to the partitions on the three separate blades 252. With triply mirrored NVRAM 204 and permanent storage protected by parity and Reed-Solomon RAID checksums, the storage system can survive the simultaneous failure of two blades 252 without losing data, metadata, or access to either.

[0121] Because the grants 168 are stateless, they can migrate between blades 252. Each grant 168 has a unique identifier. In some embodiments, the partitions of NVRAM 204 and Flash 206 are associated with the identifier of the grant 168, but not with the blade 252 on which they run. Therefore, when a grant 168 migrates, the grant 168 continues to manage the same storage partitions from its new location. When a new blade 252 is installed in one embodiment of a storage cluster, the system automatically rebalances the load by partitioning the storage of the new blade 252 for use by the grants 168 of the system, migrating the selected grants 168 to the new blade 252, starting the endpoints 272 on the new blade 252 and including them in the client connection distribution algorithm of the switch fabric 146.

[0122] From their new locations, the migrated authorities 168 maintain the contents of their NVRAM 204 partitions on flash 206, process read and write requests from other authorities 168, and fulfill client requests directed to them by endpoints 272. Similarly, if a blade 252 fails or is removed, the system reallocates its authorities 168 among the remaining blades 252 in the system. The reallocated authorities 168 continue to perform their original functions in their new locations.

[0123] Figure 2GDepicted are the authorities 168 and storage resources in the blades 252 of a storage cluster according to some embodiments. Each authority 168 is specifically responsible for a partition of the flash 206 and NVRAM 204 on each blade 252. The authorities 168 manage the content and integrity of their partitions without being affected by other authorities 168. The authorities 168 compress incoming data and temporarily hold it in their partitions of NVRAM 204, and then merge, RAID protect and hold the data in the storage segments in their partitions of flash 206. When the authorities 168 write data to the flash 206, the storage manager 274 performs the necessary flash conversion to optimize write performance and maximize media life. In the background, the authorities 168 "garbage collect" or reclaim the space occupied by the data by overwriting the data that is invalidated by the client. It should be appreciated that because the partitions of the authorities 168 are disjoint, no distributed locking is required to execute clients and writes or to perform background functions.

[0124] The embodiments described herein may utilize various software, communication and / or networking protocols. In addition, the configuration of the hardware and / or software may be adjusted to accommodate various protocols. For example, embodiments may utilize Active Directory, which is a TM A database-based system that provides authentication, directory, policy, and other services in an environment. In these embodiments, LDAP (Lightweight Directory Access Protocol) is an example application protocol for querying and modifying entries in a directory service provider such as Active Directory. In some embodiments, a network lock manager ('NLM') is used as a facility to work in conjunction with the Network File System ('NFS') to provide System V-style advisory file and record locking over a network. The Server Message Block ('SMB') protocol, a version of which is also known as the Common Internet File System ('CIFS'), can be integrated with the storage systems discussed herein. SMB operates as an application layer network protocol and is commonly used to provide shared access to files, printers, and serial ports, as well as various communications between nodes on a network. SMB also provides an authenticated inter-process communication mechanism. Amazon TMS3 (Simple Storage Service) is a web service provided by Amazon Web Services, and the system described herein can interface with Amazon S3 through web service interfaces (REST (Representational State Transfer), SOAP (Simple Object Access Protocol) and BitTorrent). The RESTful API (Application Programming Interface) decomposes transactions to create a series of small modules. Each module handles a specific underlying part of the transaction. The control or permissions provided by these embodiments, especially for object data, can include the use of access control lists ('ACLs'). ACLs are lists of permissions attached to objects, and ACLs specify which users or system processes are granted access to objects, and what operations are allowed on given objects. The system can utilize the sixth version of the Internet Protocol ('IPv6') as well as IPv4 as communication protocols, which provide identification and positioning systems for computers on the network and route traffic on the Internet. Packet routing between network systems can include equal cost multipath routing ('ECMP'), which is a routing strategy in which next-hop packet forwarding to a single destination can occur on multiple "best paths" that are tied for first place in the routing metric calculation. Multipath routing can be used in conjunction with most routing protocols because it is a per-hop decision limited to a single router. The software may support multi-tenancy, which is an architecture in which a single instance of a software application serves multiple customers. Each customer may be referred to as a tenant. In some embodiments, tenants may be given the ability to customize portions of an application, but may not customize the application's code. An embodiment may maintain an audit log. An audit log is a document that records events in a computing system. In addition to recording which resources were accessed, audit log entries typically contain destination and source addresses, timestamps, and user login information to comply with various regulations. An embodiment may support various key management policies, such as encryption key rotation. Additionally, the system may support a dynamic root password or some variant of dynamically changing passwords.

[0125] Figure 3A A diagram of a storage system 306 coupled to communicate data with a cloud service provider 302 is illustrated in accordance with some embodiments of the present disclosure. Although depicted in less detail, Figure 3A The storage system 306 depicted in FIG. 3 may be similar to the storage system 306 described above with reference to FIG. Figures 1A-1D and Figure 2A-2G In some embodiments, Figure 3AThe storage system 306 depicted in the figure may be embodied as a storage system including unbalanced active / active controllers, a storage system including balanced active / active controllers, a storage system including active / active controllers in which less than all resources of each controller are utilized such that each controller has reserve resources available to support failover, a storage system including fully active / active controllers, a storage system including controllers with separated data sets, a storage system including a two-tier architecture with a front-end controller and a back-end integrated storage controller, a storage system including a scale-out cluster of dual-controller arrays, and combinations of these embodiments.

[0126] exist Figure 3A In the depicted example, the storage system 306 is coupled to the cloud service provider 302 via a data communication link 304. Such a data communication link 304 may be entirely wired, entirely wireless, or some aggregation of wired and wireless data communication paths. In such an example, digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using one or more data communication protocols. For example, the digital information may be exchanged between the storage system 306 and the cloud service provider 302 via the data communication link 304 using the Handheld Device Transfer Protocol ('HDTP'), the Hypertext Transfer Protocol ('HTTP'), the Internet Protocol ('IP'), the Real-time Transport Protocol ('RTP'), the Transmission Control Protocol ('TCP'), the User Datagram Protocol ('UDP'), the Wireless Application Protocol ('WAP'), or other protocols.

[0127] Figure 3A The cloud service provider 302 depicted in FIG. 3 may be embodied, for example, as a system and computing environment that provides a number of services to users of the cloud service provider 302 through the sharing of computing resources via data communication links 304. The cloud service provider 302 may provide on-demand access to a shared pool of configurable computing resources, such as computer networks, servers, storage, applications, and services, among others.

[0128] exist Figure 3A In the depicted example, cloud service provider 302 may be configured to provide various services to storage system 306 and users of storage system 306 through implementations of various service models. For example, cloud service provider 302 may be configured to provide services through implementations of an infrastructure as a service ('IaaS') service model, through implementations of a platform as a service ('PaaS') service model, through implementations of a software as a service ('SaaS') service model, through implementations of an authentication as a service ('AaaS') service model, through implementations of a storage as a service model (in which cloud service provider 302 provides access to its storage infrastructure for use by storage system 306 and users of storage system 306), and so forth.

[0129] exist Figure 3A In the depicted example, cloud service provider 302 may, for example, be embodied as a private cloud, a public cloud, or a combination of private and public clouds. In embodiments where cloud service provider 302 is embodied as a private cloud, cloud service provider 302 may be dedicated to providing services to a single organization, rather than providing services to multiple organizations. In embodiments where cloud service provider 302 is embodied as a public cloud, cloud service provider 302 may provide services to multiple organizations. In alternative embodiments, cloud service provider 302 may be embodied as a mix of private and public cloud services with a hybrid cloud deployment.

[0130] although Figure 3A 306 and users of the storage system 306. For example, the storage system 306 may be coupled to (or even include) a cloud storage gateway. For example, such a cloud storage gateway may be embodied as a hardware-based or software-based device located inside the storage system 306. Such a cloud storage gateway may be used as a bridge between local applications executed on the storage system 306 and the remote cloud-based storage utilized by the storage system 306. By using a cloud storage gateway, an organization can move primary iSCSI or NAS to the cloud service provider 302, enabling the organization to save space on their internal storage systems. Such a cloud storage gateway may be configured to emulate a disk array, a block-based device, a file server, or other storage system that can convert SCSI commands, file server commands, or other appropriate commands into a REST space protocol that facilitates communication with the cloud service provider 302.

[0131] In order to enable storage system 306 and users of storage system 306 to use the services provided by cloud service provider 302, a cloud migration process may be performed during which data, applications, or other elements from an organization's local systems (or even from another cloud environment) are moved to cloud service provider 302. In order to successfully migrate data, applications, or other elements to the environment of cloud service provider 302, middleware such as cloud migration tools may be utilized to bridge the gap between the environment of cloud service provider 302 and the environment of the organization. In order to further enable storage system 306 and users of storage system 306 to use the services provided by cloud service provider 302, a cloud orchestrator may also be used to schedule and coordinate automated tasks to create an integrated process or workflow. Such a cloud orchestrator may perform tasks such as configuring various components (whether these components are cloud components or on-premises components) and managing the interconnections between these components.

[0132] exist Figure 3AIn the depicted example, as briefly described above, cloud service provider 302 may be configured to provide services to storage system 306 and users of storage system 306 by using a SaaS service model. For example, cloud service provider 302 may be configured to provide storage system 306 and users of storage system 306 with access to data analytics applications. For example, such data analytics applications may be configured to consume large amounts of telemetry data transmitted back by storage system 306. Such telemetry data may describe various operational characteristics of storage system 306 and may be analyzed for a variety of purposes, including, for example, determining the health of storage system 306, identifying workloads executing on storage system 306, predicting when storage system 306 may exhaust various resources, recommending configuration changes, hardware or software upgrades, workflow migrations, or other actions that may improve storage system 306 operations.

[0133] The cloud service provider 302 may also be configured to provide access to a virtualized computing environment to the storage system 306 and users of the storage system 306. Examples of such virtualized environments may include virtual machines created to emulate actual computers, virtualized desktop environments that separate logical desktops from physical machines, virtualized file systems that allow uniform access to different types of specific file systems, and the like.

[0134] although Figure 3A The depicted example shows storage system 306 coupled to communicate data with cloud service provider 302, but in other embodiments, storage system 306 can be part of a hybrid cloud deployment in which private cloud elements (e.g., private cloud services, on-premises infrastructure, etc.) and public cloud elements (e.g., public cloud services, infrastructure, etc. that may be provided by one or more cloud service providers) are combined to form a single solution and orchestrated between the various platforms. Such hybrid cloud deployments can utilize hybrid cloud management software, such as Microsoft Azure. TM Azure TM Arc, which centralizes the management of hybrid cloud deployments to any infrastructure and enables services to be deployed anywhere. In such an example, the hybrid cloud management software can be configured to create, update, and delete resources (both physical and virtual) that form a hybrid cloud deployment, allocate compute and storage to specific workloads, monitor workloads and resources for performance, policy compliance, updates and patches, security status, or perform a variety of other tasks.

[0135] The reader will appreciate that by pairing the storage system described herein with one or more cloud service providers, a variety of offerings can be enabled. For example, a Disaster Recovery as a Service ('DRaaS') can be provided, in which cloud resources are utilized to protect applications and data from disruptions caused by disasters, including in embodiments where the storage system can act as a primary data store. In such embodiments, total system backup can be performed to ensure business continuity in the event of a system failure. In such embodiments, cloud data backup technology (alone or as part of a larger DRaaS solution) can also be integrated into an overall solution that includes the storage system described herein and a cloud service provider.

[0136] The storage systems and cloud service providers described herein can be used to provide a number of security features. For example, the storage system can encrypt data at rest (and data can be sent to and from the storage system in encrypted form), and key management as a service ('KMaaS') can be used to manage encryption keys, keys for locking and unlocking storage devices, etc. Similarly, a cloud data security gateway or similar mechanism can be used to ensure that data stored in the storage system is not mistakenly stored in the cloud as part of a cloud data backup operation. In addition, micro-segmentation or identity-based segmentation can be used within the data center or cloud service provider containing the storage system to create security zones in data centers and cloud deployments to isolate workloads from each other.

[0137] To further explain, Figure 3B A diagram of a storage system 306 according to some embodiments of the present disclosure is illustrated. Although not depicted in detail, Figure 3B The storage system 306 depicted in FIG. 3 may be similar to the storage system 306 described above with reference to FIG. Figures 1A-1D and Figure 2A-2G The storage system described above may include many of the components described above.

[0138] Figure 3BThe storage system 306 depicted in the example may include a number of storage resources 308, which may be embodied in many forms. For example, the storage resources 308 may include nano RAM or another form of non-volatile random access memory utilizing carbon nanotubes deposited on a substrate, 3D crosspoint non-volatile memory, flash memory including single-level cell ('SLC') NAND flash, multi-level cell ('MLC') NAND flash, triple-level cell ('TLC') NAND flash, quad-level cell ('QLC') NAND flash, or others. Likewise, the storage resources 308 may include non-volatile magnetoresistive random access memory ('MRAM'), including spin transfer torque ('STT') MRAM. The example storage resources 308 may alternatively include non-volatile phase change memory ('PCM'), quantum memory that allows photonic quantum information storage and retrieval, resistive random access memory ('ReRAM'), storage class memory ('SCM'), or other forms of storage resources, including any combination of the resources described herein. The reader will appreciate that the memory system described above may utilize other forms of computer memory and storage devices, including DRAM, SRAM, EEPROM, general purpose memory, and many others. Figure 3A The storage resources 308 depicted in FIG. 3 may be embodied in various form factors, including but not limited to dual in-line memory modules ('DIMM'), non-volatile dual in-line memory modules ('NVDIMM'), M.2, U.2, etc.

[0139] Figure 3B The storage resource 308 depicted in the figure may include various forms of SCM. The SCM can effectively treat fast, non-volatile memory (e.g., NAND flash) as an extension of DRAM, so that the entire data set can be treated as an in-memory data set that resides entirely in DRAM. The SCM may include non-volatile media, such as NAND flash. Such NAND flash can be accessed using NVMe, which can use the PCIe bus as its transmission method, and the access latency is relatively low compared to the old protocol. In fact, the network protocol used for SSDs in the all-flash array may include NVMe (ROCE, NVME TCP) using Ethernet, Fibre Channel (NVMe FC), Infinite Band (iWARP), and other protocols that can treat fast, non-volatile memory as an extension of DRAM. Given that DRAM is typically byte-addressable, and fast non-volatile memory such as NAND flash is block-addressable, a controller software / hardware stack may be required to convert block data into bytes stored in the medium. For example, examples of media and software that can be used as SCM may include 3D XPoint, Intel Memory Drive Technology, Samsung's Z-SSD, and the like.

[0140] Figure 3BThe storage resource 308 depicted in FIG. 3 may also include racetrack memory (also known as domain wall memory). Such racetrack memory may be embodied as a type of non-volatile solid-state memory that relies on the inherent strength and direction of the magnetic field generated by electrons in a solid-state device as they spin, as well as their charge. By using a spin-coherent current to move magnetic domains along a nanoscale Permalloy wire, as the current passes through the wire, the magnetic domains may pass by a magnetic read / write head located near the wire, thereby changing the magnetic domains to record a bit pattern. To create a racetrack memory device, many such wires and read / write elements may be packaged together.

[0141] Figure 3B The example storage system 306 depicted in can implement various storage architectures. For example, a storage system according to some embodiments of the present disclosure can utilize block storage, where data is stored in blocks and each block essentially acts as a separate hard disk. A storage system according to some embodiments of the present disclosure can utilize object storage, where data is managed as an object. Each object can contain the data itself, a variable amount of metadata, and a globally unique identifier, where object storage can be implemented at multiple levels (e.g., device level, system level, interface level). A storage system according to some embodiments of the present disclosure utilizes file storage, where data is stored in a hierarchical structure. Such data can be stored in files and folders and presented to its storage system and retrieval system in the same format.

[0142] Figure 3B The example storage system 306 depicted in FIG. 300 may be embodied as a storage system in which additional storage resources may be added using a scale-up model, additional storage resources may be added using a scale-out model, or a combination of the two models. In a scale-up model, additional storage may be added by adding additional storage devices. However, in a scale-out model, additional storage nodes may be added to a cluster of storage nodes, wherein the storage nodes may include additional processing resources, additional network resources, and the like.

[0143] Figure 3B The example storage system 306 depicted in FIG. 3 can utilize the storage resources described above in a variety of different ways. For example, a portion of the storage resources can be used to act as a write cache, storage resources within the storage system can be used as a read cache, or tiering within the storage system can be implemented by placing data within the storage system according to one or more tiering policies.

[0144] Figure 3BThe storage system 306 depicted in the figure also includes communication resources 310, which can be used to facilitate data communication between components within the storage system 306, and data communication between the storage system 306 and computing devices external to the storage system 306, including embodiments in which these resources are separated by a relatively wide space. The communication resources 310 can be configured to facilitate data communication between components within the storage system and with computing devices external to the storage system using a variety of different protocols and data communication structures. For example, the communication resources 310 can include Fibre Channel ('FC') technology, such as FC structures and FC protocols that can transmit SCSI commands through FC networks, FC over Ethernet ('FCoE') technology (FC frames can be encapsulated by this technology and transmitted through Ethernet networks), InfiniBand ('IB') technology (in which a switching fabric topology is used to facilitate transmission between channel adapters), NVM Express ('NVMe') technology, and NVMe over Fabric ('NVMeoF') technology (through which non-volatile storage media attached via a PCI Express ('PCIe') bus can be accessed), etc. In fact, the storage system described above can directly or indirectly utilize neutrino communication technology and devices, through which information (including binary information) is transmitted using neutrino beams.

[0145] The communication resources 310 may also include mechanisms for accessing storage resources 308 within the storage system 306 utilizing Serial Attached SCSI ('SAS'), a Serial ATA ('SATA') bus interface for connecting storage resources 308 in the storage system 306 to a host bus adapter within the storage system 306, Internet Small Computer System Interface ('iSCSI') technology for providing block-level access to storage resources 308 within the storage system 306, and other communication resources that may be used to facilitate data communications between components within the storage system 306 and data communications between the storage system 306 and computing devices external to the storage system 306.

[0146] Figure 3B The storage system 306 depicted in FIG. 3 also includes processing resources 312 that can be used to execute computer program instructions and perform other computing tasks within the storage system 306. The processing resources 312 can include one or more ASICs customized for a particular purpose, as well as one or more CPUs. The processing resources 312 can also include one or more DSPs, one or more FPGAs, one or more systems on chips ('SoCs'), or other forms of processing resources 312. The storage system 306 can utilize the storage resources 312 to perform a variety of tasks, including but not limited to supporting the execution of software resources 314, which will be described in more detail below.

[0147] Figure 3BThe storage system 306 depicted in FIG. 3 also includes software resources 314 that, when executed by processing resources 312 within the storage system 306, can perform a number of tasks. The software resources 314 can include, for example, one or more computer program instruction modules that, when executed by processing resources 312 within the storage system 306, can be used to implement various data protection techniques. Such data protection techniques can be implemented, for example, by system software executed on computer hardware within the storage system, by a cloud service provider, or otherwise. Such data protection techniques can include data archiving, data backup, data replication, data snapshots, data and database cloning, and other data protection techniques.

[0148] The software resources 314 may also include software that can be used to implement software defined storage ('SDS'). In such examples, the software resources 314 may include one or more computer program instruction modules that, when executed, can be used for policy-based provisioning and data storage management, independent of the underlying hardware. Such software resources 314 can be used to implement storage virtualization to separate storage hardware from the software that manages the storage hardware.

[0149] The software resources 314 may also include software that can be used to facilitate and optimize I / O operations involving the storage system 306. For example, the software resources 314 may include software modules that perform various data reduction techniques, such as data compression, data deduplication, etc. The software resources 314 may include software modules that intelligently merge I / O operations to facilitate better use of the underlying storage resources 308, software modules that perform data migration operations to migrate from within the storage system, and software modules that perform other functions. Such software resources 314 may be embodied as one or more software containers or in many other ways.

[0150] To further explain, Figure 3C An example of a cloud-based storage system 318 according to some embodiments of the present disclosure is described. Figure 3C In the depicted example, cloud-based storage system 318 is formed entirely within cloud computing environment 316, such as Amazon Web Services ('AWS'). TM , Microsoft Azure TM , Google Cloud Management Platform TM , IBM Cloud TM , Oracle Cloud TM Etc. Cloud-based storage system 318 may be used to provide services similar to those that may be provided by the storage systems described above.

[0151] Figure 3CThe cloud-based storage system 318 depicted in FIG. 1 includes two cloud computing instances 320, 322 for supporting the execution of storage controller applications 324, 326, respectively. The cloud computing instances 320, 322 may, for example, be embodied as instances of cloud computing resources (e.g., virtual machines) that may be provided by the cloud computing environment 316 to support the execution of software applications such as the storage controller applications 324, 326. For example, each of the cloud computing instances 320, 322 may be executed on an Azure VM, where each Azure VM may include a high-speed temporary storage device that may be used as a cache (e.g., a read cache). In one embodiment, the cloud computing instances 320, 322 may be embodied as Amazon Elastic Compute Cloud ('EC2') instances. In such examples, an Amazon Machine Image ('AMI') containing the storage controller applications 324, 326 may be launched to create and configure a virtual machine that may execute the storage controller applications 324, 326.

[0152] exist Figure 3C In the example method depicted, the storage controller applications 324, 326 may be embodied as computer program instruction modules that, when executed, perform various storage tasks. For example, the storage controller applications 324, 326 may be embodied as computer program instruction modules that, when executed, perform the same tasks as described above. Figure 1A The controllers 110A, 110B in the cloud-based storage system 318 perform the same tasks, such as writing data to the cloud-based storage system 318, erasing data from the cloud-based storage system 318, retrieving data from the cloud-based storage system 318, monitoring and reporting storage device utilization and performance, performing redundancy operations (such as RAID or RAID-like data redundancy operations), compressing data, encrypting data, deduplicating data, etc. The reader will understand that because there are two cloud computing instances 320, 322, each containing a storage controller application 324, 326, in some embodiments, one cloud computing instance 320 can be used as a primary controller, as described above, and the other cloud computing instance 322 can be used as a secondary controller, as described above. The reader will understand that Figure 3C The storage controller applications 324, 326 depicted in FIG. 3 may contain the same source code that executes within different cloud computing instances 320, 322 (eg, different EC2 instances).

[0153] The reader will appreciate that other embodiments that do not include primary and secondary controllers are within the scope of the present disclosure. For example, each cloud computing instance 320, 322 may function as a primary controller for some portion of the address space supported by the cloud-based storage system 318, each cloud computing instance 320, 322 may function as a primary controller in which services involving I / O operations of the cloud-based storage system 318 are partitioned in some other manner, etc. Indeed, in other embodiments where cost savings may take precedence over performance requirements, there may be only a single cloud computing instance containing a storage controller application.

[0154] Figure 3C The cloud-based storage system 318 depicted in FIG. 3 includes cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338. The cloud computing instances 340a, 340b, 340n may, for example, embody instances of cloud computing resources that may be provided by the cloud computing environment 316 to support execution of software applications. Figure 3C The cloud computing instances 340a, 340b, 340n may be different from the cloud computing instances 320, 322 described above because Figure 3C The cloud computing instances 340a, 340b, 340n of the storage controller applications 324, 326 have local storage devices 330, 334, 338 resources, while the cloud computing instances 320, 322 supporting the execution of the storage controller applications 324, 326 do not need to have local storage resources. The cloud computing instances 340a, 340b, 340n with local storage devices 330, 334, 338 can be, for example, embodied as EC2 M5 instances including one or more SSDs, EC2 R5 instances including one or more SSDs, EC2 I3 instances including one or more SSDs, and the like. In some embodiments, the local storage devices 330, 334, 338 must be embodied as solid-state storage devices (e.g., SSDs) rather than storage devices using hard disk drives.

[0155] exist Figure 3CIn the depicted example, each of the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338 may include software daemons 328, 332, 336 that, when executed by the cloud computing instances 340a, 340b, 340n, may present themselves to the storage controller applications 324, 326 as if the cloud computing instances 340a, 340b, 340n were physical storage devices (e.g., one or more SSDs). In such examples, the software daemons 328, 332, 336 may include computer program instructions similar to those normally included on the storage devices, such that the storage controller applications 324, 326 may send and receive the same commands that the storage controller sends to the storage devices. In this way, the storage controller applications 324, 326 may include the same (or substantially the same) code that would be executed by the controller in the storage system described above. In these and similar embodiments, communication between the storage controller applications 324, 326 and the cloud computing instances 340a, 340b, 340n having local storage devices 330, 334, 338 may utilize iSCSI, NVMe over TCP, messaging, a custom protocol, or some other mechanism.

[0156] exist Figure 3C In the depicted example, each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 may also be coupled to a block storage 342, 344, 346 provided by the cloud computing environment 316, such as an Amazon Elastic Block Store ('EBS') volume. In such an example, the block storage 342, 344, 346 provided by the cloud computing environment 316 may be utilized in a manner similar to the manner in which the NVRAM devices described above are utilized, in that the software daemon 328, 332, 336 (or some other module) executed within a particular cloud computing instance 340a, 340b, 340n may initiate the writing of data to its attached EBS volume and the writing of data to its local storage 330, 334, 338 resources upon receiving a request to write data. In some alternative embodiments, data may only be written to the local storage 330, 334, 338 resources within a particular cloud computing instance 340a, 340b, 340n. In an alternative embodiment, instead of using the block storage 342, 344, 346 provided by the cloud computing environment 316 as NVRAM, actual RAM on each of the cloud computing instances 340a, 340b, 340n having local storage 330, 334, 338 can be used as NVRAM, thereby reducing the network utilization costs associated with using EBS volumes as NVRAM. In yet another embodiment, a high-performance block storage resource such as one or more AzureUltra disks can be used as NVRAM.

[0157] When a particular cloud computing instance 340a, 340b, 340n having a local storage 330, 334, 338 receives a request to write data, the software daemon 328, 332, 336 may be configured to write the data not only to its own local storage 330, 334, 338 resources and any appropriate block storage 342, 344, 346 resources, but the software daemon 328, 332, 336 may also be configured to write the data to a cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n. The cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n may be embodied, for example, as Amazon Simple Storage Service ('S3'). In other embodiments, cloud computing instances 320, 322, each including a storage controller application 324, 326, can initiate the storage of data in local storage devices 330, 334, 338 and cloud-based object storage devices 348 of cloud computing instances 340a, 340b, 340n. In other embodiments, rather than using both cloud computing instances 340a, 340b, 340n with local storage devices 330, 334, 338 (also referred to herein as 'virtual drives') and cloud-based object storage devices 348 to store data, a persistent storage layer can be implemented in other ways. For example, one or more Azure Ultra disks can be used to store data permanently (e.g., after the data has been written to the NVRAM layer). In embodiments where one or more Azure Ultra disks can be used to store data permanently, the use of cloud-based object storage devices 348 can be removed so that data is only permanently stored in Azure Ultra disks, and data is not written to the object storage layer.

[0158] Although the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n can support block-level access, the cloud-based object storage 348 attached to a particular cloud computing instance 340a, 340b, 340n only supports object-based access. Therefore, the software daemons 328, 332, 336 can be configured to retrieve data blocks, encapsulate those blocks into objects, and write the objects to the cloud-based object storage 348 attached to the particular cloud computing instance 340a, 340b, 340n.

[0159] In some embodiments, all data stored by the cloud-based storage system 318 may be stored in both: 1) a cloud-based object store 348, and 2) at least one of a local storage 330, 334, 338 resource or a block storage 342, 344, 346 resource utilized by the cloud computing instances 340a, 340b, 340n. In such embodiments, the local storage 330, 334, 338 resources and block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n may effectively be used as a cache that typically contains all data also stored in S3, such that all data reads may be served by the cloud computing instances 340a, 340b, 340n without requiring the cloud computing instances 340a, 340b, 340n to access the cloud-based object store 348. However, the reader will appreciate that in other embodiments, all of the data stored by the cloud-based storage system 318 may be stored in the cloud-based object store 348, but less than all of the data stored by the cloud-based storage system 318 may be stored in at least one of the local storage 330, 334, 338 resources or block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n. In such examples, various policies may be utilized to determine which subset of the data stored by the cloud-based storage system 318 should reside in: 1) the cloud-based object store 348, and 2) at least one of the local storage 330, 334, 338 resources or block storage 342, 344, 346 resources utilized by the cloud computing instances 340a, 340b, 340n.

[0160] One or more computer program instruction modules executing within the cloud-based storage system 318 (e.g., a monitoring module executing on its own EC2 instance) can be designed to handle the failure of one or more of the cloud computing instances 340a, 340b, 340n with local storage 330, 334, 338. In such an example, the monitoring module can handle the failure of one or more of the cloud computing instances 340a, 340b, 340n with local storage 330, 334, 338 by creating one or more new cloud computing instances with local storage, retrieving data that was stored on the failed cloud computing instance 340a, 340b, 340n from the cloud-based object storage 348, and storing the data retrieved from the cloud-based object storage 348 in the local storage of the newly created cloud computing instance. The reader will appreciate that many variations of this process can be implemented.

[0161] The reader will appreciate that various performance aspects of the cloud-based storage system 318 may be monitored (e.g., by a monitoring module executing in an EC2 instance) so that the cloud-based storage system 318 may be scaled up or down as needed. For example, if the cloud computing instance 320, 322 used to support the execution of the storage controller application 324, 326 is too small and cannot adequately service I / O requests issued by users of the cloud-based storage system 318, the monitoring module may create a new, more powerful cloud computing instance (e.g., a type of cloud computing instance that includes more processing power, more memory, etc.) containing a storage controller application that enables the new, more powerful cloud computing instance to begin operating as a primary controller. Similarly, if the monitoring module determines that the cloud computing instance 320, 322 used to support the execution of the storage controller application 324, 326 is too large and that costs can be saved by switching to a smaller, less powerful cloud computing instance, the monitoring module may create a new, less powerful (and less expensive) cloud computing instance containing a storage controller application that enables the new, less powerful cloud computing instance to begin operating as a primary primary controller.

[0162] The storage system described above can perform intelligent data backup technology, by which data stored in the storage system can be copied and stored in different locations to avoid data loss when a device failure or other form of mutation occurs. For example, the storage system described above can be configured to check each backup to avoid restoring the storage system to an undesirable state. Consider an example in which malware contaminates the storage system. In such an example, the storage system can include software resources 314 that can scan each backup to identify backups captured before the malware contaminates the storage system and backups captured after the malware contaminates the storage system. In such an example, the storage system can restore itself from a backup that does not contain malware, or at least not restore the backup portion that contains malware. In such an example, the storage system can include software resources 314 that can scan each backup to identify the presence of malware (or viruses, or some other undesirable malware), for example, by identifying write operations served by the storage system and originating from a network subnet suspected of delivering malware, by identifying write operations served by the storage system and originating from a user suspected of delivering malware, by identifying write operations served by the storage system and checking the content of the write operations according to the fingerprint of the malware, and in many other ways.

[0163] The reader will further appreciate that backups (typically in the form of one or more snapshots) may also be used to perform rapid recovery of the storage system. Consider an example in which a storage system is contaminated with ransomware that locks users out of the storage system. In such an example, software resources 314 within the storage system may be configured to detect the presence of ransomware, and may be further configured to use a retained backup to restore the storage system to a point in time prior to the point in time when the ransomware was contaminated with the storage system. In such an example, the presence of ransomware may be explicitly detected by using a software tool used by the system, by using a key (e.g., a USB drive) inserted into the storage system, or in a similar manner. Similarly, the presence of ransomware may be inferred in response to system activity satisfying a predetermined fingerprint, such as, for example, no reading or writing to the system for a predetermined period of time.

[0164] The reader will appreciate that the various components described above can be grouped into one or more optimized computing packages as a converged infrastructure. Such a converged infrastructure can contain pools of computer, storage, and network connectivity resources that can be shared by multiple applications and managed in a collective manner using policy-driven processes. Such a converged infrastructure can be implemented using a converged infrastructure reference architecture, a standalone appliance, a software-driven hyper-convergence approach (e.g., hyper-converged infrastructure), or other approaches.

[0165] The reader will appreciate that the storage systems described in this disclosure can be used to support various types of software applications. In fact, the storage system can be "application-aware" in that the storage system can obtain, maintain, or otherwise access information describing connected applications (e.g., applications utilizing the storage system) to optimize the operation of the storage system based on intelligence about the applications and their utilization patterns. For example, the storage system can optimize data layout, optimize cache behavior, optimize 'QoS' tiers, or perform other optimizations designed to improve the storage performance experienced by the application.

[0166] As an example of an application that may be supported by the storage system described herein, storage system 306 may be used to support artificial intelligence ('AI') applications, database applications, XOps projects (e.g., DevOps projects, DataOps projects, MLOps projects, ModelOps projects, PlatformOps projects), electronic design automation tools, event-driven software applications, high-performance computing applications, simulation applications, high-speed data capture and analysis applications, machine learning applications, media production applications, media serving applications, picture archiving and communication system ('PACS') applications, software development applications, virtual reality applications, augmented reality applications, and many other types of applications by providing storage resources to these applications.

[0167] Given the fact that storage systems contain compute resources, storage resources, and a variety of other resources, storage systems may be well suited to support resource-intensive applications, such as AI applications. AI applications can be deployed in a variety of areas, including: predictive maintenance in manufacturing and related fields, healthcare applications such as patient data and risk analysis, retail and marketing deployments (e.g., search advertising, social media advertising), supply chain solutions, fintech solutions such as business analytics and reporting tools, operational deployments such as real-time analytics tools, application performance management tools, IT infrastructure management tools, and many other areas.

[0168] Such AI applications enable a device to perceive its environment and take actions to maximize its chances of success in achieving a certain goal. Examples of such AI applications can include IBM Watson TM 、Microsoft Oxford TM 、Google DeepMind TM 、Baidu Minwa TM wait.

[0169] The storage systems described above may also be well suited to support other types of resource-intensive applications, such as machine learning applications. Machine learning applications can perform various types of data analysis to automate analytical model building. Using algorithms that iteratively learn from data, machine learning applications can enable computers to learn without being explicitly programmed. A specific area of ​​machine learning is called reinforcement learning, which involves taking appropriate actions in a given situation to maximize rewards.

[0170] In addition to the resources already described, the storage system described above may also include a graphics processing unit ('GPU'), sometimes also referred to as a visual processing unit ('VPU'). Such a GPU may be embodied as a dedicated electronic circuit that quickly manipulates and changes memory to accelerate the creation of images in a frame buffer for output to a display device. Such a GPU may be included in any computing device that is part of the storage system described above, including one of many separately scalable components of the storage system, where other examples of separately scalable components of such a storage system may include storage components, memory components, computing components (e.g., CPUs, FPGAs, ASICs), network connection components, software components, etc. In addition to the GPU, the storage system described above may also include a neural network processor ('NNP') for various aspects of neural network processing. Such an NNP may be used in place of (or in addition to) a GPU, or may be independently scalable.

[0171] As mentioned above, the storage system described in this article can be configured to support artificial intelligence applications, machine learning applications, big data analysis applications, and many other types of applications. The rapid growth of such applications is driven by three technologies: deep learning (DL), GPU processors, and big data. Deep learning is a computing model that uses massively parallel neural networks inspired by the human brain. Deep learning models write their own software by learning from a large number of examples, rather than having the software handcrafted by experts. Such GPUs can contain thousands of cores that are well suited to running algorithms that loosely represent the parallel nature of the human brain.

[0172] Advances in deep neural networks, including the development of multi-layer neural networks, have triggered a new wave of algorithms and tools for data scientists to mine data using artificial intelligence (AI). With the advent of improved algorithms, larger datasets, and a variety of frameworks (including open source software libraries for machine learning across a range of tasks), data scientists are tackling new use cases such as self-driving vehicles, natural language processing and understanding, computer vision, machine reasoning, strong AI, and more. The application of AI technology has been implemented in a large number of products, including speech recognition technology such as Amazon Echo, which allows users to talk to their machines; Google Translate TM , which allows machine-based language translation; Spotify's Discover Weekly, which provides recommendations for new songs and artists that users may like based on their usage and traffic analytics; Quill's text generation product, which takes structured data and turns it into a narrative story; Chatbot, which provides real-time, context-specific answers to questions in a conversational format; and many others.

[0173] Data is at the heart of modern AI and deep learning algorithms. Before training can begin, one problem that must be solved is collecting labeled data, which is critical to training accurate AI models. A comprehensive AI deployment may require the continuous collection, cleaning, transformation, labeling, and storage of large amounts of data. Adding additional high-quality data points directly translates into more accurate models and better insights. Data samples can go through a series of processing steps, including but not limited to: 1) ingesting data from external sources into the training system and storing the data in raw form, 2) cleaning and transforming the data in a format that is convenient for training, including linking data samples to appropriate labels, 3) exploring parameters and models, using smaller datasets to quickly test and iterate to converge to the most promising model to push into production clusters, 4) performing a training phase to select random batches of input data, including new and old samples, and feeding this data into production GPU servers for computation to update model parameters, and 5) evaluating the model accuracy on the held-out data, including using a held-out portion of the data not used in training. This life cycle can apply to any type of parallelized machine learning, not just neural networks or deep learning. For example, a standard machine learning framework may rely on CPUs instead of GPUs, but the data ingestion and training workflows can be the same. Readers will learn that a single shared storage data center creates a coordination point throughout the lifecycle without the need to create additional copies of data between ingestion, preprocessing, and training phases. Ingested data is rarely used for just one purpose, and shared storage provides the flexibility to train multiple different models or apply traditional analytics to the data.

[0174] The reader will understand that each stage in the AI ​​data pipeline may have different requirements for the data center (e.g., storage system or collection of storage systems). Scale-out storage systems must provide uncompromised performance for a variety of access types and patterns - from small files, metadata-heavy files to large files, from random access patterns to sequential access patterns, and from low concurrency to high concurrency. The storage systems described above can serve as ideal AI data centers because these systems can serve unstructured workloads. In the first stage, ideally, the data is ingested and stored in the same data center that will be used in subsequent stages to avoid excessive data replication. The next two steps can be completed on standard compute servers that optionally contain GPUs, and then in the fourth and final stage, the full training production job is run on powerful GPU-accelerated servers. Often, there is a production pipeline in addition to the experimental pipeline operating on the same dataset. In addition, GPU-accelerated servers can be used independently for different models, or they can be connected together to train on a larger model, or even distributed across multiple systems for training. If the shared storage layer is slow, the data must be copied to local storage at each stage, wasting time moving the data to different servers. The ideal data center for an AI training pipeline provides performance similar to data stored locally on server nodes, while also offering simplicity and the ability to enable all pipeline stages to operate simultaneously.

[0175] In order for the storage system described above to be used as part of a data center or AI deployment, in some embodiments, the storage system can be configured to provide DMA between a storage device included in the storage system and one or more GPUs used in an AI or big data analytics pipeline. One or more GPUs can be coupled to the storage system, for example, via NVMe over Fabric ('NVMe-oF'), so that bottlenecks such as the host CPU can be bypassed, and the storage system (or one of the components included therein) can directly access the GPU memory. In such an example, the storage system can utilize an API hook to the GPU to pass data directly to the GPU. For example, the GPU can be embodied as an Nvidia TM The GPU, and the storage system may support GPUDirect Storage ('GDS') software, or have similar proprietary software, enabling the storage system to pass data to the GPU via RDMA or similar mechanisms.

[0176] Although the preceding paragraphs discuss deep learning applications, the reader should understand that the storage system described herein may also be part of a distributed deep learning ('DDL') platform to support the execution of DDL algorithms. The storage system described above may also be paired with other technologies (e.g., TensorFlow, an open source software library for data flow programming across a range of tasks that can be used for machine learning applications such as neural networks) to facilitate the development of such machine learning models, applications, etc.

[0177] The storage system described above can also be used in a neuromorphic computing environment. Neuromorphic computing is a form of computing that simulates brain cells. To support neuromorphic computing, interconnected "neuron" architectures replace traditional computing models with low-power signals that pass directly between neurons to achieve more efficient computing. Neuromorphic computing can utilize very large scale integration (VLSI) systems containing electronic analog circuits to simulate the neurobiological architecture present in the nervous system, as well as analog, digital, mixed-mode analog / digital VLSI and software systems that implement models of nervous systems for perception, motor control, or multi-sensory integration.

[0178] The reader will appreciate that the storage system described above can be configured to support the storage or use of blockchains and derivatives (and other types of data), such as IBM TM The open source blockchain and related tools that are part of the Hyperledger project, permissioned blockchains that allow a certain number of trusted parties to access the blockchain, blockchain products that enable developers to build their own distributed ledger projects, etc. The blockchain and storage system described in this article can be used to support both on-chain and off-chain storage of data.

[0179] The off-chain storage of data can be implemented in a variety of ways and can occur when the data itself is not stored in the blockchain. For example, in one embodiment, a hash function can be utilized, and the data itself can be fed into the hash function to generate a hash value. In such an example, the hash value of a large block of data can be embedded in the transaction rather than the data itself. The reader will appreciate that in other embodiments, alternatives to blockchain can be used to facilitate the decentralized storage of information. For example, one alternative to blockchain that can be used is blockweave. While traditional blockchains store each transaction for verification, blockweave allows for secure decentralization without using the entire chain, thereby achieving low-cost on-chain storage of data. This blockweave can utilize a consensus mechanism based on proof of access (PoA) and proof of work (PoW).

[0180] The storage systems described above can be used alone or in combination with other computing devices to support in-memory computing applications. In-memory computing involves storing information in RAM distributed across a cluster of computers. The reader will appreciate that the storage systems described above, particularly those that are configurable with customizable amounts of processing resources, storage resources, and memory resources (e.g., those systems where the blades contain configurable amounts of each type of resource), can be configured in a manner to provide an infrastructure that can support in-memory computing. Similarly, the storage systems described above can include component portions (e.g., NVDIMMs, 3D crosspoint storage devices that provide persistent, fast random access memory) that can actually provide an improved in-memory computing environment compared to an in-memory computing environment that relies on RAM distributed across dedicated servers.

[0181] In some embodiments, the storage system described above can be configured to operate as a hybrid in-memory computing environment that includes a common interface to all storage media (e.g., RAM, flash storage devices, 3D crosspoint storage devices). In such an embodiment, users may not know the details of where their data is stored, but they can still use the same complete, unified API to address the data. In such an embodiment, the storage system can (in the background) move the data to the fastest tier available - including intelligently placing the data based on various characteristics of the data or based on some other heuristic methods. In such an example, the storage system can even leverage existing products, such as Apache Ignite and GridGain, to move data between storage tiers, or the storage system can leverage custom software to move data between storage tiers. The storage system described herein can implement various optimizations to improve the performance of in-memory computing, for example, by performing computing as close to the data as possible.

[0182] The reader will further appreciate that in some embodiments, the storage system described above can be paired with other resources to support the applications described above. For example, an infrastructure can include primary computing in the form of servers and workstations that specifically use general-purpose computing on graphics processing units ('GPGPU') to accelerate deep learning applications, which are interconnected with computing engines to train parameters for deep neural networks. Each system can have an Ethernet external connection, an InfiniBand external connection, some other form of external connection, or some combination thereof. In such an example, GPUs can be grouped for a single large training, or they can be used independently to train multiple models. The infrastructure can also include storage systems such as those described above to provide, for example, horizontally scaled all-flash file or object storage, through which data can be accessed via high-performance protocols such as NFS, S3, etc. The infrastructure can also include, for example, redundant top-of-rack Ethernet switches, which are connected via ports in MLAG port channels for storage and computing to achieve redundancy. The infrastructure can also include additional computing in the form of white box servers, optionally using GPUs, for data ingestion, preprocessing, and model debugging. The reader will appreciate that additional infrastructure is also possible.

[0183] The reader will appreciate that the storage system described above, whether used alone or in conjunction with other computing machinery, can be configured to support other AI-related tools. For example, the storage system can use tools such as ONXX or other open neural network exchange formats to make it easier to transfer models written in different AI frameworks. Similarly, the storage system can be configured to support tools such as Amazon's Gluon, which allows developers to prototype, build, and train deep learning models. In fact, the storage system described above can be part of a larger platform, such as IBM's TM Cloud Private for Data, which includes integrated data science, data engineering, and application building services.

[0184] The reader will further appreciate that the storage system described above can also be deployed as an edge solution. Such an edge solution can optimize cloud computing systems by performing data processing at the edge of the network near the data source. Edge computing can push applications, data, and computing power (i.e., services) from centralized points to the logical extremes of the network. By using an edge solution such as the storage system described above, computing tasks can be performed using the computing resources provided by such a storage system, data can be stored using the storage resources of the storage system, and cloud-based services can be accessed by using the various resources of the storage system (including network connection resources). By performing computing tasks on edge solutions, storing data on edge solutions, and primarily using edge solutions, consumption of expensive cloud-based resources can be avoided, and in fact, performance improvements can be achieved relative to a greater reliance on cloud-based resources.

[0185] While many tasks may benefit from utilizing edge solutions, certain specific uses may be particularly well suited for deployment in such environments. For example, devices such as drones, self-driving cars, robots, and the like may require extremely fast processing - so fast, in fact, that sending data to a cloud environment and back to receive data processing support may be too slow. As an additional example, some IoT devices, such as connected cameras, may not be well suited to utilizing cloud-based resources because sending data to the cloud may be impractical (not only from a privacy, security, but also from a financial perspective) simply because of the large amount of data involved. Therefore, many tasks that truly involve data processing, storage, or communication may be better suited for platforms that include edge solutions such as the storage systems described above.

[0186] The storage system described above can be used alone or in combination with other computing resources as a network edge platform that combines computing resources, storage resources, network connection resources, cloud technologies, and network virtualization technologies. As part of the network, the edge may have similar characteristics to other network facilities from customer premises and backhaul aggregation facilities to access points (PoPs) and regional data centers. The reader will understand that network workloads such as virtual network functions (VNFs) will reside on network edge platforms. Through a combination of containers and virtual machines, network edge platforms may rely on controllers and schedulers that are no longer geographically co-located with data processing resources. As microservices, these functions can be divided into control planes, user and data planes, and even state machines, allowing independent optimization and expansion techniques to be applied. Such user and data planes can be implemented through the addition of accelerators (both of which reside in server platforms such as FPGAs and smart NICs) and through commercial silicon and programmable ASICs that support SDN.

[0187] The storage systems described above can also be optimized for big data analytics use, including use as part of a composable data analytics pipeline, where a containerized analytics architecture, for example, makes analytics capabilities more composable. Big data analytics can generally be described as the process of examining large and diverse data sets to discover hidden patterns, unknown correlations, market trends, customer preferences, and other useful information that can help organizations make more informed business decisions. As part of the process, semi-structured and unstructured data, such as Internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone call detail records, IoT sensor data, and other data, can all be converted into a structured form.

[0188] The storage system described above may also support (including being implemented as a system interface) applications that perform tasks in response to human speech. For example, the storage system may support the execution of an intelligent personal assistant application, such as Amazon's Alexa. TM 、Apple Siri TM , Google Voice TM 、Samsung Bixby TM , Microsoft Cortana TM Etc. Although the example described in the previous sentence uses voice as input, the storage system described above may also support chatbot, talkbot, chatterbot, or artificial dialogue entities or other applications configured to conduct conversations via auditory or text methods. Similarly, the storage system may actually execute such applications to enable users such as system administrators to interact with the storage system via voice. Although such applications are typically capable of voice interaction, music playback, making to-do lists, setting alarms, streaming podcasts, playing audiobooks, and providing weather, traffic, and other real-time information such as news, in embodiments according to the present disclosure, such applications may be used as an interface to various system management operations.

[0189] The storage systems described above can also implement an AI platform to realize the vision of self-driving storage. Such an AI platform can be configured to provide global predictive intelligence by collecting and analyzing a large number of storage system telemetry data points, thereby enabling easy management, analysis, and support. In fact, such a storage system may be able to predict capacity and performance and generate intelligent recommendations on workload deployment, interaction, and optimization. Such an AI platform can be configured to scan all incoming storage system telemetry data against a library of problem fingerprints to predict and resolve incidents in real time before they affect customer environments, and capture hundreds of performance-related variables for predicting performance loads.

[0190] The storage system described above can support the serial or simultaneous execution of artificial intelligence applications, machine learning applications, data analysis applications, data transformations, and other tasks that can collectively form the AI ​​ladder. By combining these elements to form a complete data science pipeline, an AI ladder can be effectively formed in which there are dependencies between the elements of the AI ​​ladder. For example, AI may require that some form of machine learning has occurred, machine learning may require that some form of analysis has occurred, analysis may require that some type of data and information architecture has occurred, etc. Therefore, each element can be viewed as a rung in the AI ​​ladder, together forming a complete and complex AI solution.

[0191] The storage systems described above can also be used alone or in combination with other computing environments to provide an AI-everywhere experience, where AI permeates a wide and vast range of aspects of business and life. For example, AI may play an important role in providing deep learning solutions, deep reinforcement learning solutions and general artificial intelligence solutions, autonomous vehicles, cognitive computing solutions, commercial UAVs or drones, conversational user interfaces, enterprise classification, ontology management solutions, machine learning solutions, smart dust, smart robots, smart workplaces, etc.

[0192] The storage system described above can also be used alone or in combination with other computing environments to provide a wide range of transparent immersive experiences (including experiences using digital twins of various "things" such as people, places, processes, systems, etc.), where technology can introduce transparency between people, enterprises, and things. This transparent immersive experience can be provided through augmented reality technology, connected home, virtual reality technology, brain-computer interface, human augmentation technology, nanotube electronics, volumetric displays, 4D printing technology, or other means.

[0193] The storage system described above can also be used alone or in combination with other computing environments to support various digital platforms. For example, such digital platforms may include 5G wireless systems and platforms, digital twin platforms, edge computing platforms, IoT platforms, quantum computing platforms, serverless PaaS, software-defined security, neuromorphic computing platforms, etc.

[0194] The storage system described above can also be part of a multi-cloud environment, in which multiple cloud computing and storage services are deployed in a single heterogeneous architecture. To facilitate the operation of such a multi-cloud environment, DevOps tools can be deployed to enable orchestration across clouds. Similarly, continuous development and continuous integration tools can be deployed to standardize processes for continuous integration and delivery, new feature rollouts, and provisioning cloud workloads. By standardizing these processes, a multi-cloud strategy that can leverage the best provider for each workload can be implemented.

[0195] The storage system described above can be used as part of a platform to enable the use of cryptographic anchors that can be used to verify the origin and content of a product to ensure that it matches the blockchain record associated with the product. Similarly, as part of a set of tools to protect data stored on the storage system, the storage system described above can implement various encryption techniques and schemes, including lattice cryptography. Lattice cryptography can involve the construction of cryptographic primitives that include lattices, either in the construction itself or in security proofs. Unlike public key schemes such as RSA, Diffie-Hellman, or elliptic curve cryptography, which are vulnerable to attacks by quantum computers, some lattice-based constructions appear to be resistant to attacks by both classical and quantum computers.

[0196] A quantum computer is a device that performs quantum computing. Quantum computing is a computation performed using quantum mechanical phenomena such as superposition and entanglement. Quantum computers are different from traditional computers based on transistors because such traditional computers require data to be encoded as binary digits (bits), each of which is always in one of two definite states (0 or 1). Compared to traditional computers, quantum computers use qubits, which can be in a superposition state. A quantum computer maintains a sequence of qubits, where a single qubit can represent one, zero, or any quantum superposition of the two qubit states. A pair of qubits can be in any quantum superposition of 4 states, and three qubits are in any superposition of 8 states. A quantum computer with n qubits can typically be in any superposition of up to 2^n different states at the same time, while a traditional computer can only be in one of these states at any time. A quantum Turing machine is a theoretical model of such a computer.

[0197] The storage system described above can also be paired with an FPGA acceleration server as part of a larger AI or ML infrastructure. Such an FPGA acceleration server can be located near the storage system described above (e.g., in the same data center), or even incorporated into a device containing one or more storage systems, one or more FPGA acceleration servers, a network connection infrastructure supporting communication between one or more storage systems and one or more FPGA acceleration servers, and other hardware and software components. Alternatively, the FPGA acceleration server can reside in a cloud computing environment that can be used to perform computationally related tasks for AI and ML jobs. Any of the embodiments described above can be used together as an FPGA-based AI or ML platform. The reader will understand that in some embodiments of an FPGA-based AI or ML platform, the FPGA contained in the FPGA acceleration server can be reconfigured for different types of ML models (e.g., LSTM, CNN, GRU). The ability to reconfigure the FPGA contained in the FPGA acceleration server can accelerate ML or AI applications based on the optimal numerical precision and memory model used. The reader will understand that by treating a collection of FPGA acceleration servers as an FPGA pool, any CPU in a data center can use the FPGA pool as a shared hardware microservice, rather than limiting the server to a dedicated accelerator inserted therein.

[0198] The FPGA accelerated servers and GPU accelerated servers described above can implement a computing model in which machine learning models and parameters are fixed in high-bandwidth on-chip memory, and large amounts of data flow through high-bandwidth on-chip memory, rather than saving small amounts of data in the CPU and running long instruction streams on it as in a more traditional computing model. For this computing model, FPGAs may even be more efficient than GPUs because FPGAs can be programmed with only the instructions required to run this computing model.

[0199] The storage system described above can be configured to provide parallel storage, for example, by using a parallel file system such as BeeGFS. Such a parallel file system can include a distributed metadata architecture. For example, a parallel file system can include multiple metadata servers across which metadata is distributed, and components including services for clients and storage servers.

[0200] The system described above can support the execution of various software applications. Such software applications can be deployed in a variety of ways, including container-based deployment models. Containerized applications can be managed using various tools. For example, containerized applications can be managed using Docker Swarm, Kubernetes, etc. Containerized applications can be used to facilitate serverless, cloud-native computing deployment and management models for software applications. In order to support serverless, cloud-native computing deployment and management models for software applications, containers can be used as part of an event handling mechanism (e.g., AWS Lambdas) so that various events cause the containerized application to run as an event handler.

[0201] The systems described above may be deployed in a variety of ways, including in a manner to support fifth generation ('5G') networks. 5G networks may support much faster data communications than previous generations of mobile communications networks and may therefore result in the disaggregation of data and computing resources, as modern large-scale data centers may become less prominent and may be replaced by local micro data centers closer to mobile network towers. The systems described above may be included in such local micro data centers and may be part of or paired with a multi-access edge computing ('MEC') system. Such a MEC system may enable cloud computing capabilities and IT service environments at the edge of a cellular network. By running applications and performing related processing tasks closer to cellular customers, network congestion may be reduced and application performance may be improved.

[0202] The storage system described above can also be configured to implement NVMe partitioned namespaces. By using NVMe partitioned namespaces, the logical address space of the namespace is divided into multiple zones. Each zone provides a logical block address range that must be written sequentially and explicitly reset before being overwritten, thereby enabling the creation of a namespace that exposes the natural boundaries of the device and offloading the management of internal mapping tables to the host. To implement NVMe partitioned namespaces ('ZNS'), a ZNS SSD or other form of partitioned block device that uses zones to expose the namespace logical address space can be used. By aligning the zones with the internal physical properties of the device, several inefficiencies in data placement can be eliminated. In such an embodiment, each area can be mapped to, for example, a separate application, so that functions such as wear leveling and garbage collection can be performed on a per-zone or per-application basis rather than on the entire device. To support ZNS, the storage controller described herein can be configured to perform wear leveling and garbage collection on a per-zone or per-application basis by using, for example, Linux TM The kernel partition block device interface or other tools interact with the partition block device.

[0203] The storage system described above can also be configured to implement zoned storage in other ways, such as by using shingled magnetic recording (SMR) storage devices. In examples using zoned storage, a device management embodiment can be deployed, where the storage device hides this complexity by managing it in firmware, presenting the same interface as any other storage device. Alternatively, zoned storage can be implemented via a host-managed embodiment, which relies on the operating system to know how to handle the drive and only writes to certain areas of the drive sequentially. Zoned storage can similarly be implemented using a host-aware embodiment, in which a combination of drive management and host management implementation schemes are deployed.

[0204] The storage system described herein can be used to form a data lake. The data lake can be used as the first place to organize the flow of data, where such data can be in its original format. Metadata tags can be implemented to facilitate the search of data elements in the data lake, particularly in embodiments where the data lake contains multiple data storage areas and the format of these data is not easy to access or read (e.g., unstructured data, semi-structured data, structured data). Data can be transferred downstream from the data lake to a data warehouse, where the data can be stored in a format that is easier to process, package, and consume. The storage system described above can also be used to implement such a data warehouse. In addition, a data mart or data center can allow for easier consumption of data, where the storage system described above can also be used to provide the underlying storage resources required for the data mart or digital center. In an embodiment, querying the data lake may require a read mode method, in which the data is applied to the plan or mode when the data is pulled out of the storage location, rather than when the data enters the storage location.

[0205] The storage systems described herein may also be configured to implement a recovery point objective ('RPO'), which may be established by a user, established by an administrator, established as a system default, established as part of a storage class or service that the storage system participates in providing, or otherwise established. A "recovery point objective" is a goal for the maximum time difference between the last update of a source data set and the last recoverable update of a replicated data set, given that the update will be correctly recoverable from a continuously or frequently updated copy of the source data set. An update is correctly recoverable if the update properly accounts for all updates processed on the source data set before the last recoverable update of the replicated data set.

[0206] In synchronous replication, the RPO will be zero, meaning that under normal operation, all completed updates on the source dataset should be present and correctly recoverable on the replica dataset. In best-effort, nearly synchronous replication, the RPO can be as low as a few seconds. In snapshot-based replication, the RPO can be roughly calculated as the interval between snapshots plus the time to transfer modifications between the previously transferred snapshot and the most recent snapshot to be replicated.

[0207] If updates accumulate faster than they can be replicated, the RPO can be missed. If the data to be replicated that accumulates between two snapshots (for snapshot-based replication) exceeds the data that can be replicated between taking the snapshot and replicating the snapshot's cumulative updates to the replica, the RPO can be missed. Again, if in snapshot-based replication, data to be replicated accumulates at a faster rate than it can be delivered in the time between subsequent snapshots, replication can begin to fall further behind, which can extend the miss between the expected recovery point objective and the actual recovery point represented by the last correctly replicated update.

[0208] The storage systems described above may also be part of a shared-nothing storage cluster. In a shared-nothing storage cluster, each node of the cluster has local storage and communicates with other nodes in the cluster over a network, where the storage used by the cluster is (usually) provided only by the storage connected to each individual node. A collection of nodes that synchronously replicate data sets may be an example of a shared-nothing storage cluster, because each storage system has local storage and communicates with other storage systems over a network, where those storage systems (usually) do not use storage from elsewhere that they have shared access to through some kind of interconnect. In contrast, some of the storage systems described above are inherently built as shared storage clusters, because there are drive racks that are shared by pairs of controllers. However, other storage systems described above are built as shared-nothing storage clusters, because all storage is local to a particular node (e.g., a blade), and all communications are conducted over a network that links the compute nodes together.

[0209] In other embodiments, other forms of shared-nothing storage clusters may include embodiments in which any node in the cluster has a local copy of all storage devices it needs, and the data is mirrored to other nodes in the cluster through synchronous replication to ensure that the data is not lost or because other nodes are also using the storage devices. In such embodiments, if a new cluster node needs some data, the data can be copied from other nodes that have copies of the data to the new node.

[0210] In some embodiments, a mirror copy based shared storage cluster may store multiple copies of stored data for all clusters, wherein each subset of data is replicated to a specific set of nodes, and different subsets of data are replicated to different sets of nodes. In some variations, embodiments may store all stored data of a cluster in all nodes, while in other variations, nodes may be partitioned such that a first set of nodes will all store the same data set, and a second, different set of nodes will all store different data sets.

[0211] The reader will appreciate that a RAFT-based database (e.g., etcd) can operate like a shared-nothing storage cluster, with all RAFT nodes storing all data. However, the amount of data stored in a RAFT cluster may be limited so that additional replicas do not consume too much storage. A cluster of container servers may also be able to replicate all data to all cluster nodes, provided that the containers are not too large and their bulk data (data manipulated by applications running in containers) is stored elsewhere, such as in an S3 cluster or an external file server. In such an example, storage for containers can be provided directly by the cluster through its shared-nothing storage model, with the containers providing images that form part of the execution environment for an application or service.

[0212] To explain further, Figure 3D An exemplary computing device 350 is shown, which may be specifically configured to perform one or more of the processes described herein. Figure 3D As shown, computing device 350 may include communication interface 352, processor 354, storage device 356, and input / output ("I / O") module 358, which are communicatively connected to each other via communication infrastructure 360. Figure 3D An exemplary computing device 350 is shown in FIG. Figure 3D The components shown in the drawings are not meant to be limiting. Additional or alternative components may be used in other embodiments. Figure 3D Components of computing device 350 shown in .

[0213] Communication interface 352 may be configured to communicate with one or more computing devices. Examples of communication interface 352 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (e.g., a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0214] The processor 354 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the performance of one or more of the instructions, processes, and / or operations described herein. The processor 354 may perform operations by executing computer-executable instructions 362 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 356.

[0215] Storage device 356 may include one or more data storage media, devices, or configurations, and may employ any type, form, and combination of data storage media and / or devices. For example, storage device 356 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 356. For example, data representing computer executable instructions 362 configured to instruct processor 354 to perform any of the operations described herein may be stored in storage device 356. In some examples, the data may be arranged in one or more databases within storage device 356.

[0216] The I / O module 358 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 358 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 358 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0217] The I / O module 358 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 358 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular embodiment. In some instances, any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 350.

[0218] To explain further, Figure 3E An example of a storage system 376 for providing storage services (also referred to herein as 'data services') is shown. Figure 3EThe group of storage systems 376 depicted in FIG. 1 includes multiple storage systems 374a, 374b, 374c to 374n, each of which may be similar to the storage systems described herein. The storage systems 374a, 374b, 374c to 374n in the group of storage systems 376 may be embodied as the same storage system or different types of storage systems. For example, Figure 3E The two storage systems 374a, 374n depicted in FIG. 3 are depicted as cloud-based storage systems because the resources that together form each of the storage systems 374a, 374n are provided by different cloud service providers 370, 372. For example, the first cloud service provider 370 may be Amazon AWS. TM , and the second cloud service provider 372 is Microsoft Azure TM , but in other embodiments, one or more public clouds, private clouds, or a combination thereof may be used to provide the underlying resources for a particular storage system in the cluster that forms storage system 376.

[0219] Figure 3E The example depicted in 360 includes an edge management service 366 for providing storage services according to some embodiments of the present disclosure. The storage services provided (also referred to herein as 'data services') may include, for example, services that provide a certain amount of storage to consumers, services that provide storage to consumers according to a predetermined service tier agreement, services that provide storage to consumers according to predetermined regulatory requirements, and many other services.

[0220] Figure 3E The edge management service 366 depicted in FIG. 3 may be embodied as one or more computer program instruction modules executed, for example, on computer hardware (e.g., one or more computer processors). Alternatively, the edge management service 366 may be embodied as one or more computer program instruction modules executed on a virtualized execution environment (e.g., one or more virtual machines), in one or more containers, or in some other manner. In other embodiments, the edge management service 366 may be embodied as a combination of the embodiments described above, including embodiments in which one or more computer program instruction modules included in the edge management service 366 are distributed across multiple physical or virtual execution environments.

[0221] The edge management service 366 can operate as a gateway to provide storage services to storage consumers, where the storage services utilize storage provided by one or more storage systems 374a, 374b, 374c to 374n. For example, the edge management service 366 can be configured to provide storage services to host devices 378a, 378b, 378c, 378d, 378n that are executing one or more applications that consume storage services. In such an example, the edge management service 366 can operate as a gateway between the host devices 378a, 378b, 378c, 378d, 378n and the storage systems 374a, 374b, 374c to 374n, rather than requiring the host devices 378a, 378b, 378c, 378d, 378n to directly access the storage systems 374a, 374b, 374c to 374n.

[0222] Figure 3E The edge management service 366 exposes the storage service module 364 to Figure 3E In other embodiments, the edge management service 366 may expose the storage service module 364 to other consumers of various storage services. The various storage services may be presented to consumers via one or more user interfaces, via one or more APIs, or through some other mechanism provided by the storage service module 364. Figure 3E The storage service module 364 depicted in may be embodied as one or more computer program instruction modules executed on physical hardware, a virtualized execution environment, or a combination thereof, wherein execution of such modules enables consumers of storage services to obtain, select, and access various storage services.

[0223] Figure 3E The edge management service 366 also includes a system management service module 368 . Figure 3EThe system management service module 368 includes one or more computer program instruction modules that, when executed, coordinate with the storage systems 374a, 374b, 374c to 374n to perform various operations to provide storage services to the host devices 378a, 378b, 378c, 378d, 378n. For example, the system management service module 368 may be configured to perform tasks such as provisioning storage resources from the storage systems 374a, 374b, 374c to 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c to 374n, migrating data sets or workloads between the storage systems 374a, 374b, 374c to 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c to 374n, setting one or more tunable parameters (i.e., one or more configurable settings) on the storage systems 374a, 374b, 374c to 374n via one or more APIs exposed by the storage systems 374a, 374b, 374c to 374n, and so on. For example, many of the services described below relate to embodiments in which the storage systems 374a, 374b, 374c to 374n are configured to operate in a certain manner. In such an example, the system management service module 368 may be responsible for configuring the storage systems 374a, 374b, 374c through 374n to operate as described below using an API (or some other mechanism) provided by the storage systems 374a, 374b, 374c through 374n.

[0224] In addition to configuring the storage systems 374a, 374b, 374c to 374n, the edge management service 366 itself can also be configured to perform various tasks required to provide various storage services. Consider an example in which the storage service includes a service that, when selected and applied, causes personally identifiable information ('PII') contained in a data set to be obfuscated when the data set is accessed. In such an example, the storage systems 374a, 374b, 374c to 374n can be configured to obfuscate the PII when servicing a read request directed to the data set. Alternatively, the storage systems 374a, 374b, 374c to 374n can service the read by returning data containing the PII, but the edge management service 366 itself can obfuscate the PII when the data passes through the edge management service 366 on the way from the storage systems 374a, 374b, 374c to 374n to the host devices 378a, 378b, 378c, 378d, 378n.

[0225] Figure 3E The storage systems 374a, 374b, 374c to 374n depicted in the drawings may be embodied as described above with reference to Figure 1A-3DThe one or more storage systems described include variations thereof. In fact, the storage systems 374a, 374b, 374c to 374n may be used as a storage resource pool, wherein the individual components in the pool have different performance characteristics, different storage characteristics, etc. For example, one of the storage systems 374a may be a cloud-based storage system, another storage system 374b may be a storage system that provides block storage, another storage system 374c may be a storage system that provides file storage, another storage system 374d may be a relatively high-performance storage system, and another storage system 374n may be a relatively low-performance storage system, etc. In an alternative embodiment, there may be only one storage system.

[0226] Figure 3E The storage systems 374a, 374b, 374c to 374n depicted in the diagram may also be organized into different fault domains such that a failure of one storage system 374a should be completely unrelated to a failure of another storage system 374b. For example, each storage system may receive power from an independent power system, each storage device may communicate data over an independent data communications network, and the like. In addition, storage systems in a first fault domain may be accessed via a first gateway, while storage systems in a second fault domain may be accessed via a second gateway. For example, the first gateway may be a first instance of the edge management service 366, and the second gateway may be a second instance of the edge management service 366, including embodiments in which each instance is different or each instance is part of a distributed edge management service 366.

[0227] As an illustrative example of available storage services, a user may be presented with storage services associated with different tiers of data protection. For example, a user may be provided with storage services that, when selected and executed, assure the user that data associated with the user will be protected such that various recovery point objectives ('RPOs') may be guaranteed. For example, a first available storage service may ensure that some data set associated with a user will be protected such that any data older than 5 seconds may be recovered in the event of a failure of a primary data store, while a second available storage service may ensure that a data set associated with a user will be protected such that any data older than 5 minutes may be recovered in the event of a failure of the primary data store.

[0228] Another example of a storage service that can be presented to a user, selected by the user, and ultimately applied to a data set associated with the user can include one or more data compliance services. Such data compliance services can be embodied as, for example, data compliance services can be provided to consumers (i.e., users) to ensure that the user's data set is managed in a manner that complies with various regulatory requirements. For example, one or more data compliance services can be provided to users to ensure that the user's data set is managed in a manner that complies with the General Data Protection Regulation ('GDPR'), one or more data compliance services can be provided to users to ensure that the user's data set is managed in a manner that complies with the Sarbanes-Oxley Act of 2002 ('SOX'), or one or more data compliance services can be provided to users to ensure that the user's data set is managed in a manner that complies with some other regulatory act. In addition, one or more data compliance services can be provided to users to ensure that the user's data set is managed in a manner that complies with a certain non-governmental guidance (e.g., best practices for audit purposes), one or more data compliance services can be provided to users to ensure that the user's data set is managed in a manner that complies with the requirements of a specific customer or organization, etc.

[0229] In order to provide such a specific data compliance service, the data compliance service can be presented to the user (e.g., via a GUI) and selected by the user. In response to receiving a selection of a specific data compliance service, one or more storage service policies can be applied to the data set associated with the user to perform the specific data compliance service. For example, a storage service policy can be applied that requires that the data set be encrypted before being stored in a storage system, before being stored in a cloud environment, or before being stored elsewhere. In order to enforce this policy, requirements can be enforced that require not only that the data set be encrypted when stored, but also that the data set be encrypted before being transmitted (e.g., sending the data set to another party). In such an example, a storage service policy can also be implemented that requires that any encryption key used to encrypt the data set is not stored on the same system that stores the data set itself. The reader will appreciate that many other forms of data compliance services can be provided and implemented in accordance with embodiments of the present disclosure.

[0230] The storage systems 374a, 374b, 374c to 374n in the group of storage systems 376 may be collectively managed, for example, by one or more group management modules. The group management module may be Figure 3EThe cluster management module may be part of or separate from the system management service module 368 depicted in FIG. 374. The cluster management module may perform tasks such as monitoring the health of each storage system in the cluster, initiating updates or upgrades to one or more storage systems in the cluster, migrating workloads for load balancing or other performance purposes, and many other tasks. Thus, for many other reasons, the storage systems 374a, 374b, 374c to 374n may be coupled to each other via one or more data communication links to exchange data between the storage systems 374a, 374b, 374c to 374n.

[0231] In some embodiments, one or more storage systems or one or more elements of a storage system (e.g., features, services, operations, components, etc. of a storage system), such as any of the exemplary storage systems or storage system elements described herein, can be implemented in one or more container systems. A container system can include any system that supports the execution of one or more containerized applications or services. Such a service can be software deployed as infrastructure for building applications, operating runtime environments, and / or other services. In the discussion that follows, descriptions of containerized applications generally also apply to containerized services.

[0232] A container can combine one or more elements of a containerized software application with a runtime environment for operating software application elements bundled in a single image. For example, each such container of a containerized application can contain executable code and various dependencies, libraries and / or other components of the software application and network configurations, and is configured to access additional resources, used by elements of the software application in a specific container, so as to be able to operate these elements. A containerized application can be represented as a collection of such containers, which together represent all elements of the application in combination with various runtime environments required for the operation of all these elements. Therefore, containerized applications can be abstracted from the host operating system as a combined collection of lightweight and portable packages and configurations, wherein containerized applications can be uniformly deployed and consistently executed in different computing environments using different container-compatible operating systems or different infrastructures. In some embodiments, the containerized application shares a kernel with the host computer system and is executed as an isolated environment (an isolated collection of files and directories, processes, system and network resources, and configured to access additional resources and functions), which is isolated by the operating system of the host system in combination with the container management framework. The containerized application can provide one or more containerized workloads and / or services when executed.

[0233] The container system may include and / or utilize a node cluster. For example, the container system may be configured to manage the deployment and execution of containerized applications on one or more nodes in the cluster. Containerized applications may utilize the resources of the node, such as memory, processing, and / or storage resources provided and / or accessed by the node. Storage resources may include any of the schematic storage resources described herein, and may include on-node resources such as local files and directory trees, off-node resources such as externally networked file systems, databases, or object storage areas, or both on-node and off-node resources. Access to additional resources and functions that may be configured for containers of containerized applications may include specialized computing functions such as GPUs and AI / ML engines, or specialized hardware such as sensors and cameras.

[0234] In some embodiments, the container system may include a container orchestration system (also referred to as a container orchestrator, container orchestration platform, etc.) that is designed to make it reasonably simple and, for many use cases, automated to deploy, scale, and manage containerized applications. In some embodiments, the container system may include a storage management system configured to provide and manage storage resources (e.g., virtual volumes) for private or shared use by cluster nodes and / or containers of containerized applications.

[0235] Figure 3F An example container system 380 is shown. In this example, the container system 380 includes a container storage system 381, which can be configured to perform one or more storage management operations to organize, provide, and manage storage resources for use by one or more containerized applications 382-1 to 382-L of the container system 380. Specifically, the container storage system 381 can organize storage resources into one or more storage pools 383 of storage resources for use by the containerized applications 382-1 to 382-L. The container storage system itself can be implemented as a containerized service.

[0236] Container system 380 may include or be implemented by one or more container orchestration systems, including Kubernetes TM 、Mesos TM 、Docker Swarm TM Etc. The container orchestration system may manage the container system 380 running on the cluster 384 through services implemented by a control node depicted as 385, and may further manage the relationship between the container storage system or individual containers and their storage, memory and CPU limits, network connections, and their access to additional resources or services.

[0237] The control plane of the container system 380 can implement services including: deploying applications via the controller 386, monitoring applications via the controller 386, providing interfaces via the API server 387, and scheduling deployment via the scheduler 388. In this example, the controller 386, the scheduler 388, the API server 387, and the container storage system 381 are implemented on a single node (node ​​385). In other examples, for flexible use, the control plane can be implemented by multiple redundant nodes, where if the node providing management services for the container system 380 fails, another redundant node can provide management services for the cluster 384.

[0238] The data plane of container system 380 may include a set of nodes that provide a container runtime for executing containerized applications. Each node within cluster 384 may execute a container runtime, such as Docker. TM , and executes a container manager or node agent, such as a kubelet (not depicted) in Kubernetes that communicates with the control plane via a proxy (sometimes referred to as a proxy server, such as proxy 389) connected to a local network. Proxy 389 can route network traffic to and from containers using, for example, Internet Protocol (IP) port numbers. For example, a containerized application can request a storage class from the control plane, where the request is processed by the container manager, and the container manager transmits the request to the control plane using proxy 389.

[0239] Cluster 384 may include a set of nodes that run containers for managed containerized applications. A node may be a virtual machine or a physical machine. A node may be a host system.

[0240] Container storage system 381 may orchestrate storage resources to provide storage to container system 380. For example, container storage system 381 may provide persistent storage for containerized applications 382-1 to 382-L using storage pool 383. Container storage system 381 itself may be deployed as a containerized application by a container orchestration system.

[0241] For example, a container storage system 381 application may be deployed within a cluster 384 and perform management functions to provide storage for containerized applications 382. Management functions may include determining one or more storage pools from available storage resources, allocating virtual volumes on one or more nodes, replicating data, responding to and recovering from host and network failures, or processing storage operations. Storage pool 383 may include storage resources from one or more local or remote sources, where the storage resources may be different types of storage devices, such as block storage devices, file storage devices, and object storage devices.

[0242] Container storage system 381 can also be deployed on a set of nodes for which the container orchestration system can provide persistent storage. In some examples, container storage system 381 can be deployed on all nodes in cluster 384 using, for example, Kubernetes DaemonSet. In this example, nodes 390-1 to 390-N provide a container runtime executed by container storage system 381. In other examples, some but not all nodes in the cluster can execute container storage system 381.

[0243] The container storage system 381 can handle storage on the node and communicate with the control plane of the container system 380 to provide dynamic volumes, including persistent volumes. Persistent volumes can be mounted on the node as virtual volumes, such as virtual volumes 391-1 and 391-P. After the virtual volume 391 is mounted, the containerized application can request and use or otherwise be configured to use the storage provided by the virtual volume 391. In this example, the container storage system 381 can install a driver on the kernel of the node, where the driver handles storage operations directed to the virtual volume. In this example, the driver can receive storage operations directed to the virtual volume, and in response, the driver can perform storage operations on one or more storage resources within the storage pool 383, possibly under the guidance of or using additional logic within the container that implements the container storage system 381 as a containerized service.

[0244] The container storage system 381 can determine available storage resources in response to being deployed as a containerized service. For example, storage resources 392-1 to 392-M can include local storage devices, remote storage devices (storage devices on separate nodes in the cluster), or both local and remote storage devices. Storage resources can also include storage devices from external sources, such as various combinations of block storage systems, file storage systems, and object storage systems. Storage resources 392-1 to 392-M can include storage resources of any type and / or configuration (e.g., any of the exemplary storage resources described above), and the container storage system 381 can be configured to determine available storage resources in any suitable manner, including based on a configuration file. For example, a configuration file can specify account and authentication information for a cloud-based object storage device 348 or a cloud-based storage system 318. The container storage system 381 can also determine the availability of one or more storage devices 356 or one or more storage systems. Aggregate storage from one or more of storage 356, storage systems, cloud-based storage systems 318, edge management services 366, cloud-based object storage 348, or any other storage resources, or any combination or sub-combination of these storage resources, may be used to provide storage pool 383. Storage pool 383 is used to provide storage for one or more virtual volumes mounted on one or more nodes 390 within cluster 384.

[0245] In some embodiments, container storage system 381 can create multiple storage pools. For example, container storage system 381 can aggregate storage resources of the same type into a single storage pool. In this example, the storage type can be one of the following: storage device 356, storage array 102, cloud-based storage system 318, storage via edge management service 366, or cloud-based object storage device 348. Alternatively, it can be a storage device configured with a certain level or type of redundancy or distribution, such as a specific combination of striping, mirroring, or erasure coding.

[0246] The container storage system 381 can be executed as a containerized container storage system service within a cluster 384, where container instances implementing elements of the containerized container storage system service can operate on different nodes in the cluster 384. In this example, the containerized container storage system service can be combined with the container orchestration system operations of the container system 380 to process storage operations, mount virtual volumes to provide storage to nodes, aggregate available storage devices into a storage pool 383, allocate storage for virtual volumes from the storage pool 383, generate backup data, copy data between nodes, clusters, environments, and other storage system operations. In some examples, the containerized container storage system service can provide storage services across multiple clusters operating in different computing environments. For example, other storage system operations may include the storage system operations described herein. The persistent storage provided by the containerized container storage system service can be used to implement stateful and / or flexible containerized applications.

[0247] The container storage system 381 may be configured to perform any suitable storage operation of the storage system. For example, the container storage system 381 may be configured to perform one or more exemplary storage management operations described herein to manage storage resources used by the container system.

[0248] In some embodiments, one or more storage operations, including one or more exemplary storage management operations described herein, can be containerized. For example, one or more storage operations can be implemented as one or more containerized applications configured to perform storage operations. Such containerized storage operations can be executed in any suitable runtime environment to manage any storage system, including any exemplary storage system described herein.

[0249] The storage system described herein can support various forms of data replication. For example, two or more storage systems can synchronously replicate data sets between each other. In synchronous replication, different copies of a particular data set can be maintained by multiple storage systems, but all accesses (e.g., reads) to the data set should produce consistent results, regardless of which storage system the access is directed to. For example, reads directed to any storage system that is synchronously replicating the data set should return the same result. Therefore, although updates to data set versions do not need to occur completely simultaneously, precautions must be taken to ensure consistent access to the data set. For example, if a first storage system receives an update (e.g., a write) to a data set, the update can only be confirmed to be complete if all storage systems that synchronously replicate the data set have applied the update to their copies of the data set. In such an example, synchronous replication can be performed using I / O forwarding (e.g., a write received at the first storage system is forwarded to the second storage system), communication between storage systems (e.g., each storage system indicates that it has completed the update), or other means.

[0250] In other embodiments, a data set may be replicated using checkpoints. In checkpoint-based replication (also known as 'almost synchronous replication'), a set of updates to a data set (e.g., one or more write operations directed to the data set) may occur between different checkpoints, such that the data set is updated to a particular checkpoint only if all updates to the data set prior to the particular checkpoint have been completed. Consider an example in which a first storage system stores a real-time copy of a data set that is being accessed by a data set user. In this example, assume that the data set is replicated from the first storage system to the second storage system using checkpoint-based replication. For example, the first storage system may send a first checkpoint (at time t=0) to the second storage system, followed by a first set of updates to the data set, followed by a second checkpoint (at time t=1), followed by a second set of updates to the data set, followed by a third checkpoint (at time t=2). In such an example, if the second storage system has performed all updates in the first update set, but has not yet performed all updates in the second update set, the copy of the data set stored on the second storage system may be up to date until the second checkpoint. Alternatively, if the second storage system has performed all updates in both the first update set and the second update set, the copy of the data set stored on the second storage system may be up to date until a third checkpoint. The reader will appreciate that various types of checkpoints may be used (e.g., metadata-only checkpoints), that checkpoints may be deployed based on various factors (e.g., time, number of operations, RPO settings), etc.

[0251] In other embodiments, the data set may be replicated via snapshot-based replication (also referred to as 'asynchronous replication'). In snapshot-based replication, a snapshot of the data set may be sent from a replication source (e.g., a first storage system) to a replication target (e.g., a second storage system). In such embodiments, each snapshot may contain the entire data set or a subset of the data set, e.g., only the portion of the data set that has changed since the last snapshot was sent from the replication source to the replication target. The reader will appreciate that snapshots may be sent on demand based on a policy or otherwise that takes into account various factors (e.g., time, number of operations, RPO settings).

[0252] The storage systems described above may be configured, alone or in combination, to function as continuous data protection storage. Continuous data protection storage is a feature of a storage system that records updates to a data set in such a way that a consistent image of the previous contents of the data set can be accessed at a lower time granularity (typically in seconds or even less) and backtracked within a reasonable period of time (typically hours or days). These allow access to the latest consistent point in time for a data set, and also allow access to points in time for a data set that may have occurred just before an event, such as an event that caused a portion of the data set to be corrupted or otherwise lost, while retaining the maximum number of updates close to the event. Conceptually, they are like a series of snapshots of a data set that are taken frequently and saved for a long time, although the implementation of continuous data protection storage is typically very different from snapshots. Storage systems that implement data continuous data protection storage may also provide methods for accessing these points in time, accessing one or more points in time in the form of snapshots or cloned copies, or restoring a data set to one of these recorded points in time.

[0253] Over time, to reduce overhead, some of the time points saved in a continuous data protection storage device can be merged with other nearby time points, essentially removing some of them from the storage area. This can reduce the capacity required for storage updates. A limited number of these time points can also be converted to snapshots of longer duration. For example, such a storage might keep a low-granularity sequence of time points a few hours from now, merging or removing some time points to reduce the overhead of another day. Going back in time, some of these time points can be converted to snapshots, representing consistent point-in-time images every few hours.

[0254] Figure 44 is a block diagram illustrating an example storage system 400 according to some embodiments. The storage system 400 includes a computing device 440, a storage system controller 410 (e.g., a storage controller, a storage node, a node, a storage array controller, a storage controller node, etc.), a storage node 420, and a network 405. The system controller 410 and the storage node 420 may include a processing device 411 and one or more non-volatile memory modules 413. The processing device 411 may be a device that can execute instructions and / or can perform various operations, actions, functions, etc. (e.g., one or more CPUs, ASICs, FPGAs, multi-core processors, processing cores, circuits, etc.), as discussed in more detail below. The non-volatile memory module 413 may be a device that stores data so that the data in the device is retained even when power is no longer supplied to the device. Examples of the non-volatile memory module 413 may include, but are not limited to, an SSD, an NVME drive, a flash drive, etc. The storage system controller 410 also includes a relocation module 415, which will be discussed in more detail below.

[0255] In some embodiments, the non-volatile memory module 413 can be removed from the storage node 420 and / or the storage system controller 410. For example, the non-volatile memory module 413 can be removed and replaced with other non-volatile memory modules. The non-volatile memory module 413 can also be removable when the storage node 420 and / or the storage system controller 410 is in operation. For example, the non-volatile memory module 413 can be hot-swappable. The non-volatile memory module 413 can also be heterogeneous (e.g., non-uniform). For example, the non-volatile memory modules may have different brands, models, manufacturers, capacities, and memory types (e.g., SLC, MLC, TLC, QLC, PLC, etc.).

[0256] In one embodiment, the non-volatile memory module 413 may include a multi-plane die. The multi-plane die may be a flash die (e.g., a flash memory die, a semiconductor die) that includes multiple planes (e.g., multiple layers) of flash memory cells. Multi-plane die will be discussed in more detail below.

[0257] The network 405 can utilize various technologies, including wireless connections, direct local area network (LAN) connections, wide area network (WAN) connections such as the Internet, routers, storage area networks, Ethernet, etc. The network 405 can include one or more LANs, which can also be wireless. The network 405 can further include remote direct memory access (RDMA) hardware and / or software, transmission control protocol / Internet protocol (TCP / IP) hardware and / or software, routers, repeaters, switches, grids, etc. Protocols such as Fibre Channel, Fibre Channel over Ethernet (FCoE), iSCSI, etc. can be used in the network 405. The network 405 can be connected to a set of communication protocols for the Internet (e.g., transmission control protocol (TCP)) and Internet protocol (IP) or TCP / IP. In one embodiment, the network 405 represents a storage area network (SAN), which provides access to integrated block-level data storage devices. The SAN can be used to enhance the storage devices accessible to the computing device so that the non-volatile memory module 413 appears to the computing device 440 as a locally attached storage device.

[0258] Computing device 440 represents any number of fixed or mobile computing devices, such as desktop personal computers (PCs), servers, tablet computers, server farms, workstations, laptop computers, handheld computers, servers, personal digital assistants (PDAs), smart phones, etc. In general, computing device 440 may also include one or more processing devices, which may further include one or more processor cores. Each processor core includes a circuit system for executing instructions according to a predefined general instruction set. For example, the x86 instruction set architecture may be selected. Alternatively, the 86 instruction set architecture may be selected. or any other general purpose instruction set architecture. The processor core may access the cache memory subsystem to obtain data and computer program instructions. The cache subsystem may be coupled to a memory hierarchy including random access memory (RAM) and storage devices. The computing device may utilize the storage system controller 410 and / or the storage node 420 to read, write, store and / or access data in the storage system 400.

[0259] In one embodiment, the storage system controller 410 can communicate with other storage system controllers ( Figure 4For example, the storage system controller 410 can operate in parallel with other storage system controllers as a distributed system, where storage operations can be distributed between the storage system controller 410 and the other storage system controllers (e.g., load balancing, based on a scheduling algorithm / mechanism, or based on availability distribution, etc.). In one embodiment, the storage system controller 410 is designated as a "primary" storage system controller, which performs most or all I / O operations for the storage device group 430. However, if a software crash, hardware failure, or other error occurs, the "secondary" storage system controller ( Figure 4 ) promotes the primary host controller and takes over all responsibilities for providing services to storage device group 430.

[0260] The storage system controller 410 may include firmware, software, and / or hardware configured to provide access to the non-volatile memory module 413. For example, the storage system controller 410 may include an operating system, a processor (e.g., a processing device, a CPU, an ASIC, an FPGA), a memory (e.g., a RAM, a NVRAM, a DRAM, etc.), and / or an application ( Figure 4 not shown).

[0261] In one embodiment, when writing a data set to multiple planes of a multi-plane die, the storage system controller 410 can divide the data set into different parts (e.g., different subsets). Each part (e.g., subset) of the data set can be written to one plane of the multi-plane die. For example, if there are four planes in the multi-plane die, the data set can be divided into four parts, and each part can be written to a different plane of the multi-plane die. This can allow data to be written to the multi-plane die faster and / or more efficiently.

[0262] In one embodiment, the storage system controller 410 (e.g., the relocation module 415) can determine that the number of planes used to access data simultaneously should be changed. As described above, each non-volatile memory module 413 (e.g., SSD, flash drive, NVME drive) can include one or more dies (e.g., flash die), and the one or more dies can be multi-plane dies (e.g., dies with multiple levels or planes). The storage system controller 410 can access data (e.g., read and / or write data) using multiple planes at the same time (e.g., data can be written to multiple planes at the same time). The relocation module 415 can determine that more or fewer planes should be used to access data simultaneously. For example, the relocation module 415 can determine that more planes should be used to write data simultaneously (e.g., more planes should be written in parallel).

[0263] In one embodiment, when an additional (e.g., new) non-volatile memory module 413 is added to the storage system 400, the storage system controller 410 (e.g., the relocation module 415) can determine that the number of planes used to access data simultaneously should be changed. For example, the new non-volatile memory module 413 can be added to an existing storage node 420 and / or the storage system controller 410. In another example, a new storage system controller 410 and / or a new storage node 420 can be added to the storage system 400.

[0264] In another embodiment, the storage system controller 410 (e.g., the relocation module 415) may determine that the number of planes used to access data simultaneously should be changed when the non-volatile memory module 413 is removed from the storage system 400. For example, the non-volatile memory module 413 may be removed from an existing storage node 420 and / or the storage system controller 410. In another example, the storage system controller 410 and / or the storage node 420 may be removed from the storage system 400.

[0265] In one embodiment, the storage system controller 410 (e.g., the relocation module 415) can determine that the number of planes used to access data simultaneously should be changed based on a request indicating that the number of planes should be changed. For example, a user (e.g., a system / network administrator, an engineer, etc.) can determine that the number of planes used to access data simultaneously should be changed (e.g., increased or decreased). The user can provide user input indicating that the number of planes should be changed, and the user input can cause a request (e.g., a message or other indication to change the number of planes) to be sent to the relocation module 415.

[0266] In one embodiment, the number of planes used to access data simultaneously may be increased or decreased based on various criteria, factors, parameters, etc. For example, the number of planes used to access data simultaneously may be based on one or more of the erase block size of storage system 400 (e.g., the size of an erase block), the amount of storage space in storage system 400 (e.g., the total storage space in the storage system), the predicted useful life / lifespan of the data, the type of data (e.g., the content of the data), and / or other characteristics of the data stored within storage system 400.

[0267] In one embodiment, the number of planes used to access data simultaneously can be increased or decreased based on power usage constraints, requirements, restrictions, conditions, etc. For example, using a greater number of planes (used to access data simultaneously) can improve power efficiency, as discussed in more detail below. This allows the storage system controller 410 (e.g., the relocation module 415) to use less power based on power usage constraints. For example, if there is a specific amount of power (e.g., a maximum power) that the storage system 400 should use, the storage system controller 410 can keep the storage system 400 within the specific amount of power by increasing the number of planes used to access data simultaneously. In another example, if the price of electricity (e.g., utility rates) is more expensive during different time periods (e.g., more expensive during the day than at night), the storage system controller 410 can increase the number of planes during the more expensive time periods and reduce the number of planes during the less expensive time periods. The storage system controller 410 can automatically change (e.g., adjust) the number of planes used to access data simultaneously, and can increase or decrease the number based on power usage constraints, requirements, restrictions, conditions, etc. For example, storage system 410 may continuously monitor power usage, utility rates, or other conditions / parameters and may automatically change the number of planes used to access data simultaneously.

[0268] In one embodiment, storage system controller 410 (e.g., relocation module 415) can move, relocate, copy, shift, transfer, etc. one or more parts of one or more existing erase blocks to new erase blocks. For example, storage system controller 410 can have existing erase blocks allocated in storage system 400. The data used by computing device 440 can be stored in existing erase blocks. Relocation module 410 can identify, select, determine, etc. one or more parts of existing erase blocks (e.g., information blocks, fragments, parts, sections, etc.) should be moved to new erase blocks. Relocation module 410 can identify, select, determine, etc. one or more parts of existing erase blocks based on various standards, factors, parameters, etc. For example, relocation module 410 can identify one or more parts of existing erase blocks based on metadata. Metadata can include information about erase blocks and / or information about data characteristics in erase blocks, which will be discussed in more detail below.

[0269] In one embodiment, one or more parts of an existing erase block may include real-time data. For example, real-time data may be data that has not yet been marked as deleted / erased (e.g., data that a user or computing device has not yet indicated should be deleted). In another example, real-time data may be data that is currently in use (e.g., data that a user or computing device 440 can still access). An existing erase block may also include invalid data. For example, invalid data may be data that has been marked as deleted but has not yet been garbage collected by the storage system 400 (e.g., data that has not yet been garbage collected by one or more of the non-volatile memory module 413, the storage node 420, and the storage system controller 410). Invalid data may also be referred to as obsolete data, inactive data, stagnant data, obsolete data, etc.

[0270] In one embodiment, after the storage system controller 410 (e.g., the relocation module 415) determines that the number of planes used to access data simultaneously should be changed, one or more portions of the existing erase block (which may contain real-time data) may be moved to the new erase block at different times. For example, the relocation module 415 may move the data immediately after determining that the number of planes used to access data simultaneously should be changed. In another example, the relocation module 415 may move the data within a period of time after determining that the number of planes used to access data simultaneously should be changed (e.g., the data may be moved in the background or as part of a background process).

[0271] In one embodiment, the size of the new erase block can be different from the size of one or more existing erase blocks. For example, if the number of planes used to access data is increased simultaneously, the size of the new erase block can be greater than the size of one or more existing erase blocks. In another example, if the number of planes used to access data is reduced simultaneously, the size of the new erase block can be less than the size of one or more existing erase blocks. The size of the new erase block can also be based on various other parameters, standards, conditions, etc. For example, the size of the new erase block can be based on the characteristics (e.g., life expectancy, content, etc.) of the data to be stored in the new erase block.

[0272] In one embodiment, the storage system controller 410 (e.g., relocation module 415) can move, relocate, etc. one or more parts of one or more existing erase blocks to one or more new erase blocks during a garbage collection operation. The garbage collection operation can include reclaiming (e.g., reusing) blocks containing invalid data (or no data), and allocating / reallocating these blocks to write new data. For example, when a block is used in a new erase block (e.g., the block is erased and / or included as a block in a new erase block), the block can be garbage collected. Garbage collection can also include moving real-time data (e.g., blocks with real-time data) to other locations (e.g., to other erase blocks). The storage system controller 410 and / or the storage node 420 can perform garbage collection regularly. For example, the storage system controller 410 and / or the storage node 420 can perform garbage collection operations at regular intervals / cycles or based on a schedule.

[0273] In one embodiment, the storage system controller 410 (e.g., the relocation module 415) can move, relocate, etc. one or more portions of one or more existing erase blocks to one or more new erase blocks during a defragmentation operation (e.g., a defragmentation operation). The defragmentation operation includes moving real-time data (e.g., a block with real-time data) to other locations (e.g., to other erase blocks). The storage system controller 410 and / or the storage node 420 can perform defragmentation periodically. For example, the storage system controller 410 and / or the storage node 420 can perform defragmentation operations at regular intervals / periods or based on a schedule.

[0274] In one embodiment, using multiple planes 511 to access data simultaneously can improve Figure 4 Performance of the storage system 400 is shown. For example, accessing data from multiple planes simultaneously can enable the storage system to access data faster, as discussed in more detail below.

[0275] In one embodiment, accessing data from multiple planes (of a multi-plane die) simultaneously can reduce the amount of energy consumed and / or used by storage system 400. For example, accessing data from multiple planes simultaneously can reduce the power usage of the storage system compared to serially accessing data from multiple planes of a multi-plane die, as discussed in more detail below.

[0276] In one embodiment, accessing data from multiple planes (of a multi-plane die) simultaneously may reduce the amount of heat generated by the storage system 400. For example, accessing data from multiple planes simultaneously may reduce the amount of heat generated by the storage system compared to serially accessing data from multiple planes of a multi-plane die, as discussed in more detail below.

[0277] As data storage requirements increase and / or change, it may be useful for a data storage system (e.g., storage system 400) to operate faster and / or more efficiently. For example, the ability to read / write data faster is generally useful and / or desirable. Changing the number of planes used to access data simultaneously in a multi-plane die (e.g., increasing the number of planes) may allow storage system 400 to operate faster and / or more efficiently (e.g., write more data in a period of time). In another example, the energy usage of storage system 400 may be more efficient. However, since changing the number of planes used may also change the size of the erase block, problems may arise when allocating / deallocating erase blocks. Real-time data in an existing erase block (e.g., one or more parts of an existing erase block containing real-time data) may prevent an existing erase block from being reused in a new erase block. For example, if there is still real-time data in an existing erase block, an existing erase block (e.g., a physical block as a part of an existing erase block) may not be de-allocated, reused, etc. in other erase blocks (e.g., in a new erase block), as discussed in more detail below.

[0278] The examples, embodiments, implementation schemes, etc. described herein allow storage system 400 to allocate and / or deallocate erase blocks more effectively. For example, relocation module 415 can identify erase blocks with real-time data and move the real-time data. Moving real-time data from existing erase blocks can allow blocks in erase blocks to be used in new erase blocks. Relocation module 415 can use metadata and / or data characteristics in existing erase blocks to identify real-time data that should be moved. This can reduce and / or prevent real-time data from preventing the reuse of blocks in existing erase blocks.

[0279] Figure 5 4 is a block diagram illustrating an example non-volatile memory module 413 according to some embodiments. As discussed above, a non-volatile memory module 413 may be a device that stores data such that the data in the device is retained even when power is no longer supplied to the device. The non-volatile memory module 413 may be a device that stores data such that the data in the device is retained even when power is no longer supplied to the device. The non-volatile memory module 413 may be a device that stores data such that the data in the device is retained even when power is no longer supplied to the device. Figure 4 The non-volatile memory modules 413 may also be heterogeneous (eg, non-uniform, the brands, models and / or storage capacities of the non-volatile memory modules 413 may be different).

[0280] The non-volatile memory module 413 includes one or more multi-plane dies 510 (e.g., one or more multi-plane dies 510). The multi-plane die 510 can be a die (e.g., a flash memory die, a semiconductor die) including multiple planes 511. For example, the multi-plane die 510 can include planes 511 stacked on top of each other. Each plane 511 can include flash memory cells (e.g., single-layer cells, multi-layer cells, etc.). Each multi-plane die 510 includes four planes 511 (e.g., four layers of flash memory cells). In one embodiment, a plane 511 can be a unit on a die on which an operation (e.g., read or write) can be performed. A plane 511 can also be referred to as a layer.

[0281] Although multi-plane die 510 is shown as having four planes, in other embodiments, multi-plane die 510 may include any number of planes (e.g., may include eight planes, twenty planes, one hundred planes, etc.). Figure 5 The nonvolatile memory module 413 shown in FIG. 4 includes multi-planar dies 510 having the same number of planes 511 , but in other embodiments, the nonvolatile memory module 413 may include multi-planar dies 510 having different numbers of planes 511 .

[0282] As described above, multiple planes 511 can be used to access data simultaneously. For example, multiple planes 511 can be used to write data to multi-plane die 510 simultaneously (e.g., data can be written to two or more planes 511 simultaneously). In another example, multiple planes 511 can be used to read data from multi-plane die 510 simultaneously (e.g., data can be read from two or more planes 511 simultaneously).

[0283] In one embodiment, using multiple planes 511 to access data simultaneously can improve the storage system (e.g., Figure 4 The performance of the storage system 400 shown in FIG. 4 is improved. For example, reading data from multiple planes 511 simultaneously may allow the storage system to read data faster than reading data serially from multiple planes 511. When reading data serially from a plane 511 (e.g., one plane 511 at a time), the amount of time to read the data may be the time to access the data from the plane 511 multiplied by the number of planes 511 accessed (e.g., time*number of planes). When reading data from multiple planes 511 simultaneously, the amount of time to read the data may be the same as the time to access the data from a single plane 511 because the multiple planes 511 are accessed simultaneously.

[0284] In one embodiment, using multiple planes 511 to access data simultaneously can improve the efficiency of the storage system. For example, reading data serially from a plane 511 can use a certain amount of power / energy. Reading data from multiple planes 511 at a time can use a smaller / less amount of power / energy than reading data serially from a plane 511 because the overhead of accessing a multi-plane die can be shared among the planes 511 of the multi-plane die 510.

[0285] In one embodiment, using multiple planes 511 to access data simultaneously can reduce the amount of heat generated by the storage system. As discussed above, using multiple planes 511 to access data simultaneously can reduce the amount of power used by the storage system. Using less power also causes the storage system to generate less heat.

[0286] Fig. 6A 610A, 610B, 610C, 610D, and 620 according to some embodiments. An erase block may be a set of blocks (e.g., one or more physical blocks) that are grouped together (e.g., logically grouped together). An erase block may be a unit on which a storage system may allocate blocks for writing data. For example, a storage system may allocate one erase block at a time to write new data.

[0287] An erase block may include live data and / or failed data. For example, erase block 610A includes a portion 611 with live data and six portions 615 with failed data. Erase block 610B includes four portions 611 with live data and three portions 615 with failed data. Erase block 610C includes five portions 611 with live data and two portions 615 with failed data. Erase block 610D includes one portion 611 with live data and six portions 615 with failed data. Live data may be data that is still in use and / or has not yet been marked (e.g., labeled, tiled, etc.) for deletion. Failed data may be data that has been marked for deletion but has not yet been deleted / erased (e.g., has not yet been garbage collected).

[0288] As discussed above, the storage system can change the number of planes used to access data simultaneously in a multi-plane die. When the storage system increases the number of planes used simultaneously, the storage system can also increase the size of the erase block. For example, erase blocks 610A, 610B, 610C, and 610D can include seven parts of the same size. If the storage system increases the number of planes used simultaneously, the size of the erase block can be increased to the size of erase block 620 (for example, it can have fourteen parts). For example, the new erase block allocated by the storage system (after increasing the number of planes used simultaneously) can be larger than the old / existing erase block in the storage system. In one embodiment, after the storage system increases the number of planes used to access data simultaneously, there can be smaller erase blocks and larger erase blocks in the storage system. For example, when the storage system starts to allocate newer / larger erase blocks, there may be smaller erase blocks that have not yet been deallocated because there are still real-time data in the smaller erase blocks.

[0289] It may be beneficial for a storage system (e.g., a storage system controller, a relocation module, etc.) to track, monitor, manage, etc. real-time data in erase blocks to allow for more efficient use of blocks (e.g., data blocks, physical blocks, etc.) in the storage system. In one embodiment, the storage system controller may use metadata 650 to track, monitor, manage, etc. real-time data in erase blocks. For example, metadata 650 may include a list of erase blocks in the storage system, a list of physical blocks contained in erase blocks (e.g., a list of physical blocks for each erase block), the size of the erase blocks, and / or which parts of the erase blocks contain real-time data and which parts of the erase blocks contain failed data. Metadata 650 may also include other information, such as characteristics of the data in the erase blocks (e.g., the type / content of the data in the erase blocks, the expected life of the data in the erase blocks, etc.).

[0290] In one embodiment, a storage system (e.g., a storage system controller, a relocation module, etc.) can relocate or move real-time data to allow existing erase blocks to be deallocated and used to allocate new erase blocks (e.g., to allow blocks in existing erase blocks to be used in new erase blocks). The storage system controller can use metadata 650 to determine, identify, select, etc., portions of erase blocks that should be moved, relocated, copied, etc. to other (e.g., new) erase blocks. For example, the storage system controller can identify erase blocks that have an amount of real-time data below a threshold. The storage system controller can move or relocate the real-time data on the identified erase blocks to one or more other erase blocks. For example, the storage system controller can move / relocate real-time data in existing erase blocks to new erase blocks (which may be larger in size than existing erase blocks). Fig. 6AAs shown, erase blocks 610A and 610D each have a portion 611 (e.g., shaded portion 611) containing real-time data. The storage system (e.g., storage system controller, relocation module, etc.) can identify erase blocks 610A and 610D because the amount of real-time data in erase blocks 610A and 610D is below a threshold. The storage system controller can copy the portion 611 (e.g., shaded portion 611) containing real-time data in erase blocks 610A and 610D to new erase block 620. Erase block 620 (e.g., new erase block) also includes portion 617. Portion 617 can contain real-time data (e.g., real-time data from other existing erase blocks, data recently received from a client / computing device, etc.), and / or can contain filler data (e.g., data used to complete or fill in an erase block when not enough data is received from a user / computing device).

[0291] In one embodiment, moving the real-time (e.g., shadow) portion 611 allows the storage system to deallocate erase blocks 610A and 610D and use the blocks / portions in erase blocks 610A and 610D to allocate new erase blocks 620. This allows the storage system to use the blocks in the storage system more efficiently. For example, moving the real-time (e.g., shadow) portion 611 allows the storage system to allocate new erase blocks 620 without having to wait for the real-time (e.g., shadow) portion 611 to be deleted and / or marked as deleted. This allows the storage system to allocate new erase blocks faster without wasting portions of erase blocks 610A and 610D that contain failed data.

[0292] Figure 6B 6 is a block diagram illustrating example erase blocks 630 and 640 according to some embodiments. As discussed above, an erase block can be a set of blocks (e.g., one or more physical blocks) that are grouped together (e.g., logically grouped together). An erase block can contain live data and / or failure data. For example, erase block 630 includes a portion 611 having live data and six portions 615 having failure data.

[0293] As discussed above, the storage system can change the number of planes used to access data simultaneously in a multi-plane die. When the storage system reduces the number of planes used simultaneously, the storage system can also reduce the size of the erase block. For example, erase block 630 can include fourteen parts of the same size. If the storage system reduces the number of planes used simultaneously, the size of the erase block can be reduced to the size of erase block 640 (for example, it can have seven parts). For example, the new erase block allocated by the storage system (after reducing the number of planes used simultaneously) can be smaller than the old / existing erase block in the storage system. In one embodiment, after the storage system reduces the number of planes used to access data simultaneously, there can be smaller erase blocks and larger erase blocks in the storage system. For example, the storage system begins to allocate newer / larger erase blocks, and there may be larger erase blocks that have not yet been deallocated because there is still live data in the larger erase blocks.

[0294] It may be beneficial for a storage system (e.g., a storage system controller, a relocation module, etc.) to track, monitor, manage, etc. live data in erase blocks to allow for more efficient use of blocks (e.g., data blocks, physical blocks, etc.) in the storage system. In one embodiment, a storage system controller may use metadata 650 to track, monitor, manage, etc. live data in erase blocks. As discussed above, metadata 650 may include a list of erase blocks, a list of physical blocks contained in an erase block, the size of the erase block, which portions of the erase block contain live data and which portions of the erase block contain failed data, and / or other information such as characteristics of the data in the erase block.

[0295] In one embodiment, the storage system (e.g., storage system controller, relocation module, etc.) can relocate or move live data to allow existing erase blocks to be deallocated and used to allocate new erase blocks. The storage system controller can use metadata 650 to determine, identify, select, etc., portions of erase blocks that should be moved, relocated, copied, etc. to other (e.g., new) erase blocks. The storage system controller can move or relocate live data on the identified erase blocks to one or more other erase blocks. Figure 6B As shown, erase block 630 has two portions 611 (e.g., shaded portions 611) containing real-time data. The storage system (e.g., storage system controller, relocation module, etc.) can identify erase block 630 because the amount of real-time data in erase block 630 is below a threshold. The storage system controller can copy portion 611 (e.g., shaded portion 611) of erase block 630 containing real-time data to new erase block 640.

[0296] In one embodiment, moving active (e.g., shadow) portions 611 allows the storage system to deallocate erase blocks 630 and use blocks / portions in erase blocks 630 to allocate new erase blocks 640 (and other erase blocks). This allows the storage system to use blocks in the storage system more efficiently. For example, moving live (e.g., shadow) portions 611 allows the storage system to allocate new erase blocks 640 without having to wait for live (e.g., shadow) portions 611 to be deleted and / or marked for deletion. This allows the storage system to allocate new erase blocks faster without wasting portions of erase blocks 630 that contain failed data.

[0297] Figure 7 1 is a flow chart illustrating a method 700 for performing a storage operation according to some embodiments of the present disclosure. The method 700 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processing device to perform hardware simulation), firmware, or a combination thereof. In one embodiment, the method 700 may be performed by a storage system controller, a relocation module, a processing device, and / or an authorization of a storage system as shown in FIGS. 1-4.

[0298] The method begins at block 705, where method 700 may determine whether to change the number of planes (in a multi-plane die) used to access data simultaneously. For example, method 700 may determine whether a new non-volatile memory module (e.g., a new SSD, a new flash drive, a new NVME drive, etc.) has been added to the storage system. In another example, method 700 may determine whether a request to change the number of planes has been received.

[0299] At block 710, method 700 may optionally identify one or more existing erase blocks. For example, method 700 may use metadata to identify erase blocks containing an amount of real-time data below a threshold. At block 715, method 700 may move (e.g., copy, relocate, etc.) one or more portions of an erase block to a new erase block. For example, method 700 may copy a portion of an erase block having real-time data to a new erase block. The new erase block may have a different size than an existing erase block (e.g., may be larger or smaller than an existing erase block).

[0300] At block 720, method 700 may optionally write data (e.g., additional data) to the storage system using an erase block of the same size as the new erase block. For example, if the size of the erase block has been increased to a larger size, the data may be written to the non-volatile memory module using an erase block of the larger size.

[0301] Although some embodiments are described primarily in the context of storage systems, those skilled in the art will recognize that embodiments of the present disclosure may also take the form of a computer program product disposed on a computer-readable storage medium for use with any suitable processing system. Such a computer-readable storage medium may be any storage medium for machine-readable information, including magnetic media, optical media, solid-state media, or other suitable media. Examples of such media include magnetic disks in hard drives or floppy disks, optical disks in optical drives, tapes, and other media that will be contemplated by those skilled in the art. Those skilled in the art will immediately recognize that any computer system with appropriate programming means is capable of performing the steps embodied in the computer program product described herein. Those skilled in the art will also recognize that although some embodiments described in this specification are directed to software installed and executed on computer hardware, alternative embodiments implemented as firmware or hardware are fully within the scope of the present disclosure.

[0302] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided according to the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and / or the computing device to perform one or more operations, including one or more operations described herein. Any of a variety of known computer-readable media may be used to store and / or transmit such instructions.

[0303] As referred to herein, non-transitory computer-readable media may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0304] The advantages and features of the present disclosure can be further described by the following statements:

[0305] Statement 1. A storage system comprising:

[0306] a plurality of non-volatile memory modules, a non-volatile memory module comprising a multi-plane die;

[0307] A storage system controller operably coupled to a plurality of storage devices, the storage system controller comprising a processing device configured to:

[0308] determining that a number of planes of the multi-plane die used to access data simultaneously should be changed;

[0309] In response to determining that the number of the planes of the multi-plane die used to access data simultaneously should be changed, one or more portions are moved from an existing erase block to a new erase block, the existing erase block having a size different from a size of the new erase block.

[0310] Statement 2. The storage system of Statement 1, wherein to determine that the number of said planes for simultaneously accessing data of said multi-plane die should be changed, said processing device is further configured to:

[0311] It is determined that an additional non-volatile memory module has been added to the memory system.

[0312] Statement 3. The storage system of Statements 1-2, wherein to determine that the number of planes for simultaneously accessing data of the multi-plane die should be changed, the processing device is further configured to:

[0313] A request is received to change the number of the planes of the multi-plane die used to access data simultaneously.

[0314] Statement 4. The storage system of Statements 1 to 3, wherein the one or more portions of the existing erase blocks include live data and other portions of the existing erase blocks include failed data.

[0315] Statement 5. The storage system of Statements 1 to 4, wherein the processing device is further configured to:

[0316] The one or more portions of the existing erase block are identified based on metadata, the metadata indicating real-time data within the existing erase block.

[0317] Statement 6. The storage system of Statements 1-5, wherein the size of the new erase block is larger than the size of the existing erase block.

[0318] Statement 7. The storage system of Statements 1-6, wherein the number of the planes of the multi-plane die used to access data simultaneously is increased.

[0319] Statement 8. The storage system of Statements 1 to 7, wherein the processing device is further configured to:

[0320] Data is written to the non-volatile memory module, wherein different portions of the data are written to different planes of the multi-plane die, and wherein the number of different portions is equal to the number of planes.

[0321] Statement 9. The storage system of Statements 1-8, wherein moving one or more portions from the existing erase block to the new erase block is performed during a garbage collection operation of the storage system.

[0322] Statement 10. The storage system of Statements 1-9 wherein moving one or more portions from the existing erase block to the new erase block is performed during a defragmentation operation of the storage system.

[0323] Statement 11. The memory system of Statements 1-10, wherein simultaneously accessing data from the multiple planes of the multi-plane die reduces the amount of power used by the memory system as compared to serially accessing data from the multiple planes of the multi-plane die.

[0324] Statement 12. The memory system of Statements 1-11, wherein simultaneously accessing data from the multiple planes of the multi-plane die reduces the amount of heat generated by the memory system as compared to serially accessing data from the multiple planes of the multi-plane die.

[0325] Statement 13. A method comprising:

[0326] determining that a number of planes used simultaneously to access data of a multi-plane die that is part of a non-volatile memory module of a plurality of non-volatile memory modules of a storage system should be changed;

[0327] In response to determining that the number of the planes of the multi-plane die used to access data simultaneously should be changed, one or more portions are moved from an existing erase block to a new erase block, the existing erase block having a size different from a size of the new erase block.

[0328] Statement 14. The method of Statement 13, wherein determining that the number of said planes for simultaneously accessing data of said multi-plane die should be changed comprises:

[0329] It is determined that an additional non-volatile memory module has been added to the memory system.

[0330] Statement 15. The method of Statements 13-14, wherein the one or more portions of the existing erase blocks include live data and other portions of the existing erase blocks include failed data.

[0331] Statement 16. The method according to statements 13 to 15, further comprising:

[0332] The one or more portions of the existing erase block are identified based on metadata, the metadata indicating real-time data within the existing erase block.

[0333] Statement 17. The method of Statements 13-16, wherein moving one or more portions from the existing erase block to the new erase block is performed during a garbage collection operation of the storage system.

[0334] Statement 18. The method of Statements 13-17, wherein moving one or more portions from the existing erase block to the new erase block is performed during a defragmentation operation of the storage system.

[0335] Statement 19. The method of Statements 13-18, wherein simultaneously accessing data from the multiple planes of the multi-plane die reduces the amount of power used by the memory system as compared to serially accessing data from the multiple planes of the multi-plane die.

[0336] Statement 20. A non-transitory computer-readable storage medium storing instructions that, when executed, cause a processing device of a storage system controller to:

[0337] determining that a number of planes used simultaneously to access data of a multi-plane die that is part of a non-volatile memory module of a plurality of non-volatile memory modules of a storage system should be changed;

[0338] In response to determining that the number of the planes of the multi-plane die used to access data simultaneously should be changed, one or more portions are moved from an existing erase block to a new erase block, the existing erase block having a size different from a size of the new erase block.

[0339] One or more embodiments may be described herein by means of method steps illustrating the execution of specified functions and their relationships. For ease of description, the boundaries and order of these function building blocks and method steps are arbitrarily defined herein. As long as the specified functions and their relationships are properly performed, alternative boundaries and orders may be defined. Therefore, any such alternative boundaries or orders are within the scope and spirit of the claims. In addition, for ease of description, the boundaries of these function building blocks are arbitrarily defined. As long as certain important functions are properly performed, alternative boundaries may be defined. Similarly, flow chart blocks may also be arbitrarily defined herein to illustrate certain important functionality.

[0340] To the extent used, the flow chart block boundaries and order may be defined in other ways and still perform certain important functionality. Therefore, this alternative definition of functional building blocks and flow chart blocks and order is within the scope and spirit of the claims. Those of ordinary skill in the art will also recognize that the functional building blocks and other illustrative blocks, modules and components herein may be implemented as depicted, or by discrete components, application specific integrated circuits, processors executing appropriate software, etc., or any combination thereof.

[0341] Although specific combinations of various functions and features of one or more embodiments have been expressly described herein, other combinations of these features and functions are also possible. The present disclosure is not limited to the specific examples disclosed herein, and expressly incorporates these other combinations.

Claims

1. A storage system, comprising: a plurality of non-volatile memory modules, a non-volatile memory module comprising a multi-plane die; A storage system controller operably coupled to a plurality of storage devices, the storage system controller comprising a processing device configured to: determining that a number of planes of the multi-plane die used to access data simultaneously should be changed; In response to determining that the number of the planes of the multi-plane die used to access data simultaneously should be changed, blocks are allocated with a new block size by combining sets of erase blocks at the same address in separate planes based on the newly determined number of planes.

2. The storage system of claim 1 , wherein in order to determine that the number of the planes for simultaneously accessing data of the multi-plane die should be changed, the processing device is further configured to: It is determined that an additional non-volatile memory module has been added to the memory system.

3. The storage system of claim 1 , wherein in order to determine that the number of the planes for simultaneously accessing data of the multi-plane die should be changed, the processing device is further configured to: A request is received to change the number of the planes of the multi-plane die used to access data simultaneously.

4. The storage system according to claim 1, further comprising: One or more portions are moved from an existing erase block to a new erase block, the existing erase block having a new block size.

5. The storage system according to claim 4, wherein the processing device is further configured to: The one or more portions of the existing erase block are identified based on metadata, the metadata indicating real-time data within the existing erase block. The storage system of claim 1 , wherein the new block size is larger than a previous block size.

7. The storage system of claim 1, wherein the number of the planes of the multi-plane die used to access data simultaneously is increased.

8. The storage system according to claim 1, wherein the processing device is further configured to: Data is written to the non-volatile memory module, wherein different portions of the data are written to different planes of the multi-plane die, and wherein the number of different portions is equal to the number of planes.

9. The storage system of claim 4, wherein moving one or more portions from the existing erase block to the new erase block is performed during a garbage collection operation of the storage system.

10. The storage system of claim 4, wherein moving one or more portions from the existing erase block to the new erase block is performed during a defragmentation operation of the storage system.

11. The memory system of claim 1 , wherein simultaneously accessing data from the multiple planes of the multi-plane die reduces an amount of power used by the memory system compared to serially accessing data from the multiple planes of the multi-plane die.

12. The memory system of claim 1, wherein simultaneously accessing data from the multiple planes of the multi-plane die reduces an amount of heat generated by the memory system compared to serially accessing data from the multiple planes of the multi-plane die.

13. A method comprising: determining that a number of planes used simultaneously to access data of a multi-plane die that is part of a non-volatile memory module of a plurality of non-volatile memory modules of a storage system should be changed; In response to determining that the number of the planes of the multi-plane die used to access data simultaneously should be changed, blocks are allocated with a new block size by combining sets of erase blocks at the same address in separate planes based on the newly determined number of planes.

14. The method of claim 13, wherein determining that the number of the planes for simultaneously accessing data of the multi-plane die should be changed comprises: It is determined that an additional non-volatile memory module has been added to the memory system.

15. The method according to claim 13, further comprising: One or more portions are moved from an existing erase block to a new erase block, the existing erase block having a new block size.

16. The method according to claim 15, further comprising: The one or more portions of the existing erase block are identified based on metadata, the metadata indicating real-time data within the existing erase block.

17. The method of claim 15, wherein moving one or more portions from the existing erase block to the new erase block is performed during a garbage collection operation of the storage system.

18. The method of claim 15, wherein moving one or more portions from the existing erase block to the new erase block is performed during a defragmentation operation of the storage system.

19. The method of claim 13, wherein simultaneously accessing data from the multiple planes of the multi-plane die reduces an amount of power used by the memory system compared to serially accessing data from the multiple planes of the multi-plane die.

20. A non-transitory computer-readable storage medium storing instructions that, when executed, cause a processing device of a storage system controller to: determining that a number of planes used simultaneously to access data of a multi-plane die that is part of a non-volatile memory module of a plurality of non-volatile memory modules of a storage system should be changed; In response to determining that the number of the planes of the multi-plane die used to access data simultaneously should be changed, blocks are allocated with a new block size by combining sets of erase blocks at the same address in separate planes based on the newly determined number of planes.

Images