Graph-based storage management

Abstract

Techniques are described in which storage nodes within a storage cluster are configured to support graph-based storage management. For example, a storage cluster includes a network and a plurality of computing systems, each interconnected over the network, where the plurality of computing systems includes a plurality of storage nodes. A computing system of a plurality of computing systems is configured to allocate a storage volume within a storage cluster, generate a volume graph for the volume, where the volume graph represents one or more functional elements in a data plane of the volume, and manage the volume based on the volume graph.

Description

Technical Field

[0001] The present disclosure relates to networks and storage systems, and more particularly, to management of storage systems within a data center network. Background Art

[0002] In a typical cloud-based data center, a large number of interconnected servers provide computing resources and / or storage capacity for the execution of various applications. For example, a data center may include a facility that hosts applications and services for subscribers (e.g., customers or tenants of the data center). The data center may, for example, host all infrastructure equipment, such as computing nodes, network and storage systems, power systems, and environmental control systems. In a typical data center, storage system clusters and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. The size of data centers varies greatly, with some public data centers containing hundreds of thousands of servers and often distributed across multiple geographic locations for redundancy.

[0003] Many devices in computer networks (e.g., storage / computing servers, firewalls, intrusion detection devices, switches, routers, and other network-attached devices) often use general-purpose processors, including multi-core processing systems, to process data (e.g., network or storage data). However, general-purpose processing cores and multi-processing systems are generally not designed for the high-volume network and storage workloads of modern networks and may be relatively poor at performing packet flow processing. In addition, in large-scale configurations, storage systems may become unavailable from time to time due to hardware errors, software errors, or other reasons. Data persistence procedures may be employed to provide more reliable access to critical data. Summary of the invention

[0004] The present disclosure describes a technology in which storage nodes within a storage cluster are configured to support graph-based storage management. In some examples, the present disclosure describes operations performed by a computing node, a storage node, a computing system, a network device, and / or a storage cluster according to one or more aspects of the present disclosure. In a specific example, the present disclosure describes a method comprising: allocating a storage volume within a storage cluster by a storage cluster having multiple storage nodes; generating a volume map of the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and managing the volume based on the volume map.

[0005] In another example, the present disclosure describes a computing system comprising: a network interface for interconnecting the computing system with at least one other computing system to form multiple computing systems on a network; and at least one storage node, wherein the at least one storage node is part of a storage cluster consisting of multiple computing systems, and wherein the computing system is configured to: allocate a storage volume within the storage cluster; generate a volume map for the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and manage the volume based on the volume map.

[0006] In another example, the present disclosure describes a computer-readable storage medium comprising instructions that, when executed, cause one or more processors to: allocate a storage volume within a storage cluster; generate a volume map for the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and manage the volume based on the volume map.

[0007] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

[0008] This summary introduces some of the concepts that are further described in the detailed description below in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is a block diagram illustrating an example system including one or more computing devices that support graph-based storage management, configured in accordance with one or more aspects of the present disclosure.

[0010] Figure 1B is a simplified block diagram illustrating an example storage cluster in accordance with one or more aspects of the present disclosure.

[0011] Figure 2A is a block diagram illustrating a system with a data processing unit (DPU) configured to support graph-based storage management, in accordance with the techniques described in this disclosure.

[0012] Figure 2B is a block diagram illustrating hardware components of an example DPU, according to techniques described in this disclosure.

[0013] Figure 3A is a conceptual diagram illustrating an example object model of a data plane for an allocated volume, according to techniques described in this disclosure.

[0014] Figure 3Bis a table describing information associated with an example set of abstraction layers according to techniques described in this disclosure.

[0015] Figure 4 In accordance with the techniques described in this disclosure, an example graph-based representation of functional elements in a data plane of a volume is illustrated.

[0016] Figure 5 In accordance with the techniques described in this disclosure, an example graph-based representation of a volume snapshot is illustrated.

[0017] Figure 6 is a block diagram illustrating example interactions of a volume with a clone of a user volume when performing read and write operations according to the techniques described in this disclosure.

[0018] Figure 7A-7B is a block diagram illustrating an example of performing a failover operation using a volume map according to the techniques described in this disclosure.

[0019] Figure 8 is a flow chart illustrating an example method for graph-based storage management according to the techniques described in this disclosure. DETAILED DESCRIPTION

[0020] Described herein is a technique in which storage nodes within a storage cluster are configured to support graph-based storage management. For example, storage cluster resources (e.g., storage nodes and / or storage devices) can be allocated for a "volume," which in some examples can refer to a conceptual abstraction of a storage unit in a storage cluster. The volume is a logical storage device represented by a collection of one or more storage nodes, wherein the volume provides an abstraction layer for physical storage devices and, in addition, presents block-level storage operations on the logical storage. The volume can be designed as a horizontal expansion model so that a pool of local storage devices in a storage cluster can operate as a common storage device pool. The design enables the storage system to be persistent and redundant. The horizontal expansion model of the volume is implemented through a data plane on a fully meshed structure of interconnected storage nodes, and the management and control planes of the volume can logically combine the operations of the storage nodes together.

[0021] Storage nodes within a storage cluster can be configured to generate a graph-based representation of functional elements in the data plane of the volume to which they are assigned. The graph representation of the volume, also referred to as a "volume graph," can be used to manage the volume. For example, a storage node assigned to a volume can perform various functions, such as storage functions and functions offloaded from a server, such as security functions (e.g., encryption), compression and regular expression processing, data persistence functions, data storage functions, and network operations. The volume graph for the volume may include nodes in the graph (referred to as "function nodes") that represent functions of the storage node. The volume graph also includes leaf nodes that represent resources (e.g., storage nodes and / or storage devices) assigned to the function. In addition, the volume graph also includes a root node that represents a tenant or customer of the storage cluster associated with the volume.

[0022] As described in the present disclosure, a storage node may use a volume map for managing volumes, such as resource allocation, event management, and large-scale recovery (e.g., failover) for volumes. In some examples, a function node in a volume map that represents a function of a volume (e.g., a data persistence scheme) may be replaced by another function node to implement a different type of function for the volume. In some examples, if a first resource for function allocation fails, fails to meet quality of service standards, or is otherwise no longer suitable for the volume, the function node in the volume map that represents the first resource may be replaced by another function node that represents a second resource for the function.

[0023] Multiple techniques described herein may provide one or more technical advantages. For example, by generating a graph-based representation of functional elements in the data plane of a volume, the management complexity of the volume is reduced. For example, in order to implement different types of functions (e.g., data persistence schemes) for a volume, a functional node representing a certain data persistence scheme (e.g., erasure code) is simply replaced in the volume graph with another functional node representing another data persistence scheme (e.g., replication), and one or more leaf nodes representing resources allocated for the new data persistence scheme are mapped. In some examples, the volume graph can be used to rebuild the volume by simply replacing leaf nodes representing resources that have failed, exceeded quality of service standards, or are otherwise no longer suitable for use with the volume. In some examples, failover can be achieved by simply rebuilding the volume graph on a secondary storage node so that it assumes the role of a primary storage node.

[0024] Figure 1Ais a block diagram illustrating an example system 108 according to one or more aspects of the present disclosure, the system 108 including one or more storage nodes configured to support graph-based storage management. The multiple techniques described herein can enable a storage node to generate a graph-based representation of functional elements within a data plane of a volume in a storage cluster. Management of the volume can be performed using the graph representation of the volume. The storage nodes described herein may also be referred to as data processing units (DPUs) or devices including DPUs. Other devices within a network, such as routers, switches, servers, firewalls, gateways, etc., can be easily configured to utilize the data processing techniques described herein.

[0025] The system 108 includes a data center 101 capable of providing data processing and data storage. In some examples, the data center 101 can represent one of many geographically distributed network data centers. Typically, the data center 101 provides an operating environment for applications and services of tenants 11 (e.g., customers) coupled to the data center 101. The data center 101 can host infrastructure equipment, such as computing nodes, network and storage systems, redundant power supplies, and environmental control devices. In the described example, the tenant 11 is coupled to the data center 101 through a service provider network 7 and a gateway device 20. The service provider network 7 can be coupled to one or more networks managed by other providers and can therefore form part of a large-scale public network infrastructure (e.g., the Internet).

[0026] exist Figure 1A In the example of FIG. 1 , data center 101 is a facility that provides information services to tenants 11. Tenants 11 may be collective entities, such as businesses and governments, or individuals. For example, a network data center may host network services for multiple businesses and end users. Other exemplary services may include data storage, virtual private networks, file storage services, data mining services, scientific or supercomputing services, etc.

[0027] Figure 1AThe controller 130 shown in the data center 101 of the embodiment of the present invention can be one or more computing devices that are responsible for managing the configuration and operation aspects of the data center 101. In some examples, the controller 130 can operate as a high-level controller, or can be a software-defined network (SDN) controller to configure and manage the routing and switching infrastructure of the data center 101. In this example, the controller 130 can provide a logically (and in some cases physically) centralized controller for facilitating the operation of one or more virtual networks in the data center 101. The controller 130 can operate independently or according to a signal received by an administrator device (not shown) in response to an administrator operation. The controller 130 can provide application programming interface ("API") support for various cluster services, which can include orchestration, storage services and / or storage management capabilities. This capability can also include resource balancing, infrastructure discovery, registration and initialization, role-based access control, multi-tenancy and resource partitioning, application workload deployment and orchestration, flexible network control, identity management, and hardware lifecycle management and monitoring.

[0028] The controller 130 may also be responsible for allocating and accounting resources for volumes, which may refer to a conceptual abstraction of a storage unit within a storage cluster. In this example, a volume may be a storage container divided into fixed-size blocks and may be allocated and released by the controller 130, and may also be written to and read by nodes or other devices within the data center 101.

[0029] In the illustrated example, data center 101 includes a set of storage systems and application servers 12 interconnected via a switching structure 114. In some examples, servers 12 are arranged into multiple different server groups, each group including any number of servers, such as up to "N" servers 12A to 12N. As used throughout this disclosure, N can be different in each instance. For example, N can be different for different server groups. Servers 12 are used to provide computing and storage facilities for applications and data associated with tenants 11, and can be physical (bare metal) servers, virtual machines running on physical servers, virtualized containers running on physical servers, or combinations thereof.

[0030] exist Figure 1AIn the example of , one or more servers 12 may couple one or more nodes 17 to the switch fabric 114 for processing information flows, such as network packets or storage packets. For example, the nodes 17 may be assigned respective IP addresses and provide routing operations to the servers 12. The nodes 17 may interface with and utilize the core switches within the switch fabric 14 to provide a fully meshed (any-to-any) interconnection, such that any node 17 (or server 12) may use any number of parallel data paths within the data center 101 to communicate packet data to any node 17 for a given packet flow.

[0031] For the example architecture of Node 17, this article refers to Figure 2A and Figure 2B For some or all of these examples, the architecture of each node 17 includes a multi-core processor system that represents a high-performance, hyper-converged network, storage, data processor, and input / output hub. The architecture of each node 17 can be optimized for high-performance and high-efficiency stream processing. As an example, the DPU corresponding to or within each node 17 can execute an operating system, such as a general-purpose operating system (e.g., Linux or Unix) or a dedicated operating system, which provides an execution environment for data plane software for data processing.

[0032] More details about how Node 17 operates can be found in U.S. Patent No. 10,841,245, entitled “WorkUnit Stack Data Structures in Multiple Core Processor System”; U.S. Patent No. 10,540,288, entitled “EFFICIENT WORK UNIT PROCESSING IN A MULTICORE SYSTEM,” filed on February 2, 2018; and U.S. Patent No. 10,659,254, entitled “Access Node Integrated Circuit for Data Centers which Includes a Networking Unit, a Plurality of Host Units, Processing Clusters, a Data Network Fabric, and a Control Network Fabric.”

[0033] Back to Figure 1AIn the example shown, the nodes 17 are logically or physically organized into node groups 19, units 40, and racks 70. The nodes 17 can be arranged into multiple different node groups 19, each node group 19 including any number of nodes, such as up to "N" nodes 17A-17N (representing any number of nodes 17). For example, rack 70A includes one or more node groups 19, each node group 19 including a group of nodes 17 and storage devices 127. The node group 19 and the server group 12 supported by the nodes 17 of the node group 19 are referred to herein as a network computing storage unit (NCSU) 40. Figure 1A NCSU 40A to NCSU 40N are shown in the figure, representing any number of NCSUs. In the use of the present disclosure, "N" can represent any number and can be changed in the diagram and its description. For example, nodes 17A-17N and NCSUs 40A-40N may have different "Ns". For ease of illustration, only the components of NCSU 40A are shown. In some examples, the data center 101 may include many NCSUs, and multiple NCSUs 40 may be organized into racks 70, which may be logical racks and / or physical racks within the data center 101. For example, in some implementations, two NCSUs may form a logical rack and four NCSUs may form a physical rack. Other arrangements are also possible. Such other arrangements may include nodes 17 within racks 70 being relatively independent and not logically or physically included in any node group or NCSU 40.

[0034] Some nodes 17 are shown connected to one or more servers 12, which can be used to offload (from the servers 12) some of the work of packet processing and other network-related functions. For example, each node group 19 of the rack 70A can be configured to operate as a high-performance I / O hub designed to aggregate and process network and / or storage I / O for multiple servers 12. As further described below, the collection of nodes 17 within each node 19 in the node group can provide programmable dedicated I / O processing circuits for the server 12 to handle network and communication operations. In addition, in some examples, each node group 19 in the node group may include a storage device 127, such as a solid-state drive (SSD) and / or a hard disk drive (HDD), which is configured to provide network-accessible storage that can be used by applications executing on the server 12. In some examples, one or more of the SSDs may include non-volatile memory (NVM) or flash memory. Although logically illustrated as being located within node group 19 and external to nodes 17 , storage device 127 may additionally or alternatively be included within one or more nodes 17 or within one or more servers 12 .

[0035] One or more nodes 17 may serve as storage nodes ("storage targets"), which may not be directly connected to any server 12. For example, Figure 1A A rack 70B is shown, which includes nodes 17A to 17N (representing any number of nodes 17). The nodes 17 of the rack 70B can store data in one or more storage devices 127 (including within these nodes 17 or connected to these nodes 17) according to the technology described herein. The nodes 17 in the rack 70B can be regarded as network interface subsystems, which are configured as data storage nodes to store data in the storage devices 127. In the illustrated example, the nodes 17 in the rack 70B are not organized into groups or units, but are relatively independent, and each node is capable of performing the storage functions described herein. However, in other examples, the nodes 17 of the rack 70B can be logically or physically organized into groups, units and / or logical racks.

[0036] Rack 70C is illustrated as being implemented in a manner similar to rack 70B, wherein node 17 is a storage node configured to store data in storage device 127 (included within or connected to the node 17). For ease of illustration, Figure 1A Only racks 70A, 70B, 70C to 70N are shown in the figure. It can be easily understood that any number of racks 70 can be included in the data center 101 and any configuration can be adopted, including the configuration described herein. In addition, although Figure 1A The diagram illustrates a rack 70A whose nodes 17 support servers 12, and other racks 70B, 70C whose nodes 17 serve as storage nodes, but any number and combination of racks may be implemented. In addition, any rack 70 may include a combination of nodes 17 supporting servers 12 and nodes 17 serving as storage nodes. Further, although the data center 101 is described in the context of nodes 17 being arranged within racks 70, other logical or physical arrangements of nodes 17 may be used. For example, other implementations may involve groups, units, or other logical or physical arrangements that do not involve racks.

[0037] Node 17 of rack 70B (or rack 70C) may be a device or system that is the same or similar to node 17 of rack 70A. In other examples, node 17 of rack 70B may have different capabilities and / or may be implemented differently than rack 70A. Figure 1A In the example shown, the nodes 17 of rack 70B may be more capable (e.g., more computing power, more memory capacity, more storage capacity, and / or other additional functionality) than the nodes 17 of rack 70A. For example, each node 17 of rack 70B may be implemented using a pair of nodes 17 of rack 70A. To reflect this example, Figure 1AThe nodes 17 of racks 70B and 70C illustrated in FIG. 7 are larger than the nodes 17 of rack 70A.

[0038] As further described herein, in one example, each node 17 may be a highly programmable I / O processor specifically designed to perform storage functions and / or offload certain functions from the server 12. Each node 17 may be implemented as a component (e.g., an electronic chip) within a device (e.g., a computing node, an application server, or a storage server) and may be deployed on the device's motherboard or within a removable card, such as a storage and / or network interface card. In some examples, each node 17 may be implemented as one or more application specific integrated circuits (ASICs) or other hardware and software components, each supporting a subset of the storage devices 127 or a subset of the servers 12.

[0039] In some examples, each node 17 includes multiple internal processor clusters, each cluster including two or more processing cores, and is equipped with hardware engines that can offload certain functions from the server 12, such as security functions (e.g., encryption), acceleration (e.g., compression) and regular expression (RegEx) processing, data persistence functions (e.g., erasure coding, replication, etc.), data storage functions, and network operations. One or more nodes 17 may include a data persistence module or unit, referred to as an "accelerator" unit, which can be implemented as a dedicated module or unit for performing data persistence functions. For example, one or more computing devices may include a node that includes one or more data persistence, data reliability, and / or erasure coding accelerator units.

[0040] In some examples, one or more nodes 17 are configured to establish volumes for system 108. Multiple volumes may be established for a given system. A volume represents a logical storage device, including a group of one or more nodes 17, wherein the logical storage device provides an abstraction level from the physical storage device 127 and, in addition, provides block-level storage operations on the logical storage. Volumes may provide persistent storage of data blocks and may enable seamless hardware-based data persistence capabilities enabled by the hardware engine of the node 17. An example system architecture for volumes (otherwise referred to as "persistent block devices") is described in U.S. Patent No. 10,949,303, entitled "DURABLE BLOCK STORAGE IN DATA CENTER ACCESS NODES WITH INLINEERASURE CODING," filed on December 10, 2018.

[0041] The multiple techniques described herein enable graph-based management of a storage cluster. One or more nodes 17 may generate a graph-based representation of functional elements in a data path of a volume of the storage cluster. As further described below, the volume graph may include various abstract layers of volumes that represent one or more data storage schemes (e.g., data persistence, data reliability, etc.). One or more nodes 17 and storage devices 127 may be used to implement one or more data storage schemes. Figure 1A In an example, the controller 130 may include a map generation module 131 for generating a volume map for a target node, for example, a volume map 135 representing storage functions and / or functions uninstalled from the server 12 and resources allocated for the function, such as node 17 and / or storage device 127.

[0042] Node 17A of rack 70B may use volume map 135 to manage the storage cluster. For example, node 17A of rack 70B may pass information to each DPU based on volume map 135 to manage the storage cluster. In some examples, node 17A of rack 70B may use volume map 135 for modifying existing volumes, resource allocation, event management, dynamic rebalancing of resources, and / or volume attribute modification for the storage cluster. Examples of volume maps and the use of volume maps are described in conjunction with Figure 4 Give a description.

[0043] Figure 1B is a simplified block diagram illustrating an example storage cluster in accordance with one or more aspects of the present disclosure. Figure 1B A storage cluster 102 is shown, which may be Figure 1A An example storage cluster included in data center 101. Figure 1B The storage cluster 102 in Figure 1A The schematic diagram of the data center 101 is similar and includes Figure 1A However, Figure 1A The elements are already in Figure 1B 1 to help illustrate certain aspects of how storage cluster 102 is implemented within data center 101.

[0044] exist Figure 1B In the example of , storage cluster 102 includes controller 130, one or more initiator nodes 110, and one or more storage nodes 120, all of which can communicate via switch fabric 114. Storage cluster 102 may include one or more volumes 121 (e.g., volume 121J and volume 121K), each of which may be viewed as a conceptual abstraction of a storage unit in storage cluster 102. Volumes may be associated with different tenants or customers of data center 101 for storage cluster 102. For example, in Figure 1BIn the illustrated example, volume 121J has been allocated to tenant J for use, and volume 121K has been allocated to tenant K for use. Figure 1B The dashed lines extending from volumes 121J and 121K in FIG. 1 are intended to illustrate that each of these volumes 121 is stored across multiple storage nodes 120. Figure 1B Only two volumes are shown, and the storage cluster 102 can support any number of volumes 121 for any number of tenants. Figure 1B Only a single tenant is illustrated for each volume in the volumes 121 , but a volume may also be allocated to multiple tenants for use.

[0045] like Figure 1A , Figure 1B The controller 130 in the storage cluster 102 can provide cluster management orchestration of storage resources within the storage cluster 102. Figure 1A , Figure 1B The controller 130 in the embodiment may be implemented by any suitable computer system, including one or more computing nodes within the data center 101 or the storage cluster 102. Figure 1B Although illustrated as a single system in the storage cluster 102 of the data center 101, the controller 130 may be implemented as multiple systems and / or as a distributed system located both inside and outside the data center 101 and / or the storage cluster 102. In other examples, some or all aspects of the controller 130 may be implemented outside the data center 101, such as in a cloud-based implementation.

[0046] exist Figure 1B In the example of , the controller 130 includes a storage service module 132 and a data store 133. The storage service module 132 of the controller 130 can perform functions related to establishing, allocating, and enabling read and write access to one or more volumes 121 within the storage cluster 102. Generally, the storage service module 132 can perform functions characterized as "cluster services" or "storage services", which can include allocating, creating and / or deleting volumes. In some examples, the storage service module 132 can also provide services to help meet the quality of service standards of the volumes 121 within the storage cluster 102. In some examples, the storage service module 132 can also manage input from one or more administrators (e.g., an operations administrator device 134). Generally, the storage service module 132 can have a comprehensive understanding of all resources within the storage cluster 102 and how the resources are allocated among the various volumes 121.

[0047] The data storage 133 may represent any suitable data structure or storage medium for storing information related to resources within the storage cluster 102 and how these resources are allocated within the storage cluster 102 and / or between volumes 121. The data storage 133 may be primarily maintained by the storage service module 132.

[0048] Each of the initiating nodes 110 may correspond to or be represented by Figure 1A One or more implementations of the server 12 and / or node 17 illustrated in FIG. Figure 1B In the example of FIG. 1 , each of the initiating nodes 110 is shown as including at least one server 112 , where the initiating nodes 110A– 110C each include a DPU 118 . Figure 1B Each server 112 within the initiating node 110 can communicate with Figure 1A Corresponding to one or more servers in the server 12. Similarly, Figure 1B Each DPU 118 within the initiating nodes 110A-110C may communicate with Figure 1A Therefore, in combination with Figure 1A The description provided with respect to server 12 and node 17 can also be applied to Figure 1B The server 112 and DPU 118 in.

[0049] Figure 1B The initiator nodes 110 shown in the figure may be involved in causing or initiating read and / or write operations with the storage cluster represented by the storage cluster 102. The DPU 118 within each of the initiator nodes 110 may serve as a data path center for each of the initiator nodes 110, connecting each of the initiator nodes 110 (and the storage nodes 120) through the switch fabric 114. In some examples, one or more of the one or more initiator nodes 110 may be an x86 server that may execute NVMe (Non-Volatile Memory Express) over a communication protocol (e.g., TCP). In some examples, other protocols may be used, including the "FCP" described in U.S. Patent No. 11,178,262, entitled "FABRICCONTROL PROTOCOL FOR DATA CENTER NETWORKS WITH PACKET SPRAYING OVER MULTIPLEALTERNATE DATA PATH".

[0050] Each storage node in the storage node 120 may be composed of Figure 1A Implementation of the node 17 and storage device 127 shown in FIG. Accordingly, Figure 1A The description of the node 17 and the storage device 127 in Figure 1B The DPU 117 and storage device 127 in the. Figure 1B Storage nodes 120 are illustrated in to emphasize that in some examples, each of storage nodes 120 may serve as a storage target for initiating node 110 .

[0051] Figure 1B Also included is a conceptual diagram of volumes 121J and 121K. Within storage cluster 102, volume 121 can serve as a storage container for data associated with a tenant of storage cluster 102, where each volume is an abstraction intended to represent a set of data stored on one or more storage nodes 120. In some examples, each volume in volume 121 can be divided into fixed-size blocks and can support multiple operations. Typically, this operation generally includes a read operation (e.g., reading one or more fixed-size blocks from a volume) and a write operation (e.g., writing one or more fixed-size blocks to a volume). Other operations are also possible and fall within the scope of the present disclosure.

[0052] exist Figure 1B In the illustrated example, the controller 130 may receive a request to allocate a volume. Figure 1B In the described example, controller 130 can detect an input that it determines corresponds to a request to create a new volume. In some examples, the input originates from one or more of initiating nodes 110 seeking to create a new volume for a tenant (e.g., Figure 1B In other examples, the input may originate from an administrator device (e.g., administrator device 134), which may be operated by an administrator seeking to allocate new storage space for a tenant representing storage cluster 102. In other examples, the input may originate from a different device.

[0053] Controller 130 may allocate volumes based on one or more rules (or criteria). The rules may be based on one or more metrics, such as availability of input / output operations per second (“IOPs”), storage capacity availability, failure domains, quality of service criteria, and / or volume types, such as durability schemes (e.g., erasure coding, replication). For example, controller 130 may receive information describing one or more rules, where the information comes from or originates from input from an administrator (e.g., via administrator device 134). In other examples, such input may originate from a representation of a tenant (e.g., via a client device, Figure 1B ), where the representation is used to select or specify rules for the storage cluster 102. The controller 130 can output information regarding a request to allocate a new volume to the storage service module 132, which evaluates the information and determines that the request is for a new volume to be allocated for a specific tenant (e.g., tenant "J" in the example being described). The storage service module 132 also determines information about the volume type and the quality of service associated with the new volume based on input received by the controller 130. The storage service module 132 accesses the data store 133 and determines which storage nodes 120 can be allocated to support the new volume. Any number of storage nodes can be assigned to support a volume. Figure 1B In the example shown, volume 121J is allocated using three storage nodes 120. In some examples, such a determination may involve evaluating which DPUs 117 and storage devices 127 within a storage node 120 are available to service read and write requests for the new volume. For example, the storage service module 132 may determine which DPUs 117 and / or storage devices 127 have sufficient IOPs required for the volume. The storage service module 132 may additionally or alternatively determine which storage devices 127 have sufficient storage capacity for the volume.

[0054] In some examples, the storage service module 132 may determine which DPUs 117 and storage devices 127 within the storage node 120 may provide data protection for the new volume. For example, to determine which DPUs 117 and storage devices 127 may be assigned to support the new volume, the storage service module 132 may determine whether the DPUs 117 and storage devices 127 are located in different failure domains and reduce the possibility of losing or being unable to use multiple DPUs and storage devices 127 at the same time.

[0055] In some examples, the storage service module 132 may determine the usage of the DPUs 117 and storage devices 127 within the storage node 120 and be used to allocate resources for the new volume in a manner that balances the load of the DPUs 117 and storage devices 127 within the storage node 120. For example, the storage service module 132 may determine the input / output operation (IOP) usage of the DPUs 117 and storage devices 127, apply a cost to each DPU 117 and storage device 127 based on their IOP usage, and then allocate DPUs 117 and storage devices to the new storage volume based on the IOP usage cost (e.g., the DPU 117 and storage device 127 with the lowest cost may be allocated). Other criteria may also be used. In some examples, the storage service module 132 may determine the cost of each storage device 127 based on its storage capacity usage, and allocate DPUs 117 and storage devices 127 based on the determined costs (e.g., the DPU 117 and storage device 127 with the lowest cost may be allocated).

[0056] In some examples, the controller 130 may assign one or more storage nodes 120 as a primary node, a secondary node, and / or a plex node for a volume 121J. This assignment may be used for various functions, including data reliability and / or data persistence schemes. For example, data persistence schemes (e.g., replication, erasure coding, etc.) may be implemented by storing data in multiple storage nodes 120 and / or multiple failure domains. In some implementations, a primary target node (e.g., a primary storage node) may be used as a communication or interaction point for other devices within the storage cluster 102 (e.g., an initiating node 110). For example, the storage service module 132 may designate storage node 120A as a "primary" target node that serves as the primary target or interaction node for operations involving the volume, while one or more storage nodes 120A, 120B, and 120D (including storage nodes within the volume) serve as plex nodes for storing data associated with the volume.

[0057] A plex node may be used to store data associated with a volume and may be managed by a primary target node. A "plex" may refer to a unit of data (e.g., located on a separate drive) that is a member of a particular volume (e.g., an erasure coded volume). In this example, volume 121J may include one or more plex nodes located locally or remotely from a storage node (e.g., storage nodes 120A, 120B, and 120D). Storage node 120 may have plex nodes for multiple volumes. For example, storage node 120B may have one or more plex nodes for volume 121J and one or more plex nodes for volume 121K.

[0058] The storage service module 132 may also specify a "secondary" target node (e.g., a secondary storage node) that will serve as a failover target or interactive node when the primary storage node is unavailable or inoperable. The "secondary" target node may be a storage node within the same volume as the primary storage node. For example, the storage service module 132 may specify storage node 120B as a "secondary" target node that will serve as a failover target or interactive node when the primary storage node 120A is unavailable or inoperable. In these examples, volume 121J may provide logging to provide data reliability, wherein an intent log (i.e., a log) including data and metadata of the primary target node (e.g., storage node 120A) is copied to the secondary target node (e.g., storage node 120B) so that any data of the host server (e.g., server 112) that has been confirmed to be written to the application can be reliably written to the underlying storage medium when the primary target node responds to a failure.

[0059] Typically, the storage service module 132 ensures that the designated primary target node (e.g., storage node 120A) and secondary target node (e.g., storage node 120B) are assigned to different storage nodes 120 or failure domains. The plex nodes may also be stored in different storage nodes 120 or failure domains. In some implementations, the same storage node 120 may be used as both a plex node and a primary target node (or, alternatively, as a plex node and a secondary target node).

[0060] In some examples, volume 121J may use log structuring and a data persistence scheme, such as aggregating multiple data blocks into larger data blocks (or dividing large data blocks into smaller data blocks) before storing across multiple storage devices for a data persistence scheme.

[0061] The management of volumes can be performed using various techniques. In some implementations, a volume map of functional elements in a volume (e.g., volume 121J) data plane is used to manage the volume within a storage cluster (e.g., storage cluster 102). The volume map can be generated in a variety of ways. For example, the controller 130 may include a map generation module 131 for generating a volume map for a target node representing storage functions and / or functions uninstalled from the server 112 and resources allocated to these functions (e.g., storage nodes 120 and / or storage devices 127). More specifically, the map generation module 131 can be used to generate a volume map 135A for the primary target node (e.g., storage node 120A) of volume 121J, wherein the volume map 135A can represent functional elements in the volume 121J data plane. The volume map 135 can be configured in various graph structures. In some examples, the volume map 135 can represent a tree structure, including functional nodes representing functions associated with the volume, and leaf nodes representing resources allocated for the functions. For example, volume map 135A may include a root node representing a host server associated with volume 121J, intermediate nodes (eg, function nodes) representing data persistence operations implemented by volume 121J, and one or more leaf nodes representing resources allocated for the data persistence operations.

[0062] The storage nodes 120 may be assigned to volumes to perform various functions. For example, the volume map 135A may include storage nodes 120 assigned to a data persistence scheme (e.g., an erasure coding scheme) for volume 121J. In this example, the erasure coding block size of volume 121J may be represented as m+n, where the variable m is the amount of original data and the variable n is the additional or redundant data added to provide fault protection. For example, the storage service module 132 may allocate DPUs 117 and storage devices 127 within the storage nodes 120A, 120B, and 120D according to the erasure coding scheme (e.g., m+n). In this example, the graph generation module 131 may generate a volume map 135A representing the DPUs 117 and storage devices 127 assigned to the erasure coding scheme for volume 121J within the storage nodes 120A, 120B, and 120D. For example, the volume map 135A may include a functional node representing an erasure coded volume for volume 121J, and a leaf node representing resources allocated for the erasure coded volume.

[0063] In some implementations, the volume may additionally or alternatively provide a replication scheme. For example, the block size of the replicated volume of volume 121J may be represented by a replication factor "X", which represents a factor by which data is replicated to provide high data availability to protect a device or node from failures. The graph generation module 131 may generate a volume graph 135A representing the DPUs 117 and storage devices 127 allocated within the storage nodes 120A, 120B, and 120D for the replication scheme of volume 121J. For example, the volume graph 135A may include a functional node representing the replicated volume of volume 121J, whose leaf nodes represent resources allocated for the replicated volume.

[0064] In some implementations, the volume may also or alternatively provide a journaling scheme to provide data reliability in the event of a failure. For example, volume map 135A may include a functional node representing a journal volume for volume 121J, whose leaf nodes represent a journal copy (e.g., in non-volatile memory (NVM)) stored in a primary target node (e.g., storage node 112A) and another journal copy stored in a secondary target node (e.g., storage node 112B).

[0065] In some implementations, the volume may provide log structuring before executing the data persistence scheme. For example, the volume map 135A may include a function node representing a log-structured volume of volume 121J, whose leaf nodes represent data persistence volumes.

[0066] The volume map may include one or more root nodes, each root node representing a host server executing an application that initiates read and write operations to the volume. For example, volume map 135A may include a root node representing DPU 118A of initiating node 110A, which may initiate read and write requests for an application executing on server 112A corresponding to volume 121J.

[0067] Volume map 135 may be used to manage volumes. For example, volume map 135A may be used to allocate resources to volume 121J, modify the allocated resources of volume 121J, manage events associated with resources allocated to volume 121J, dynamically rebalance the resources allocated to volume 121J, and / or manage volume attribute modifications of volume 121J. As an example, volume map 135 may be used to modify the data persistence scheme of the volume. For example, volume 121J may have initially implemented erasure coding. To modify the data persistence scheme of volume 121J from erasure coding to replication, graph generation module 131 may modify volume 121J by replacing a functional node representing an erasure coded volume and its leaf nodes, the replaced functional nodes representing a replicated volume, and its leaf nodes representing the resources allocated to the replicated volume.

[0068] In some examples, the volume map 135 may be used for event management. Events may include a storage node 120 crash, a storage device 127 being unplugged, a volume being deleted, and any other event related to resources allocated to the volume map 135. The volume map 135 may be used to rebuild a volume based on an event. For example, a network device including the storage device 127 may detect that one of the storage devices 127 has failed and generate an event indicating that the failed storage device has failed. The event may propagate upward in the layers of the volume map 135. For example, such an event may propagate from a leaf node representing the failed storage device 127 to the parent node of the leaf node, such as a functional node representing a data persistence volume. In response to the functional node representing the data persistence volume receiving the event, the storage node 120 (including the failed storage device 127) may send a message to the controller 130 to notify the controller 130 about the situation of the failed storage device 127. Then, the map generation module 131 may use another storage device to rebuild one or more leaf nodes of the data persistence volume.

[0069] In some examples, the volume map 135 may be used to dynamically rebalance the resources of the volume 121. For example, the storage device and / or the node may obtain one or more indicators, including storage usage, IOP usage, health of the storage device, bandwidth of the node, etc. The storage device and / or the node may compare these indicators with thresholds and generate an alarm when the indicator reaches a certain (multiple) thresholds. Based on the alarm, the map generation module 131 can rebalance the load of the resource based on the alarm. As an example, the storage capacity of the storage device 127 may exceed the storage usage threshold. In response, the storage device 127 can generate an alarm. The alarm can be propagated upward in the layers of the volume map 135. For example, the alarm can be propagated from a leaf node representing the storage device 127 to the parent node of the leaf node, for example, a functional node representing a data persistence volume. In response to receiving an event for a node representing a data persistence volume, the storage node 120 can rebalance the load on the storage device 127, for example, moving the load on the storage device 127 to another storage device.

[0070] In some examples, volume 121 may be modified using volume map 135 when one or more parameters for allocating volume 121 are changed. Volume 121 may be allocated using a specified set of one or more parameters (e.g., block size, encryption key, compression scheme, volume size, data protection scheme, etc.). For example, a volume may be allocated using a specified data protection scheme (e.g., erasure coded replication, no protection, etc.). In some examples, one or more parameters may be changed after volume 121 is created. In these examples, the new parameters are verified and a clone of the volume is created using the new parameters. Storage node 120 may then connect to the clone of the volume created using the new parameters.

[0071] Figure 1B The modules illustrated in (e.g., storage service module 132) and / or illustrated or described elsewhere in this disclosure may perform operations described using software, hardware, firmware, or a combination of hardware, software, and firmware located on and / or executed on one or more computing devices. For example, a computing device may use multiple processors or multiple devices to execute one or more of these modules. A computing device may execute one or more of these modules as a virtual machine executing on underlying hardware. One or more of these modules may be executed as one or more services of an operating system or computing platform. One or more of these modules may be executed as one or more executable programs on an application layer of a computing platform. In other examples, the functionality provided by the modules may be implemented by dedicated hardware devices.

[0072] Although some modules, data storage, components, programs, executable files, data items, functional units and / or other projects included in one or more storage devices may be described separately, one or more of these projects may be combined and run as a single module, component, program, executable file, data item or functional unit. For example, one or more modules or data storages may be combined or partially combined so that they run or provide functions as a single module. In addition, one or more modules may interact with each other and / or combine operations, for example, a module runs as a service or extension of another module. Similarly, each module, data storage, component, program, executable file, data item, functional unit or other project described in a storage device may include multiple undescribed components, subcomponents, modules, submodules, data storage and / or other components or modules or data storages.

[0073] In addition, each module, data store, component, program, executable file, data item, functional unit, or other item illustrated in the storage device can be implemented in a variety of ways. For example, each module, data store, component, program, executable file, data item, functional unit, or other item shown in the storage device can be implemented as a downloadable or pre-installed application or "app". In other examples, each module, data store, component, program, executable file, data item, functional unit, or other item illustrated in the storage device can be implemented as part of an operating system executed on a computing device.

[0074] Figure 2A is a block diagram illustrating a system 201 having a data processing unit (DPU) 210 configured to support graph-based storage management in accordance with various techniques described in this disclosure. Figure 2A In the illustrated example, system 201 also includes a CPU 240 that is communicatively coupled to DPU 210. Each of DPU 210 and CPU 240 generally represents a hardware chip implemented in digital logic circuits. The operation of DPU 210 can be related to Figure 1A Node 17 and Figure 1B Any one of the DPUs 117 is substantially similar.

[0075] DPU 210 may be implemented as a highly programmable I / O processor having multiple processing cores (as described below, for example, with respect to Figure 2B ).exist Figure 2AIn the illustrated example, the DPU 210 includes a network interface (e.g., an Ethernet interface) to directly connect to a network, and multiple host interfaces (e.g., a PCI-e interface) to directly connect to one or more application processors (e.g., CPU 240) and one or more storage devices (e.g., multiple SSDs). The DPU 210 also includes a data plane operating system (OS) 212 executed on two or more of the multiple processing cores. The data plane OS 212 provides a data plane 214 as an execution environment for running to complete software functions called on the data plane OS 212 to process work units. The work unit is associated with one or more flow data units (e.g., packets in a packet flow), and specifies a software function for processing the flow data unit and at least one processing core of the multiple processing cores for executing the software function.

[0076] The software function called to process the work unit may be one of a plurality of software functions for processing stream data. The software function may be included in a library 220 provided by the data plane OS 212. In the illustrated example, the library 220 includes a network function 222, a storage function 224, a security function 226, and an analysis function 228. For example, the network function 222 may include network I / O data processing functions related to Ethernet, network overlay, network protocol, encryption, and firewall. The storage function 224 may include storage I / O data processing functions related to NVME (non-volatile memory expression), compression, encryption, replication, erasure coding, and pooling. The security function 226 may include security data processing functions related to encryption, regular expression processing, and hash processing. The analysis function 228 may include analysis data processing functions related to a customizable data conversion pipeline.

[0077] Typically, the data plane OS 212 can be implemented as a low-level, run-to-completion operating system that runs on the bare metal DPU 212, which runs hardware threads for data processing and manages work units. As described in more detail below, the data plane OS 212 may include queue manager logic for managing work unit interfaces, enqueuing and dequeuing work units from a queue, and calling software functions specified by the work unit on a processing core specified by the work unit. In a run-to-completion programming model, the data plane OS 212 is configured to dequeue a work unit from a queue, process the work unit on a processing core, and return the results of processing the work unit to the queue.

[0078] DPU 210 also includes a multi-tasking control plane operating system 232 that executes on one or more of the plurality of processing cores. In some examples, multi-tasking control plane operating system 232 may include Linux, Unix, or a proprietary operating system. Figure 2AAs shown, in some examples, the control plane OS 232 provides a control plane 216, which includes a control plane software stack executed on the data plane OS 212. As shown, the control plane software stack includes a hypervisor 230, a multi-tasking control plane OS 232 executed on the hypervisor 230, and one or more control plane service agents 234 executed on the control plane OS 232. The hypervisor 230 can operate to isolate the control plane OS 232 from the work unit and data processing executed on the data plane OS 212. The control plane service agent 234 executed on the control plane OS 232 includes application-level software that is configured to perform the setup and teardown of software structures to support the work unit processing performed by the software functions executed on the data plane OS 212. In the example of packet processing, the control plane service agent 234 is configured to set up a packet flow for packet processing performed by the software functions on the data plane OS 212, and tear down the packet flow after the packet processing is completed. In this manner, DPU 210 comprises a highly programmable processor that can run application-level processing while leveraging the underlying work unit data structure for parallel stream processing.

[0079] In another example, the multi-tasking control plane operating system may not run on top of the data plane OS 212, but instead run on one or more independent processing cores dedicated to the control plane operating system, which are different from the processing cores that execute the data plane OS 212. In this example, if there is an independent processing core dedicated to the control plane operating system at the hardware level, then the control plane software stack may not include a hypervisor. Instead, the control plane software stack running on the independent processing core may include a multi-tasking control plane operating system and one or more control plane service agents executed on the control plane operating system.

[0080] CPU 240 is an application processor with one or more processing cores for handling computationally intensive tasks. Figure 2A In the illustrated example, the CPU 240 includes multiple host interfaces (e.g., PCI-e interfaces) to directly connect to the DPU 210. The CPU 240 includes a hypervisor / operating system 242 that supports one or more service agents 246 and one or more drivers 247. Figure 2AAs shown, the CPU 240 may also include a virtual machine (VM) operating system 244 executing on top of a hypervisor / operating system 242, which supports one or more drivers 248. Application-level software executing on the operating system 242 of the CPU 240, such as an agent 246 or a driver 247, or a driver 248 executing on the virtual machine operating system 244, may determine which data processing tasks should be offloaded from the CPU 240 to the DPU 210. For example, the hypervisor / operating system 242 of the CPU 240 may use multiple physical functions (PFs) and / or multiple virtual functions (VFs) of a PCIe link to offload data processing tasks to the DPU 210. In some implementations, the virtual machine operating system 244 of the CPU 240 may use multiple virtual functions of a PCIe link to offload data processing tasks to the DPU 210.

[0081] exist Figure 2A In the illustrated example, the system 201 also includes a controller 200 that communicates with the DPU 210 and the CPU 240 via a control application programming interface (API). The controller 200 may provide a high-level controller for configuring and managing application-level software executed on the control plane operating systems of the DPU 210 and the CPU 240, respectively. For example, the controller 200 may configure and manage which data processing tasks should be offloaded from the CPU 240 to the DPU 210. In some examples, the controller 200 may include a software defined network (SDN) controller that may operate in a manner similar to that of a controller 200. Figure 1A and 1B Controller 200 may be substantially similar to controller 130 in FIG. In some examples, controller 200 may operate in response to configuration input received by a network administrator via an orchestration API.

[0082] The data plane operating system 212 of the DPU 210 is configured to receive flow data units for processing on behalf of application-level software executing on the hypervisor / operating system 242 of the CPU 240. In the example of packet processing, the flow data units may include packets of a packet flow. In this example, the received packet flow may include any network packet flow, storage packet flow, security packet flow, analytics packet flow, or any combination thereof. The data plane operating system 212 executing on a processing core of the DPU 210 may receive a packet from a network unit, a host unit, or another processing core of the processing core of the DPU 210 (e.g., as discussed below with respect to FIG. 1 ). Figure 2B) receives each packet flow in the form of one or more work units. Each work unit in the work units of the received packet flow may be associated with one or more packets of the packet flow. After receiving the packet flow, the data plane operating system 212 may look up in the flow table to determine that the packet flow is legal, and may map the packet flow to one of the processing cores of the DPU 210 so as to serialize the packets of the packet flow. The flow table may include a hardware-implemented flow table that is updated and maintained by the control plane 216 for legal packet flows and is used to assign processing cores to packet flows.

[0083] In the event that the data plane operating system 212 cannot identify the received packet flow (e.g., the packet flow has not been set up in the flow table), the data plane operating system 212 may send the packet flow through the slow path of the control plane 216 for setup. The control plane service agent 224 executing on the control plane operating system 232 may then determine whether the packet flow is legitimate and send instructions to the data plane operating system 212 to set the packet flow in the flow table.

[0084] Once the control plane service agent 234 sets up the packet flow, the data plane operating system 212 can assign the packet flow to a specific processing core of the DPU 210, which can perform flow processing for the packet flow. As an example, the data plane operating system 212 can execute a queue manager configured to receive work units associated with one or more packets of the packet flow, enqueue the work units to a work unit queue associated with the processing core for the packet flow, dequeue the work units from the work unit queue to the processing core, and invoke a software function specified by the work unit on the processing core for processing the work unit.

[0085] The data plane operating system 212 also provides an interface for one or more hardware accelerators of the DPU 212 (as described below, for example, with respect to Figure 2B ), these hardware accelerators are configured to perform acceleration of various data processing functions. The data plane operating system 212 can use hardware accelerators to process one or more portions of a packet flow, namely one or more work units, which are arranged as a work unit (WU) stack. In a work unit stack, a work unit can include an identifier of a subsequent work unit in the work unit stack to further process the packet after the work unit is completed. In order to perform stream processing on a packet flow, the hardware accelerator can be configured to execute one or more hardware commands included in the work unit stack as input parameters of the first work unit, and after completing the one or more hardware commands, proceed to the subsequent work unit in the work unit stack identified by the current work unit.

[0086] As described herein, the DPU utilizes fine-grained work units, work unit queues, and queue managers executed on the data plane operating system of each processing core to serialize packet processing so that packets of the same packet flow are processed by the same processing core. In this way, the DPU is able to process any type of packet flow at fine granularity between processing cores and reduce processing overhead. In contrast, other multi-core systems may use shared memory and locks to provide memory consistency. The granularity of these lock schemes can be (multiple) orders of magnitude larger than the work unit scheme described herein. For example, in some implementations, the processing overhead associated with the work unit scheme can be less than 100 clock cycles. The processing overhead may include the number of cycles required to implement the work unit, and the number of cycles required to dequeue and deploy the work unit to a given processing core for processing. As described herein, serializing packet processing on a given run-to-completion hardware thread to maintain synchronization can generate approximately the same overhead as the lock scheme used in a traditional multi-core system.

[0087] According to various techniques described in this disclosure, the controller 200 may include a graph generation module 280 for generating a volume (e.g., Figure 1B The operation of the graph generation module 280 may be similar to Figure 1A and Figure 1B The graph generation module 280 may be substantially similar to the graph generation module 131 in . In this example, the graph generation module 280 may generate a volume graph including function nodes representing one or more functions (such as storage function 224) and leaf nodes representing resources allocated for the functions. The control plane 216 of the DPU 210 may include a graph module 284 configured to communicate information to each DPU based on the volume graph to manage the storage cluster and to manage volumes using the volume graph. As described herein, the functionality of a volume may be changed by replacing a function node with another function node in the volume graph to implement different data protection, data availability, data compression schemes, and / or any other functionality.

[0088] In some examples, the volume map enables a volume to fail over. For example, DPU 210 may operate as a secondary storage node for the volume. In response to a primary storage node failure, map generation module 280 may generate a volume map for DPU 210 to enable it to assume the role of the primary storage node for the volume.

[0089] In some examples, graph module 284 may use the volume graph for resource allocation. For example, graph module 284 may apply one or more rules to the leaf nodes of the volume graph to allocate resources to the volumes. The rules may include resource availability (e.g., IOP availability, storage capacity availability), data protection (e.g., resources in different fault domains), and / or load balancing rules (e.g., based on IOP usage, storage usage, etc.).

[0090] In some examples, the graph module 284 may use the volume graph for event management. Events may include a storage node crash, a storage device being unplugged, a volume being deleted, and any other event associated with an allocated volume represented by the volume graph. For example, the graph module 284 may determine that an event generated by a leaf node of the graph (e.g., a storage device) will propagate upward along the volume stack hierarchy of the volume graph to a functional node representing a data persistence operation of the volume. In response, the graph module 284 may instruct the data processing unit 210 to send a message to the controller 200 to notify the controller 200 of the event. After receiving the message, the graph generation module 280 of the controller 200 may rebuild the volume graph by replacing one or more leaf nodes of the functional node representing the data persistence operation with another storage device.

[0091] In some examples, the graph module 284 can use the volume graph to modify the volume based on the change in the volume parameters. As described herein, a volume can be created using a specified set of one or more parameters (e.g., block size, encryption key, compression scheme, volume size, data protection scheme, etc.). For example, a volume can be created using a specified data protection scheme (e.g., erasure coded replication, no protection, etc.). In some examples, the volume parameters may be changed after the volume is created. In these examples, the graph module 284 can use the volume graph to generate a clone of the volume with the modified parameters and switch the host connection to the clone of the volume.

[0092] In some examples, the graph module 284 can use the volume graph to dynamically rebalance resources for the volume. For example, the graph module 284 can rebalance resources for the volume based on an alert generated by a leaf node of the volume graph. For example, the storage usage of the storage device represented by the leaf node in the volume graph may exceed a storage usage threshold and may generate an alert that propagates up the hierarchy of the volume stack of the volume graph to the function node representing the data persistence operation of the volume, and in response, the graph module 284 can move the load from the storage device to a new storage device.

[0093] Figure 2B 2 is a block diagram of hardware components of an example data processing unit (DPU) 210 illustrating various techniques of the present disclosure. DPU 210 generally represents a hardware chip implemented in digital logic circuits. The operation of DPU 210 may be similar to Figure 1A Any node 17 and Figure 1BThus, DPU 210 may be communicatively coupled to a CPU, a GPU, one or more network devices, a server device, a random access memory, a storage medium (e.g., multiple SSDs), a data center structure, etc. via PCI-e, Ethernet (wired or wireless), or other such communication medium.

[0094] exist Figure 2B In the illustrated example, DPU 210 includes a plurality of programmable processing cores 250A-250N ("cores 250"). The operation of DPU 210 may be related to Figure 2A 2 is substantially similar to the DPU 210 in FIG. 2 , wherein a data plane operating system 212 that runs to completion is executed on each core 250. Each of the cores 250 includes a corresponding level 1 (L1) cache in L1 caches 251A–251N (“cache 251”). Cache 251 may include an L1 buffer cache for caching non-coherent data (e.g., flow data units, such as packets or other data for flow processing). The L1 buffer cache may store short-term cached data for quick access to the data.

[0095] exist Figure 2B In the illustrated example, DPU 210 also includes a network unit 252, a coherent memory manager 254, an incoherent memory manager 255, one or more host units 256, a plurality of accelerators 258A-258N ("accelerators 258"), a queue manager 260, and a plurality of work unit (WU) queues 262. Figure 2B Not shown, each of core 250, network unit 252, coherent memory manager 254, incoherent memory manager 255, host unit 256, accelerator 258, queue manager 260, and work unit queue 262 may be communicatively coupled to one another.

[0096] exist Figure 2B In the example of , DPU 210 represents a high-performance, hyper-converged network, storage, data processing, and input and output hub. For example, network unit 252 can be configured to send and receive flow data units with one or more external devices (such as network devices). Network unit 252 can perform network interface card functions, packet switching, etc. Network unit 252 can use large forwarding tables and provide programmability. Network unit 252 can expose a network interface (such as Ethernet) port for connecting to a network, such as Figure 1AThe network 7 and / or switch fabric 114 in the DPU 210. Each host unit 256 can expose one or more host interface (e.g., PCI-e) ports to send and receive streaming data units with an application processor (e.g., an x86 processor of a server device or a local CPU or GPU of a device hosting the DPU 210) and / or multiple data storage devices (e.g., multiple SSDs). The DPU 210 can also include one or more high-bandwidth interfaces for connecting to off-chip external memory ( Figure 2B (not shown in the figure).

[0097] Each accelerator in accelerator 258 can be configured to accelerate various data processing functions, such as lookup, matrix multiplication, encryption, compression, regular expression processing, etc. For example, accelerator 258 may include hardware implementations of a lookup engine, a matrix multiplier, an encryption engine, a compression engine, a regular expression interpreter, etc. In some examples, accelerator 258 may include a data persistence unit that can be used to implement one or more data persistence and / or data reliability schemes. In some examples, such a data persistence unit can be configured to perform matrix multiplication operations commonly performed in erasure coding schemes (e.g., Reed-Solomon erasure coding schemes). Such a data persistence unit can be configured to efficiently perform operations, such as those related to Galois field mathematics, which may be difficult to perform and / or inefficient using commonly available processors or other processing hardware. In addition, such a data persistence unit can design and / or implement multiple different types of data persistence schemes by configuring specific different matrices for each implementation.

[0098] Queue manager 260 is configured to maintain and manage work unit queues 262. At least one of work unit queues 262 may be associated with at least one of cores 250 and may be configured to store a plurality of work units queued for processing on its associated respective (multiple) cores 250. In some examples, each of cores 250 may have a dedicated work unit queue 262 for storing work units processed by the respective core 250. In other examples, each of cores 250 may have two or more dedicated work unit queues 262 for storing work units of different priorities processed by the respective core 250. Figure 2B In the illustrated example, queue manager 260 and work unit queue 262 are both physically centralized with respect to core 250. However, work unit queue 262 may be logically decentralized.

[0099] In some examples, data processing unit 210 may utilize two types of on-chip memory or storage devices, namely, coherent cache memory and non-coherent buffer memory (not shown in FIG. Figure 2B). The coherent memory manager 254 is configured to control access to the coherent cache memory, and the incoherent memory manager 255 is configured to control access to the incoherent buffer memory. In this way, any component of the data processing unit 210 (e.g., the core 250, the network unit 252, the host unit 256, the accelerator 258, the queue manager 260, and the work unit queue 262) and any number of external devices (e.g., network devices, servers, external storage devices, or similar devices) can access the coherent cache memory via the coherent memory manager 254 and access the incoherent buffer memory via the incoherent memory manager 255. The memory managers 254, 255 can be configured to perform a series of operations to perform memory management according to the present disclosure. For more details on a bifurcated memory system that may be included in the DPU 217, see U.S. Patent No. 10,565,112, filed on April 10, 2018, entitled “Relay Consistent Memory Management in a Multiple Processor System” (Attorney Docket No1242-008US01).

[0100] Core 250 may include one or more of a MIP (microprocessor without internal interlocked pipeline stages) core, an ARM (Advanced RISC (Reduced Instruction Set Computing) Machine) core, a PowerPC (Performance Optimization through Enhanced RISC - High Performance Computing) core, a RISC-V (RISC Five) core, or a Complex Instruction Set Computing (CISC or x86) core. Each of cores 250 may be programmed to process one or more events or activities associated with a given packet flow, such as a network packet flow, a storage packet flow, a security packet flow, or an analysis packet flow. Each of cores 250 may be programmed using a high-level programming language such as C, C++, or a similar language.

[0101] In some examples, the plurality of cores 250 execute instructions for processing a plurality of events associated with each packet in a packet stream, which packets may be received by the network unit 252 or the host unit 256 according to an order associated with one or more work units associated with the packets. As described above, a work unit is a data set exchanged between the core 250 and the network unit 252 or the host unit 256, wherein each work unit may represent one or more of the events associated with a given packet. More specifically, a work unit is associated with one or more packets. A work unit specifies a software function for processing a packet, and further specifies one of the cores 250 for executing the software function.

[0102] Typically, to process a work unit, a core 250 designated by the work unit may be configured to retrieve data packets associated with the work unit from memory and execute software functions designated by the work unit to process the data packets. For example, the core 250 may retrieve data packets from a non-coherent memory buffer via a non-coherent memory manager 255 and may cache the data packets in a cache 251 within the respective core 250.

[0103] In a more detailed example, receiving a work unit may be indicated by receiving a message in a work unit receive queue (e.g., one of the work unit queues 262). Each of the work unit queues 262 is associated with at least one of the cores 250 and is addressable in the header of a work unit message. When a work unit message is received from the network unit 252, one of the host units 256, or another of the cores 250, the queue manager 260 queues the work unit to one of the work unit queues 262 associated with the core 250 specified by the work unit. After the queue manager 260 dequeues the work unit from the work unit queue 262, it passes the work unit to the associated core 250. The queue manager 260 then calls the software function specified by the work unit on the associated core 250 to process the work unit.

[0104] To process a work unit, a core 250 receives a work unit from a work unit queue 262. The core 250 then retrieves data packets associated with the work unit from the cache 251 within the respective core 250 and executes the called software function to process the data packets. The core 250 then outputs the corresponding results obtained from processing the work unit back to the work unit queue 262. For example, when processing a work unit, the core 250 may generate a new work unit message by executing stored instructions corresponding to addresses mapped to a work unit send queue (e.g., another one of the work unit queues 262). The stored instructions write the contents of the message to the queue. The release of the work unit message from the core 250 may be linked to (controlled by) the flushing of dirty data from the associated cache 251.

[0105] As an example use case, stream processing may be divided into work units that are executed on multiple intermediate processors between a source and a destination. Depending on the amount of work to be performed at each stage, the number and type of intermediate processors involved may vary. When processing multiple events associated with each data packet, a first core 250 (e.g., core 250A) may process a first event of the multiple events. In addition, the first core 250A may provide a first work unit of the work units to a second core 250 (e.g., core 250B). Moreover, the second core 250B may process a second event of the multiple events in response to receiving the first work unit from the first core 250A.

[0106] Figure 3A is a conceptual diagram of an example object model of a data plane for a volume according to multiple techniques of the present disclosure. Figure 3A Figure, volume (e.g., Figure 1B The volume 121J in the data center provides an abstraction layer from multiple solid-state drives (SSDs) 350. The SSD 350 can be a logically shared pool of storage devices created by interconnected nodes in a data center storage cluster and can be accessed by all nodes coupled to the data center switch fabric. The abstraction layer of the allocated volume includes: multiple segment (extent) layers 352, each segment layer maps to a portion of an SSD in the SSD 350; multiple raw volume layers 354, each raw volume layer maps to multiple segments 352; multiple raw volume set layers 356, each raw volume set layer maps to multiple raw volumes 354; persistent volume layer 358, mapped to multiple raw volume sets 356; log structure logical volume layer 360, mapped to persistent volume 358; and user volume 362, mapped to log structure logical volume 360. The distribution of data may occur on two boundaries: 1) user volume 362 to log structure logical volume 360, and 2) raw volume set 356 to raw volume 354. In some examples, the allocated volume can configure the user volume 362 to split / join the user volume to multiple logical volumes, thereby providing a 1:N mapping of the user volume to the logical volume. For example, this can facilitate expansion and recovery of the user volume across multiple data processing units and storage device failures.

[0107] Figure 3BMore information about each layer of abstraction from application to device is set forth and described in U.S. Patent No. 10,949,303, entitled “DURABLE BLOCK STORAGE IN DATA CENTER ACCESSNODES WITH INLINE ERASURE CODING,” filed on December 10, 2018, and claims the benefit of U.S. Provisional Patent Application No. 62 / 597,185, filed on December 11, 2017. Although not shown in the example of Table 1, as noted above, in some implementations, a user volume may be mapped to 1:N logical volumes.

[0108] exist Figure 3A and Figure 3B In the illustrated example, a node may implement three types of roles in a tiered approach. Based on the deployment topology (e.g., hyper-converged vs. decoupled) and the location of the data, a given node may play one or all roles for a given volume. One role that a node may implement is an additional node 364. The additional node 364 is a node that is attached to a volume as an application running in a virtual machine or container on a server. For example, the additional node 364 may be a node to which a PCIe link is attached to a server, or a node where a NVMeoF (non-volatile memory expression over architecture) connection to a volume is terminated. The user volume 362 functionality runs on an additional node.

[0109] Another role that a node may fulfill is a primary / backup controller node 366. The volume includes a primary controller node and, in some examples, for persistent volumes (e.g., erasure coded volumes, replicated volumes, etc.), at least one backup controller node. The primary and backup schemes can be used to provide data consistency and availability in the event of node failures. The primary controller node creates a transaction log that is replicated to one or more backup controller nodes. If the primary controller node fails, the control plane (e.g., from Figure 2A The control plane 216) can elect a new primary controller node from one or more backup controller nodes and roll transactions forward before allowing blocks with pending transactions to be requested by users. Figure 3A In the illustrated example, the functionality of the log-structured logical volume 360 ​​, the persistent volume 358 , and the raw volume set 356 runs on a controller node 366 .

[0110] Another role that a node may fulfill is a storage node 368. The storage node 368 is a node to which a storage device is attached. The volume may include multiple storage nodes 368. For example, multiple SSDs 350 may be divided into segments 352 (e.g., 1GB) and accessed via storage nodes 368 attached to the SSDs 350 (via PCIe Extents). The segments 352 are configured into raw volumes 354, which can be remotely accessed by other nodes in the cluster. Figure 3A In the illustrated example, the functionality of the original volume 354 and the segment 352 runs on the storage node 368 .

[0111] Volumes can be configured via the management plane and the control plane as follows: Each configuration step includes communication between the management plane and one or more nodes, indicating the role(s) that the (multiple) nodes play with respect to the volume being created. Each volume can have a globally unique identifier for communication so that each node can identify the correct volume. The management plane can use a variety of methods to determine which nodes to select to play the different roles for a given volume. Typically, the management plane may select nodes within the cluster that are not in the same failure zone so that multiple nodes supporting the volume are less likely to fail at the same time. An example method for configuring a volume is described below.

[0112] As a first step, the management plane receives a top-level specification from a management console (e.g., OpenStack Cinder) that defines volume parameters, including block size, volume size (number of blocks) (also known as "capacity"), quality of service (QoS), encryption, compression, failure domains, and durability schemes (e.g., replication factor or erasure coding scheme).

[0113] As a second step, the management plane creates a raw volume 354 on each storage node. The raw volume 354 can be created by allocating segments 352 from the available multiple SSDs 350. The segments 352 can be statically sized (e.g., 1GB) during deployment. This step can be done statically or dynamically (e.g., thin provisioning), depending on how the storage node accesses storage space.

[0114] As a third step, the management plane creates raw volume sets 356 on each controller node. The number of raw volumes in each raw volume set 356 may depend on the durability scheme specified in the volume top-level specification (e.g., X for replication factor and m+n for erasure coding). The number of raw volume sets 356 may depend on the size of the raw volume 354 and the volume size specified in the top-level specification.

[0115] As a fourth step, the management plane creates a persistent volume 356 on each controller node. The parameters of the persistent volume 356 may include a persistence scheme (e.g., replication or erasure coding) and / or a volume size (including additional space for log compression).

[0116] As a fifth step, the management plane creates a log-structured logical volume 360 ​​on each controller node. Parameters of the log-structured logical volume 360 ​​may include block size, volume size, and compression-related parameters. The log-structured logical volume 360 ​​may be a volume type that is compatible with logical volume management (LVM) in the data plane storage stack. The log-structured logical volume 360 ​​provides fault tolerance and flash optimization for the volume, and may also provide flash optimization for raw block devices that are not fault-tolerant. The log-structured logical volume 360 ​​may be used to aggregate multiple data blocks into larger data blocks for use by the persistent volume 356 before storing the data on multiple storage nodes. Note that the primary controller node and the backup controller node may be managed by a control plane (e.g., separate from the management plane) that is separate from the management plane. Figure 2A It is managed by the control plane 216 in.

[0117] As a sixth step, the management plane creates a user volume 362 on each additional node. The user volume 362 can receive read and write requests for data blocks from applications running on the additional servers. The read and write requests can be passed to the log structure volume 360 ​​for processing. Parameters for the user volume 362 can include block size, encryption key, compression scheme, and volume size.

[0118] Once created, the volume may rely on a distribution protocol to exchange data between the associated nodes. For example, NVMeoF may be used as the underlying protocol. The network bond may be based on the Transmission Control Protocol (TCP) or various forms of reliable datagrams. In some examples, the network bond may be TCP with congestion control based on the Fibre Channel Control Protocol (FCP).

[0119] Various objects may have universally unique identifiers (UUIDs) that enable them to be addressed in the network via a distribution protocol. For example, the log-structured logical volume 360 ​​may be accessed directly using a UUID obtained from an attached node or via an NVMeoF client. In another example, the original volume 354 may receive an I / O request from the original volume set 356 for storing a replica or erasure coded fragment in the original volume 354 identified by the UUID. In some examples, authentication may be included as part of NVMeoF so that "bad actors" (unauthorized parties) on the network cannot access these remotely addressable entities.

[0120] In some examples, the volume design described in the present disclosure can support a scale-down model all the way down to a single node, in which case the original volume 354 can be allocated from the local node, resulting in a device similar to a RAID (Redundant Array of Independent Disks) or embedded erasure coding implementation that is still able to tolerate SSD failures.

[0121] Figure 4 Illustrated is an example of a graph-based representation of functional elements in a data plane of a volume, according to various techniques described in this disclosure. Figure 4 The volume diagram 400 in FIG. 4 is described in conjunction with an example implementation of a volume, such as the one described above in conjunction with Figure 1B Described volume 121J. Volume map 400 can be used for volume management, such as resource allocation, event management, and large-scale recovery. Volume map 400 can span multiple nodes, enabling horizontal expansion of capacity, redundancy, and performance.

[0122] exist Figure 4 In the example of , volume map 400 provides a graphical representation of the abstraction layers of a volume. For example, volume map 400 may include an allocation layer 402, a persistence layer 404, a mode layer 406, and an aggregation layer 408 of a volume. The allocation layer 402 of volume map 400 may graphically represent the resources allocated to one or more persistence volumes, such as erasure coded volumes 416 and log volumes 418. Log volume 418 provides crash recovery capabilities for the volume. For example, a log-structured logical volume 422 may use an intent log (i.e., a log) stored in NVM 420 of a DPU so that any writes acknowledged to an application's host server can be reliably executed to the underlying storage media in the presence of a component failure. The log may be replicated to another DPU to provide resilience to node failures. Volume map 400 may graphically represent the nodes allocated to log volume 418. For example, volume map 400 may include a leaf node for log volume 418 that represents a node operating as a primary storage node (e.g., dpu_0) that includes a first copy of a log stored in NVM 420A. Volume map 400 may also include another leaf node for log volume 418 that represents a node operating as a secondary storage node (e.g., dpu_1) that includes a second copy of a log stored in NVM 420B. In this example, the primary storage node (e.g., dpu_0) may represent Figure 1B DPU 117A in the storage node 120A stores a log copy of volume 121J in the NVM of the storage node 120A, and the secondary storage node can represent Figure 1B DPU 117B in , which stores a log copy of volume 121J in the NVM of storage node 120B.

[0123] Erasure coded volumes 416 provide data protection techniques in which data is broken down into fragments, expanded, and encoded using a configurable number of redundant data slices that can be stored in a set of storage devices. Erasure codes provide reliability by enabling data that is damaged in a storage location to be reconstructed using data information stored in another storage location. In this example, volume map 400 can graphically represent nodes that operate as storage nodes, which provide access to the original volumes created by erasure coded volumes 416 on each storage node. For example, volume map 400 can include one or more leaf nodes of erasure coded volumes 416, which represent nodes that operate as storage nodes (e.g., dpu_2 to dpu_m+n+1), which provide access to the original volumes created by allocating segments from storage devices 410A-410N (collectively referred to as "storage devices 410"). Storage device 410 can represent storage device 127 within storage nodes 120A, 120B, and 120D. In volume map 400, storage nodes that provide access to raw volumes are illustrated as multiple local thin blocks (BLTs) 412. The number of raw volumes in each raw volume set may depend on the persistence scheme (e.g., "m+n" for erasure coding or "X" for replication), which can be specified in the top-level specification of the volume. In some examples, volume map 400 may include one or more nodes for erasure coded volumes 416, which represent remote datagram sockets (RDS), such as RDS 414, each RDS providing remote access to raw storage volumes. Although erasure coded volume 416 is illustrated as including RDS 414, in some examples, erasure coded volume 416 may include RDS 414 and BLT 412, only BLT 412, or a combination of the two. In some examples, log volume 418 may include one or more leaf nodes, which represent RDSs that provide remote access to log copies stored in NVM 420.

[0124] The persistence layer 404 of the volume map 400 may graphically represent one or more persistent volumes of a volume, for example, Figure 4In the example of , erasure coded volume 416 and log volume 418. Although volume map 400 is illustrated as including erasure coded volume 416 and log volume 418, volume map 400 may also additionally or alternatively include other persistent volumes, such as replicated volumes, in which data blocks are replicated multiple times based on a replication factor and distributed among storage devices to provide high data availability to protect devices or nodes from failures. In this example, volume map 400 includes a functional node representing erasure coded volume 416, which is connected to a leaf node representing an original volume created by storage device 410 (or to an RDS that provides remote access to the original volume). Volume map 400 also includes a functional node representing log volume 418, which is connected to a node representing non-volatile memory (NVM) 420 including a log copy (or to an RDS that provides remote access to the log copy).

[0125] The architectural layer 406 of the volume map 400 can graphically represent a log-structured logical volume 422 created on each node operating as a controller node. As described above, the log-structured logical volume 422 can be used to aggregate multiple data blocks into larger blocks for inline erasure coding by the erasure coded volume 416 before storage across multiple storage nodes. The volume map 400 includes a functional node representing the log-structured logical volume 422, which is connected to nodes representing persistent volumes (e.g., the erasure coded volume 416 and the log volume 418). Although the volume map 400 is illustrated as including the log-structured logical volume 422, the volume map 400 can also include other volumes, such as direct-mapped volumes or the like.

[0126] Aggregation layer 408 of volume map 400 can graphically represent partitioned volume group 424. Partitioned volume group 424 can combine multiple log-structured logical volumes (not shown) to create a storage volume, or split a log-structured logical volume into multiple storage volumes. Volume map 400 includes a functional node representing partitioned volume group 424, which is connected to a node representing log-structured logical volume 422.

[0127] Volume map 400 includes nodes representing user volumes in the map, for example, as additional nodes (e.g., Figure 1B 4. A user volume 426 is created on a node running on the initiating node 110 in the volume map 400. For ease of explanation, the volume map 400 is described for a single user volume 426. In some examples, the volume map 400 may include multiple user volumes. As described above, the user volume 426 receives read and write requests for data blocks from applications running on additional servers, and passes these read and write requests to the log structure volume 422 for processing. In some examples, multiple user volumes may use the volume represented by the volume map 400. In this example, the volume map 400 may include multiple root nodes, which represent multiple user volumes that use the volume.

[0128] The control plane may configure volume maps using APIs that map parameters for creating volumes to volume maps, such as volume map 400. These APIs are referred to herein as "volume creation intent APIs." For example, a volume creation intent API may include a volume type parameter that specifies the type of volume to be created. The volume type parameter may specify a replicated volume (e.g., VOL_TYPE_BLK_REPLICA), an erasure coded volume (VOL_TYPE_BLK_EC), a block local thin volume (VOL_TYPE_BLK_LOCAL_THIN), a raw volume (VOL_TYPE_BLK_RF1), and the like. In some examples, the volume creation intent API may include a data protection parameter that specifies parameters of a data protection technology to be implemented for the volume, such as the number of storage nodes that can be tolerated after going offline (Num_redundant_dpus), the number of data disks for erasure coded volumes (Num_data_disks), the number of media failures that can be tolerated (Num_failed_disks), and the like. In some examples, the volume creation intent API may include a compression parameter that specifies whether compression is enabled for the volume. In some examples, the volume creation intent API may include an encryption parameter that specifies whether encryption is enabled for the volume. In some examples, the volume creation intent API may include a capacity parameter that specifies the size of the volume. In some examples, the volume creation intent API may include a QoS parameter that specifies the QoS requirements for the volume. In some examples, the volume creation intent API may include a failure domain parameter that specifies the failure domain in which the volume may or may not be created. In some examples, the volume creation intent API may include a cyclic redundancy check (CRC) parameter that specifies whether CRC is enabled for the volume.

[0129] As described below, the control plane can use volume map 400 for resource allocation, event management, large-scale recovery, and various other functions. In some examples, the control plane can replace a functional node in volume map 400 with another functional node to implement a different data protection, data availability, and / or data compression scheme. For example, erasure coded volume 416 in volume map 400 can be replaced by a functional node representing a replicated volume in order to apply a different type of data protection scheme for the volume. In this example, the control plane can reconfigure volume map 400 to include a functional node representing a replicated volume and connect the replicated volume to a leaf node representing the resources allocated to the replicated volume without affecting other nodes within volume map 400. In some examples, the composability of volume map 400 enables failover for volumes. As described below Figure 7A-7BAs further described, upon determining that the primary storage node (e.g., dpu_0) has failed, the control plane can reconfigure the volume map 400 to replace the connection to the failed primary storage node with a connection to a secondary storage node (e.g., dpu_1) to allow the secondary storage node to assume the role of the primary storage node for the volume.

[0130] In some examples, the control plane can use volume map 400 for resource allocation. For example, the control plane can apply one or more rules to the leaf nodes of volume map 400 for allocating resources to volumes. The rules can include resource availability, data protection, and / or load balancing rules.

[0131] In some examples, the control plane can apply resource availability rules to allocate resources based on one or more rules. These rules can be based on one or more metrics, such as the availability of input / output operations per second (IOPs) (e.g., the speed at which the system can read and write commands in one second). For example, to allocate storage devices and / or storage nodes to erasure coded volumes 416, the control plane can determine whether a given storage device or a given storage node has sufficient IOPs for use by erasure coded volumes 416, such as by comparing the IOPs of a given storage device or a given storage node to an IOP availability threshold. The control plane can select one or more storage devices or nodes that are determined to have sufficient available IOPs (e.g., meet the IOP availability threshold) to allocate to erasure coded volumes 416. The volume map 400 can be configured so that the leaf nodes (multiple) of the erasure coded volumes 416 represent the allocated storage devices (multiple) and / or storage nodes (multiple) that are determined to have sufficient available IOPs.

[0132] In some examples, the control plane can apply resource availability rules to allocate resources based on one or more rules (such as storage capacity availability). For example, to allocate a storage device for an erasure coded volume 416, the control plane can determine whether a given storage device has sufficient storage capacity for use by the erasure coded volume 416, such as by comparing the storage capacity availability of the given storage device to a storage capacity availability threshold. The control plane can select one or more storage devices determined to have sufficient storage capacity available (e.g., satisfying the storage capacity availability threshold) to allocate for the erasure coded volume 416. The volume map 400 can be configured such that the leaf node(s) of the erasure coded volume 416 represent the allocated storage device(s) determined to have storage capacity available.

[0133] In some examples, the control plane can apply data protection rules to assign storage nodes that are located in different fault domains within the cluster so that multiple storage nodes used to support the volume are less likely to fail simultaneously. For example, to assign storage nodes for log volume 418, the control plane can determine the fault domain (e.g., power domain or chassis) to which the storage nodes belong. In response to determining that certain storage nodes belong to different fault domains, the control plane can select one storage node as a primary storage node and select another storage node located in a different fault domain as a secondary storage node. Figure 4 In the illustrated example, it can be determined that dpu_0 and dpu_1 belong to different fault domains. In this case, the control plane can select dpu_0 as the primary storage node for log volume 418 and select dpu_1 as the secondary storage node for log volume 418. Volume map 400 can be configured so that the leaf nodes of log volume 418 represent the selected primary storage node and secondary storage node. In addition, in order to assign the node on which the storage node for erasure coded volume 416 runs, the control plane can determine the fault domain to which the storage node belongs.

[0134] In some examples, the control plane may apply load balancing rules to allocate resources based on one or more rules. These rules may be based on one or more indicators, such as IOP usage and / or storage usage. For example, in order to allocate storage devices and / or storage nodes for erasure code volume 416, the control plane may determine the IOP usage of a given storage device and / or a given storage node. The control plane may add a cost value based on the IOP usage of a given storage device and / or a given storage node. Then, the control plane may select a storage device and / or storage node with the lowest IOP usage cost for erasure code volume 416. In some examples, a storage device and / or storage node with an IOP usage cost below a certain threshold may be selected. Volume map 400 may be configured so that the (multiple) leaf nodes of erasure code volume 416 represent the selected (multiple) storage devices and / or (multiple) storage nodes. In some examples, the control plane may determine the storage usage of a given storage device and add a cost value based on the storage usage of a given storage device. Then, the control plane may select a storage device with the lowest storage usage cost for erasure code volume 416. In some examples, storage devices and / or storage nodes whose storage usage costs are below a certain threshold are selected. Volume map 400 can be configured such that leaf node(s) of erasure coded volume 416 represent the selected storage device(s).

[0135] In some examples, the control plane can use volume map 400 for event management and rebuild volumes based on events. Events may include storage node crashes, storage devices being unplugged, volumes being deleted, and any other events associated with allocated volumes represented by volume map 400. For example, a network device including storage device 410A may detect a failure of storage device 410A and generate an event indicating that storage device 410A has failed. The event may propagate up the volume stack layers of volume map 400, such as to BLT 412, which then propagates the event to erasure coded volume 416. In response to receiving the event at erasure coded volume 416, the control plane may instruct the DPU to send a request to a controller (e.g., Figure 2A The controller 200 in the storage device 410A sends a message to notify the controller 200 of the event. In response to the received message, the graph generation module 280 can rebuild the volume graph by replacing one or more leaf nodes of the erasure coded volume 416 representing the failed storage device 410A with another storage device.

[0136] As another example, a network device including a storage node (e.g., dpu_0) may detect a failure of the storage node and generate an event indicating that the storage node has failed. The event may propagate up the volume stack layers of volume map 400 to log volume 418, which in turn propagates the event to log structured logical volume 422. In response to receiving the event at log structured logical volume 422, the control plane may instruct the DPU to send a request to a controller (e.g., Figure 2A The controller 200 in the example embodiment sends a message to notify the controller 200 of the event. In response to receiving the message, the graph generation module 280 may rebuild the volume graph by replacing one or more leaf nodes of the log structured logical volume 422 representing the failed storage node with another storage node, for example, by connecting the log volume 418 to a leaf node representing another storage node that has a log copy.

[0137] In some examples, the control plane can use volume map 400 to modify a volume based on changes in volume parameters. As described above, a volume can be assigned a specified set of one or more parameters (e.g., block size, encryption key, compression scheme, volume size, data protection scheme, etc.). For example, a volume can be assigned a specified data protection scheme, such as erasure coded replication, no protection, etc. In some examples, volume parameters may be changed after the volume is created. In this example, the new parameters are verified and a clone of the volume with the modified parameters can be created. The clone of a volume is an independent volume, but until the clone of the volume is fully hydrated, it relies on the source volume for reads. As an example, the block size of an erasure coded volume 416 may be increased. In this example, the control plane can create a clone of the volume with an increased block size for the erasure coded volume, and can switch the host connection to the clone of the volume. In some examples, when the clone of the volume is fully saturated, the original volume is deleted to free up space. More examples of snapshots and clones of volumes are described below. Figure 5 and Figure 6 Described in.

[0138] In some examples, the control plane can use volume map 400 to dynamically rebalance resources for volumes. For example, storage devices and / or storage nodes allocated for persistent volumes can generate alerts based on metrics associated with resources allocated for volumes reaching certain thresholds. Metrics may include storage usage, IOP usage, health of storage devices, bandwidth of storage nodes, etc. The control plane can rebalance the load of resources based on the alert. For example, the bandwidth of a storage node running as a primary storage node (such as dpu_0) may exceed a bandwidth threshold. In response, the storage node can generate an alert that propagates up the volume stack layer of volume map 400 to log volume 418, and in response, the control plane can move the load from the storage node running as the primary storage node to a different storage node and rebuild the leaf nodes of log volume 418 with different storage nodes. In another example, the storage usage of a storage device may exceed a storage usage threshold and generate an alert that propagates up the volume stack layers of volume map 400 to erasure coded volume 416, in response to which the control plane may move load from the storage device to a different storage device and use the different storage device to rebuild the leaf nodes of erasure coded volume 416. In some examples, the system may use a machine learning algorithm to predict resource usage over time and determine whether to rebalance resources based not only on current usage but also on predicted future usage of the resources.

[0139] In some examples, the control plane can use volume map 400 to rebalance resources when storage devices and / or storage nodes are added or removed from the system. For example, the control plane can determine storage usage in response to each storage device after adding the storage device. For example, the control plane can determine whether there are storage devices with low storage usage (e.g., less than 20% storage usage), medium (e.g., more than 50% storage usage), or high storage usage (e.g., more than 80% storage usage). Data in the storage device with the highest storage usage can be migrated.

[0140] In some examples, the control plane may determine which storage device to migrate data from by calculating a ratio (r1) of physical usage to allocated space (e.g., allocated segments) in the storage device, calculating a ratio (r2) of allocated space (e.g., allocated segments) to the physical size of the storage device, multiplying r1 and r2, and selecting the segment with the highest value for migration. For example, the physical capacity of the storage device may be 900GB. The storage device may allow 200% of the capacity, and a total of 14 segments are located on the storage device. For example, 4 400GB segments and 10 10GB segments are allocated on the storage device. Assume, for example, that there are two 400GB segments undergoing input and output operations, and that the two segments use a total of 600GB. The first segment may occupy 100% of its allocated storage space, while the second segment may occupy 50% of its allocated storage space. Assume that there are four 10GB extents undergoing I / O operations, three of which may be occupying 100% of their allocated storage space, while one may be occupying 50% of its allocated storage space. The control plane can arrange the usage of these extents into a list (e.g., [400,400,400,400,10,10,10,10,10,10,10,10,10,10]) and calculate the physical usage ratio (r1) of each extent (e.g., [1.0,0.5,0,0,1.0,1.0,1.0,0.5,0,0,0,0,0,0]). The control plane can then multiply r1 by the ratio of allocated space to the physical size of the storage device (r2) (e.g., [0.44, 0.22, 0, 0, 0.01, 0.01, 0.005, 0, 0, 0, 0, 0]), where, for a 400GB segment, r2 is 400 / 900 = 0.44. In this example, the segment with the highest value (e.g., 0.44) may be selected for migration.

[0141] In some examples, the control plane can use volume map 400 to rebalance storage nodes when new storage nodes are added (or re-enabled). For example, the control plane can determine the bandwidth usage (or other QoS requirements) of the storage nodes after adding the new storage nodes, and if the bandwidth usage of any storage node exceeds a bandwidth usage threshold, the control plane can move the log-structured logical volumes mapped to the storage nodes that exceed the bandwidth usage threshold to the new storage node. In some examples, volumes that are not mounted due to unavailability of volatile memory (e.g., double data rate (DDR) memory) can be mounted on the newly added storage node.

[0142] In some examples, the control plane can use volume map 400 to rebalance volumes when a failed storage node comes back online. For example, when a storage node running as a primary storage node (e.g., dpu_0) fails or degrades (or is otherwise unavailable), the persistent volumes on the storage node can be moved to a secondary storage node (e.g., dpu_1). When the original primary storage node (dpu_0) comes back online, the control plane can use volume map 400 to rebalance the persistent volumes back to the original primary storage node, such as by resynchronizing NVM 420A with the log copy, configuring the old primary storage node (e.g., dpu_0) as a new secondary storage node, resynchronizing the leaf nodes of the new standby controller node, changing the log structured logical volume 422 to an online state, unmapping the log structured logical volume 422 from the current primary storage node (e.g., dpu_1), and mapping the log structured logical volume 422 to the new secondary controller node. Partitioned volume groups 424, user groups 426, and / or snapshots can then be mapped to the log structured logical volume 422.

[0143] In some examples, the control plane can use volume map 400 to monitor volumes that are in the process of being deleted, and restart the storage service or operating system (e.g., as a microservice) executed on the storage node. If the storage service is restarted, the control plane can determine whether there are any marked (multiple) volumes in the database, including the status of the volume (such as being in the process of being deleted), and whether the volume has been deleted in the operating system. If the volume has been deleted in the operating system, the control plane can remove the volume from the database. If the volume has not been deleted in the operating system, the control plane can resume the deletion process (e.g., restore the deletion in progress flag in the database).

[0144] Figure 5 An example graph-based representation of a snapshot of a volume according to various techniques described in this disclosure is illustrated. Graph 500 may represent Figure 4 An example implementation of a snapshot of a user volume 426 in FIG. Figure 5In the example of FIG. 5 , a user volume 426 can be associated with one or more snapshots. A snapshot of a user volume 426 (e.g., snapshot 502) can include the contents of the user volume 426 at a particular point in time. There can be multiple snapshots of a user volume 426. In some examples, in addition to the user volume block 506, the snapshot 502 of the user volume 426 is also included as a snapshot block (e.g., snapshot block 508) in the log-structured logical volume 422. As described below, a clone of the user volume can be created from the snapshot 502.

[0145] Figure 6 is a block diagram illustrating an example interaction of a volume with a clone of a user volume when performing read and write operations according to the various techniques described in this disclosure. Figure 6 In the example of , snapshot 502 may include the contents of user volume 426 at a particular point in time. Clone 602 may represent a clone of user volume 426 created from snapshot 502. Clone 602 of user volume 426 is an independent volume, but it depends on the source volume (e.g., user volume 426) for reads until clone 602 is sufficiently saturated. In this example, clone filter 604 is used for conditional reads.

[0146] As described above, clone 602 can be used to switch a host connection (e.g., user volume 426) to a volume that has been modified with different volume parameters. In these examples, the new parameters are verified and a clone of user volume 426 is created using the modified parameters (e.g., clone 602). In this example, when clone 602 is sufficiently saturated, the host connection is switched to clone 602. In some examples, when the clone of the volume is sufficiently saturated, the original volume is deleted to free up space. In some examples, a snapshot of clone 602 (e.g., snapshot clone 606) can include the contents of clone 602 at a particular point in time.

[0147] Figure 7A-7B A block diagram illustrating an example of a failover operation using a volume map according to multiple techniques described in this disclosure. To simplify the description, Figure 7A-7B Will combine Figure 4 The volume diagram 400 in FIG.

[0148] Fig. 7A An example is illustrated in which a host 702 accesses a volume 706 via a connection to a storage node 704 operating as a primary storage node (e.g., dpu_0) and to one or more storage nodes 704 operating as secondary storage nodes (e.g., dpu_1 through dpu_n). In some examples, the primary storage node and the secondary storage node may be selected from nodes having a copy of the log (e.g., Figure 4The control plane may select dpu_0 as the primary storage node to access volume 706 based on a determination that dpu_0 has the most available bandwidth and / or accesses the fewest volumes.

[0149] Until the primary storage node fails, access to volume 706 via one or more secondary storage nodes may be inactive. In response to the failure of the primary storage node, the control plane may rebuild the volume map to enable failure transfer to the secondary storage node, such as Figure 7B As further described and illustrated in Figure 7B In the example of , the control plane rebuilds volume map 700 by connecting host 702 to a secondary storage node (eg, dpu_1 ), which can provide host 702 with access to volume 706 .

[0150] Figure 8 is a flow chart of an example method of storage management based on a graph according to multiple techniques described in the present disclosure. At step 802, method 800 includes allocating storage volumes within a storage cluster. In some examples, a volume represents a logical storage device and provides an abstraction layer with physical storage. A volume can be allocated by a storage cluster having multiple storage nodes. Resources for a volume can be allocated based on various rules. In some examples, one or more resources can be allocated for a volume based on one or more metrics associated with one or more resources. Example metrics include input output operations per second (IOP) capacity, storage capacity, fault / failure domains, IOP usage, or storage usage.

[0151] At step 804, method 800 includes generating a volume map for the volume. The volume map can be generated using various methods, including using a map generation module. The volume map represents one or more functional elements in the data plane of the volume. The volume map can be configured in various ways. For example, the volume map can include a tree map. In some examples, the volume map includes at least one root node representing at least one tenant of the storage cluster. The volume map may include one or more functional nodes or intermediate nodes, each node specifying a function implemented by one or more resources allocated for the volume. The volume map may include one or more leaf nodes, each node specifying a resource allocated for a given function.

[0152] At step 806, method 800 includes managing volumes based on the volume map. Managing volumes may include performing various tasks, such as resource allocation, event management, and large-scale recovery (e.g., failover) for the volumes. Managing volumes may include modifying function nodes with different function nodes. For example, function nodes may be replaced by function nodes that specify different functions. In some examples, managing volumes includes modifying leaf nodes with different leaf nodes. For example, a leaf node of a first resource that specifies a function may be replaced by a leaf node of a different resource that specifies the function. In some examples, managing volumes includes dynamically rebalancing one or more resources allocated to the volume based on one or more modified metrics associated with one or more resources allocated to the volume. In some examples, managing volumes includes receiving an event associated with one or more leaf nodes. Based on the event, various actions may be performed. For example, the event may allocate different resources to a given function by replacing the leaf node with a different leaf node that represents different resources. In some examples, managing volumes includes determining that one or more parameters of the volume have changed. For example, one or more resources may be allocated to the volume based on one or more parameters of the volume. Examples of such parameters include block size, encryption key, compression scheme, volume size, data protection scheme, and the like. After determining that one or more parameters of the volume have changed, a clone of the volume may be generated based on the changed parameter(s).

[0153] The following paragraphs provide additional support for the claims of the subject application. In one aspect, a method for graph-based storage management is provided, the method comprising: allocating a storage volume within a storage cluster having a plurality of storage nodes; generating a volume map of the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and managing the volume based on the volume map. In this aspect, additionally or alternatively, the volume map of the volume comprises a tree map, the tree map comprising: a function node specifying a function implemented by one or more resources allocated to the volume; and one or more leaf nodes of the function node, wherein each leaf node specifies a resource in the one or more resources allocated for the function. In this aspect, additionally or alternatively, the tree map also comprises a root node representing at least one tenant of the storage cluster. In this aspect, additionally or alternatively, managing the volume based on the volume map comprises: modifying the function node with a second function node specifying a second function. In this aspect, additionally or alternatively, the one or more leaf nodes comprise a first leaf node of a first resource specifying a function, and wherein managing the volume based on the volume map comprises: modifying the first leaf node of the first resource specifying the function with a second leaf node of a second resource specifying the function. In this aspect, additionally or alternatively, one or more resources are allocated to the volume based on applying one or more rules to one or more leaf nodes, wherein the one or more rules include one or more of resource availability rules, data protection rules, or load balancing rules. In this aspect, additionally or alternatively, managing the volume based on the volume map includes dynamically rebalancing the one or more resources allocated to the volume based on one or more modified metrics associated with the one or more resources allocated to the volume. In this aspect, additionally or alternatively, managing the volume based on the volume map includes: receiving, by a function node, an event associated with one or more leaf nodes of the function node; and allocating different resources to the function by replacing one or more leaf nodes of the function node with leaf nodes representing different resources for the function. In this aspect, additionally or alternatively, one or more resources are allocated to the volume based on one or more parameters of the volume, and wherein managing the volume based on the volume map includes: determining that one or more parameters of the volume have changed; and generating a clone of the volume based on the changed parameters. In this aspect, additionally or alternatively, the one or more parameters include one or more of a block size, an encryption key, a compression scheme, a volume size, or a data protection scheme.

[0154] On the other hand, a computing system for graph-based storage management is provided, the computing system comprising: a network interface for interconnecting the computing system with at least one other computing system to form a plurality of computing systems on a network; and at least one storage node, wherein at least one storage node is part of a storage cluster formed by a plurality of computing systems, and the computing system is configured to: allocate a storage volume within the storage cluster; generate a volume map of the volume, wherein the volume map represents one or more functional elements in the data plane of the volume; and manage the volume based on the volume map. In this aspect, additionally or alternatively, the volume map of the volume includes a tree map, the tree map including: a function node specifying a function implemented by one or more resources allocated to the volume; and one or more leaf nodes of the function node, wherein the one or more leaf nodes each specify a resource allocated for the function in one or more resources. In this aspect, additionally or alternatively, managing the volume based on the volume map includes modifying the function node with a second function node specifying a second function. In this aspect, additionally or alternatively, the one or more leaf nodes include a first leaf node specified for a first resource for the function, and in order to manage the volume based on the volume map, the computing system is further configured to modify the first leaf node specified for the first resource for the function with a second leaf node specified for the second resource for the function. In this regard, additionally or alternatively, one or more resources are allocated to the volume based on applying one or more rules to one or more leaf nodes, wherein the one or more rules include one or more of: resource availability rules, data protection rules, or load balancing rules.

[0155] On the other hand, a computer-readable storage medium for graph-based storage management is provided, the computer-readable storage medium including instructions that, when executed, cause one or more processors to: allocate a storage volume within a storage cluster; generate a volume map of the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and manage the volume based on the volume map. In this aspect, additionally or alternatively, the volume map of the volume includes a tree map, the tree map including: a function node specifying a function implemented by one or more resources allocated to the volume; and one or more leaf nodes of the function node, wherein each leaf node specifies a resource of the one or more resources allocated for the function. In this aspect, additionally or alternatively, managing the volume based on the volume map includes modifying the function node with a second function node specifying a second function. In this aspect, additionally or alternatively, the one or more leaf nodes include a first leaf node specified for a first resource for the function, and wherein to manage the volume based on the volume map, the computing system is further configured to modify the first leaf node specified for the first resource for the function with a second leaf node specified for the second resource for the function. In this regard, additionally or alternatively, one or more resources are allocated to the volume based on applying one or more rules to one or more leaf nodes, wherein the one or more rules include one or more of: resource availability rules, data protection rules, or load balancing rules.

[0156] For the processes, devices, and other examples or illustrations described herein, including in any flowchart or flow chart, certain operations, behaviors, steps, or events included in any described technology may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described behaviors or events are necessary to practice these technologies). In addition, in some examples, operations, behaviors, steps, or events may be performed concurrently, for example, through multithreading, interrupt processing, or multiple processors, rather than sequentially. Further, certain operations, behaviors, steps, or events may be automatically executed even if they are not specifically identified as being automatically executed. Similarly, certain operations, behaviors, steps, or events described as being automatically executed may not be automatically executed, but in some examples, are executed in response to input or other events.

[0157] For ease of explanation, only a limited number of devices (e.g., initiator node 110, storage node 120, controller 130, etc.) are shown in the accompanying drawings and / or other illustrations cited herein. However, multiple techniques according to one or more aspects of the present disclosure can be performed with more such systems, components, devices, modules, and / or other items. Collective references to these systems, components, devices, modules, and / or other items can represent any number of such systems, components, devices, modules, and / or other items.

[0158] The drawings included herein each illustrate at least one example implementation of aspects of the present disclosure. However, the scope of the present disclosure is not limited to such implementations. Accordingly, other examples or alternative implementations of the systems, methods, or techniques described herein may be appropriate in other circumstances, in addition to those illustrated in the drawings. Such implementations may include a subset of the devices and / or components included in the drawings, and may include accessory devices and / or components not shown in the drawings.

[0159] The detailed description described above is intended to describe various configurations and is not intended to represent the only configuration in which the concepts described herein can be practiced. Specific details are included in the detailed description for the purpose of providing sufficient information for a full understanding of the various concepts. However, these concepts can also be practiced without these specific details. In some cases, in order to avoid obscuring these concepts, known structures and components are shown in the form of block diagrams in the referenced drawings.

[0160] Accordingly, although one or more implementations of various systems, devices, and / or components may be described with reference to specific figures, these systems, devices, and / or components may be implemented in a variety of different ways. For example, a device illustrated herein as a separate device may alternatively be implemented as a single device; a component illustrated as a separate component may alternatively be implemented as a single component. Similarly, in some examples, a device illustrated herein as a single device may alternatively be implemented as multiple devices; a component illustrated as a single component may alternatively be implemented as multiple components. These multiple devices and / or components may be directly coupled via wired or wireless communications, or remotely coupled via one or more networks. In addition, one or more devices or components that may be illustrated in various figures herein may alternatively be implemented as part of another device or component not shown in these figures. In this manner and in other ways, some functions described herein may be performed via distributed processing of two or more devices or components.

[0161] In addition, some operations, techniques, features, and / or functions may be described herein as being performed by specific components, devices, and / or modules. In other examples, these operations, techniques, features, and / or functions may be performed by different components, devices, or modules. Accordingly, some operations, techniques, features, and / or functions that may be described herein as being attributed to one or more components, devices, or modules may be attributed to other components, devices, and / or modules in other examples, even if not specifically described in this manner herein.

[0162] Although specific advantages have been identified when describing some examples, various other examples may include some, none, or all of the advantages listed. Other advantages, whether technical or otherwise, may emerge from the present disclosure as will occur to those of ordinary skill in the art. Furthermore, although specific examples have been disclosed herein, various aspects of the present disclosure may be implemented using any number of technologies, whether currently known or unknown. Accordingly, the present disclosure is not limited to the examples specifically described and / or illustrated in the present disclosure.

[0163] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium and / or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, including any medium that can facilitate the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0164] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, or optical disk storage, disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is transmitted from a website, server or other remote source via a wired (e.g., coaxial cable, fiber optic cable, twisted pair) or wireless (e.g., infrared, radio and microwave) connection, then the wired or wireless connection is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but rather point to non-transient, tangible storage media.

[0165] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" or "processing circuit" used herein may refer to any of the above structures or any other structure suitable for implementing the described techniques, respectively. In addition, in some examples, the described functionality may be provided in dedicated hardware and / or software modules. Furthermore, these techniques may be implemented entirely in one or more circuits or logic elements.

[0166] The multiple techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handheld devices, mobile or non-mobile computing devices, wearable or non-wearable computing devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize the functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Instead, as described above, the various units may be combined in hardware units or provided by hardware units working in coordination, including one or more processors described above, in combination with appropriate software and / or firmware.

[0167] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as there may be many variations. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions illustrated and / or described may be performed in the order illustrated and / or described, may be performed in other orders, may be performed in parallel, or may be omitted. Similarly, the order of the processes described above may also be changed.

[0168] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, and all equivalents thereof.

Claims

1. A method for graph-based storage management, the method comprising: Allocating storage volumes within the storage cluster by a storage cluster having a plurality of storage nodes; generating a volume map for the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; as well as The volume is managed based on the volume map.

2. The method of claim 1 , wherein the volume map of the volume comprises a tree map, the tree map comprising: a function node, the function node specifying a function implemented by one or more resources allocated to the volume; as well as One or more leaf nodes to the function node, wherein each of the one or more leaf nodes specifies a resource of the one or more resources that is allocated for the function. 3 . The method of claim 2 , wherein the tree map further comprises a root node representing at least one tenant of the storage cluster.

4. The method of claim 2, wherein managing the volume based on the volume map comprises: The function node is modified with a second function node specifying a second function.

5. The method of claim 2, wherein the one or more leaf nodes include a first leaf node that specifies a first resource for the function, and wherein managing the volume based on the volume map includes: The first leaf node designated for the first resource for the function is modified with a second leaf node designated for a second resource for the function.

6. A method according to claim 2, wherein the one or more resources used for the volume are allocated based on applying one or more rules to the one or more leaf nodes, wherein the one or more rules include one or more of the following: resource availability rules, data protection rules, or load balancing rules.

7. The method of claim 6, wherein managing the volume based on the volume map comprises: The one or more resources allocated to the volume are dynamically rebalanced based on the one or more modified metrics associated with the one or more resources allocated to the volume.

8. The method of claim 2, wherein managing the volume based on the volume map comprises: receiving, by the functional node, an event associated with the one or more leaf nodes to the functional node; as well as The different resources are allocated to the function by replacing the one or more leaf nodes to the function node with leaf nodes representing different resources for the function.

9. The method of claim 2, wherein the one or more resources allocated to the volume are allocated based on one or more parameters of the volume, and wherein managing the volume based on the volume map comprises: determining that the one or more parameters of the volume have changed; as well as A clone of the volume is generated based on the one or more parameters that have changed.

10. The method of claim 9, wherein the one or more parameters include one or more of: a block size, an encryption key, a compression scheme, a volume size, or a data protection scheme.

11. A computing system for graph-based storage management, the computing system comprising: a network interface for interconnecting the computing system with at least one other computing system to form a plurality of computing systems on a network; as well as at least one storage node, wherein the at least one storage node is part of a storage cluster formed by the plurality of computing systems, and wherein the computing system is configured to: Allocating storage volumes within the storage cluster; generating a volume map for the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and The volume is managed based on the volume map.

12. The computer system of claim 11, wherein the volume map of the volume comprises a tree map, the tree map comprising: a function node, the function node specifying a function implemented by one or more resources allocated to the volume; as well as One or more leaf nodes to the function node, wherein each of the one or more leaf nodes specifies a resource of the one or more resources that is allocated for the function.

13. The computer system of claim 12, wherein managing the volume based on the volume map comprises: The function node is modified with a second function node specifying a second function.

14. A computer system according to claim 12, wherein the one or more leaf nodes include a first leaf node designated for a first resource of the function, and wherein the volume is managed based on the volume map, and the computing system is further configured to: modify the first leaf node designated for the first resource of the function with a second leaf node designated for a second resource of the function.

15. A computer system according to claim 12, wherein the one or more resources used for the volume are allocated based on applying one or more rules to the one or more leaf nodes, wherein the one or more rules include one or more of the following: resource availability rules, data protection rules, or load balancing rules.

16. A computer-readable storage medium for graph-based storage management, the computer-readable storage medium comprising instructions that, when executed, cause one or more processors to: Allocate storage volumes within the storage cluster; generating a volume map for the volume, wherein the volume map represents one or more functional elements in a data plane of the volume; and The volume is managed based on the volume map.

17. The computer-readable storage medium of claim 16, wherein the volume map of the volume comprises a tree map, the tree map comprising: a function node, the function node specifying a function implemented by one or more resources allocated to the volume; as well as One or more leaf nodes to the function node, wherein each of the one or more leaf nodes specifies a resource of the one or more resources that is allocated for the function.

18. The computer-readable storage medium of claim 17, wherein managing the volume based on the volume map comprises: The function node is modified with a second function node specifying a second function.

19. A computer-readable storage medium according to claim 17, wherein the one or more leaf nodes include a first leaf node designated for a first resource of the function, and wherein, in order to manage the volume based on the volume map, the computing system is further configured to: modify the first leaf node designated for the first resource of the function with a second leaf node designated for a second resource of the function.

20. The computer-readable storage medium of claim 17, wherein the one or more resources for the volume are allocated based on applying one or more rules to the one or more leaf nodes, wherein the one or more rules comprise one or more of: resource availability rules, data protection rules, or load balancing rules.

Images