Storage Management in a Data Management Platform for Cloud-Native Workloads

Abstract

A data management platform is provided that includes a compute server and a storage server. The storage server manages a plurality of storage devices communicatively coupled to the storage server. The compute server and the storage server are communicatively coupled via a network. The plurality of storage devices managed by the storage server are separated from the compute server such that storage capacity of the plurality of storage devices can be scaled independently of the compute server.

Description

[Technical field]

[0001] Claiming priority This application claims priority under 35 U.S.C. 365(c) to U.S. patent application Ser. No. 16 / 729,075, entitled "STORAGE MANAGEMENT IN A DATA MANAGEMENT PLATFORM FOR CLOUD-NATIVE WORKLOADS," filed Dec. 27, 2019, which is incorporated herein in its entirety. [Background technology]

[0002] Cloud computing provides access to servers, storage, databases, and a wide range of application services over the Internet. Cloud service providers offer cloud services, such as network services and business applications, hosted on servers in one or more data centers that can be accessed by companies or individuals over the Internet. Hyperscale cloud service providers typically have hundreds of thousands of servers. Each server in a hyperscale cloud includes storage devices for storing user data, for example, user data for business intelligence, data mining, analytics, social media, and microservices. Cloud service providers generate revenue from businesses and individuals (also called tenants) that use cloud services. For example, tenants can store data by paying a monthly fee to the cloud service provider based on the amount of storage allocated to the tenant.

[0003] Most enterprise data centers today do not have the capacity in scale and performance to effectively manage and process petabytes of data. Data-intensive applications and tools such as artificial intelligence (AI) inference and analytics generate and consume exponential amounts of data and telemetry that must be moved, stored, and processed in a more secure, faster, and scalable manner. Hyperscale data centers typically do this by adding additional servers to the data center. However, depending on the workloads running in the data center, one type of component of these servers may be overutilized while another type of component may be underutilized, meaning customers and service providers are not optimizing the use of their investment. Summary of the Invention

[0004] Features of the embodiments of the claimed subject matter will become apparent as the following detailed description proceeds and by reference to the drawings, in which like numerals refer to like parts and in which: [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a conceptual diagram of one embodiment of a data management platform (DMP). [Diagram 2] FIG. 2 is a block diagram of a physical cluster embodiment of the data management platform shown in FIG. 1. [Diagram 3] 3 is a block diagram of an embodiment of one of the calculation servers in the data management platform shown in FIG. 2. [Figure 4] FIG. 3 is a block diagram of one embodiment of an accelerator server in the data management platform shown in FIG. 2. [Diagram 5] A logical diagram illustrating access to solid state drives from the computational servers in the data management platform shown in Figure 2. [Figure 6]FIG. 5 is a block diagram of an accelerator server shown in FIG. 4 that performs resource control of storage services. [Figure 7] FIG. 1 illustrates data transfer between a network interface controller and a solid-state drive in an accelerator server. [Figure 8] 1 is a flow diagram illustrating a method for moving data from a solid state drive through an L3 cache and a network interface controller to a data plane. [Figure 9] FIG. 1 illustrates an embodiment for configuring a last level cache to isolate N sets of cache ways of the last level cache that are shared by solid state drives and a network interface controller. [Figure 10] 1 is a flow diagram illustrating an embodiment of a method for configuring a set of cache ways of a last level cache shared by a solid state drive and a network interface controller. [Figure 11] FIG. 3 is a block diagram of an embodiment of a rack in a data management platform in the physical cluster shown in FIG. 2 during normal operation on a healthy system. [Figure 12] 3 is a block diagram of an embodiment of a rack in a data management platform in the physical cluster shown in FIG. 2 in the case of degraded operation in a failed system. [Figure 13] A block diagram showing a metrics exporter in a container in a pod in a node that is used by the data management platform to detect node status / failure. [Figure 14] 1 is a flow diagram illustrating a method for managing a hardware failure of a data management platform in a physical cluster. [Figure 15] FIG. 2 is a block diagram illustrating hardware events and measurements within a node used by the data management platform to monitor and manage performance thresholds to detect node health and failures. [Figure 16] 1 is a flow diagram illustrating a method implemented in a storage node in a rack for monitoring performance of the storage node. [Figure 17] FIG. 2 is a block diagram of an embodiment of a compute node. [Figure 18] FIG. 2 is a block diagram of another embodiment of a computing node. [Figure 19] 3 is a block diagram of an embodiment of a rack in a data management platform in the physical cluster shown in FIG. 2 including a resource manager that automatically adds and removes logical resources. [Figure 20] 20 is a flow diagram of a method for automatically adding or removing logical resources in response to detecting stress in a rack in the data management platform shown in FIG. 19. [Figure 21] FIG. 2 is a block diagram of one embodiment of a physical cluster in a data management platform including a storage self-repair mechanism. [Figure 22] FIG. 22 illustrates one embodiment of a mapping of workloads in the cluster illustrated in FIG. 21. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The following detailed description proceeds with reference to exemplary embodiments of the claimed subject matter, many alternatives, modifications, and variations of which will be apparent to those skilled in the art. It is therefore intended that the claimed subject matter be broadly construed and defined as set forth in the appended claims.

[0007] The data management platform includes accelerator servers and compute servers. The storage devices managed by the accelerator servers are separate from the compute servers so that storage capacity can be scaled independently of the compute.

[0008] Various embodiments and aspects of the invention will be described with reference to the details discussed below, and the accompanying drawings illustrate various embodiments. The following description and drawings are illustrative of the invention and should not be construed as limiting the invention. Numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. However, in some instances, well-known or conventional details are not described in order to provide a concise discussion of the embodiments of the invention.

[0009] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0010] Figure 1 is a conceptual diagram of one embodiment of a data management platform (DMP) 100. In the embodiment shown in Figure 1, the data management platform 100 is a rack-centric physical cluster having racks 106 interconnected via a routing interconnect 110. The routing interconnect 110 can be an Ethernet fabric arranged in a multi-tier Clos topology, or any other Open Systems Interconnect (OSI) layer 3 routing interconnect.

[0011] Racks 106 in a data center are a type of physical steel and electronic framework designed to house servers, networking devices, cables, and other data center computing equipment. Each rack 106 connects to a routing interconnect 110 and can include one or more compute servers 112, accelerator servers 114, utility servers 118, and infrastructure servers 116. The servers can also be referred to as nodes.

[0012] A utility server 118 is used to initialize the physical cluster. During initialization, the utility server 118 performs orchestration and scheduling functions. In one embodiment, Kubernetes (K8) is used to perform the functions for the orchestrator / scheduler 102. Kubernetes is an open source container-orchestration system for automating application deployment, scaling, and management. The Kubernetes control plane is hosted on the infrastructure servers 116. A Kubernetes Host Agent runs on all compute servers 112 and accelerator servers 114.

[0013] Application deployment can also be automated through the use of virtual machines. Other examples of orchestrator / schedulers 102 include OpenShift (a Red Hat platform as a service (PaaS) built on Docker and Kubernetes) and Pivotal Container Service (PKS).

[0014] The control plane manager 104 can perform functions to create, manage, and update infrastructure resources such as virtual machines. The control plane manager 104 can also initialize physical machines and network switches. Examples of control plane managers 104 include Fleet, Red Hat Satellite, Teraform, and Metal As A Service (MaaS).

[0015] Each of the compute servers 112, the accelerator servers 114, the utility servers 118, and the infrastructure servers 116 includes a Baseboard Management Controller (BMC) 120. The BMC 120 is a dedicated service processor that provides services to monitor the physical state of the compute servers 112, the accelerator servers 114, the utility servers 118, and the infrastructure servers 116 and to monitor and control their operation via a Management API 108. Examples of the Management API 108 include IPMI (Intelligent Platform Management Interface), Redfish® (a Distributed Management Task Force (DMTF) standard), and Dell® Open Manage Enterprise (OME).

[0016] 2 is a block diagram of an embodiment of the data management platform (DMP) 100 shown in FIG. 1 in a physical cluster 200. The physical cluster 200 has N racks 106, 106-1, ... 106-N. In one embodiment, N is 16. Each rack 106 includes a compute server 112 and at least one accelerator server 114. Each compute server 112 and accelerator server 114 is communicatively coupled to a data switch 206 and a management switch 208. The data switch 206 of each rack 106 provides a data plane 204 (also referred to as a data fabric) between the compute servers 112 and accelerator servers 114 in the same rack 106, the compute servers 112 and accelerator servers 114 in other racks 106, and infrastructure servers 116 shared by multiple racks 106. The management switch 208 of each rack 106 provides a control plane 202 (also called a management network) between the rack 106 and a utility server 118 shared by multiple racks 106 .

[0017] Figure 3 is a block diagram of an embodiment of one of the compute servers 112 in the physical cluster 200 shown in Figure 2. In the illustrated embodiment, the compute server 112 includes a system-on-chip 306, a network interface controller 302, and compute server control logic 304. The network interface controller 302 is communicatively coupled to the data plane 204 shown in Figure 2. An embedded network interface controller 308 in the system-on-chip 306 is communicatively coupled to the control plane 202 shown in Figure 2.

[0018] Figure 4 is a block diagram of an embodiment of one of the accelerator servers 114 in the physical cluster 200 shown in Figure 2. In the illustrated embodiment, the accelerator server 114 performs storage processing tasks and may be referred to as a storage server 400.

[0019] The storage server 400 includes a system-on-chip 306, a storage server control logic 402 communicatively coupled to a network interface controller 302, and one or more solid state drives 404. In one embodiment, the storage server control logic 402 is communicatively coupled to the solid state drives 404 and the network interface controller 302 using a Peripheral Component Interconnect (PCIe) protocol. An embedded network interface controller 308 in the system-on-chip 306 is communicatively coupled to the control plane 202 shown in FIG. 2.

[0020] The storage server control logic 402 performs storage processing tasks offloaded by the system-on-chip 306, enabling decomposition of computation and storage into independent, scalable resources.

[0021] Figure 5 is a logical diagram illustrating access from the compute servers 112 to the solid state drives 404 in the physical cluster 200 shown in Figure 2. A virtual routing function 508 in the kernel space 502 of the operating system provides a relational database management system 506 in the user space 500 access to data stored on the solid state drives 404 of the storage servers 400 (Figure 4) via the data plane 204. The virtual routing function 508 includes a forwarding information base (FIB) 510 and a flow table 512 that stores routes and policies.

[0022] Routers 504 provide secure network connectivity for virtual managers and containers. An example of a router 504 is Calico. Calico provides secure network connectivity for container and virtual machine workloads. Calico uses Layer 3 (network layer) of the Open System Interconnection (OSI) model and Border Gateway Protocol (BGP) to build routing tables. Calico creates a flat Layer 3 network and assigns fully routable Internet Protocol (IP) addresses to every rack 106. Workloads can communicate without IP encapsulation or network address translation for bare metal performance. Calico uses Felix (a per-node domain daemon) to configure routes and enforce network policies.

[0023] Shared resources such as last level cache (LLC) and main memory bandwidth have a significant impact on workload performance in data management platforms (DMPs). Closer monitoring and management of these resources enables deployments to meet more demanding workload demands, including increasingly stringent performance service-level agreements (SLAs).

[0024] FIG. 6 is a block diagram of the storage server 400 shown in FIG. 4 that performs resource control of the storage service.

[0025] The storage server 400 includes a system-on-chip (SOC or SoC) 604 that integrates a processor, memory, and input / output (I / O) control logic into a single SoC package. The SoC 604 includes at least one central processing unit (CPU) module 608 and a memory controller 614. In other embodiments, the memory controller 614 may be external to the SoC 604. The CPU module 608 includes at least one processor core 602 that includes a level 1 (L1) and a level 2 (L2) cache 620, and a level 3 (L3) cache 606 that is shared with the other processor cores 602 in the CPU module 608.

[0026] Although not shown, each of the processor cores 602 may include internally an execution unit, a prefetch buffer, an instruction queue, a branch address calculation unit, an instruction decoder, a floating point unit, a retirement unit, etc. The CPU module 608, according to one embodiment, may correspond to a single-core or multi-core general-purpose processor such as those offered by Intel® Corporation.

[0027] Within I / O subsystem 612 reside one or more I / O interfaces 616 that translate host communication protocols utilized within processor core 602 into protocols compatible with particular I / O devices. Some of the protocols that can utilize an I / O interface for translation include Peripheral Component Interconnect (PCI) Express (PCIe), Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), and Institute of Electrical and Electronics Engineers (IEEE) 1594 "Firewire."

[0028] The I / O interface 616 may communicate with one or more solid state drives 404 and the network interface controller 302 via memory 630 and / or L3 cache 606. The solid state drives 404 may be communicatively and / or physically coupled to one another via one or more buses using one or more of a variety of protocols, including, but not limited to, Serial Attached SCSI (Small Computer System Interface) (SAS), Peripheral Component Interconnect Express (PCIe), NVM Express (NVMe) over PCIe (Peripheral Component Interconnect Express), and Serial ATA (Advanced Technology Attachment) (SATA). In other embodiments, other storage devices, such as hard disk drives (HDDs), may be used in place of the solid state drives 404, and the hard disk drives and / or solid state drives may be configured as a Redundant Array of Independent Disks (RAID).

[0029] The Non-Volatile Memory Express (NVMe) standard defines a register-level interface for host software to communicate with a non-volatile memory subsystem (e.g., a solid-state drive 404) over a high-speed serial computer expansion bus, Peripheral Component Interconnect Express (PCIe). The NVM Express standard is available at www.nvmexpress.org. The PCIe standard is available at www.pcisig.com.

[0030] In one embodiment, memory 630 is a volatile memory and memory controller 614 is a volatile memory controller. Volatile memory is memory whose state (and therefore the data stored therein) is indeterminate if power to the device is interrupted. Dynamic volatile memory requires the data stored in the device to be refreshed in order to maintain its state. An example of dynamic volatile memory includes DRAM (Dynamic Random Access Memory) or some variant of SDRAM (Synchronous DRAM). The memory subsystem described herein may be compatible with several memory technologies such as DDR3 (Double Data Rate version 3, original release by JEDEC (Joint Electronic Device Engineering Council) on June 27, 2007). "DDR4" (DDR version 4, the initial specification published by JEDEC in September 2012), DDR4E (DDR version 4), LPDDR3 (Low Power DDR version 3, JESD209-3B, published by JEDEC in August 2013), LPDDR4 (LPDDR version 4, JESD209-4, first published by JEDEC in August 2014), WIO2 (Wide Input / Output version 2, JESD229-2, first published by JEDEC in October 2014), HBM (Wide Bandwidth Memory, JESD325, first published by JEDEC in October 2013), DDR5 (DDR version 5, currently under review by JEDEC), LPDDR5, HBM2 (HBM version 2), currently under review by JEDEC, etc., or any other combination of memory technologies, and any technologies based on any derivative or extension of such specifications. JEDEC standards are available at www.jedec.org.

[0031] In another embodiment, memory 630 is a non-volatile memory (NVM) and memory controller 614 is a non-volatile memory controller. A non-volatile memory device is a memory whose state is determined even when power to the device is interrupted. Non-volatile memory devices may include byte-addressable write-in-place 3D cross point memory devices such as single-level or multi-level phase change memory (PCM) or switched phase change memory (PCMS) or other byte-addressable write-in-place NVM devices (also referred to as persistent memory), NVM devices using chalcogenide phase change materials (e.g., chalcogenide glasses), resistive memories including metal oxide-based, oxygen vacancy-based and conductive bridge random access memories (CB-RAM), nanowire memories, ferroelectric random access memories (FeRAM, FRAM®), magnetoresistive random access memories (MRAM) incorporating memristor technology, spin transfer torque (STT) MRAM, spintronic magnetic junction memory-based devices, magnetic tunnel junction (MTJ)-based devices, DW (domain wall) and SOT (spin orbit transfer)-based devices, thyristor-based memory devices, or any combination of the above or other memories.

[0032] In yet another embodiment, memory 630 includes both byte-addressable write-in-place NVM devices and volatile memory devices that may be included in one or more memory modules.

[0033] Shared resources that have a significant impact on workload performance include processor cache and memory bandwidth resources, which can have a significant impact on application performance and runtime determinism. Closer monitoring and management of these resources enables deployments to meet more demanding workload demands, including increasingly stringent performance service-level agreements (SLAs) to support new workloads such as network function virtualization (NFV).

[0034] 7 illustrates data transfer between the network interface controller 302 and the solid state drives 404 of the storage server 400. The I / O adapter 616 includes a first PCIe adapter 702 for managing communications from the solid state drives 404 over a first PCIe interface 708 and a second PCIe interface 704 for managing communications over the first PCIe interface 708 to the network interface controller 302. The network interface controller 302 can exchange data using remote direct memory access (RDMA), e.g., direct memory access from the L3 cache 606 and / or memory 630 to memory of a compute server 112 or another accelerator server 114 communicatively coupled to the data plane 204.

[0035] The solid state drive 404 and the network interface controller 302 exchange data through an L3 cache 606 and / or a memory 630. The L3 cache 606 may also be referred to as a last level cache (LLC). In addition to being shared with other processor cores 602 in the CPU module 608, the level 3 (L3) cache 606 is also shared with a first PCIe interface 708 and a second PCIe interface 710.

[0036] With multiple agents (processor core 602, first PCIe interface 708, and second PCIe interface 710) all competing to access the same L3 cache 606, there may be more variability in cache misses in the L3 cache 606, cache evictions from the L3 cache 606 to memory 630, and latency variations in agent transactions. If the bandwidth of the network interface controller 302 is closely matched to the bandwidth of the solid state drive 404 and the L3 cache 606 is of sufficient size, the majority of data transfers between the solid state drive 404 and the network interface controller 302 will be through the L3 cache 606 without eviction ("spill") to memory 630 via path 714.

[0037] To minimize spillover from the L3 cache 606 to memory 630, a configurable portion (a subset of the cache ways) of the L3 cache 606 is dedicated to storing data transferred between the solid state drive 404 and the network interface controller 302. In the embodiment shown in FIG. 6, there are N processor cores 602-1, .. 602-N. A first subset of the cache ways 712 of the L3 cache 606 is dedicated to both the processor core 602-1 and the processor core 602-2. A second subset of the cache ways 712 of the L3 cache 606 is dedicated to the processor core 602-3, the solid state drive 404, and the network interface controller 302.

[0038] In one embodiment, Intel® Cache Allocation Technology (CAT) can be used to dedicate a subset of the cache ways of the L3 cache 606 to specific processor cores 602-1,..602-N and / or I / O memory space (PCIe) and control which agents share / contend for a particular subset of the cache ways (or portions) of the L3 cache 606. All other agents are excluded from using a second set of cache ways 706 of the L3 cache 606 that exclusively stores data transferred between the solid state drive 404 and the network interface controller 302.

[0039] The use of a configurable second subset of cache ways 706 of the L3 cache 606 reduces workload variability and provides more accurate and predictable resource allocation of the storage service to other co-located workloads, allowing for more accurate service level predictions for the storage service co-located with other workloads. Three quality of service mechanisms (cache, core / input / output memory, and logical volume bandwidth throttling) combine to provide tunable resource sharing, isolation, and variability reduction.

[0040] Containers (e.g., Kubernetes containers or virtual machine containers) and threads associated with storage services and networking are assigned effective allocations to a subset of cache ways or buffer space and a subset of memory bandwidth (memory transaction credits-memory bandwidth enforcement in one embodiment) to constrain network / storage functionality to the subset of cache / buffer ways and / or memory bandwidth. With proper bandwidth matching between the storage server network allocations and the bandwidth / size allocations of the L3 cache 606, the storage server 400 can support storage to / from network data flows that pass entirely or nearly entirely through the second portion of the cache ways 706 of the L3 cache 606 with little or no spillage of data to memory 630.

[0041] Additionally, there are operating system mechanisms (e.g., the device mapper in a Linux operating system) to partition the access bandwidth of a logical volume (LVM) on a single storage device (such as, but not limited to, an NVMe solid state drive) to a configured bandwidth, e.g., 200 Megabytes per second (MB / s). Combining logical volume rate quality of service control with the configurable subset of cache ways described above provides an overall system solution that provides storage services workloads that coexist with other workloads in a more tunable and predictable manner.

[0042] 8 is a flow diagram illustrating a method for moving data from a solid state drive 404 to the data plane 204 via the L3 cache 606 and the network interface controller 302. Data can also be moved in the opposite direction from the network interface controller 302 to the solid state drive 404, through the L3 cache 606 to the data plane 204.

[0043] In block 800, a fixed number of cache ways (e.g., a second subset of cache ways 706) are allocated in the L3 cache 606 to store data shared by the solid state drive 404 and the network interface controller 302. The fixed number of cache ways is tunable based on system performance requirements. The number of cache ways allocated in the second subset of cache ways 706 for use only to store data transferred between the solid state drive 404 and the network interface controller 302 reduces the number of cache ways in the L3 cache available for use by other cores, resulting in reduced performance for the other cores.

[0044] In one embodiment, the selected fixed number is not dynamically changed during operation. The N cache ways of the configurable portion of L3 cache 606 are also shared by one or more processor cores 602 (e.g., 602-2). A second subset of cache ways 706 of L3 cache 606 are isolated from other agents. The remaining cache ways in L3 cache 606 (e.g., the first set of cache ways 712) may be used / shared by other agents.

[0045] In block 802, the solid state drive 404 is configured to write data directly (via direct memory access) to a second subset of the cache ways 706 of the L3 cache 606, and the network interface controller 302 is configured to read data directly (via direct memory access) from the second subset of the cache ways 706 of the L3 cache 606. The solid state drive 404 writes the data to the second subset of the cache ways 706 of the L3 cache 606 while the network interface controller 302 reads the data from the second subset of the cache ways 706 of the L3 cache 606.

[0046] At block 804, if the rate at which the solid state drive 404 is writing to the second subset of the cache ways 706 of the L3 cache 606 does not match the rate at which the network interface controller 302 is reading data from the second subset of the cache ways 706 of the L3 cache 606, processing proceeds to block 806. If the rate at which the solid state drive 404 is writing to the second subset of the cache ways 706 of the L3 cache 606 matches the rate at which the network interface controller 302 is reading data from the second subset of the cache ways 706 of the L3 cache 606, processing proceeds to block 802.

[0047] In block 806, data is evacuated from the L3 cache 606 to memory 630 to allow the solid state drive 404 to continue writing data to the second subset of the cache ways 706 of the L3 cache 606. The evacuation of data from the L3 cache may be referred to as a cache spill.

[0048] FIG. 9 illustrates an embodiment of configuring the last level cache to isolate N sets of cache ways 906 of the L3 cache 606 (also referred to as the last level cache) shared by the solid state drive 404 and the network interface controller 302. Intel® Cache Allocation Technology (CAT) includes classes of service (CLOS) that act as resource control tags by which threads / apps / virtual memories (VMs) / containers can be grouped. Each service class has an associated resource capacity bitmask (CBM) that indicates how much of the last level cache may be used by a given service class. In the embodiment illustrated in FIG. 9, the service class table 902 has four service classes, labeled CLOS1 through CLOS4. In other embodiments, there may be more or less than four service classes. Each CLOS register has a bit for each processor core 602, and the state of the bit indicates whether the core is part of a particular service class. The force mask 904 is circuitry / logic that restricts the cache mapping to the cache way bitmap encoded in the respective CLOS register.

[0049] In the illustrated embodiment, each service class CLOS1-CLOS4 has an associated capacity bit mask labeled MASK1-MASK4. The value of each bit in the capacity bit mask indicates the amount of L3 cache 606 available to the service class and indicates how many of the cache ways in the set of cache ways 906 are shared by service classes CLOS1-CLOS4.

[0050] In one embodiment, the capacity bitmask associated with the service class exclusively stores data shared by the network interface controller, and a portion of the set of cache ways 906 is shared by the solid state drive 404 and the network interface controller 302.

[0051] FIG. 10 is a flow diagram illustrating one embodiment of a method for configuring and using a set of cache ways in a last level cache shared by a solid state drive 404 and a network interface controller 302.

[0052] Cache allocation technology allows resource allocation based on application priority or service class (COS or CLOS). The processor exposes a set of service classes to which applications (or individual threads) can be assigned. Cache allocation for each application or thread is limited based on the class they are associated with. Each service class can be configured with a capacity bitmask that represents the capacity, indicating the degree of overlap and isolation between classes. Each logical processor has an exposed register that allows the operating system / virtual machine manager to specify the service class when an application, thread, or virtual machine is scheduled. Service class usage is consistent across resources, and service classes may be tagged with multiple resource control attributes, reducing software overhead during context swaps. Rather than adding a new type of service class tag for each resource, the overhead of service class management is constant. Cache allocation for a directed application / thread / container / VM is automatically controlled by the hardware based on the class and the bitmask associated with that class. The bitmask can be set via the mode status register for the L3 cache.

[0053] At block 1000, the cache allocation technology allows an operating system (OS), hypervisor / virtual machine manager (VMM), or similar system service management agent to specify the amount of cache space an application can fill. Enumeration support is provided to query the capabilities of a particular cache allocation technology, such as what levels of the cache hierarchy are supported and the maximum allocation bitmask size.

[0054] In block 1002, an operating system or hypervisor sets the amount of resources available to a particular service class via a list of allocation bitmasks. The bit length of the available capacity mask depends on the configuration of the L3 cache.

[0055] At block 1004, if there is a context switch, processing continues at block 1006. Otherwise, processing continues at block 1008.

[0056] At block 1006, the service class of the currently executing application is communicated to the execution environment (operating system / virtual). If the service class of the new thread is different from the currently executing application service class, the different service class can be loaded. Processing proceeds to block 1008.

[0057] At block 1008, if there is a memory request, processing continues at block 1010. Otherwise, processing continues at block 1004.

[0058] At block 1010, the service class associated with the memory access is used to perform cache allocation. Processing continues at block 1004.

[0059] Returning to FIG. 2, recovery of hardware failures of the data management platform 100 in the physical cluster 200 can be performed by a hardware or software load balancer along with health checks. Hardware failures can include hardware failures of the compute servers 112, accelerator servers 114, data switches 206, management switches 208, infrastructure servers 116, and utility servers 118. Each of the servers (compute servers 112, accelerator servers 114, infrastructure servers 116, and utility servers 118) can also be referred to as a node. Typically, a logging system is used to flag events and an operator manually intervenes to remove or replace the failed hardware when a particular event is logged.

[0060] However, current load balancers do not account for degradation of hardware components and / or performance degradation based on hardware failure or degradation of data management platform 100. Additionally, current load balancers are difficult to scale in enterprise data centers.

[0061] FIG. 11 is a block diagram of an embodiment of a rack 106 in the data management platform 100 in the physical cluster 200 shown in FIG. 2 during normal operation in a system with no hardware failures or degradation. The method and system for transparent system service repair of hardware failures and degradations allows direct and efficient exposure of processor hardware events and measurements. Telemetry works with a unified interface to the Routing Information Base (RIB) 1112, Forwarding Information Base (FIB) 510 (FIG. 5), Filtering System (FS) 1114, and Internet Protocol Anycast to selectively allow or suppress dynamic routes from the servers (compute servers 112, accelerator servers 114, infrastructure servers 116, and utility servers 118) based on hardware events. The Routing Information Base (RIB) 1112 is a data table that stores routes to specific network destinations.

[0062] Dynamic routing is a networking technique that provides optimal data routing. Dynamic routing allows routers to select paths according to the logical network layout that changes in real time. In dynamic routing, the routing protocols running on the router are responsible for creating, maintaining, and updating the dynamic route table. Dynamic routing is the process by which network traffic to an endpoint can be forwarded through different routes based on the environment.

[0063] A failed component in a server (node) 1100a-e may affect the functionality and performance of one or more applications running on the server (node) 1100a-e. Examples of failed components in a server (node) 1100a-e include a solid state drive, a memory module, or a power distribution unit. If a server (node) 1100a-e in the data management platform 100 is degraded, the failed component event is detected by a node failure detector in a filtering system (FS) 1114, and the route associated with the service is withdrawn from the route table 1102. In one embodiment, the route is withdrawn by a router 504 (FIG. 5), e.g., Calico in Kubernetes.

[0064] In one embodiment, operating system (OS) 1106 is a Linux operating system. Border Gateway Protocol (BGP) clients on servers (nodes) 1100a-e read routing state from FIB 510 and distribute it to other BGP clients running on other servers (nodes) 1100a-e. Routes in FIB 510 are set up by agents in response to requests to provision connectivity for specific workloads. BGP clients distribute updated routes to BGP clients running on other servers (nodes) 1100a-e in response to updates to FIB 510.

[0065] In one embodiment, the agent that sets up routes in the FIB 510 is Felix and the BGP client is BIRD. BIRD is an open source implementation for routing Internet Protocol packets on Unix-like operating systems. Felix is ​​a per-node domain daemon for setting routes and enforcing network policies.

[0066] A node failure detector in the filtering system 1114 monitors node hardware metrics and generates alerts (e.g., failed component events). In one embodiment, failures or degradations are detected via platform telemetry and failure / degradation events are communicated to an open collector, e.g., "collectd", and then to an event handler that takes corrective action. One example of a corrective action is removing a route.

[0067] In an Internet Protocol Anycast implementation, the route associated with the failed server service is withdrawn, triggering removal of the route from connected peers, which are all network devices (both servers and switches) in the data plane network of the data management platform 100. Flows are transparently redirected to healthy or available server nodes 1100a-e.

[0068] In the illustrated embodiment, a rack 106 is shown containing a number of nodes (N) 1110, with five of the N nodes 1110a-1110e shown. Each node is a physical server that may be a compute server 112, an accelerator server 114, an infrastructure server 116, or a utility server 118. The utility server 118 may also be referred to as a control plane server node that performs management tasks in the data management platform.

[0069] In one embodiment, the physical cluster 200 has up to 16 racks 106 and up to 20 nodes 1110 per rack. In other embodiments, there can be more than 16 racks and 20 nodes per rack. The first three racks have one utility server 118 per node, the second and third racks have one infrastructure server 116 per rack, the first three racks 106 have up to 14 compute servers 112 per rack 106, the next thirteen racks 106 have up to 16 compute servers, and up to four accelerator servers 114 per rack 106. The accelerator servers 114 perform storage processing tasks and can be referred to as storage servers 400 (FIG. 4).

[0070] In one embodiment, each node 1110a-1110e includes a pod 1104 and an operating system (OS) 1106 (e.g., a Red Hat Enterprise Linux (RHEL) operating system). A pod 1104 is the basic execution unit of a Kubernetes application and is the smallest and simplest unit in the Kubernetes object model that can be created or deployed. A pod 1104 represents the unit of deployment, i.e., a single instance of an application in Kubernetes, which can include either a single container or a small number of containers that are tightly coupled and share resources.

[0071] A pod 1104 is a group of one or more containers with shared storage / network. Containers within a pod 1104 share an Internet Protocol (IP) address and port space and can communicate with other pods 1104 using standard inter-process communication. Containers in different pods 1104 have separate Internet Protocol addresses and communicate with each other using the pod's 1104 IP address.

[0072] Anycast is a network addressing and routing method in which a single destination address has multiple routing paths to two or more endpoint destinations. The router 504 selects paths between the nodes 1110a-e based on the number of hops, distance, lowest cost, latency measurements, or based on the least congested route. In normal operation, each node 1110a-e in the rack 106 advertises the same Internet Protocol (IP) address (the Anycast address) for distributed common services.

[0073] Referring to the example shown in Figure 11, a service is advertised from each of the nodes 1110a-e, and the anycast address (IP address) associated with the service is the same across all six nodes 1100a-e. In this example, the IP address is "10.1.1.1". Each node 1110a-e has a unique Ethernet address stored in the route table 1102 of the data switch 206. The data switch 206 in the rack 106 can also be referred to as a top-of-rack (TOR) switch.

[0074] For example, the IP address of node 1100a is "10.1.1.1" and the Ethernet address of node 1100a is 192.168.1.1. In the absence of hardware failures or degradation events, the route table 1102 managed by the orchestrator / scheduler 102 (e.g., Kubernetes) allows all routes (through all nodes 1100a-e in the rack 106) to be advertised. The data switch 206 sees a single IP address (10.1.1.1) and six paths (through one of the nodes 1100a-e) to reach the destination. The destination is an application instance. In one embodiment, the application instance is a Kubernetes service. An application can be spawned as multiple application instances to load balance network traffic within the data management platform 100 and provide access to the application and data.

[0075] The data switch 206 can select one of the paths to nodes 1100a-e using built-in load balancing methods, such as Equal-Cost Multipath Routing (ECMP), a routing strategy in which next-hop packet forwarding to a single destination can occur via multiple "best paths" that tie together in a routing metric calculation. Multipath routing can be used in conjunction with most routing protocols because it is a per-hop decision limited to a single router.

[0076] FIG. 12 is a block diagram of an embodiment of a rack 106 in the data management platform 100 in the physical cluster 200 shown in FIG. 2 for degraded operation in the event of a failed system.

[0077] During a failure or degradation event (e.g., a failed network interface controller 302 in a compute server 112, a failed solid state drive 404 in a storage node 400, or an unstable operating system 1106 in a node 1100a-e), the affected node 1100a-e suppresses advertisements of routes associated with the affected application (also called a service).

[0078] In an embodiment where the orchestrator / scheduler 102 is Kubernetes, if the failure or degradation event is related to network connectivity, the event is handled by the Kubernetes\OpenShift and Kubernetes network components. Kubernetes detects that nodes 1100a-e are not available on the network. The network components update the route tables 1102 across the physical cluster 200.

[0079] If the failure or degradation event is not related to network connectivity, the event is handled by a Logging, Monitoring, and Alerting (LMA) stack in the pod 1104. Exporters spawned on each node 1100a-e periodically provide metrics data to the LMA stack. Based on the metrics, the node 1100a-e can be marked with additional labels and potentially additional actions can be taken. For example, a container running on a failed or degraded node 1100a-e can be rescheduled on another node 1100a-e.

[0080] 12, node 1100a has a degradation or failure event and suppresses (never advertises) or withdraws the route associated with the affected service via route table 1102. In this example, the anycast address is 10.1.1.1 for each node 1100a-e in rack 106. When a route to node 1100a is withdrawn, e.g., node 1100a stops advertising the route and data switch 206 removes the route from route table 1102.

[0081] Of the six available paths to the anycast address (10.1.1.1), the path to node 1100a (destination 10.1.1.1, next hop 192.168.1.1) is removed from the route table 1102 of the data switch 206. The next hop is a unique Internet Protocol (IP) address associated with each of the nodes 1100a-e. The anycast IP address is the IP address of an application instance or pod 1104 on each of the nodes 1100a-e. The network traffic is forwarded to the pod 1104 on the nodes 1100a-e and then to the application instance in the nodes 1100a-e. All remaining traffic flows are distributed to the remaining available paths via the route table 1102 of the data switch 206.

[0082] In an embodiment of the system using the Linux® operating system, during a failure or degradation event, such as a value derived from raw sensor data (e.g., “critical maximum power match” via the Linux® “libsensors” library via the “sysfs” interface), event action detectors and managers in pods 1104 of the affected nodes 1100a-e suppress advertisements of routes associated with the affected Kubernetes service.

[0083] A Kubernetes service is an abstraction that defines a logical set of pods 1104 that run within a cluster 200, all of which provide the same functionality. When created, each Kubernetes service is assigned a unique Internet Protocol (IP) address that it is routed to, also known as a clusterIP. The assigned IP address does not change for the life of the Kubernetes service.

[0084] Pods 1104 can be configured to communicate with Kubernetes services in the orchestrator / scheduler 102. Communications to a Kubernetes service are automatically load balanced to Pods 1104 that are members of the Kubernetes service. Multiple nodes can advertise the same service IP, which is called "Anycast." An example of a Kubernetes service is a ClusterIP backing a Pod 1104 or set of Pods 1104 hosting applications such as NGINX (an open source, high performance HTTP server and reverse proxy) and an IMAP / POP3 proxy server, DNS (Domain Name System) and Apache (an open source web server).

[0085] After an event is detected, an event action detector and manager in the pod 1104 of the affected node 1100a initiates a script that blackholes (suppresses and does not advertise alternative paths) the route associated with the service on the affected node 1100a.

[0086] When the failed node (in this case, node 1100a) becomes functional again, it automatically advertises the anycast IP address (10.1.1.1) associated with the previously failed service and is transparently reinserted into the physical cluster 200. The data switch 206 discovers another path for node 1100a (destination (IP address of the pod) 10.1.1.1, next hop (IP address of node 1100a) 192.168.1.1) and adds this to the existing route table 1102 as a multipath destination.

[0087] 13 is a block diagram illustrating metrics exporters in containers of pods 1104 (FIG. 11) of nodes 1100a-e that are used by data management platform 100 to detect node status and failures. In the illustrated embodiment, there are four metrics exporters and another exporter 1314. Each metrics exporter is in a separate container of pod 1104 (FIG. 11).

[0088] The device mapper exporter 1310 collects low-level metrics from the device mapper volumes. Examples of low-level metrics collected from the device mapper volumes include average read / write time, average latency, utilization, queue size, writes / reads per second, read / write size per second, and merged reads / writes per second.

[0089] The storage exporter 1312 collects low-level metrics from the solid-state drive. Examples of low-level metrics collected include program and erase failures to the non-volatile memory in the solid-state drive, end-to-end error detection counts, cyclic redundancy check (CRC) error counts, timed workload timers, thermal throttling conditions, retry buffer overflow counts, wear leveling counts, timed workload media wear, timed workload host read / write ratio, power outage imminent (pli) lock loss counts, bytes written to non-volatile memory in the solid-state drive, bytes written by the host to the solid-state drive, and remaining system area life remaining.

[0090] The memory bandwidth exporter 1308 collects low-level metrics based on the memory bandwidth monitor. The Processor Counter Monitor (PCM) is an application programming interface (API) and a set of API-based tools for monitoring performance and energy metrics of Intel® processors. The memory bandwidth exporter 1308 collects low-level metrics using the processor counter monitor. Examples of low-level metrics related to memory bandwidth that are collected include Channel Read / Write, Memory Read / Write Mega Byte / sec, Read / Write, Memory, and Memory Mega Byte / sec.

[0091] The exporters of the network interface controller 1306 collect low-level metrics from the network interface controllers. Examples of low-level metrics collected include transmit queue drops, transmit queue stalls, receive outside buffer, transmit errors, receive through buffer threshold, and receive / transmit signal integrity.

[0092] Other exporters included in the pod 1104 include a server chassis exporter 1316, a node exporter 1318, and a black box exporter 1320. The server chassis exporter 1316 collects low-level metrics from the server chassis. The node exporter 1318 collects operating system level metrics. The black box exporter 1320 collects metrics related to Hyper Text Transfer Protocol (HTTP) / Transmission Control Protocol (TCP) endpoints.

[0093] Some exporters (device mapper 1310 and storage 1312) are only used in storage nodes 1302 because they only monitor metrics on solid state drives 404. As shown in FIG. 13, non-storage nodes 1304 (e.g., compute servers 112, utility servers 118, infrastructure servers 116, and non-storage accelerator servers 114) do not include device mapper 1310 and storage 1312 exporters. Data management platform 100 can detect and react to such events by redirecting traffic to application instances on healthy nodes based on metrics from the exporters (device mapper, solid state drive, memory bandwidth, and network interface controller).

[0094] If all nodes in data management platform cluster 200 are healthy, there is no limit to spawning application instances on multiple nodes. In that case, traffic from the network operates as described in connection with FIG. 12. If one of the exporters detects metrics that indicate a node has a hardware problem, the LMA (Logging, Monitoring, and Alerting) in pod 1104 takes action to exclude the unhealthy node and block network traffic as described in connection with FIG. 13.

[0095] FIG. 14 is a flow diagram illustrating a method for managing a hardware failure of the data management platform 100 in a physical cluster 200.

[0096] In block 1400, the exporters described in connection with Figure 13 (network interface exporter 1306, memory bandwidth exporter 1308, device mapper exporter 1310, storage exporter 1312, and other exporters 1314) continuously monitor metrics for nodes 1100a-e. LMAs in pods 1104 of nodes 1100a-e collect the metrics from the exporters.

[0097] At block 1402, if all metrics collected from the exporters are good, processing continues to block 1404. Otherwise, processing continues to block 1410.

[0098] In block 1404, if the node 1100a-e is operational, the node is marked as operational and all metrics are good, indicating the node 1100a-e is operating without errors. If the node recovers from a non-operational state and was previously marked as non-operational, the node is marked as operational.

[0099] In block 1406, all nodes 1100a-e in the rack 106 are operational. The RIB 1112 is updated to restore routes to the previously non-operational nodes 1100a-e and to restore traffic to application instances on the recovered nodes 1100a-e.

[0100] At block 1408, traffic is resumed to all application instances on the operational nodes 1100a-e in the rack 106. Processing continues at block 1400.

[0101] In block 1410, if all of the metrics collected from the exporters are not good, indicating a failure or degradation event in node 1100a-e, node 1100a-e is marked as non-operational. Anycast service advertisements and ECMP forward access to the application via other application instances on other nodes 1100b-e.

[0102] In block 1412, the data network is not available to the inactive node 1100a and access to the application instances on the inactive node 1100a is not available. The RIBs of all nodes in the cluster are updated.

[0103] At block 1414, traffic to the application instances on the active nodes 1100b-e is resumed for all active nodes 1100b-e in the rack 106. Traffic to the application instances is not sent to the inactive node 1100a. Processing proceeds to block 1400.

[0104] Current load balancers (software or hardware), along with health checks, scripting or monitoring systems, do not dynamically react when performance thresholds are exceeded (e.g., central processing unit (CPU) utilization exceeds 90%). Closer monitoring and management of these performance thresholds will enable deployments to meet more stringent workload demands, including increasingly stringent performance service-level agreements (SLAs) to support new workloads such as network functions virtualization (NFV).

[0105] In one embodiment, dynamic and transparent scaling in response to performance thresholds that provide indications of stress and performance degradation is done per Kubernetes service level based on defined performance thresholds, allowing dynamic detection and transparent service scaling based on triggered performance thresholds, enabling a more optimized and scalable Kubernetes implementation.

[0106] As mentioned above, the Kubernetes control plane is hosted on the infrastructure servers 116, and the Kubernetes host agent runs on all compute servers 112 and accelerator servers 114. Direct and efficient exposure of hardware events and measurements is provided along with a unified interface to the Routing and Information Base (RIB) 1112. Examples of hardware events and measurements include telemetry such as raw sensor data exposed through the Linux “libsensors” library via the “sysfs” interface. Nodes are Kubernetes worker machines, formerly known as minions. Nodes can be virtual machines or physical machines (servers), depending on the cluster. Each node contains the services required to run pods 1104. Kubernetes services on a node include the container runtime (software that runs containers and manages container images on the node), the kubelet (which runs pods 1104), and the kube-proxy (a network proxy that runs on each node in the cluster and implements part of the Kubernetes services that maintain network rules on the node).

[0107] FIG. 15 is a block diagram illustrating hardware events and measurements in a pod 1104 of nodes 1500 (FIG. 11) that are used by the data management platform 100 to monitor and manage performance thresholds 1502 to detect node status and failures.

[0108] Exposure of hardware events and measurements is provided through pods 1104 and IP anycast event detectors and monitors 1504. Exposure of hardware events and measurements allows for selectively allowing or suppressing dynamic routes from servers (nodes of data management platform 100) based on defined performance thresholds such as CPU utilization. These thresholds can be set before or during runtime.

[0109] Examples of performance thresholds include CPU utilization, Input / Output Operation IOPS per second of solid state drive 404, or bandwidth utilization. Performance thresholds are associated with specific Kubernetes services, providing per-Kubernetes service granularity. After a performance threshold event is detected or a metric is matched, event detectors and monitors 1504 blackhole routes associated with the service on the affected node.

[0110] In networking, a black hole refers to a place in a network where incoming or outgoing traffic is silently discarded (or "dropped") without informing the source that the data did not reach its intended recipient. Black holes themselves are not visible when examining the topology of a network and can only be detected by monitoring the lost traffic.

[0111] The utilization threshold specifies a percentage of a resource over a set period of time. For example, if the resource is the bandwidth of the NVMe interface to the solid-state drive 404, the bandwidth utilization threshold may be 75% of the maximum bandwidth of the NVMe interface of the solid-state drive 404 (e.g., 25 gigabits per second). If the utilization threshold is met by the filtering system 1114, the route associated with that given service is withdrawn. In an embodiment using IP anycast, the route associated with the failed node service is withdrawn, triggering removal of the route from connected peers. Flows are transparently redirected to nodes that are within the "operational range".

[0112] A triggered utilization threshold specifies a percentage of a resource that, if exceeded for a set period of time, triggers a threshold notification. Each node has a set of routes that it advertises. The set of routes can be viewed in the node's route table 1102.

[0113] In the absence of triggered performance thresholds, all active routes are advertised (default mode of operation). The connected data switch 206 sees one anycast IP address, e.g., anycast IP address (10.1.1.1), and five paths are available to reach the destination, in this case a Kubernetes service, via one of the five nodes 1100a-e. A load balancing method, e.g., ECMP, can be used to select one of the paths.

[0114] During a triggered performance threshold event, the affected node suppresses advertisements of routes associated with the affected service. The node suppresses advertisements of routes by blackholing routes associated with the affected node's service.

[0115] Referring to FIG. 12, node 1100a has a matching performance threshold event and suppresses (never advertises) or withdraws the route associated with the affected service. In this example, the address is anycast IP address 10.1.1.1. The route (path) is withdrawn, i.e., node 1100a stops advertising the path for anycast IP address 10.1.1.1, and data switch 206 removes the path through node 1100a for anycast IP address 10.1.1.1 from route table 1102. Of the five available paths to that address, the path to node 1100a (destination 10.1.1.1, next hop 192.168.1.1) is removed from route table 1102 of data switch 206. This removes the connection to the affected node. All remaining traffic flows are distributed to the remaining paths.

[0116] After the “performance affected” node (in this case, node 1100a) functions normally, node 1100a automatically advertises the next hop (the Ethernet address (destination 10.1.1.1, next hop 192.168.1.1) associated with the previously failed service). Node 1100a is transparently reinserted into the cluster by removing the previously installed blackhole route associated with the service on node 1100a. The data switch 206 detects the blackholed (i.e., suppressed and not advertising another path for that address (destination 10.1.1.1, next hop 192.168.1.1)) route and adds it to the route table 1102 as a multipath destination.

[0117] FIG. 16 is a flow diagram illustrating a method implemented in a storage node 400 in a rack 106 for monitoring the performance of the storage node 400.

[0118] In block 1600, the event detector and monitor 1504 described in connection with FIG. 15 continuously monitors the performance thresholds 1502 within the node.

[0119] At block 1602, if the performance threshold does not match the predetermined threshold maximum, processing continues at block 1604. If they match, processing continues at block 1610.

[0120] If the node meets the service level agreement parameters, the node is marked as compliant, at block 1604. If the node was previously marked as non-compliant, the node is marked as compliant.

[0121] In block 1606, all nodes in the cluster are now compliant and the Routing Information Base (RIB) is updated to restore routes to previously non-compliant nodes and to restore traffic to application instances on such nodes.

[0122] Traffic is resumed to all application instances on compatible nodes in the cluster at block 1608. Processing continues to block 1600.

[0123] At block 1610, if the node does not meet the SLA parameters, the node is marked as non-compliant.

[0124] The data network is unavailable to the non-compliant nodes, and access to the application instances is unavailable, in block 1612. The RIBs of all nodes in the cluster are updated.

[0125] Traffic is resumed to all compliant nodes in the cluster at block 1614. Traffic is not sent to non-compliant nodes. Processing continues at block 1600.

[0126] Returning to FIG. 2, as previously mentioned, each of the servers (compute servers 112, accelerator servers 114, infrastructure servers 116, and utility servers 118) can also be referred to as a node. The orchestrator / scheduler 102 manages a fixed number of nodes. The number of nodes is selected to accommodate peaks in traffic in the datacenter and is typically over-provisioned. In current datacenters, when a workload is under pressure, the orchestrator / scheduler 102 can throttle the workload or prevent the scheduling of additional workload on the nodes where the workload is under pressure which would degrade the performance of the datacenter.

[0127] Typically, manual datacenter resizing is performed when the load in a datacenter reaches capacity in terms of CPU, memory, or storage. Datacenter resizing involves adding, provisioning, and configuring new nodes. When the load decreases, the datacenter is further overprovisioned.

[0128] In one embodiment, the total cost of ownership (TCO) of a data center can be lowered by reducing oversubscription of resources in the data center by monitoring various stress conditions within the data center managed by the orchestrator and requesting resizing of existing nodes with additional logical resources.

[0129] 17 is a block diagram of an embodiment of a compute node 1704. The compute node 1704 includes a system-on-chip (SOC or SoC) 604 that integrates a processor, memory, and input / output (I / O) control logic into a single SoC package. The SoC 604 includes at least one central processing unit (CPU) module 608 and a memory controller 614.

[0130] In the illustrated embodiment, the SoC 604 also includes an internal graphics processing unit (GPU) 1700. The internal GPU 1700 may include one or more GPU cores and a GPU cache that may store graphics-related data for the GPU cores. The GPU cores may include one or more execution units and one or more instruction and data caches internally. Additionally, the internal graphics processing unit (GPU) 1700 may include other graphics logic units not shown in FIG. 17, such as one or more vertex processing units, rasterization units, media processing units, and codecs.

[0131] In other embodiments, the memory controller 614 may be external to the SoC 604. The CPU module 608 includes at least one processor core 102 that includes a level 1 (L1) and a level 2 (L2) cache 620, and a level 3 (L2) cache 606 that is shared with other processor cores 102 in the CPU module 608.

[0132] In one embodiment, memory 630 is a volatile memory. In yet another embodiment, memory 630 includes both byte addressable write-in-place NVM devices and volatile memory devices that may be included in one or more memory modules. Resource manager agents 1706 and workloads 1708 are stored in memory 630.

[0133] The compute node 1704 also includes persistent memory 1702. The persistent memory 1702 may include a byte-addressable write-in-place three-dimensional cross-point memory device, or other byte-addressable write-in-place non-volatile memory device, or other memory. An example of a byte-addressable write-in-place three-dimensional cross-point memory device is 3DXPoint (e.g., Intel® Optane® and Micron® QuantX®).

[0134] 18 is a block diagram of another embodiment of a compute node 1804. The compute node 1804 includes a system-on-chip (SOC or SoC) 604 that integrates a processor, memory, and input / output (I / O) control logic into a single SoC package. The SoC 604 includes at least one central processing unit (CPU) module 608 and a memory controller 614.

[0135] The compute node 1804 also includes a field programmable gate array (FPGA) 1800 and an accelerator 1802 communicatively coupled to an input / output (I / O) subsystem 612 within the SoC 604. In one embodiment, the FPGA 1800 is an Intel® Agilex® FPGA device.

[0136] 19 is a block diagram of an embodiment of a rack 106 in the data management platform 100 in the physical cluster shown in FIG. 2 including a resource manager 1950 for automatically adding and removing logical resources. The rack 106 includes multiple compute nodes 1904-1, 1904-2, 1904-3 and an accelerator node 1902. The compute nodes 1904-1, 1904-2, 1904-3 may include the compute nodes 1704 described in connection with FIG. 17 or the compute nodes 1804 described in connection with FIG. 18.

[0137] The resource manager 1950 monitors the metrics to determine when to automatically attach and configure logical resources. In one embodiment, the resource manager 1950 is in the orchestrator / scheduler 102. In other embodiments, the resource manager 1950 is included in one of the compute nodes 1904-1, 1904-2, 1904-3, or another component in the data management platform 100. The resource manager 1950 has access to the accelerator nodes 1902 and all of the metrics of the data management platform 100. The resource manager 1950 in the data management platform 100 automatically attaches, detaches, and configures logical resources (e.g., memory, storage volumes, graphics processing units (GPUs), and field programmable gate array (FPGA) logical resources) without user intervention.

[0138] 19, there are three compute nodes 1904-1, 1904-2, 1904-3 and one accelerator node 1902 in a rack 106. The orchestrator / scheduler 102 monitors the workloads and processes on each of the compute nodes 1904-1, 1904-2, 1904-3 in the rack 106.

[0139] The data management platform 100 includes orchestrator metrics 1911, node metrics 1910, and workload metrics 1912. The orchestration metrics 1911 are managed by the orchestrator / scheduler 102. The workload metrics 1912 are accessible by the resource manager 150. The workload metrics 1912 may be exposed by the workload and queried by the resource manager 1950, or by the metrics aggregator 1914, which is queried by the resource manager 1950. The node metrics 1910 are exposed by the node exporter (e.g., github.com / prometheus / node_exporter). The node metrics 1910 may be queried by a resource manager 1950 or by a metrics aggregator 1914 , which is queried by a resource manager 1950 .

[0140] The orchestrator / scheduler 102 stores basic node metrics in node metrics 1910. The basic node metrics include the number of workloads per compute node, the number of processes per compute node, the pressure state, the CPU utilization per compute node, and the memory utilization per compute node. The pressure state indicates whether the compute node 1904 is under pressure.

[0141] A compute node 1904 is under pressure if the compute node 1904 is experiencing high resource utilization that impacts performance of the workload 1708 executing on the compute node 1904. To determine if a compute node 1904 is under pressure, additional node metrics are monitored and stored in node metrics 1910. The additional node metrics include CPU utilization per process, memory bandwidth utilization per process, memory utilization per process, storage latency per process, storage utilization per process, storage input / output per second per process, GPU and / or FPGA utilization per process, and GPU and / or FPGA latency per process.

[0142] The orchestrator / scheduler 102 also monitors and stores workload metrics 1912. The workload metrics 1912 include the number of clients, the average response latency, and percentile metrics. An example of a percentile metric is the 99th percentile latency or the 99.9th percentile latency, which is the maximum latency for 99% or 99.9% of the workload.

[0143] The resource manager 1950 aggregates metrics (node ​​metrics 1910, workload metrics 1912, and orchestrator metrics (1911)) to detect stress conditions when they occur. The resource manager 1950 also aggregates metrics to detect stress conditions before they occur. Time series analysis algorithms can be used to detect stress conditions before they occur. Time series analysis algorithms include Markov sequence / chain algorithms or artificial intelligence algorithms (e.g., neural networks or genetic algorithms). Additionally, the resource manager 1950 aggregates metrics to detect which resources (memory / disk / GPU / FPGA) are under stress and requests the addition of more resources to one or more of the compute nodes 1904-1, 1904-2, 1904-3.

[0144] FIG. 20 is a flow diagram of a method for automatically adding or removing logical resources in response to detecting a squeeze in a rack 106 in the data management platform 100 shown in FIG.

[0145] In general, pressure detection relies on multiple input sources. Pressure detection can be based on events occurring post-factum, e.g., 50th, 99th, or 99.9th percentile latency spikes (workload metrics). Pressure detection can also be based pre-factum, i.e., before a 50th, 99th, or 99.9th percentile latency spike is detected, based on increased resource utilization in compute nodes 1904-1, 1904-2, 1904-3 and workloads 1708-1, ..., 1708-9.

[0146] If the pressure detection is post-hoc, orchestrator metrics 1911 and node metrics 1910 are used to detect compute nodes 1904-1, 1904-2, 1904-3 and resources associated with the pressure detection. If the pressure detection is pre-hoc, a prediction is made that pressure exists based on increased resource utilization at nodes 1904-1, 1904-2, 1904-3 and workloads 1708-1, ..., 1708-9.

[0147] In block 2000, the resource manager 1950 monitors system metrics. The system metrics monitored include orchestrator metrics 1911, node metrics 1910, and workload metrics 1912.

[0148] At block 2002, a stress state occurs when compute nodes 1904-1, 1904-2, 1904-3 are under stress. While monitoring system metrics, resource manager 1950 can detect whether the status of the stress state is active or inactive. A stress state is active if a stress state is about to occur, if a stress state is about to end, or if a stress state is in progress. If an active stress state is detected by resource manager 1950, processing proceeds to block 2004. Otherwise, processing proceeds to block 2000 to continue monitoring metrics.

[0149] At block 2004, an active stress condition is detected. Examples of a stress condition are 99th percentile latency or 99.9th percentile latency spikes (workload metrics). Applications running on compute nodes 1904-1, 1904-2, 1904-3 that are affected by the active stress condition are determined. Processing proceeds to block 2006.

[0150] At block 2006, the compute nodes 1904-1, 1904-2, 1904-3 affected by the active squeeze state are determined.

[0151] In block 2008, the detected active compression state may be a compression state about to occur, about to end, or in progress. If a compression state is about to occur or is in progress, processing proceeds to block 2012. If a compression state is about to end, processing proceeds to block 2010.

[0152] If the stress state is about to end, logical resources are removed from the compute nodes 1904-1, 1904-2, 1904-3 at block 2010. Processing continues at block 2000 to continue monitoring the metrics.

[0153] At block 2012, if a stress condition is about to occur or is in progress, logical resources are added to the compute nodes 1904-1, 1904-2, 1904-3. In one embodiment, more logical resources are added to the compute nodes 1904-1, 1904-2, 1904-3 that are experiencing the stress condition. The additional logical resources may be used by all workloads 1708 on the compute nodes 1904-1, 1904-2, 1904-3, or may be restricted to be used only by specific workloads 1708 on the compute nodes 1904-1, 1904-2, 1904-3. Logical resources that may be added to relieve stress include storage, memory, accelerators, and field programmable gate array (FPGA) resources.

[0154] A pressure condition on a disk (e.g., solid state drive 404 (FIG. 4)) can be due to a lack of disk space or increased input / output latency to the disk. Upon detecting a pressure condition on the disk (post-mortem or pre-mortem), resource manager 1905 requests accelerator node 1902 to create a new volume and logically attach the newly created volume to each of compute nodes 1904-1, 1904-2, 1904-3. Resource manager agent 1706 on each of compute nodes 1904-1, 1904-2, 1904-3 performs file system expansion on the newly created volume and directly mounts the newly created volume for the workload 1708 running on one of compute nodes 1904-1, 1904-2, 1904-3.

[0155] Pressure on memory (e.g., memory 630 or persistent memory 1702 (FIG. 17)) may result from high memory bandwidth usage, low free memory on compute nodes 1904-1, 1904-2, 1904-3, or spikes in memory usage of workload 1708 on compute nodes 1904-1, 1904-2, 1904-3. Resource manager 1950 can allocate new pools of memory to compute nodes 1904-1, 1904-2, 1904-3 using persistent memory 1702 (FIG. 17), simple storage service (S3) endpoints, or remote solid state drives 404 in storage node 400. Simple storage services can be accessed via network interface controllers 302 on compute nodes 1904-1, 1904-2, 1904-3. The Simple Storage Service presents one endpoint with multiple remote drives accessible through the network interface controller 302. The Simple Storage Service is an Application Programming Interface (API) that provides object storage through a web services interface. The Amazon® Simple Storage Service is the de facto standard in object storage solutions. Examples of interfaces compatible with the Amazon Simple Storage Service include Ceph RADOS Gateway, OpenIO, Scality, and MinIO. In one embodiment, the Simple Storage Service is provided by MinIO. The remote solid state drives 404 are on the storage node 400 in the same rack 106 as the compute nodes 1904-1, 1904-2, 1904-3 and the workload 1708.

[0156] The new pool of memory is accessible for utilization by the compute nodes 1904-1, 1904-2, 1904-3 via the resource manager agent 1706. The resource manager agent 1706 maps the new pool of memory as an extension of memory already allocated to the compute nodes 1904-1, 1904-2, 1904-3. The new pool of memory allocated in persistent memory 1702 is directly accessible by the workloads 1708. The new pool of memory allocated by the simple storage service is exposed to the workloads 1708 via kernel functions (e.g., the “userfaultfd” function) that enable on-demand paging from user space 500.

[0157] The pool of newly allocated memory in the persistent memory 1702 or the remote solid-state drive 404 is used as a warm tier of logical memory, and the memory 630 contains volatile memory and is a cache for the warm tier of logical memory. The local memory 630 is a hot tier of logical memory with low latency and high bandwidth. The persistent memory 1702 has a larger capacity, higher latency, and lower bandwidth than the memory 630. The solid-state drive 302 has a larger capacity, higher latency, and lower bandwidth than the persistent memory 1702.

[0158] A stressed state for an accelerator 1802 or FPGA 1800 results in a workload 1708 being affected by a lack of sufficient resources in the accelerator 1802 or FPGA 1800. Job queues for resources of the accelerator 1802 or FPGA 1800 used by one or more workloads 1708 may fill during the stressed state.

[0159] In one embodiment, the resource manager 1850 detects an increase in percentile latency due to CPU processing stalls. For example, if the requested data is not in the CPU cache, the requested data must be fetched from distant memory or storage. This results in a variance in the average response time (i.e., deviation from the average). In response to detecting the latency increase, the resource manager 1850 requests new accelerator server or FPGA resources from the storage node 404 using a Remote Direct Memory Access (RDMA) based communication protocol.

[0160] Examples of RDMA-based communication protocols include NVMeOF (NVM Express over Fabric) or FPGA resources over Fabric (e.g., FPGAs accessible using RDMA over Fabric). NVM Express over Fabric defines a common architecture that supports a set of storage networking fabrics for NVMe block storage protocols over storage networking fabrics. This includes enabling front-side interfaces to storage systems, scaling out to large numbers of NVMe devices, and extending the distances within the data center over which NVMe devices and NVMe subsystems can be accessed.

[0161] The new logic accelerator or FPGA is connected to one of the compute nodes 1904-1, 1904-2, 1904-3 and used by the workload 1708. Processing continues to block 2000 to continue monitoring the metrics.

[0162] Self-healing storage layer A shared-nothing architecture (SN) is a distributed computing architecture in which an update request is satisfied by a single node. The node can be a compute node, a memory node, or a storage node. The intention is to eliminate contention between nodes. Each node accesses memory and storage independently. Nodes do not share memory or storage.

[0163] Shared-nothing architecture systems can be scaled by adding more nodes because there is no central resource that bottlenecks the system. Another term for shared-nothing architecture is sharding. A database shard is a horizontal partition of data in a database or search engine. An individual partition is called a shard or database shard. Each shard is stored on a separate database server instance to distribute the load.

[0164] Distributed applications used in systems with a shared-nothing architecture require their shards stored on database server instances to be persistent. Examples of distributed applications with a shared-nothing architecture include Structured Query Language (SQL) databases, Simple Storage Service (S3) object stores, and time-series databases. Structured Query Language is a domain-specific language used in programming, designed to manage data stored in a relational database management system (RDBMS) or for stream processing in a relational data stream management system (RDSMS).

[0165] A failure of a database server instance or a group of database server instances affects users of the database server. The failure may increase the latency of requests for data stored in the database server instance or cause requests for data stored in the database server instance to fail. Furthermore, recovery from a failure may take time and resources because data associated with the failed database server instance or group of database server instances must be restored.

[0166] Manual intervention is required to perform database server instance recovery with knowledge of the application topology and the failure domains in which the application is deployed. In a scale-out architecture, an application consists of several processes, each running in a Kubernetes pod. These pods are distributed across failure domains, i.e. racks in the data management platform 100, such that failures do not affect the availability of the application or the durability of the data the application manages. The distribution of these pods is the application topology.

[0167] In one embodiment, the storage self-healing mechanism, which may also be referred to as storage self-healing logic or circuitry, monitors the storage subsystem and monitors the workloads that use the storage subsystem (storage nodes and solid-state drives) to ensure that all workloads are distributed across available failure domains.

[0168] 21 is a block diagram of one embodiment of a physical cluster 2100 in a data management platform 100 that includes a storage self-healing mechanism 2108. The physical cluster 2100 includes an orchestrator / scheduler 102 and a rack 106. In one embodiment, the storage self-healing mechanism 2108 is in the orchestrator / scheduler 102. In other embodiments, the storage self-healing mechanism may be in another component of the data management platform 100.

[0169] 21, there is one rack 106 having one data switch 206, three compute nodes 2110a-c, and two storage nodes 2102a-b. The compute nodes 2110a-b and the storage nodes 2102a-b are communicatively coupled to the data switch 206.

[0170] The physical cluster 2100 has multiple failure domains for applications that use the storage subsystem (storage nodes 2102a-b and solid state drives 2106a-d). The first failure domain is the data switch 206, the second failure domain is the compute nodes 21000a-b on which the workloads 2104a-c run, the third failure domain is the storage nodes 2102a-b, and the fourth failure domain is the solid state drives 2106a-b.

[0171] The storage self-healing mechanism 2108 periodically performs health checks on each of the multiple failure domains. One of the health checks performed by the storage self-healing mechanism for the data switch 206 is to determine whether the orchestrator / scheduler 102 can access the data switch 206. In one embodiment, the Internet Control Notification Protocol (ICMP) can be used to determine whether the orchestrator 102 can access the data switch. For example, an ICMP echo request / reply or an extended echo request / reply message can be used to determine whether the orchestrator / scheduler 102 can access the data switch 206. The Internet Control Notification Protocol is an error reporting protocol and an extension of the Internet Protocol (IP) defined by Request for Comments (RFC) 792.

[0172] Another health check performed by the storage self-healing mechanism 2108 for the data switch 206 is to determine whether routes are available at the data switch. In one embodiment, the "ip rg" command (a Linux utility command) checks whether a route to a compute node 2100a-c or storage node 2102a-b bound to a specified IP address is visible, and whether a route to a pod is visible.

[0173] One of the health checks performed by the storage self-healing mechanism 2108 on the compute nodes 2100a-c or storage nodes 2102a-b is to determine whether the compute nodes 2110a-c or storage nodes 2102a-b are reachable through the data switch 206. In one embodiment, Internet Control Message Protocol (ICMP) may be used to determine whether the compute nodes 2110a-c or storage nodes 2102a-b have access to the data switch.

[0174] Another health check for the compute nodes 2110a-c or storage nodes 2102a-b performed by the storage self-healing mechanism 2108 is to check whether the orchestrator 102 reports the compute nodes 2110a-c or storage nodes 2102a-b as ready. The orchestrator 102 checks the health of the compute nodes 2110a-c or storage nodes 2102a-b and reports whether the respective compute nodes 2110a-c or storage nodes 2102a-b are ready to accept workload. The health of the compute nodes 2110a-c or storage nodes 2102a-b may include memory and CPU checks from an operating system perspective, as well as network connectivity between the orchestrator and the respective compute nodes 2110a-c or storage nodes 2102a-b.

[0175] One of the health checks performed by the storage self-healing mechanism 2108 on the solid-state drives 2106a-d is write amplification. For example, write amplification ratios, free user space and over-provisioning can be used to predict failures in the solid-state drives 2106a-d. Another health check performed by the storage self-healing mechanism 2108 on the solid-state drives 2106a-d is to check the health of the solid-state drives 2106a-d.

[0176] The health of the solid state drives 2106a-d can be monitored using Self-Monitoring, Analysis and Reporting Technology (SMART). SMART is a monitoring system included in solid state drives that monitors and reports indicators of the reliability of the solid state drive and can be used to take preventative measures to prevent data loss. Examples of SMART metrics for the NAND-based solid state drives 2106a-d include program failures, erase failures, wear leveling, end-to-end error detection, cyclic redundancy code (CRC) error count, timed workload-media wear, timed workload-host, read / write ratio, timed workload timer, thermal throttle status, retry buffer overflows, PLI lock loss, NAND byte writes, host byte writes, and system area life remaining.

[0177] In a scaled-out shared-nothing architecture, a workload has multiple instances. The minimum number of acceptable failed instances is workload specific. Simple Storage Service uses erasure coding and checksums to protect data against hardware failures and silent data corruption.

[0178] Erasure correction codes are mathematical algorithms for reconstructing lost or corrupted data. Simple Storage Service divides objects into data blocks and parity blocks. With eight data blocks and four parity blocks, data can be recovered in the presence of up to four instance failures. In a database with three replicas (copies of the database), up to two instances can recover data.

[0179] The self-healing mechanism 2108 can detect incoming failures and can trigger and schedule regeneration of data stored on failed solid-state drives 2106a-d. The storage self-healing mechanism 2108 can trigger automated actions on the storage subsystem (storage nodes 2102a-b and solid-state drives 2106a-d) using metrics obtained via the health checks described above.

[0180] Figure 22 illustrates an embodiment of a mapping of workloads in the cluster 2100 shown in Figure 21. The storage self-healing mechanism 2108 tracks the mapping of workloads 2104a-c to storage subsystems. For example, the mapping tracks the solid state drives 2106a-d to which logical volumes are mapped.

[0181] As shown in Figure 22, Workload A (Instance 1) is mapped to Compute 1 in Rack 1, Accelerator 1, Solid State Drive 1, Volume A.

[0182] In a system including the storage self-healing mechanism 2108, no administrator / operator involvement is required to recover an application, and the latency and bandwidth impact of a failure of a clustered application instance on overall system performance is also reduced.

[0183] Upon detecting a failure of the data switch 206 or a failure associated with all of the storage nodes 2102a-b and compute nodes 2110a-c in a rack 106, multiple workloads are affected. The storage self-healing mechanism 2108 detects the workloads affected by the failure. Another rack 106 is selected from the available racks in the physical cluster 2100. The storage self-healing mechanism 2108 in the orchestrator 102 disconnects the volumes on the solid state drives and removes the volumes from the failed compute and storage nodes via the NVMe over Fabric interface.

[0184] Resources for the workloads that were running on the compute nodes of the failed rack are created in the other rack. The workloads are rescheduled to run on the compute nodes of the other rack. After the rescheduled workload instances are rescheduled, the storage self-healing mechanism 2108 in the orchestrator 102 triggers the workload "repair / recovery" mechanism of the workloads for all data on the volumes of the other rack after the storage is recreated in the other rack, and repairs the blocks upon detecting errors in the respective blocks.

[0185] When the storage self-healing mechanism 2108 detects a compute node failure or a compute node reports a failure, the storage self-healing mechanism 2108 detects which workloads were affected on the failed compute node. The affected workloads are rescheduled to run on another compute node in the same rack that is not already hosting an instance of that workload. The orchestrator 102 storage self-healing mechanism 2108 requests the disconnection of the volume from the failed compute node and the connection of the volume to the other compute node. In an embodiment where the communication path between the solid state drives and the storage node is through NVMeOF, the disconnect and connect requests are sent through the NVMeOF interface. After the workload instance is resumed on the other compute node, the orchestrator 102 triggers the workload "repair / recovery" mechanism for all of the data on the volume to repair any blocks in error.

[0186] If a failure of a storage node 2102a-b affecting multiple workloads and data on solid-state drives cannot be recovered, the storage self-healing mechanism 2108 in the orchestrator 102 determines the affected workloads and volumes used by the failed storage node, reschedules all affected workloads onto different compute nodes 2110a-c in the rack 102, creates new volumes on the solid-state drives of another storage node 2100a-b, and connects the new volumes on the solid-state drives to the new compute node 2110a-c via the NVMeOF interface.

[0187] If the compute nodes 2110a-c in the rack 106 are already hosting the same type of workload, the orchestrator 102 selects another rack 106 in the physical cluster 2100, and the orchestrator 102 reschedules all affected workloads to run on the compute nodes 2100a-c in the other rack 102. The orchestrator 102 creates new volumes on the solid-state drives of another storage node 2100a-b in the other rack 106 and connects them to the new compute nodes 2110a-c in the other rack 106 via the NVMeOF interface. After the workload instance is resumed on the other compute node, the orchestrator 102 triggers the workload "repair / recovery" mechanism for all data on the volume and repairs any blocks in error.

[0188] When one or more solid-state drives of a storage node 2102a-d fail, multiple workloads are affected. The storage self-healing mechanism 2108 determines the affected workloads, i.e., the workloads using logical volumes on the failed solid-state drives. The storage self-healing mechanism 2108 creates new volumes on other operational solid-state drives in the same storage node 2102a-b or in another storage node 2102a-b in the same rack 106. The new volumes are attached to the compute node via the NVMeOF interface and the old volumes are detached. After the workload instance is resumed on the other compute node, the orchestrator 102 triggers the workload “repair / recovery” mechanism for all data on the volumes to repair any blocks in error.

[0189] The flow diagrams shown herein provide examples of sequences of various process actions. The flow diagrams may depict operations performed by software or firmware routines as well as physical operations. In one embodiment, the flow diagrams may depict states of a finite state machine (FSM) that may be implemented in hardware and / or software. Although shown in a particular sequence or order, unless otherwise specified, the order of actions may be changed. Thus, the illustrated embodiments should be understood as examples, and processes may be performed in different orders and some actions may be performed in parallel. Additionally, one or more actions may be omitted in various embodiments, and thus not all actions are required in all embodiments. Other process flows are possible.

[0190] To the extent that various operations or functions are described herein, they may be described or defined as software code, instructions, settings, and / or data. The content may be directly executable ("object" or "executable" format), source code, or differential code ("delta" or "patch" code). The software content of the embodiments described herein may be provided via an article of manufacture in which the content is stored, or via a method of operating a communications interface to transmit data through the communications interface. A machine-readable storage medium may cause a machine to perform the described functions or operations and includes any mechanism for storing information in a form accessible by a machine (e.g., computing device, electronic system, etc.), such as recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). A communications interface may include any mechanism for interfacing with either hardwired, wireless, optical media, etc. to communicate with another device, such as a memory bus interface, a processor bus interface, an Internet connection, a disk controller, etc. The communications interface may be configured by providing configuration parameters and / or sending signals to prepare the communications interface for providing data signals describing the software content. The communications interface may be accessed via one or more commands or signals sent to the communications interface.

[0191] Various components described herein may be means for performing the described operations or functions. Each component described herein includes software, hardware, or a combination thereof. A component may be implemented as a software module, a hardware module, dedicated hardware (e.g., application specific hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), etc.), an embedded controller, a hardwired circuit, etc.

[0192] In addition to those described herein, various modifications can be made to the disclosed embodiments and implementations of the invention without departing from the scope of the invention.

[0193] Therefore, the illustrations and examples herein should be construed in an illustrative rather than a limiting sense, and the scope of the present invention should be determined solely by reference to the claims that follow.

[0194] Generally, with respect to the description herein, in one example, an apparatus includes a compute server and a storage server managing a plurality of storage devices communicatively coupled to the storage server, where the compute server and the storage server are communicatively coupled via a network, and where the plurality of storage devices managed by the storage server are separate from the compute server such that the storage capacity of the plurality of storage devices can be scaled independently of the compute server.

[0195] In one example, the network interface controller is communicatively coupled to a network, and the system-on-chip comprises a plurality of cores and a last level memory, the plurality of cores are communicatively coupled to a last level cache memory, the last level cache memory comprises a plurality of cache ways, and a portion of the plurality of cache ways are allocated for exclusive use by logical volumes in a plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller.

[0196] In one example, a portion of the multiple cache ways are allocated during initialization.

[0197] In one example, a logical volume stores data for use by a compute server.

[0198] In one example, data transferred between a logical volume and multiple cache ways in the last level cache is transferred between the storage server and the compute server via a network communicatively coupled to the network interface controller.

[0199] In one example, at least one of the plurality of cores is assigned for exclusive use by logical volumes in a plurality of storage devices and a network interface controller to transfer data between the logical volume and the network interface controller.

[0200] In one example, an external memory coupled to the system-on-chip temporarily stores data transferred between the logical volume and the network interface once all of a plurality of cache ways in a portion of the last level cache are allocated for exclusive use by the logical volume and the network interface controller to store the data transferred.

[0201] In general, with respect to the description herein, in one example, an apparatus includes a plurality of servers, each of which monitors metrics at the servers, a data switch to which the plurality of servers are communicatively coupled, the data switch having a route table including a plurality of dynamic routes to services at the plurality of servers, and a filtering system that dynamically allows or suppresses routes to services at one of the plurality of servers based on the monitored metrics.

[0202] In one example, a route to a service is removed from the route table to suppress the route.

[0203] In one example, a route to the service is added to a route table to allow the route.

[0204] In one example, the monitored metrics include metrics related to stress or performance degradation across multiple servers.

[0205] In one example, the metrics related to the stress state include workload metrics.

[0206] In one example, the metrics related to performance degradation include metrics related to memory bandwidth.

[0207] In one example, the monitored metrics include metrics related to degraded hardware components within a plurality of servers.

[0208] In one example, metrics related to degraded hardware components include node metrics, orchestrator metrics, and workload metrics.

[0209] In one example, the server is a storage server and the metrics relating to degraded hardware components include solid state drive metrics.

[0210] Generally, with respect to the description herein, in one example, a method includes monitoring metrics at a plurality of servers, storing a plurality of dynamic routes to services at the plurality of servers in a route table of a data switch, and dynamically allowing or suppressing the route to the service at one of the plurality of servers based on the monitored metrics.

[0211] In one example, a route to a service is removed from the route table to suppress the route.

[0212] In one example, a route to the service is added to a route table to allow the route.

[0213] In one example, the monitored metrics include metrics related to stress or performance degradation across multiple servers.

[0214] In one example, the metrics related to the stress state include workload metrics.

[0215] In one example, the metrics related to performance degradation include metrics related to memory bandwidth.

[0216] In one example, the monitored metrics include metrics related to degraded hardware components within a plurality of servers.

[0217] In general, with respect to the description herein, in one example, a data management platform includes a rack including a plurality of servers, each of which monitors metrics on the servers, a data switch to which the plurality of servers are communicatively coupled, the data switch including a route table including a plurality of dynamic routes to services on the plurality of servers, and a filtering system that dynamically allows or suppresses routes to services on one of the plurality of servers based on the monitored metrics.

[0218] In one example, a route to a service is removed from the route table to suppress the route.

[0219] In one example, a route to the service is added to a route table to allow the route.

[0220] In one example, the monitored metrics include metrics related to stress or performance degradation across multiple servers.

[0221] In general, with respect to the description herein, in one example, an apparatus includes a plurality of compute nodes, each of which executes a workload using logical resources, and an orchestrator communicatively coupled to the plurality of compute nodes, the orchestrator monitoring metrics to detect an active stress state in the compute nodes, and in response to detecting the active stress state, attaching new logical resources to the compute nodes while the stress state is active.

[0222] In one example, the metrics include orchestrator metrics, node metrics, and workload metrics.

[0223] In one example, the node metrics include CPU utilization per compute node and memory utilization per compute node.

[0224] In one example, the workload metrics include the number of clients, the average response latency, and percentile metrics.

[0225] In one example, the orchestrator detaches the new logical resource from the compute node when the compression state is not active.

[0226] In one example, the new resources are restricted for use only by a particular workload on the compute node while the pressure state is active.

[0227] In one example, the new resources are used by all workloads on the compute node while the pressure state is active.

[0228] In general, with respect to the description herein, in one example, an apparatus includes a data management platform comprising a rack comprising: a plurality of compute nodes, each of which executes a workload using logical resources; a storage subsystem comprising a plurality of storage nodes communicatively coupled to a plurality of storage devices; and a storage self-healing mechanism communicatively coupled to the plurality of compute nodes and the storage subsystem, the storage self-healing mechanism performing health checks on the storage subsystem, detecting failures in the storage subsystem using metrics obtained via the health checks, and managing repair of failures in the storage subsystem.

[0229] In one example, the failure is in a compute node and the storage self-healing mechanism reschedules the workload running on the failed compute node to another compute node in the rack.

[0230] In one example, the failure is in a compute node, other compute nodes in the rack are hosting the same type of workload, and the storage self-healing mechanism reschedules the workload to run on another compute node in another rack.

[0231] In one example, the failure is in a storage device and the storage self-healing mechanism determines the workload associated with the storage device to create a volume on another storage device in the storage node.

[0232] In one example, the rack further comprises a data switch, the failure is in the data switch, and the storage self-healing mechanism determines the workload in the rack, selects another rack, and reschedules the workload in the other rack. [Other possible items] [Item 1] Multiple servers, each monitoring server metrics; a data switch communicatively coupled to the plurality of servers, the data switch having a route table including a plurality of dynamic routes to services at the plurality of servers; a filtering system that dynamically allows or suppresses routes to the service on one of the plurality of servers based on the monitored metrics; An apparatus comprising: [Item 2] 2. The apparatus of claim 1, wherein the route to the service is removed from the route table to suppress the route. [Item 3] 2. The apparatus of claim 1, wherein the route to the service is added to the route table to allow the route. [Item 4] 2. The apparatus of claim 1, wherein the monitored metrics include metrics related to stress or performance degradation in the plurality of servers. [Item 5] 5. The apparatus of claim 4, wherein the metrics related to the stress state include workload metrics. [Item 6] 5. The apparatus of claim 4, wherein the metrics related to the performance degradation include a metric related to memory bandwidth. [Item 7] 2. The apparatus of claim 1, wherein the monitored metrics include metrics related to degraded hardware components within the plurality of servers. [Item 8] 8. The apparatus of claim 7, wherein the metrics associated with the degraded hardware components include node metrics, orchestrator metrics, and workload metrics. [Item 9] 8. The apparatus of claim 7, wherein one of the plurality of servers is a storage server and the metrics related to the degraded hardware components include solid state drive metrics. [Item 10] monitoring metrics on a plurality of servers; storing a plurality of dynamic routes to services at the plurality of servers in a route table of a data switch; dynamically allowing or throttling routes to the service on one of the plurality of servers based on the monitored metrics; The method includes: [Item 11] Item 11. The method of item 10, wherein the route to the service is removed from the route table to suppress the route. [Item 12] Item 11. The method of item 10, wherein the route to the service is added to the route table to allow the route. [Item 13] 11. The method of claim 10, wherein the monitored metrics include metrics related to stress or performance degradation in the plurality of servers. [Item 14] 14. The method of claim 13, wherein the metric related to the stress state comprises a workload metric. [Item 15] 14. The method of claim 13, wherein the metrics related to the performance degradation include a metric related to memory bandwidth. [Item 16] 11. The method of claim 10, wherein the monitored metrics include metrics related to degraded hardware components within the plurality of servers. [Item 17] Multiple servers, each monitoring server metrics; a data switch communicatively coupled to the plurality of servers, the data switch including a route table including a plurality of dynamic routes to services at the plurality of servers; a filtering system that dynamically allows or suppresses routes to the service on one of the plurality of servers based on the monitored metrics; A rack having A data management platform that provides: [Item 18] 20. The data management platform of claim 17, wherein the route to the service is removed from the route table to suppress the route. [Item 19] 20. The data management platform of claim 17, wherein the route to the service is added to the route table to allow the route. [Item 20] 20. The data management platform of claim 17, wherein the monitored metrics include metrics related to stress or performance degradation in the plurality of servers. [Item 21] A calculation server; and a storage server that manages a plurality of storage devices communicatively coupled to the storage server, wherein the computation server and the storage server are communicatively coupled via a network, and the plurality of storage devices managed by the storage server are separate from the computation server such that a storage capacity of the plurality of storage devices can be scaled independently of the computation server. [Item 22] The storage server, a network interface controller communicatively coupled to the network; a system-on-chip including a plurality of cores and a last level memory, the plurality of cores being communicatively coupled to the last level cache memory, the last level cache memory including a plurality of cache ways, a portion of the plurality of cache ways being assigned for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller; 22. The device of claim 21, further comprising: [Item 23] 23. The apparatus of claim 22, wherein the portion of the plurality of cache ways is allocated during initialization. [Item 24] 24. The apparatus of claim 23, wherein the logical volume stores data for use by the computational server. [Item 25] 25. The apparatus of claim 24, wherein the data transferred between the logical volume and the plurality of cache ways in the last level cache is transferred between the storage server and the compute server via a network communicatively coupled to the network interface controller. [Item 26] 23. The apparatus of claim 22, wherein at least one of the plurality of cores is assigned for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller. [Item 27] 23. The apparatus of claim 22, further comprising: an external memory coupled to the system-on-chip, the external memory temporarily storing data transferred between the logical volume and the network interface when all of the plurality of cache ways in a portion of a last level cache are allocated for exclusive use by the logical volume and the network interface controller to store the transferred data. [Item 28] managing, by a storage server, a plurality of storage devices communicatively coupled to said storage server; communicatively coupling the storage server to a computation server via a network, the plurality of storage devices managed by the storage server being separate from the computation servers such that storage capacity of the plurality of storage devices can be scaled independently of the computation servers; The method includes: [Item 29] The storage server, a network interface controller communicatively coupled to the network; a system-on-chip including a plurality of cores and a last level memory, the plurality of cores being communicatively coupled to the last level cache memory, the last level cache memory including a plurality of cache ways, a portion of the plurality of cache ways being assigned for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller; 29. The method of claim 28, further comprising: [Item 30] 30. The method of claim 29, wherein the portion of the plurality of cache ways is allocated during initialization. [Item 31] 31. The method of claim 30, wherein the logical volume stores data for use by the computational server. [Item 32] 32. The method of claim 31, wherein the data transferred between the logical volume and the plurality of cache ways in the last level cache is transferred between the storage server and the compute server via a network communicatively coupled to the network interface controller. [Item 33] 30. The method of claim 29, wherein at least one of the plurality of cores is assigned for exclusive use by a logical volume in the plurality of storage devices and the network interface controller to transfer data between the logical volume and the network interface controller. [Item 34] 30. The method of claim 29, further comprising an external memory coupled to the system-on-chip, the external memory temporarily storing data transferred between the logical volume and the network interface when all of the plurality of cache ways in a portion of a last level cache are assigned for exclusive use by the logical volume and the network interface controller to store the transferred data. [Item 35] A number of compute nodes, each of which uses logical resources to run workloads; an orchestrator communicatively coupled to the plurality of compute nodes, the orchestrator monitoring metrics to detect an active stressed state on a compute node, and in response to detecting the active stressed state, attaching new logical resources to the compute node while the stressed state is active; An apparatus comprising: [Item 36] Item 36. The apparatus of item 35, wherein the metrics include orchestrator metrics, node metrics, and workload metrics. [Item 37] Item 37. The apparatus of item 36, wherein the node metrics include CPU utilization per compute node and memory utilization per compute node. [Item 38] Item 37. The apparatus of item 36, wherein the workload metrics include number of clients, average response latency, and percentile metrics. [Item 39] Item 36. The apparatus of item 35, wherein the orchestrator detaches the new logical resource from the compute node when the pressure state is not active. [Item 40] Item 37. The apparatus of item 36, wherein the new logical resource is restricted for use only by a particular workload on the compute node while the stressed state is active. [Item 41] Item 37. The apparatus of item 36, wherein the new logical resource is used by all workloads on the compute node while the pressured state is active. [Item 42] executing a workload using the logical resources by each compute node of the plurality of compute nodes; monitoring metrics by an orchestrator communicatively coupled to the plurality of compute nodes to detect active stress conditions on the compute nodes; in response to detecting the active compression state, the orchestrator attaching new logical resources to the compute node while the compression state is active; The method includes: [Item 43] Item 43. The method of item 42, wherein the metrics include orchestrator metrics, node metrics, and workload metrics. [Item 44] Item 44. The method of item 43, wherein the node metrics include a CPU utilization rate per compute node and a memory utilization rate per compute node. [Item 45] Item 44. The method of item 43, wherein the workload metrics include number of clients, average response latency, and percentile metrics. [Item 46] Item 43. The method of item 42, wherein the orchestrator detaches the new logical resource from the compute node when the pressure state is not active. [Item 47] 43. The method of claim 42, wherein the new logical resource is restricted for use only by a particular workload on the compute node while the pressure state is active. [Item 48] Item 43. The method of item 42, wherein the new logical resource is used by all workloads on the compute node while the pressure state is active. [Item 49] A number of compute nodes, each of which uses logical resources to run workloads; a storage subsystem including a plurality of storage nodes communicatively coupled to a plurality of storage devices; a storage self-healing mechanism communicatively coupled to the plurality of compute nodes and the storage subsystem, the storage self-healing mechanism performing health checks on the storage subsystem, detecting failures in the storage subsystem using metrics obtained via the health checks, and managing repair of the failures in the storage subsystem; A rack having A data management platform that provides: [Item 50] 50. The data management platform of claim 49, wherein the failure is in a compute node and the storage self-healing mechanism reschedules workloads running on the failed compute node to another compute node in the rack. [Item 51] 50. The data management platform of claim 49, wherein the failure is in a compute node, other compute nodes in the rack are hosting the same type of workload, and the storage self-healing mechanism reschedules the workload to run on another compute node in another rack. [Item 52] 50. The data management platform of claim 49, wherein the failure is in a storage device and the storage self-healing mechanism determines the workload associated with the storage device to create a volume on another storage device within the storage node. [Item 53] 50. The data management platform of item 49, wherein the rack further comprises a data switch, the failure is in the data switch, and the storage self-repair mechanism determines the workload in the rack, selects another rack, and reschedules the workload in the other rack. [Item 54] 1. A method performed in a rack within a data management platform, comprising: executing a workload using logical resources by each of a plurality of computing nodes in the rack; performing a health check on the storage subsystem by a storage self-healing mechanism communicatively coupled to the plurality of compute nodes and a storage subsystem in the rack including a plurality of storage nodes communicatively coupled to a plurality of storage devices in the storage subsystem; detecting, by the storage self-healing mechanism, failures in the storage subsystem using the metrics obtained via the health check, and managing repair of the failures in the storage subsystem; The method includes: [Item 55] Item 55. The method of item 54, wherein the failure is in a compute node and the storage self-healing mechanism reschedules workloads running on the failed compute node to another compute node in the rack. [Item 56] Item 55. The method of item 54, wherein the failure is in a compute node, other compute nodes in the rack are hosting the same type of workload, and the storage self-healing mechanism reschedules the workload to run on another compute node in another rack. [Item 57] Item 55. The method of item 54, wherein the failure is in a storage device and the storage self-healing mechanism determines the workload associated with the storage device to create a volume on another storage device within the storage node. [Item 58] determining, in response to detecting a fault in a data switch in the rack, the workload in the rack by the storage self-healing mechanism; selecting another rack by the storage self-repair mechanism; rescheduling the workload on the other rack by the storage self-healing mechanism; 55. The method of claim 54, further comprising:

Claims

1. An apparatus comprising: A calculation server; a storage server that manages a plurality of storage devices communicatively coupled to the storage server; Equipped with the computation server and the storage server are communicatively coupled via a network, and the plurality of storage devices managed by the storage server are separate from the computation server such that storage capacity of the plurality of storage devices can be scaled independently of the computation server; The storage server, a network interface controller communicatively coupled to the network; a system-on-chip including a plurality of cores and a last level cache memory, the plurality of cores being communicatively coupled to the last level cache memory, the last level cache memory including a plurality of cache ways, a portion of the plurality of cache ways being allocated for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller; An apparatus having the above configuration.

2. 2. The apparatus of claim 1, wherein said portion of said plurality of cache ways is allocated during initialization.

3. The apparatus of claim 2 , wherein the logical volume stores data for use by the compute server.

4. 4. The apparatus of claim 3, wherein the data transferred between the logical volume and the plurality of cache ways in the last level cache memory is transferred between the storage server and the compute server via a network communicatively coupled to the network interface controller.

5. 2. The apparatus of claim 1, wherein at least one of the plurality of cores is assigned for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller.

6. 2. The apparatus of claim 1, further comprising: an external memory coupled to the system-on-chip for temporarily storing data transferred between the logical volume and the network interface controller when all of the plurality of cache ways in the portion of a last level cache are allocated for exclusive use by the logical volume and the network interface controller to store the transferred data.

7. A method comprising: managing, by a storage server, a plurality of storage devices communicatively coupled to said storage server; communicatively coupling the storage server to a computation server via a network, the plurality of storage devices managed by the storage server being separate from the computation servers such that storage capacity of the plurality of storage devices can be scaled independently of the computation servers; Including, The storage server, a network interface controller communicatively coupled to the network; a system-on-chip including a plurality of cores and a last level cache memory, the plurality of cores being communicatively coupled to the last level cache memory, the last level cache memory including a plurality of cache ways, a portion of the plurality of cache ways being allocated for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller; The method comprising:

8. 8. The method of claim 7, wherein said portion of said plurality of cache ways is allocated during initialization.

9. The method of claim 8 , wherein the logical volume stores data for use by the compute server.

10. 10. The method of claim 9, wherein the data transferred between the logical volume and the plurality of cache ways in the last level cache memory is transferred between the storage server and the compute server via a network communicatively coupled to the network interface controller.

11. 8. The method of claim 7, wherein at least one of the plurality of cores is assigned for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller.

12. 8. The method of claim 7, wherein an external memory coupled to the system-on-chip temporarily stores data transferred between the logical volume and the network interface controller when all of the plurality of cache ways in the portion of a last level cache are allocated for exclusive use by the logical volume and the network interface controller to store the transferred data.

13. Apparatus comprising means for carrying out the method according to any one of claims 7 to 12.

14. A computer program comprising a plurality of instructions, In response to being executed by the system, the system further comprises: causing a storage server to manage a plurality of storage devices communicatively coupled to said storage server; communicatively coupling the storage server to a computation server via a network; the plurality of storage devices managed by the storage server are separate from the computation servers such that the storage capacity of the plurality of storage devices can be scaled independently of the computation servers; The storage server, a network interface controller communicatively coupled to the network; a system-on-chip including a plurality of cores and a last level cache memory, the plurality of cores being communicatively coupled to the last level cache memory, the last level cache memory including a plurality of cache ways, a portion of the plurality of cache ways being allocated for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller; 1. A computer program comprising:

15. 15. The computer program product of claim 14, wherein the portion of the plurality of cache ways is allocated during initialization.

16. 16. The computer program product of claim 15, wherein the logical volume stores data for use by the compute server.

17. 17. The computer program product of claim 16, wherein the data transferred between the logical volume and the plurality of cache ways in the last level cache memory is transferred between the storage server and the compute server over a network communicatively coupled to the network interface controller.

18. 15. The computer program product of claim 14, wherein at least one of the plurality of cores is assigned for exclusive use by logical volumes in the plurality of storage devices and the network interface controller to transfer data between the logical volumes and the network interface controller.

19. 15. The computer program product of claim 14, wherein an external memory coupled to the system-on-chip temporarily stores data transferred between the logical volume and the network interface controller when all of the plurality of cache ways in the portion of a last level cache are allocated for exclusive use by the logical volume and the network interface controller to store the transferred data.

20. At least one computer readable storage medium storing a computer program according to any one of claims 14 to 19.

Images