Method and apparatus for quality of service based throttling in fabric architecture

Abstract

The invention discloses a method and apparatus for quality of service based throttling in a fabric architecture. Technologies for quality of service based throttling in a fabric architecture include a network node of a plurality of network nodes interconnected across the fabric architecture via an interconnect fabric. The network node includes a host fabric interface (HFI) configured to facilitate data transfer to / from the network node, monitor a quality of service level of resources of the network node used to process and transfer data, and detect a throttling condition based on a result of the monitored quality of service level. The HFI is further configured to generate and transmit a throttling message to one or more of the interconnected network nodes in response to the throttling condition having been detected. The HFI is additionally configured to receive a throttling message from another of the network nodes and perform a throttling action on one or more of the resources based on the received throttling message. Other embodiments are described herein.

Description

[0001] CROSS-REFERENCE TO RELATED U.S. APPLICATIONS This application claims priority to U.S. utility patent application serial number 15 / 088,948, filed on April 1, 2016, entitled “TECHNOLOGIES FOR QUALITY OF SERVICEBASED THROTTLING IN FABRIC ARCHITECTURES.” Background Art

[0002] The demand for increased computing performance and storage capacity of computing devices by individuals, researchers, and enterprises has led to various computing technologies being developed to meet these demands. For example, computing-intensive applications such as enterprise cloud-based applications (e.g., software as a service (SaaS) applications), data mining applications, data-driven modeling applications, scientific computing problem-solving applications, etc., often rely on complex, large-scale computing environments (e.g., high-performance computing (HPC) environments, cloud computing environments, etc.) to execute computing-intensive applications and store large amounts of data. Such large-scale computing environments can include thousands (e.g., enterprise systems) to tens of thousands (e.g., HPC systems) of multi-processor / multi-core network nodes connected via high-speed interconnects (e.g., interconnects using a unified fabric).

[0003] In order to implement such processor-intensive computations, various computing techniques have been implemented to distribute the workload across different network computing devices, such as parallel computing, distributed computing, etc. In support of the operation of such distributed workloads, multi-processor hardware architectures (e.g., multiple multi-core processors sharing a memory) have been developed to facilitate multiprocessing (i.e., coordinated, simultaneous processing by multiple processors) across local and remote shared memory systems using various parallel computer memory design architectures, such as non-uniform memory access (NUMA), and other distributed memory architectures.

[0004] Thus, memory requests from multiple interconnected network nodes may occupy the same shared buffer (e.g., superqueue, request table, etc.) as local memory requests of a specific network node. However, such shared buffers are limited in size (e.g., containing dozens of entries), which may cause other memory requests to be queued until data for those memory requests currently in the shared buffer is returned from the memory subsystem. As such, entries of the shared buffer tend to be occupied by those memory requests for memory that provide high latency access (e.g., memory requests received from remote network nodes) or those memory requests for memory that is being overutilized. Therefore, other requests for faster or non-congested memory (e.g., local memory requests) (i.e., memory requests that will be serviced faster) may become starved in the core because there are no available shared buffer entries available to execute the memory requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The concepts described herein are shown in the accompanying drawings as examples rather than as limitations. For simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. Where deemed appropriate, reference numerals have been repeated in the drawings to indicate corresponding or similar elements.

[0006] Figure 1 is a simplified block diagram of at least one embodiment of a system for throttling based on quality of service in a fabric architecture, the system comprising a plurality of interconnected network nodes communicatively coupled via an interconnect fabric; Figure 2 yes Figure 1 A simplified block diagram of at least one embodiment of one of the network nodes of a system; Figure 3 yes Figure 2 A simplified block diagram of another embodiment of a network node; Figure 4 It can be done by Figure 2 A simplified block diagram of at least one embodiment of an environment established by a network node; Figure 5 It can be done by Figure 2 A simplified flow chart of at least one embodiment of a method for processing a local storage request from a remote network node performed by a network node of a remote network; Figure 6 It can be done by Figure 2 A simplified flow chart of at least one embodiment of a method for accessing a memory of a remote network node performed by a network node of a remote network; Figure 7 It can be done by Figure 2A simplified flow chart of at least one embodiment of a method for generating a throttling message for external transmission to one or more remote network nodes, performed by a network node of a remote network; and Figure 8 It can be done by Figure 2 A simplified flow chart of at least one embodiment of a method performed by a network node of a remote network for processing a throttling message received from a remote network node. DETAILED DESCRIPTION

[0007] Although the concept of the present invention is susceptible to various modifications and alternative forms, its specific embodiments have been shown as examples in the drawings and will be described in detail herein. However, it should be understood that there is no intention to limit the concept of the present invention to the specific forms disclosed, but on the contrary, it is intended to cover all modifications, equivalents and alternatives consistent with the present invention and the appended claims.

[0008] References in this specification to "one embodiment," "an embodiment," "illustrative embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may or may not necessarily include the particular feature, structure, or characteristic. In addition, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other embodiments. In addition, it should be appreciated that items included in a list in the form of "at least one of A, B, and C" may mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C). Similarly, items listed in the form of "at least one of A, B, or C" may mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).

[0009] In some cases, the disclosed embodiments may be implemented using hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media (e.g., memory, data storage, etc.), which may be read and executed by one or more processors. The machine-readable storage medium may be implemented as any storage device, mechanism, or other physical structure (e.g., volatile or non-volatile memory, media disk, or other media device) for storing or transmitting information in a form readable by a machine.

[0010] In the accompanying drawings, some structural or method features are shown in a specific arrangement and / or ordering. However, it should be appreciated that such specific arrangement and / or ordering may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or ordering different from that shown in the accompanying drawings. In addition, the inclusion of structural or method features in a specific figure does not mean that such features are required in all embodiments, and in some embodiments, such features may not be included or such features may be combined with other features.

[0011] Reference now Figure 1 In an illustrative embodiment, a system 100 for throttling based on quality of service in a fabric architecture includes a plurality of interconnected network nodes 102 communicatively coupled via an interconnect fabric 114. The illustrative system 100 includes various types of network nodes 102, including a plurality of compute nodes 104 and storage nodes 112. The illustrative compute nodes 104 include a first compute node designated as compute node (1) 106, a second compute node designated as compute node (2) 108, and a third compute node designated as compute node (N) 110 (i.e., the “Nth” compute node of the compute nodes 104, where “N” is a positive integer and designates one or more additional compute nodes 104). It should be appreciated that in other embodiments, any number of compute nodes 104 and / or storage nodes 112 may be present. Illustratively, the interconnect fabric 114 includes a network switch 116 and a plurality of fabric interconnects 118 for communicatively coupling the network nodes 102. However, it should be appreciated that while only a single network switch 116 is shown, any number of network switches 116 may be present in other interconnect fabric embodiments.

[0012] In use, the network node 102 monitors the quality of service level associated with local resources (e.g., physical and / or virtual components) to detect throttling conditions associated with such resources (e.g., congestion, saturation, over-utilization, unfairness in workload distribution, etc.) and, upon detecting such throttling conditions, transmits throttling messages to other network nodes 102 of the fabric, requesting throttling actions to be performed by the receiving network node 102. The throttling messages may include various types of throttling requests for throttling specific resources of the network node 102. For example, the throttling messages may include memory throttling requests, I / O throttling requests, accelerator throttling processing requests, HFI saturation throttling requests, etc. It should be appreciated that the throttling messages are periodically transmitted during the time period when the throttling conditions are detected. In other words, the network node 102 continues to transmit throttling messages until the corresponding throttling conditions disappear.

[0013] To do so, unlike in current techniques where the network node 102 does not transmit throttling messages externally, thereby limiting throttling to only those resources that are local to the network node 102, components of the network node 102 and associated interconnect structure 114 are extended to transmit throttling information (e.g., generate new throttling messages, propagate existing throttling signals, etc.) to other network nodes 102 that are currently requesting access to a shared structure (e.g., a shared buffer) of a corresponding one of the network nodes 102 that has detected a throttling condition.

[0014] In an illustrative example, an Intel Certain coherence protocols for the cache agent and home agent of the coherence protocol include a proxy entity configured to initiate transactions into the coherent memory (e.g., via the cache agent) and service the coherent transactions (e.g., via the home agent). Such proxy entities are currently configured to detect certain conditions local to a corresponding one of the network nodes 102 and issue a local processor core throttling signal to throttle one or more cores of the processor. However, contention in the fabric architecture can occur not only in shared paths within each network node 102, but also in shared paths of the interconnect fabric 114, such as shared buffers (e.g., super queues in processor cores, request tables in cache / home agents, etc.).

[0015] In an illustrative example, compute node (1) 106 may be accessing memory of compute node (2) 108, which may be configured to monitor memory access requests (e.g., locally received memory accesses, memory accesses received from another compute node in compute node 104, etc.) and memory utilization levels. Under certain conditions, compute node (2) 108 may experience high and unequal contention due to memory request queue entries being occupied by requests for slower memory (e.g., non-cacheable memory) of compute node (2) 108 that have been received from compute node (1) 106. Therefore, under such conditions, compute node (2) 108 is configured to transmit a throttling message to compute node (1) 106 indicating that the memory of compute node (2) 108 is currently saturated, which compute node (1) may use to reduce the injection rate of memory requests directed to compute node (2) 108.

[0016] In some embodiments, the network nodes 102 are configured to use the transport layer (i.e., layer 4 (L4)) of the open systems interconnect (OSI) model to disclose current node throttling techniques between different network nodes 102 of the system 100. Thus, new and / or existing throttling signals originating from one of the network nodes 102 (e.g., from a caching agent, a home agent, an input / output operation, a scheduler, etc.) may be propagated through the fabric interconnect 118 to other network nodes 102, such as those network nodes requesting access to a shared fabric of the network node 102 from which the throttling signal originated.

[0017] The network node 102 may be implemented as any type of network traffic (e.g., network packets, messages, data, etc.) computing and / or storage computing device capable of performing the functions described herein, such as, but not limited to, a server (e.g., stand-alone, rack-mounted, blade, etc.), a network device (e.g., physical or virtual), a switch (e.g., rack-mounted, stand-alone, fully managed, partially managed, full-duplex, and / or half-duplex communication mode enabled, etc.), a router, a web appliance, a distributed computing system, and / or a multi-processor-based system. As previously described, the illustrative network node 102 includes a computing node 104 and a storage node 112; however, it should be appreciated that the network node 102 may include additional and / or alternative network nodes, such as a controller node, a network node, a tool node, etc., which are not shown for clarity of description.

[0018] like Figure 2 As shown, the illustrative network node 102 includes a first processor designated as processor (1) 202, a second processor designated as processor (2) 208, an input / output (I / O) subsystem 214, a main memory 216, a data storage device 218, and communication circuitry 220. It should be appreciated that Figure 1 The computing nodes 104 and / or storage nodes 112 of the illustrative network nodes 102 may include Figure 2 Components described in .

[0019] Of course, in other embodiments, network node 102 may include other or additional components, such as those typically found in computing devices. Additionally, in some embodiments, one or more of the illustrative components may be incorporated into another component, or otherwise form a portion thereof. For example, in some embodiments, cache memory 206 or portions thereof may be incorporated into one or both of processors 202, 208. Additionally, in some embodiments, one or more of the illustrative components may be omitted from network node 102. For example, although illustrative network node 102 includes two processors 202, 208, in other embodiments, network node 102 may include a greater number of processors.

[0020] Each of the processors 202, 208 (i.e., physical processor packages) may be implemented as any type of multi-core processor capable of performing the functions described herein, such as, but not limited to, a single physical multi-processor core chip, or package. The illustrative processor (1) 202 includes multiple processor cores 204, while the illustrative processor (2) 208 similarly includes multiple processor cores 210. As previously described, each of the processors 202, 208 includes more than one processor core (e.g., 2 processor cores, 4 processor cores, 8 processor cores, 16 processor cores, etc.). Each of the processor cores 204, 210 is implemented as an independent logical execution unit capable of executing programmed instructions. In some embodiments, the processor cores 204, 210 may include a portion of a cache memory (e.g., an L1 cache) and a functional unit that can be used to independently execute a program or thread. It should be appreciated that in some embodiments of the network node 102 (e.g., a supercomputer), the network node 102 may include thousands of processor cores. Each of the processors 202, 208 may be connected to a physical connector or socket on a motherboard (not shown) of the network node 102 configured to accept a single physical processor package (i.e., a multi-core physical integrated circuit).

[0021] The illustrative processor (1) 202 additionally includes a cache memory 206. Similarly, the illustrative processor (2) 208 also includes a cache memory 212. Each cache memory 206, 212 can be implemented as any type of cache memory that the corresponding processor 202, 208 can access faster than accessing the main memory 216, such as an on-die cache or an on-processor cache. In other embodiments, the cache memories 206, 212 can be off-die caches, but reside on the same system on chip (SoC) as the corresponding processor 202, 208. It should be appreciated that in some embodiments, the cache memories 206, 212 can have a multi-level architecture. In other words, in such multi-level architecture embodiments, the cache memories 206, 212 can be implemented as, for example, L1, L2, or L3 caches.

[0022] The main memory 216 may be implemented as any type of volatile or non-volatile memory or data storage device capable of performing the functions described herein. In operation, the main memory 216 may store various data and software used during the operation of the network node 102, such as operating systems, applications, programs, libraries, and drivers. The main memory 216 is communicatively coupled to the processors 202, 208 via the I / O subsystem 214, which may be implemented as circuits and / or components for facilitating the operation of input / output with the processors 202, 208, the main memory 216, and other components of the network node 102. For example, the I / O subsystem 214 may be implemented as or otherwise include a memory controller hub, an input / output control hub, a firmware device, a communication link (i.e., a point-to-point link, a bus link, a wire, a cable, an optical guide, a printed circuit board trace, etc.) and / or other components and subsystems for facilitating the operation of input / output. In some embodiments, I / O subsystem 214 may form part of a SoC and be incorporated on a single integrated circuit chip along with one or both of processors 202 , 208 , main memory 216 , and / or other components of network node 102 .

[0023] The data storage device 218 may be implemented as any type of device or devices configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard drives, solid-state drives, or other data storage devices. It should be appreciated that the data storage device 218 and / or the main memory 216 (e.g., computer-readable storage media) may store various data as described herein, including operating systems, applications, programs, libraries, drivers, instructions, etc. that are executable by a processor (e.g., processor 202, processor 208, etc.) of the network node 102.

[0024] The communication circuit 220 may be implemented as any communication circuit, device, or collection thereof that enables communication between the network node 102 and other computing devices (e.g., computing node 104, storage node 112, etc.) over a network. The communication circuit 220 may be configured to use any one or more communication technologies (e.g., wireless or wired communication technologies) and related protocols (e.g., Internet Protocol (IP), Ethernet, WiMAX, LTE, 5G, etc.) to achieve such communications.

[0025] The illustrative communication circuitry 220 includes a host fabric interface (HFI) 222. The HFI 222 may be implemented as one or more built-in boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that may be used by the network node 102. For example, in some embodiments, the HFI 222 may be integrated with one or both of the processors 202, 208 (e.g., on a consistent fabric within one or both of the processors 202, 208), implemented as an expansion card coupled to the I / O subsystem 214 via an expansion bus (e.g., PCI Express (PCIe)), part of a SoC that includes one or more processors, or included on a multi-chip package that also includes one or more processors. Additionally or alternatively, in some embodiments, the functionality of the HFI 222 may be integrated into one or more components of the network node 102 at the board level, the socket level, the chip level, and / or other levels. The HFI 222 is configured to facilitate the transfer of data / messages to enable tasks executing on the processors 202 , 208 to access shared structures (eg, shared physical memory) of other network nodes 102 , such as may be necessary during parallel or distributed computing operations.

[0026] It should be appreciated that those network nodes 102 implemented as storage nodes 112 may generally include more data storage device 218 capacity than those network nodes 102 implemented as compute nodes 104. Similarly, it should also be appreciated that those network nodes 102 implemented as compute nodes 104 may generally include more processor power than those network nodes 102 implemented as storage nodes 112. In other words, storage nodes 112 may be implemented as a physical server that includes a number of hard disk drives (HDDs) or solid state drives (SDDs) related to the number of storage devices of compute nodes 104, while compute nodes 104 may be implemented as a physical server that includes a number of processors with a number of cores related to the number of processors of storage nodes 112. However, it should be further appreciated that any network node 102 may be implemented as a compute node 104 and / or a storage node 112 regardless of the configuration of components associated with the other network nodes 102.

[0027] Reference again Figure 1, the interconnect fabric 114 (illustratively a combination of a network switch 116 and a fabric interconnect 118) can be implemented as one or more buses, switches, and / or networks that are configured to support the transmission of network traffic as a function of various interconnect protocols and / or network protocols. In use, the interconnect fabric 114 is used by the network nodes 102 (e.g., via corresponding HFIs 222) to communicate with other network nodes 102 (i.e., across the interconnect fabric 114). Thus, the network switch 116 can be implemented as any type of switching device (e.g., a crossbar switch) capable of forwarding network traffic via the fabric interconnect 118 in a switched or switching fabric architecture.

[0028] Now refer to Figure 3 , in an illustrative embodiment, Figure 2 The network node 102 includes one or more non-uniform memory access (NUMA) domains 300 communicatively coupled to the HFI 222. The illustrative NUMA domains 300 include a first NUMA domain designated as NUMA domain (1) 302 and a second NUMA domain designated as NUMA domain (2) 308. Each domain in the NUMA domains 300 includes a plurality of allocated processor cores of a physical processor package, which is referred to herein as a processor. As shown in the illustrative embodiment, the NUMA domain (1) 302 includes the processor core 204 of the processor (1) 202 and the NUMA domain (2) 308 includes the processor core 210 of the processor (2) 208. However, it should be appreciated that in some embodiments, the processor core 204 of the processor 202 and / or the processor core 210 of the processor 208 may be partitioned, and each group of partitioned processor cores may be assigned to a different NUMA domain 300. It should be appreciated that each group of allocated processor cores assigned to a corresponding one of the NUMA domains 300 may be referred to as a socket core. In other words, the number of allocated cores of a physical processor package can be referred to as a socket.

[0029] In addition, each NUMA domain 300 corresponds to a specific memory type (e.g., double data rate (DDR) memory, disk, etc.) and includes a portion of the local memory of that memory type (e.g., main memory 216) that has been allocated to the processor core of the corresponding NUMA domain 300. In addition, the local memory is directly linked to the physical processor package where the processor core resides. In the illustrative embodiment, NUMA domain (1) 302 includes local memory (1) 304, and NUMA domain (2) 308 includes local memory (2) 310. In some embodiments, the local memory can be connected via interconnect 314 (e.g., Ultra Path Interconnect (UPI), Quick Path Interconnect (QPI), The interconnect 314 may be a unified media interface (UMI) interconnect, or the like, to transfer data between the NUMA domains 300. The local memory of one of the NUMA domains 300 is considered to be remote or foreign to the other NUMA domains 300. Therefore, it should be appreciated that network traffic transferred across the interconnect 314 may introduce load / contention, increase overall bandwidth usage, and reduce latency associated with accessing remote memory compared to processing data using local memory.

[0030] Each illustrative processor 202, 208 further includes an on-die interconnect (e.g., on-die interconnect 306 of processor 202 and on-die interconnect 312 of processor 208) configured to interface with HFI 222 via point-to-point interface 316, which can facilitate data transfer between HFI 222 and processors 202, 208. In some embodiments, NUMA domains 300 can be defined internally in HFI 222. In the illustrative example, one of the NUMA domains 300 (e.g., NUMA domain (1) 302) of one of the network nodes 102 (e.g., compute node (1) 106) can correspond to a transaction processed by HFI 222 from another network node 102 (e.g., compute node (2) 108). Thus, HFI 222 of compute node (1) 106 may send a throttling message to compute node (2) 108 when it is determined by compute node (1) 106 that compute node (2) 108 is transmitting too many requests to compute node (1) 106. In some embodiments, such throttling messages may include information propagated from a cache agent of processor 202 received by HFI 222 via point-to-point interface 316.

[0031] Reference now Figure 4 , in an illustrative embodiment, an environment 400 established during operation by one of the network nodes 102. The illustrative environment 400 includes a communication management module 410, a quality of service (QoS) monitoring module 420, a throttle message transmission module 430, a throttle message receiving module 440, and a throttle response execution module 450. The various modules of the environment 400 may be implemented as hardware, firmware, software, or a combination thereof. As such, in some embodiments, one or more of the modules of the environment 400 may be implemented as a circuit or collection of electrical devices (e.g., communication management circuit 410, QoS monitoring circuit 420, throttle message transmission circuit 430, throttle message receiving circuit 440, throttle response execution circuit 450, etc.).

[0032] It should be appreciated that in such embodiments, one or more of the communication management circuitry 410, the QoS monitoring circuitry 420, the throttle message transmission circuitry 430, the throttle message reception circuitry 440, and the throttle response execution circuitry 450 may form part of one or more processors (e.g., Figure 2 102 and processor (2) 208), I / O subsystem 214, communication circuit 220, and / or other components of network node 102. Additionally, in some embodiments, one or more of the illustrative modules may form part of another module and / or one or more of the illustrative modules may be independent of one another. Additionally, in some embodiments, one or more of the modules of environment 400 may be implemented as virtualized hardware components or emulated architectures that may be established and maintained by one or more processors and / or other components of network node 102.

[0033] In the illustrative environment 400, the network node 102 further includes network node data 402, monitoring results data 404, request monitoring data 406, and NUMA identification data 408, each of which may be stored in the main memory 216 and / or the data storage device 218 of the network node 102. Moreover, each of the network node data 402, monitoring results data 404, request monitoring data 406, and NUMA identification data 408 may be accessed by various modules and / or sub-modules of the network node 102. Additionally, it should be appreciated that in some embodiments, the data stored in or otherwise represented by each of the network node data 402, monitoring results data 404, request monitoring data 406, and NUMA identification data 408 may not be mutually exclusive of one another.

[0034] For example, in some implementations, data stored in network node data 402 may also be stored as part of monitoring results data 404, and / or vice versa. As such, although various data utilized by network node 102 are described herein as specific discrete data, in other embodiments, such data may be combined, aggregated, and / or otherwise formed as part of a single or multiple data sets, including duplicate copies. It is further appreciated that network node 102 may include additional and / or alternative components, subcomponents, modules, submodules, and / or devices typically found in computing devices, which are not described herein for clarity of description. Figure 4 Shown in.

[0035] The communication management module 410 (which may be implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or combinations thereof as described above) is configured to facilitate inbound and outbound wired and / or wireless network communications (e.g., network traffic, network packets, network flows, etc.) to and from the network node 102. To do so, the communication management module 410 is configured to receive and process network packets from other network nodes 102 via the interconnect fabric. Additionally, the communication management module 410 is configured to prepare and transmit network packets to other network nodes 102 via the interconnect fabric. Thus, in some embodiments, at least a portion of the functionality of the communication management module 410 may be performed by the communication circuitry 220 of the network node 102, or more specifically, by the HFI 222 of the communication circuitry 220. In some embodiments, data (such as IP address information, traffic information, etc.) that may be used to communicate with other network nodes 102 of the fabric architecture may be stored in the network node data.

[0036] The QoS monitoring module 420 (which may be implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or combinations thereof as described above) is configured to monitor various characteristics of the network node 102. To do so, the illustrative QoS monitoring module 420 includes a resource utilization monitoring module 422, a load balancing monitoring module 424, and an HFI saturation monitoring module 426. It should be appreciated that each of the resource utilization monitoring module 422, the load balancing monitoring module 424, and the HFI saturation monitoring module 426 of the QoS monitoring module 420 may be individually implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or combinations thereof. For example, the resource utilization monitoring module 422 may be implemented as a hardware component, while the load balancing monitoring module 424 and / or the HFI saturation monitoring module 426 may be implemented as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, emulation architecture, and / or combinations thereof.

[0037] The resource utilization monitoring module 422 is configured to monitor the utilization level of the resources (i.e., physical and / or virtual components) of the network node 102. In an illustrative example, the resource utilization monitoring module 422 can be configured to monitor the memory utilization level. To do so, in some embodiments, the resource utilization monitoring module 422 can be configured to receive a throttling signal currently generated by one or more local cache agents of the processor of the network node 102, which can be used to slow down or otherwise reduce the injection rate of a given memory type indicated by the throttling signal. Additionally or alternatively, the resource utilization monitoring module 422 can be configured to identify the current usage level of the resource to determine the saturation level of the monitored resource.

[0038] The load balancing monitoring module 424 is configured to monitor the distribution of workloads across the resources (i.e., physical and / or virtual components) of the network node 102. The HFI utilization monitoring module 426 is configured to monitor the utilization of the HFI 222. Therefore, the HFI utilization monitoring module 426 can detect the saturation of the HFI 222 even if the resources attached to it have not become saturated. In an illustrative example, one of the computing nodes 104 can saturate the HFI 222 of one of the storage nodes 112 when accessing the storage device of the storage node 112. Under such conditions, the HFI 222 of the storage node 112 may become saturated, while the storage device of the storage node 112 may not be fully utilized (i.e., saturated). In some embodiments, the monitoring results (e.g., current / historical utilization values, current / historical load balancing information, etc.) can be stored in the monitoring results data 404.

[0039] The throttling message transmission module 430 (which may be implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or a combination thereof as described above) is configured to generate and transmit throttling messages to other network nodes 102. As previously described, certain conditions (i.e., throttling conditions) may exist on a network node 102 such that resource access requests generated by the network node 102 requesting access to local resources of the network node 102 may be starved due to other network nodes 102 maintaining an unimpeded injection rate to the local throttled resources. Therefore, unlike the prior art that only provides local throttling, the network node 102 is configured to detect such throttling conditions and generate throttling messages for transmission to other network nodes 102 that are responsible for or otherwise contribute to the throttling conditions.

[0040] To generate and transmit throttling messages to other network nodes 102, the illustrative throttling message transmission module 430 includes a throttling condition detection module 432 and a transmission mode determination module 434. It should be appreciated that each of the throttling condition detection module 432 and the transmission mode determination module 434 of the QoS monitoring module 420 can be individually implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or a combination thereof. For example, the throttling condition detection module 432 can be implemented as a hardware component, while the transmission mode determination module 434 is implemented as a virtualized hardware component or implemented as some other combination of hardware, firmware, software, virtualized hardware, emulation architecture, and / or a combination thereof.

[0041] The throttling condition detection module 432 is configured to detect whether a throttling condition exists. To do so, the throttling condition detection module 432 can be configured to compare the current quality of service condition (e.g., which can be determined by the QoS monitoring module 420) with a corresponding threshold. For example, the throttling condition detection module 432 can be configured to compare the current memory utilization level with the memory saturation threshold. Therefore, the throttling condition detection module 432 can detect the throttling condition when it is determined that the current memory utilization level exceeds the memory saturation threshold. Additionally or alternatively, the throttling condition detection module 432 can be configured to process the throttling signal generated inside the network node 102. As previously described, an agent entity (e.g., a cache agent, a home agent, etc.) can generate a local throttling request for a cluster (such as storage or I / O) on a specific die. Therefore, the throttling condition detection module 432 is configured to interpret such local throttling requests to determine whether they indicate a throttling condition, whereby one or more network nodes in other network nodes 102 should be notified to take appropriate throttling actions.

[0042] The transmission mode determination module 434 is configured to determine which transmission mode to use to transmit a throttling message generated in response to a detected throttling condition (such as detected by the throttling condition detection module 432). To do so, the transmission mode determination module 434 is configured to detect which one or more of the network nodes 102 to transmit the throttling message to based on the identification of other network nodes 102 that are responsible for or otherwise contribute to the throttling condition. For example, the transmission mode determination module 434 may determine that a single network node 102 is issuing too many memory access requests, in which case the transmission mode determination module 434 may determine to use a unicast mode to transmit the generated throttling message. Otherwise, if the transmission mode determination module 434 determines that more than one of the network nodes 102 is responsible for or otherwise contributes to the current throttling condition, the transmission mode determination module 434 may determine to use a multicast mode to transmit the generated throttling message.

[0043] As previously described, the throttling message transmission module 430 is configured to transmit a throttling message requesting another network node 102 to take action (e.g., throttling the processor cores of a specific NUMA domain) in response to receipt of a throttling message. Each NUMA domain 300 of each network node 102 has a corresponding NUMA domain identifier, which can be used by the throttling message transmission module 430 to determine which NUMA domain 300 is to be throttled. Therefore, the network node 102 includes a NUMA domain identifier for the NUMA domain 300 local to the network node 102 and NUMA domain identifiers for the NUMA domains of other network nodes 102. However, in some embodiments, the NUMA domain identifiers may not be well-known, such as in a distributed tag directory scheme. In such embodiments, the throttling message transmission module 430 can predict which NUMA domain 300 the receiving network node 102 will perform a response action on.

[0044] To do so, the throttling messaging module 430 may be further configured to predict which NUMA domain 300 the receiving network node 102 will take action on based on the principle that applications accessing the NUMA domain will operate within a certain range of memory addresses in the NUMA domain. Figure 3 One of the point-to-point interfaces 316 of the on-die interconnect interface 316 is configured to generate a request to a proxy entity (such as a cache agent). Therefore, the on-die interconnect interface can be extended to use the domain prediction table to determine which NUMA domain corresponds to the throttling message, and whether the NUMA domain (e.g., the processor core of the NUMA domain) is currently in distress (i.e., has been throttled for the NUMA domain). If a component of the NUMA domain is currently in distress, a throttling message may not be issued (i.e., injected) to the proxy entity until the distress no longer exists and is confirmed by the proxy entity. Therefore, the use of the domain prediction table can allow the network node 102 to infer the affected NUMA domain 300 of another network node 102. In some embodiments, the data of the domain prediction table can be stored in the NUMA identification data 408.

[0045] The domain prediction table may include an identifier for each agent entity, a NUMA level known by each network node 102, a last address range accessed for each NUMA domain 300 and / or agent entity (e.g., formatted as a bit mask), and a granularity that may be configurable per NUMA domain 300. In an illustrative embodiment, the granularity is 4GB for a specific NUMA domain 300, and the last address (e.g., 0x78C9657FA) sent to a specific agent entity for a specific NUMA level belongs to the address range 0x700000000-0x700000000+4GB. As previously described, an application accessing a NUMA domain will operate within a specific range of memory addresses in that NUMA domain. As such, by appropriately specifying the granularity, it may produce more accurate predictions, resulting in a high hit rate, and returning results within a few cycles. Therefore, in some embodiments, to predict the NUMA domain of a throttle message for a specific address and a specific proxy entity, the throttle messaging module 430 may be configured to access a domain prediction table to retrieve the predicted NUMA domain as a content addressable memory (CAM) structure. For example, if the use case for the application is to allocate a 10GB memory block of the storage node 112 via memory exposed to a NUMA domain, then if the selected granularity is GM, the domain prediction request is likely to hit on the prediction table.

[0046] In some embodiments, the process for a throttle message to a specific proxy entity to a specific address may include determining the modulus and granularity of the last memory address to predict which NUMA domain the last memory address belongs to. If the predicted NUMA domain request returns NULL (i.e., no NUMA domain matches), it can be assumed that the closest NUMA domain is NUMA level 0. As previously described, when the distress signal for the NUMA domain is valid, the processor core does not send transactions to the proxy entity, and only issues transactions after the distress signal is invalid and confirmed by the proxy entity. Therefore, if the distress signal is valid for the result of the predicted NUMA domain, the processor core does not send transactions to the proxy entity until the distress signal is invalid and confirmed by the proxy entity. In addition, in some embodiments, the proxy entity can perform system address decoding, update appropriate counters (e.g., throttling request counters), and generate distress signals when necessary. In addition, the prediction table is updated based on the feedback received based on the proxy entity returning confirmation and the NUMA domain identifier for the prediction request for the specific domain.

[0047] It should be appreciated that the target fiber architecture for the QoS-based throttling scheme described herein is for enterprise systems with a scale of hundreds of network nodes 102. Therefore, in such embodiments with a larger scale (such as embodiments with a scale of high performance computing (HPC) of thousands of nodes), due to the large number of messages that can be transmitted therein, the multicast mode may not be ideal for implementation. However, a subdomain of the network nodes 102 of the fabric architecture (e.g., consisting only of those network nodes 102 connected to a specific network switch 116) can be defined, such as by using a specific multicast topology, so as to propagate the throttling message only to a subset of the network nodes 102. It should be further appreciated that the multicast mode can be an unreliable multicast. As previously described, the throttling message is periodically transmitted for the duration that the throttling condition exists, thereby eliminating the need to confirm the receipt of the throttling message. Reliability can be improved, such as by adding a receipt confirmation in the process; however, such reliability improvements may add more stress to the structure.

[0048] In some embodiments, the throttle message transmission module 430 (e.g., the throttle condition detection module 432 and / or the transmission mode determination module 434) may utilize the request monitoring table to determine when to generate a throttle message and / or to which network nodes 102 the generated throttle message is to be transmitted. In an illustrative example, the throttle message transmission module 430 may be configured to consider external transactions to the local NUMA domain 300 of the specific network node 102. As previously described, each NUMA domain 300 has a corresponding NUMA domain identifier, which is used by the throttle message transmission module 430 to determine which NUMA domain 300 a received throttle message corresponds to, as well as a request counter that is incremented with each access.

[0049] In some embodiments, the NUMA domain identifier, the value of the request counter, and other values ​​(e.g., an enumeration value of the throttle message request type) may be stored in a model specific register (MSR). Thus, the throttle message transmission module 430 may be configured to read the value of the request counter to determine whether the request counter exceeds a threshold. It should be appreciated that the MSR value may be configured during operation or boot time (e.g., using a ring zero function) and may be exposed to the operating system of the network node 102.

[0050] In some embodiments, the request counter may be stored in a request monitoring table that includes an identifier of the network node 102 from which the throttling message is received, a current value of the request counter, a NUMA domain identifier, and a threshold value. In some embodiments, the data of the request monitoring table may be stored in the request monitoring data 406. If the request counter exceeds the threshold value, the throttling message transmission module 430 may be configured to generate a throttling message for transmission in unicast mode (i.e., to the network node 102 that is only responsible for the current state of the request counter). In addition, the throttling message transmission module 430 may be configured to generate a throttling message for transmission in multicast mode (i.e., to all other network nodes 102 that issue transactions to a specific NUMA domain) when a throttling signal is received internally (such as from a cache agent). As previously described, the throttling message transmission module 430 is configured to periodically generate throttling messages when a throttling condition is detected.

[0051] It should be appreciated that system configuration, such as NUMA domain identifiers, MSRs in different network nodes 102, etc., should be performed holistically to ensure consistency. Therefore, system configuration should be enforced at system boot time (e.g., when the routing system address decoding scheme is executed) to ensure that the information conveyed in the throttling messages is consistent across different network nodes 102. For example, in an embodiment where the NUMA domain identifier of compute node (1) 106 is being propagated to compute node (2) 108, compute node (2) 108 should already know which NUMA domain identifier corresponds to the specific NUMA domain 300 of compute node (1) 106. In some embodiments, the NUMA domain identifiers of other network nodes 102 may be stored in NUMA identification data 408.

[0052] The throttling message receiving module 440 (which may be implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or a combination thereof as described above) is configured to receive and process throttling messages from other network nodes 102. To do so, the illustrative throttling message receiving module 440 includes a throttling type identification module 442 and a NUMA target identification module 444. It should be appreciated that each of the throttling type identification module 442 and the NUMA target identification module 444 of the throttling message receiving module 440 may be individually implemented as hardware, firmware, software, virtualized hardware, emulation architecture, and / or a combination thereof. For example, the throttling type identification module 442 may be implemented as a hardware component, while the NUMA target identification module 444 is implemented as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, emulation architecture, and / or a combination thereof.

[0053] The throttling type identification module 442 is configured to identify the type associated with the received throttling message. As previously described, the throttling message request type associated with the throttling message may include a memory throttling request, an I / O throttling request, an accelerator throttling processing request, an HFI saturation throttling request, etc. In some embodiments, the throttling message request type may be an enumeration type so that they can be mapped to specific actions. In addition, in some embodiments, the enumeration values ​​of the throttling message request type may be stored in a throttling action table that maps the enumeration values ​​to corresponding actions. The NUMA target identification module 444 is configured to identify the NUMA domain target or its component associated with the received throttling message.

[0054] The throttling response execution module 450 (which may be implemented as hardware, firmware, software, virtualized hardware, simulation architecture, and / or a combination thereof as described above) is configured to take action in response to having received a throttling message from another network node 102. To do so, the illustrative throttling response execution module 450 includes a processor core throttling execution module 452, a software interrupt execution module 454, and an HFI throttling execution module 456. It should be appreciated that each of the processor core throttling execution module 452, the software interrupt execution module 454, and the HFI throttling execution module 456 of the throttling response execution module 450 may be individually implemented as hardware, firmware, software, virtualized hardware, simulation architecture, and / or a combination thereof. For example, the processor core throttling execution module 452 may be implemented as a hardware component, while the software interrupt execution module 454 and / or the HFI throttling execution module 456 may be implemented as a virtualized hardware component or as some other combination of hardware, firmware, software, virtualized hardware, simulation architecture, and / or a combination thereof.

[0055] The processor core throttling execution module 452 is configured to throttle the processor core in response to receiving the propagated throttling message. To do so, the processor core throttling execution module 452 is configured to convert the received throttling message into a corresponding on-die interconnect throttling signal supported by the network node 102 architecture to reduce the injection rate of the externally transmitted access request. The software interrupt execution module 454 is configured to execute a software interrupt in response to the received software interrupt request throttling message. To do so, in such embodiments, the throttling message is propagated to the software heap via the software interrupt, wherein the software heap supports load balancing and injection control mechanisms.

[0056] The HFI throttling execution module 456 is configured to throttle injection at the HFI 222 based on the type of throttling message received. In other words, the HFI 222 is responsible for reducing the injection rate or stopping the injection completely. Therefore, such a response may be a suitable solution for throttling message types that are not supported by the structural architecture of the network node 102. It should be appreciated that the processor cores and other injectors of the network node 102 are not throttled.

[0057] Reference now Figure 5 In use, a network node 102 (eg, Figure 1 1 ) may perform a method 500 for processing a local memory request from a remote network node (i.e., another one of the network nodes 102 of the fabric). The method 500 begins at block 502, where the network node 102 determines whether a memory access request has been received from the remote network node. If not, the method 500 loops back to block 502 to determine whether a memory access request has been received from the remote network node; otherwise, the method 500 proceeds to block 504. In block 504, the network node 102 inserts the received remote memory access request into a shared buffer of the network node 102. It should be appreciated that under certain conditions, the shared buffer may be full for a period of time before the network node 102 can insert the received remote memory access request into the shared buffer.

[0058] In block 506, the network node 102 determines whether to process the received request (e.g., pop the corresponding entry from the shared buffer and process the request). If so, the method 500 proceeds to block 508, where the network node 102 performs an action in response to the received remote memory access request. For example, in block 510, the network node 102 transmits the request data in response to the remote memory access request having requested that the data be stored in the memory of the network node 102 (e.g., the main memory 216). Alternatively, in block 512, the network node 102 may store the data received with the remote memory access request. In some embodiments, in block 514, the network node 102 transmits an acknowledgment in response to the remote memory access request having been received / processed.

[0059] Reference now Figure 6 In use, a network node 102 (eg, Figure 1 6 . The method 600 begins at block 602, where the network node 102 determines whether to access memory located in another network node 102. For example, the network node 102 may be retrieving data replicated in a remote memory (i.e., the memory of the remote network node), executing an application that utilizes a distributed data structure on one or more remote network nodes, employing log shipping (i.e., relying on a log or micro-log stored on a remote network node for failure recovery), or performing some other operation that requires access to the memory of a remote network node.

[0060] If not, the method 600 loops back to block 602 to again determine whether to access a memory located in another network node 102; otherwise, the method 600 proceeds to block 604. In block 604, the network node 102 generates a remote memory access request, which includes memory address information that can be used to retrieve or store data of the remote memory access request. In addition, in block 606, the network node 102 includes source identification information of the network node 102. In block 608, the network node 102 inserts the memory access request into the message transmission queue.

[0061] In block 610, the network node 102 determines whether the injection rate corresponding to the component from which the remote memory access request is requesting access is due to a throttling message received from the remote network node (see, e.g., Figure 7 If not, the method 600 branches to block 612, where the network node 102 transmits the remote memory access request at the unthrottled injection rate; otherwise, the method 600 branches to block 614, where the network node 102 transmits the remote memory access request at the throttled rate.

[0062] Reference now Figure 7 In use, a network node 102 (eg, Figure 1 102) may perform a method 700 for generating a throttling message for external transmission to one or more remote network nodes (i.e., one or more of the other remote network nodes). The method 700 begins at block 702, where the network node 102 monitors a quality of service level of the network node 102. For example, in block 704, in some embodiments, the network node 102 monitors a utilization level of resources (e.g., memory, processors, components of a NUMA domain, etc.) of the network node 102. Additionally or alternatively, in block 706, in some embodiments, the network node 102 monitors a distribution of workloads distributed across components of the network node 102. In block 708, in some embodiments, the network node 102 additionally or alternatively monitors a saturation level of the HFI 222 of the network node 102. As previously described, in some embodiments, the network node 102 may rely on a request monitoring table to determine when to generate a throttling message for a specific NUMA domain 300.

[0063] As also previously described, certain conditions (i.e., throttling conditions) may exist on the network node 102 such that resource access requests generated by the network node 102 requesting access to local resources of the network node 102 may become starved due to the remote network node 102 maintaining an unimpeded injection rate to the locally throttled resources. Therefore, in block 710, the network node 102 determines whether a throttling condition (e.g., congestion, saturation, over-utilization, unfair workload distribution, etc. of a component of the network node 102) has been detected (i.e., currently exists) as a result of the monitoring of quality of service performed in block 702.

[0064] If the network node 102 determines that there is no throttling condition, the method 700 loops back to block 702 to continue monitoring the quality of service level of the network node 102; otherwise, the method 700 proceeds to block 712, where the network node 102 generates a throttling message. In block 714, the network node 102 includes a throttling message request type indicator with the throttling message. As previously described, the throttling message request type associated with the throttling message may include a memory throttling request, an I / O throttling request, an accelerator throttling processing request, an HFI saturation throttling request, etc. In addition, in block 716, the network node 102 includes a throttling message source indicator. The throttling message source indicator may include an identifier of a component for which a throttling condition has been detected (e.g., a NUMA domain identifier, an HFI identifier) ​​and / or an identifier of the network node 102.

[0065] In block 718, the network node 102 identifies one or more target network nodes (i.e., one or more of the other network nodes 102 of the fabric) to receive the throttling message generated in block 712. As previously described, in some embodiments, the network node 102 may rely on a request monitoring table to determine the one or more target network nodes. In block 720, the network node 102 transmits the generated throttling message to the one or more target network nodes identified in block 718. To do so, in block 722, the network node 102 transmits the generated throttling message based on a cyclic rate corresponding to each target network node.

[0066] Depending on the number of target network nodes identified in block 718, the network node 102 may transmit the generated throttling message via a multicast transmission (i.e., more than one target network node) in block 724 or via a unicast transmission (i.e., a single target network node) in block 726. Additionally, in some embodiments, in block 728, the network node 102 may transmit the generated throttling message via the transport layer of the OSI model. To do so, in some embodiments, the fabric may be extended with a new type of virtual channel that facilitates the transfer of throttling messages, so as to isolate the throttling messages from the fabric's existing channels. Such embodiments may be implemented via a new type of physical line that selects the fastest path within the fabric, so as to deliver the throttling message as quickly as possible.

[0067] As previously described, throttling messages are transmitted periodically during a detected throttling condition.As such, method 700 can monitor the quality of service level relative to that particular throttling condition and iterate method 700 due to that particular quality of service level monitoring.

[0068] Reference now Figure 8 In use, a network node 102 (eg, Figure 1 A method 800 for processing a throttling message received from a remote network node (i.e., one of the other network nodes 102) may be performed by the network node 102. The method 800 begins at block 802, where the network node 102 determines whether a throttling message has been received from the remote network node. If not, the method 800 loops back to block 802 to again determine whether a throttling message has been received from the remote network node; otherwise, the method 800 proceeds to block 804, where the network node 102 identifies information associated with the throttling message received in block 802.

[0069] For example, in box 806, the network node 102 identifies the type of throttling message. As previously described, the throttling message request type associated with the throttling message may include a memory throttling request, an I / O throttling request, an accelerator throttling processing request, an HFI saturation throttling request, etc. In addition, in box 808, the network node 102 identifies the source of the throttling message. The source of the throttling message may include information identifying the target network node from which the throttling message is received. In addition, the source of the throttling message may include a component identifier (e.g., a NUMA identifier) ​​identifying a component of a remote network node from which a throttling task was received. In some embodiments, the throttling message may additionally include component information of the receiving network node 102, which may be used to identify which network node 102 resources (from which remote memory access is being requested) are to be throttled.

[0070] In block 810, the network node 102 performs an action based on the received throttling message, such as an action that may be performed based on the type of throttling message identified in block 806. For example, in block 812, the network node 102 may reduce the injection rate of shared resource access requests being transmitted to (i.e., for) a remote network node through a self-throttling request by the HFI 222 of the network node 102. In another example, in block 814, the network node 102 may throttle the network node 102 processor core by using an existing throttling scheme. To do so, the network node 102 may propagate the received throttling message to a proxy entity (e.g., a cache proxy) via a corresponding on-die interconnect (e.g., on-die interconnect 306 of processor 202, on-die interconnect 312 of processor 208, etc.) to throttle the processor core of the network node 102 by using an existing throttling scheme. In yet another example, in block 816, in such embodiments where the software stack supports load balancing and injection control mechanisms, the network node 102 may propagate the received throttling message to the software stack via a software interrupt.

[0071] It should be appreciated that at least a portion of one or more of the methods 500, 600, 700, and 800 may be performed by the HFI 222 of the network node 102. It should be further appreciated that in some embodiments, one or more of the methods 500, 600, 700, and 800 may be implemented as various instructions stored on a computer-readable medium, which may be executed by a processor (e.g., processor 202, processor 208, etc.), the HFI 222, and / or other components of the network node 102 to cause the network node 102 to perform the methods 500, 600, 700, and 800. The computer-readable medium may be implemented as any type of medium that can be read by the network node 102, including but not limited to the main memory 216, the data storage device 218, the secure memory (not shown) of the HFI 222, other memory or data storage devices of the network node 102, a portable medium readable by a peripheral device of the network node 102, and / or other media.

[0072] Example The following provides illustrative examples of the techniques disclosed herein. An embodiment of these techniques may include any one or more of the examples described below, and any combination.

[0073] Example 1 includes a network node for throttling based on quality of service in a fabric architecture, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, the network node comprising a processor; a host fabric interface (HFI), the HFI being used to facilitate data transfer between the plurality of interconnected network nodes through an interconnect structure of the fabric architecture; and one or more data storage devices storing a plurality of instructions which, when executed by the processor, cause the network node to monitor a quality of service level of the network node; detect a throttling condition based on a result of the monitored quality of service level; generate a throttling message based on a request type associated with the detected throttling condition in response to the throttling condition having been detected; and transmit the generated throttling message to one or more of the plurality of interconnected network nodes communicatively coupled to the network node via the interconnect structure.

[0074] Example 2 includes the subject matter of Example 1, and wherein monitoring the quality of service level of the network node comprises monitoring a utilization level of one or more resources of the network node.

[0075] Example 3 includes the subject matter of any of Examples 1 and 2, and wherein the one or more resources of the network node include at least one of the processor, the one or more data storage devices, or the HFI.

[0076] Example 4 includes the subject matter of any of Examples 1-3, and wherein the one or more resources of the network node include multiple non-uniform memory access (NUMA) domains, wherein each of the multiple NUMA domains includes an allocated portion of a processor core of the processor and an allocated portion of the one or more data storage devices, and wherein each of the multiple NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0077] Example 5 includes the subject matter of any of Examples 1-4, and wherein monitoring the quality of service level of the network node comprises monitoring workload distribution.

[0078] Example 6 includes the subject matter of any of Examples 1-5, and wherein monitoring the quality of service level of the network node comprises monitoring a saturation level of the HFI.

[0079] Example 7 includes the subject matter of any of Examples 1-6, and wherein monitoring the quality of service level of the network node comprises monitoring a throttling signal received from a proxy entity of the processor.

[0080] Example 8 includes the subject matter of any of Examples 1-7, and wherein the plurality of interconnected network nodes includes one or more compute nodes and one or more storage nodes.

[0081] Example 9 includes the subject matter of any of Examples 1-8, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to one of the plurality of interconnected network nodes via unicast transmission.

[0082] Example 10 includes the subject matter of any of Examples 1-9, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to more than one of the plurality of interconnected network nodes via multicast transmission.

[0083] Example 11 includes the subject matter of any of Examples 1-10, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes via a transport layer of an open systems interconnection (OSI) model.

[0084] Example 12 includes the subject matter of any of Examples 1-11, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes at a periodic injection rate for a duration of the detected throttling condition.

[0085] Example 13 includes a network node for throttling based on quality of service in a fabric architecture, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, the network node comprising a quality of service monitoring circuit for monitoring the quality of service level of the network node between a host fabric interface (HFI) of the network node and one or more HFIs of other interconnected network nodes on an interconnect structure of the fabric architecture; a throttling message transmission circuit for (i) detecting a throttling condition based on a result of the monitored quality of service level, (ii) in response to the throttling condition having been detected, generating a throttling message based on a request type associated with the detected throttling condition, and (iii) transmitting the generated throttling message to one or more of the plurality of interconnected network nodes communicatively coupled to the network node via the interconnect structure.

[0086] Example 14 includes the subject matter of Example 13, and wherein monitoring the quality of service level of the network node comprises monitoring a utilization level of one or more resources of the network node.

[0087] Example 15 includes the subject matter of any of Examples 13 and 14, and wherein the one or more resources of the network node include at least one of a processor of the network node, one or more data storage devices of the network node, or the HFI.

[0088] Example 16 includes the subject matter of any of Examples 13-15, and wherein the one or more resources of the network node include multiple non-uniform memory access (NUMA) domains, wherein each of the multiple NUMA domains includes an allocated portion of a processor core of the processor and an allocated portion of the one or more data storage devices, and wherein each of the multiple NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0089] Example 17 includes the subject matter of any of Examples 13-16, and wherein monitoring the quality of service level of the network node comprises monitoring workload distribution.

[0090] Example 18 includes the subject matter of any of Examples 13-17, and wherein monitoring the quality of service level of the network node comprises monitoring a saturation level of the HFI.

[0091] Example 19 includes the subject matter of any of Examples 13-18, and wherein monitoring the quality of service level of the network node comprises monitoring a throttling signal received from a proxy entity of the processor.

[0092] Example 20 includes the subject matter of any of Examples 13-19, and wherein the plurality of interconnected network nodes comprises one or more compute nodes and one or more storage nodes.

[0093] Example 21 includes the subject matter of any of Examples 13-20, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to one of the plurality of interconnected network nodes via unicast transmission.

[0094] Example 22 includes the subject matter of any of Examples 13-21, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to more than one of the plurality of interconnected network nodes via multicast transmission.

[0095] Example 23 includes the subject matter of any of Examples 13-22, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes via a transport layer of an open systems interconnection (OSI) model.

[0096] Example 24 includes the subject matter of any of Examples 13-23, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes at a periodic injection rate for the duration of the detected throttling condition.

[0097] Example 25 includes a method for throttling based on quality of service across a fabric architecture, the method comprising monitoring a quality of service level of a network node through a host fabric interface (HFI) of the network node of the fabric architecture, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, wherein each of the plurality of interconnected network nodes is interconnected through an interconnect structure of the fabric architecture; detecting a throttling condition through the HFI based on a result of the monitored quality of service level; generating a throttling message through the HFI based on a request type associated with the detected throttling condition in response to the throttling condition having been detected; and transmitting the generated throttling message through the HFI to one or more of the plurality of interconnected network nodes communicatively coupled to the network node via the interconnect structure.

[0098] Example 26 includes the subject matter of Example 25, and wherein monitoring the quality of service level of the network node comprises monitoring a utilization level of one or more resources of the network node.

[0099] Example 27 includes the subject matter of any of Examples 25 and 26, and wherein the utilization level of one or more resources of the network node comprises monitoring at least one of a processor of the network node, one or more data storage devices of the network node, or the HFI.

[0100] Example 28 includes the subject matter of any of Examples 25-27, and wherein monitoring the utilization level of one or more resources of the network node comprises monitoring one or more of a plurality of non-uniform memory access (NUMA) domains, wherein each of the plurality of NUMA domains comprises an allocated portion of a processor core of a processor of the network node and an allocated portion of one or more data storage devices of the network node, and wherein each of the plurality of NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0101] Example 29 includes the subject matter of any of Examples 25-28, and wherein monitoring the quality of service level of the network node comprises monitoring workload distribution.

[0102] Example 30 includes the subject matter of any of Examples 25-29, and wherein monitoring the quality of service level of the network node comprises monitoring a saturation level of the HFI.

[0103] Example 31 includes the subject matter of any of Examples 25-30, and wherein monitoring the quality of service level of the network node comprises monitoring a throttling signal received from a proxy entity of the processor.

[0104] Example 32 includes the subject matter of any of Examples 25-31, and wherein the plurality of interconnected network nodes comprises one or more compute nodes and one or more storage nodes.

[0105] Example 33 includes the subject matter of any of Examples 25-32, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to one of the plurality of interconnected network nodes via unicast transmission.

[0106] Example 34 includes the subject matter of any of Examples 25-33, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to more than one of the plurality of interconnected network nodes via multicast transmission.

[0107] Example 35 includes the subject matter of any of Examples 25-34, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes via a transport layer of an open systems interconnection (OSI) model.

[0108] Example 36 includes the subject matter of any of Examples 25-35, and wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes at a periodic injection rate for the duration of the detected throttling condition.

[0109] Example 37 includes a network node, the network node comprising a processor; and a memory having a plurality of instructions stored thereon, the plurality of instructions, when executed by the processor, causing the network node to perform the method of any of Examples 25-36.

[0110] Example 38 includes one or more machine-readable storage media, including a plurality of instructions stored on the one or more machine-readable storage media, the plurality of instructions in response to being executed causing the network node to perform the method of any one of Examples 25-36.

[0111] Example 39 includes a network node for throttling based on quality of service in a fabric architecture, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, the network node comprising a component for monitoring the quality of service level of the network node at a host fabric interface (HFI) of the network node, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, wherein each of the plurality of interconnected network nodes is interconnected via an interconnect structure of the fabric architecture; a component for detecting a throttling condition based on a result of the monitored quality of service level; a component for generating a throttling message based on a request type associated with the detected throttling condition in response to the throttling condition having been detected; and a component for transmitting the generated throttling message to one or more of a plurality of interconnected network nodes communicatively coupled to the network node via the interconnect structure.

[0112] Example 40 includes the subject matter of Example 39, and wherein means for monitoring the quality of service level of the network node comprises means for monitoring a utilization level of one or more resources of the network node.

[0113] Example 41 includes the subject matter of any of Examples 39 and 40, and the component for the utilization level of one or more resources of the network node includes a component for monitoring at least one of a processor of the network node, one or more data storage devices of the network node, or the HFI.

[0114] Example 42 includes the subject matter of any of Examples 39-41, and wherein the means for monitoring the utilization level of one or more resources of the network node includes means for monitoring one or more of a plurality of non-uniform memory access (NUMA) domains, wherein each of the plurality of NUMA domains includes an allocated portion of a processor core of a processor of the network node and an allocated portion of one or more data storage devices of the network node, and wherein each of the plurality of NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0115] Example 43 includes the subject matter of any of Examples 39-42, and wherein the means for monitoring the quality of service level of the network node comprises means for monitoring workload distribution.

[0116] Example 44 includes the subject matter of any of Examples 39-43, and wherein the means for monitoring the quality of service level of the network node comprises means for monitoring a saturation level of the HFI.

[0117] Example 45 includes the subject matter of any of Examples 39-44, and wherein the means for monitoring the quality of service level of the network node comprises means for monitoring a throttling signal received from a proxy entity of the processor.

[0118] Example 46 includes the subject matter of any of Examples 39-45, and wherein the plurality of interconnected network nodes comprises one or more compute nodes and one or more storage nodes.

[0119] Example 47 includes the subject matter of any of Examples 39-46, and wherein the means for transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises means for transmitting the generated throttling message to one of the plurality of interconnected network nodes via unicast transmission.

[0120] Example 48 includes the subject matter of any of Examples 39-47, and wherein the means for transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises means for transmitting the generated throttling message to more than one of the plurality of interconnected network nodes via multicast transmission.

[0121] Example 49 includes the subject matter of any of Examples 39-48, and wherein the means for transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises means for transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes via a transport layer of an open systems interconnection (OSI) model.

[0122] Example 50 includes the subject matter of any of Examples 39-49, and wherein the means for transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes includes means for transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes at a periodic injection rate for the duration of the detected throttling condition.

[0123] Example 51 includes a network node for throttling based on quality of service in a fabric architecture, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, the network node comprising a processor; a host fabric interface (HFI) for facilitating data transfer between the plurality of interconnected network nodes; and one or more data storage devices storing a plurality of instructions which, when executed by the processor, cause the network node to transmit an access request for access to a shared resource of one of the plurality of interconnected network nodes communicatively coupled to the network node via an interconnect structure of the fabric architecture; receive a throttling message from one of the plurality of interconnected network nodes; identify information associated with the received throttling message; and perform a throttling action on one or more resources of the network node based on the identified information.

[0124] Example 52 includes the subject matter of Example 51, and wherein receiving the throttling message comprises receiving the throttling message via a transport layer of an open systems interconnection (OSI) model.

[0125] Example 53 includes the subject matter of any of Examples 51 and 52, and wherein identifying the information associated with the received throttling message comprises identifying at least one of a request type of the received throttling message and a source of the received throttling message.

[0126] Example 54 includes the subject matter of any of Examples 51-53, and wherein the request type of the received throttling message comprises one of a memory throttling request, an I / O throttling request, an accelerator throttling processing request, or an HFI saturation throttling request.

[0127] Example 55 includes the subject matter of any of Examples 51-54, and wherein performing the throttling action comprises reducing an injection rate of shared resource access requests to one of the plurality of interconnected network nodes.

[0128] Example 56 includes the subject matter of any of Examples 51-55, and wherein performing the throttling action comprises throttling a processor core of the processor of the network node.

[0129] Example 57 includes the subject matter of any of Examples 51-56, and wherein throttling the processor core of the processor of the network node comprises propagating a received throttling message to a proxy entity of the processor for conversion to an existing throttling signal.

[0130] Example 58 includes the subject matter of any of Examples 51-57, and wherein performing the throttling action comprises propagating the received throttling message to a software stack via a software interrupt.

[0131] Example 59 includes the subject matter of any of Examples 51-58, and wherein the one or more resources of the network node include at least one of the processor, the one or more data storage devices, or the HFI.

[0132] Example 60 includes the subject matter of any of Examples 51-59, and wherein the one or more resources of the network node include multiple non-uniform memory access (NUMA) domains, wherein each of the multiple NUMA domains includes an allocated portion of a processor core of the processor and an allocated portion of the one or more data storage devices, and wherein each of the multiple NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0133] Example 61 includes a network node for throttling based on quality of service in a structural architecture, wherein the network node is one of a plurality of interconnected network nodes of the structural architecture, the network node comprising a communication management circuit, the communication management circuit being used to transmit an access request, the access request being used to access a shared resource of one of the plurality of interconnected network nodes communicatively coupled to the network node via an interconnect structure of the structural architecture; a throttling message receiving circuit, the throttling message receiving circuit being used to (i) receive a throttling message from a host fabric interface (HFI) of one of the plurality of interconnected network nodes; and (ii) identify information associated with the received throttling message; and a throttling corresponding execution circuit, the throttling corresponding execution circuit being used to perform a throttling action on one or more resources of the network node based on the identified information.

[0134] Example 62 includes the subject matter of Example 61, and wherein receiving the throttling message comprises receiving the throttling message via a transport layer of an open systems interconnection (OSI) model.

[0135] Example 63 includes the subject matter of any of Examples 61 and 62, and wherein identifying the information associated with the received throttling message comprises identifying at least one of a request type of the received throttling message and a source of the received throttling message.

[0136] Example 64 includes the subject matter of any of Examples 61-63, and wherein the request type of the received throttling message comprises one of a memory throttling request, an I / O throttling request, an accelerator throttling processing request, or an HFI saturation throttling request.

[0137] Example 65 includes the subject matter of any of Examples 61-64, and wherein performing the throttling action comprises reducing an injection rate of shared resource access requests to one of the plurality of interconnected network nodes.

[0138] Example 66 includes the subject matter of any of Examples 61-65, and wherein performing the throttling action comprises throttling a processor core of the processor of the network node.

[0139] Example 67 includes the subject matter of any of Examples 61-66, and wherein throttling the processor core of the processor of the network node comprises propagating a received throttling message to a proxy entity of the processor for conversion to an existing throttling signal.

[0140] Example 68 includes the subject matter of any of Examples 61-67, and wherein performing the throttling action comprises propagating the received throttling message to a software stack via a software interrupt.

[0141] Example 69 includes the subject matter of any of Examples 61-68, and wherein the one or more resources of the network node include at least one of the processor, the one or more data storage devices, or the HFI.

[0142] Example 70 includes the subject matter of any of Examples 61-69, and wherein the one or more resources of the network node include multiple non-uniform memory access (NUMA) domains, wherein each of the multiple NUMA domains includes an allocated portion of a processor core of the processor and an allocated portion of the one or more data storage devices, and wherein each of the multiple NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0143] Example 71 includes a method for throttling based on quality of service across a structural architecture, the network node including transmitting an access request through a host fabric interface (HFI) of the network node, the access request being used to access a shared resource of one of a plurality of interconnected network nodes communicatively coupled to the network node via an interconnect fabric of the structural architecture; receiving a throttling message from one of the plurality of interconnected network nodes through the HFI; identifying information associated with the received throttling message through the HFI; and performing a throttling action on one or more resources of the network node based on the identified information through the HFI.

[0144] Example 72 includes the subject matter of Example 71, and wherein receiving the throttling message comprises receiving the throttling message via a transport layer of an open systems interconnection (OSI) model.

[0145] Example 73 includes the subject matter of any of Examples 71 and 72, and wherein identifying the information associated with the received throttling message comprises identifying at least one of a request type of the received throttling message and a source of the received throttling message.

[0146] Example 74 includes the subject matter of any of Examples 71-73, and wherein identifying the request type of the received throttling message comprises identifying one of a memory throttling request, an I / O throttling request, an accelerator throttling processing request, or an HFI saturation throttling request.

[0147] Example 75 includes the subject matter of any of Examples 71-74, and wherein performing the throttling action comprises reducing an injection rate of shared resource access requests to one of the plurality of interconnected network nodes.

[0148] Example 76 includes the subject matter of any of Examples 71-75, and wherein performing the throttling action comprises throttling a processor core of a processor of the network node.

[0149] Example 77 includes the subject matter of any of Examples 71-76, and wherein throttling the processor core of the processor of the network node comprises propagating a received throttling message to a proxy entity of the processor for conversion to an existing throttling signal.

[0150] Example 78 includes the subject matter of any of Examples 71-77, and wherein performing the throttling action comprises propagating the received throttling message to a software stack via a software interrupt.

[0151] Example 79 includes the subject matter of any of Examples 71-78, and wherein performing the throttling action on the one or more resources of the network node includes performing the throttling action on at least one of a processor of the network node, one or more data storage devices of the network node, or the HFI.

[0152] Example 80 includes the subject matter of any of Examples 71-79, and wherein performing the throttling action on the one or more resources of the network node includes performing the throttling action on at least one of a plurality of non-uniform memory access (NUMA) domains, wherein each of the plurality of NUMA domains includes an allocated portion of a processor core of a processor of the network node and an allocated portion of one or more data storage devices of the network node, and wherein each of the plurality of NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

[0153] Example 81 includes a network node, the network node comprising a processor; and a memory having a plurality of instructions stored thereon, the plurality of instructions, when executed by the processor, causing the network node to perform the method of any one of Examples 71-80.

[0154] Example 82 includes one or more machine-readable storage media, including a plurality of instructions stored on the one or more machine-readable storage media, the plurality of instructions, in response to being executed, causing the network node to perform the method of any one of Examples 71-80.

[0155] Example 83 includes a network node for throttling based on quality of service in a structural architecture, wherein the network node is one of a plurality of interconnected network nodes of the structural architecture, and the network node includes a component for transmitting an access request through a host fabric interface (HFI) of the network node, the access request being used for accessing a shared resource of one of the plurality of interconnected network nodes communicatively coupled to the network node via an interconnect fabric of the structural architecture; a component for receiving a throttling message from one of the plurality of interconnected network nodes through the HFI; a component for identifying information associated with the received throttling message through the HFI; and a component for performing a throttling action on one or more resources of the network node based on the identified information through the HFI.

[0156] Example 84 includes the subject matter of Example 83, and wherein means for receiving the throttling message comprises receiving the throttling message via a transport layer of an open systems interconnection (OSI) model.

[0157] Example 85 includes the subject matter of any of Examples 83 and 84, and wherein the means for identifying the information associated with the received throttling message includes means for identifying at least one of a request type of the received throttling message and a source of the received throttling message.

[0158] Example 86 includes the subject matter of any of Examples 83-85, and wherein the means for identifying the request type of the received throttling message includes means for identifying one of a memory throttling request, an I / O throttling request, an accelerator throttling processing request, or an HFI saturation throttling request.

[0159] Example 87 includes the subject matter of any of Examples 83-86, and wherein the means for performing the throttling action comprises means for reducing an injection rate of shared resource access requests to one of the plurality of interconnected network nodes.

[0160] Example 88 includes the subject matter of any of Examples 83-87, and wherein the means for performing the throttling action comprises means for throttling a processor core of a processor of the network node.

[0161] Example 89 includes the subject matter of any of Examples 83-88, and wherein the means for throttling the processor core of the processor of the network node comprises means for propagating a received throttling message to a proxy entity of the processor for conversion into an existing throttling signal.

[0162] Example 90 includes the subject matter of any of Examples 83-89, and wherein the means for performing the throttling action comprises means for propagating the received throttling message to a software stack via a software interrupt.

[0163] Example 91 includes the subject matter of any of Examples 83-90, and wherein the component for performing the throttling action on the one or more resources of the network node includes a component for performing the throttling action on at least one of a processor of the network node, one or more data storage devices of the network node, or the HFI.

[0164] Example 92 includes the subject matter of any of Examples 83-91, and wherein the means for performing the throttling action on the one or more resources of the network node includes means for performing the throttling action on at least one of a plurality of non-uniform memory access (NUMA) domains, wherein each of the plurality of NUMA domains includes an allocated portion of a processor core of a processor of the network node and an allocated portion of one or more data storage devices of the network node, and wherein each of the plurality of NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

Claims

1. A network node for throttling based on quality of service in a fabric architecture, wherein the network node is one of a plurality of interconnected network nodes of the fabric architecture, the network node comprising: processor; a host fabric interface (HFI) for facilitating data transfer between the plurality of interconnected network nodes via an interconnect structure of the fabric; as well as One or more data storage devices having a plurality of instructions stored therein, which, when executed by the processor, cause the network node to: monitoring a quality of service level of the network node; detecting a throttling condition based on results of the monitored quality of service levels; In response to having detected the throttling condition, generating a throttling message based on a request type associated with the detected throttling condition; as well as The generated throttling message is transmitted to one or more of the plurality of interconnected network nodes communicatively coupled to the network node via the interconnect fabric.

2. The network node of claim 1 , wherein monitoring the quality of service level of the network node comprises monitoring a utilization level of one or more resources of the network node, and wherein the one or more resources of the network node comprise at least one of the processor, the one or more data storage devices, or the HFI.

3. The network node of claim 2, wherein the one or more resources of the network node include a plurality of non-uniform memory access (NUMA) domains, wherein each of the plurality of NUMA domains includes an allocated portion of a processor core of the processor and an allocated portion of the one or more data storage devices, and wherein each of the plurality of NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

4. The network node of claim 1, wherein monitoring the quality of service level of the network node comprises monitoring at least one of workload distribution, a saturation level of the HFI, or a throttling signal received from an agent entity of the processor.

5. The network node of claim 1 , wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises one of: transmitting the generated throttling message to one of the plurality of interconnected network nodes via unicast transmission, or transmitting the generated throttling message to more than one of the plurality of interconnected network nodes via multicast transmission.

6. The network node of claim 1 , wherein transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes comprises transmitting the generated throttling message to the one or more of the plurality of interconnected network nodes at a periodic injection rate for a duration of the detected throttling condition.

7. A method for throttling based on quality of service across a fabric architecture, the method comprising: monitoring a quality of service level of a network node via a host fabric interface (HFI) of the network node of the fabric, wherein the network node is one of a plurality of interconnected network nodes of the fabric, wherein each of the plurality of interconnected network nodes is interconnected via an interconnect structure of the fabric; detecting a throttling condition based on the results of the monitored quality of service level by the HFI; In response to detecting the throttling condition, generating, by the HFI, a throttling message based on a request type associated with the detected throttling condition; as well as The generated throttling message is transmitted by the HFI to one or more of the plurality of interconnected network nodes communicatively coupled to the network node via the interconnect structure.

8. The method of claim 7, wherein monitoring the quality of service level of the network node comprises monitoring a utilization level of one or more resources of the network node, wherein the one or more resources of the network node comprise at least one of a processor of the network node, one or more data storage devices of the network node, or the HFI.

9. The method of claim 8, wherein the one or more resources of the network node include a plurality of non-uniform memory access (NUMA) domains, wherein each of the plurality of NUMA domains includes an allocated portion of a processor core of the processor and an allocated portion of the one or more data storage devices, and wherein each of the plurality of NUMA domains is communicatively coupled to the HFI via an on-die interconnect of the processor.

10. The method of claim 7, wherein monitoring the quality of service level of the network node comprises monitoring at least one of workload distribution, a saturation level of the HFI, or a throttling signal received from an agent entity of a processor of the network node.

Images