Multi-node system low power management

By adopting point-to-point communication and detection filter mechanisms in multi-node systems, the problem of performance degradation in multi-node systems during power down and wake-up is solved, and efficient power management and performance improvement is achieved.

CN111684426BActive Publication Date: 2024-11-05ADVANCED MICRO DEVICES INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201880088614.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-12-21
Filing Date
2018-09-19
Publication Date
2024-11-05
Estimated Expiration
2038-09-19

AI Technical Summary

Technical Problem

In modern computing devices, multi-node systems have a problem of performance degradation during power down and wake-up, especially when increasing the number of sockets to improve performance, the scale of the system becomes difficult, routing multiple connections increases the semaphore and increases the size of the semaphore. The interface area is significant, and the processing node power down and wake-up delays are significant.

Method used

Using a point-to-point communication structure between multiple nodes, efficient power management is achieved through broadcast detection and miss response mechanisms, including maintaining the detection filter when serving memory requests, updating dirty cache lines connected to system memory, and When an interrupt is detected, arbitration and fixed interrupts are processed using unicast messages to prevent broadcast detection from waking up the power-down node.

Benefits of technology

It realizes efficient power management of multi-node systems, reduces power down and wake-up delays, improves system performance, and reduces system cost and interface area.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111684426B_ABST
    Figure CN111684426B_ABST
Patent Text Reader

Abstract

Various systems, devices and methods for performing efficient power management on a multi-node computing system are disclosed. A computing system including multiple nodes utilizes a non-uniform memory access (NUMA) architecture. A first node receives a broadcast probe from a second node. The first node fakes a miss response for a powered-off third node, the miss response preventing the third node from waking up in response to the broadcast probe. Before powering off, the third node flushes its probe filter and cache and updates its system memory with the received dirty cache line. The computing system includes a master node, the master node being used to store interrupt priorities of multiple cores in the computing system for arbitration interrupts. The core stores an indication of a fixed interrupt identifier for each core in the computing system. Arbitration interrupts and fixed interrupts are handled by the core using point-to-point unicast messages rather than broadcast messages.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly to multi-node system low power management. Background Art

[0002] Description of Related Technology

[0003] The power consumption of modern integrated circuits (ICs) has increasingly become a design issue for each generation of semiconductor chips. As power consumption increases, more expensive cooling systems (such as larger fans and heat sinks) must be used to remove excess heat and prevent IC failure. However, cooling systems increase system costs. IC power dissipation constraints are not only a problem for portable computers and mobile communication devices, but also for desktop computers and servers using high-performance microprocessors. These microprocessors include multiple processor cores or cores and multiple pipelines within a core.

[0004] A variety of computing devices (such as a variety of servers) utilize heterogeneous integration that integrates multiple types of ICs to provide system functions. Multiple functions are placed in a processing node, and the multiple functions include audio / video (A / V) data processing, other high-speed data parallel applications in the medical and commercial fields, processing instructions of a general instruction set architecture (ISA), digital, analog, mixed signal, and radio frequency (RF) functions, etc. There are multiple options for placing a processing node in a system package to integrate multiple types of ICs. The processing node includes multiple ICs, communication structures, power controllers, interrupt controllers, link interfaces, etc.

[0005] In some computing devices, a system on a chip (SOC) is used, while in other computing devices, smaller and higher yield chips of a processing node are packaged as large chips in a multi-chip module (MCM). Some computing devices include three-dimensional integrated circuits (3D ICs) that utilize die stacking technology with silicon interposers, through silicon vias (TSVs), and other mechanisms to vertically stack and electrically connect two or more dies in a system-in-package (SiP).

[0006] Regardless of which system package is chosen, powering off a computing system with multiple sockets, each with a copy of the selected package, is complicated. Each package includes a power controller, and therefore, the system has multiple power controllers. If each power controller is connected to every other power controller in the system, it becomes easier to communicate when the system is powered off. However, if each power controller is to be directly connected to every other power controller in the system, it becomes difficult to scale the system (such as increasing the number of sockets to improve performance). In addition, routing multiple connections increases the amount of signals between sockets and increases the interface area. In addition, powering off a processing node includes multiple steps. When a given processing node is powered off and the given processing node is awakened shortly thereafter, a significant delay occurs before the given processing node is ready to process tasks. Therefore, when the node is powered off and powered back on shortly thereafter, performance is reduced.

[0007] In view of the foregoing, there is a need for efficient methods and systems for performing efficient power management for multi-node computing systems. Summary of the invention

[0008] In a first aspect, an embodiment of the present application provides a computing system comprising: a plurality of nodes, each node comprising: one or more clients configured to process an application; a plurality of links coupled to other nodes among the plurality of nodes; and a power controller; and wherein in response to receiving a broadcast detection from a first node among the plurality of nodes, a second node among the plurality of nodes is configured to: generate a miss response for a third node among the plurality of nodes that is directly connected to the second node in response to determining that the third node has lost power; and send the miss response to the first node.

[0009] In one embodiment, in response to generating the broadcast probe, the first node of the plurality of nodes is configured to: generate a miss response for a fourth node of the plurality of nodes directly connected to the first node in response to determining that the fourth node is powered off.

[0010] In one embodiment, the first node is configured to: maintain a probe filter when servicing memory requests; and in response to determining that each of the one or more clients is idle: invalidate each entry in the probe filter; and update a system memory connected to the first node with dirty cache lines received as a result of invalidating the entries in the probe filter; and power down the multiple links of the first node and the one or more clients.

[0011] In one embodiment, wherein at least one of the one or more clients includes a plurality of cores, and wherein a master node of the plurality of nodes is configured to store interrupt priorities of the plurality of cores of the plurality of nodes for arbitrating interrupts.

[0012] In one embodiment, in response to detecting an arbitration interrupt, the first node is configured to: send a request to the master node to identify a given kernel having the lowest priority and a given node including the given kernel; and based on a response from the master node identifying the given kernel and the given node, send a request to the given node to handle the arbitration interrupt with the given kernel.

[0013] In one embodiment, in response to determining that the priority of the kernel for arbitrating interrupts is updated in the first node, the first node is configured to send the updated priority to the master node for updating the priority of the kernel for arbitrating interrupts stored on the master node.

[0014] In one embodiment, wherein at least one of the one or more clients includes a plurality of cores, and wherein at least the first node is configured to store an indication of a fixed interrupt identifier (ID) for each core of the plurality of cores.

[0015] In one embodiment, in response to detecting a fixed interrupt, the first node is configured to: identify a given node including a given core having a fixed interrupt ID that matches the fixed interrupt ID of the detected fixed interrupt; and send a request to the given node to process the fixed interrupt with the given core.

[0016] In a second aspect, an embodiment of the present application provides a method, which includes: processing an application by multiple nodes, each node including: one or more clients, the one or more clients configured to process the application; multiple links, the multiple links coupled to other nodes among the multiple nodes; and a power controller; and in response to receiving a broadcast detection from a first node among the multiple nodes, a second node among the multiple nodes is configured to: generate a miss response for a third node among the multiple nodes in response to determining that a third node directly connected to the second node has lost power; and send the miss response to the first node by the second node.

[0017] In one embodiment, in response to generating the broadcast probe: generating, by the first node, a miss response for a fourth node of the plurality of nodes that is directly connected to the first node in response to determining that the fourth node is powered off.

[0018] In one embodiment, the method further includes: maintaining a probe filter by the first node when servicing memory requests; and in response to determining that each of the one or more clients in the first node is idle: invalidating each entry in the probe filter; and updating a system memory connected to the first node with a dirty cache line received as a result of invalidating the entry in the probe filter; and powering down the multiple links and the one or more clients of the first node.

[0019] In one embodiment, wherein at least one of the one or more clients includes a plurality of cores, the method further comprises: storing, by a master node among the plurality of nodes, interrupt priorities of the plurality of cores among the plurality of nodes for arbitrating interrupts.

[0020] In one embodiment, in response to detecting an arbitration interrupt, the method further includes: sending, by the first node, a request to the master node to identify a given kernel having the lowest priority and a given node including the given kernel; and sending, by the first node, a request to the given node to handle the arbitration interrupt with the given kernel based on a response from the master node identifying the given kernel and the given node.

[0021] In one embodiment, in response to determining that the priority of the kernel for arbitrating interrupts has been updated in the first node, the method also includes: the first node sending the updated priority to the master node for updating the priority of the kernel for arbitrating interrupts stored on the master node.

[0022] In one embodiment, wherein at least one of the one or more clients includes a plurality of cores, and the method further comprises: storing, by at least the first node, an indication of a fixed interrupt identifier (ID) for each core of the plurality of cores.

[0023] In one embodiment, in response to detecting a fixed interrupt, the method also includes: identifying, by the first node, a given node that includes a given kernel, the given kernel having a fixed interrupt ID that matches the fixed interrupt ID of the detected fixed interrupt; and sending, by the first node, a request to the given node to process the fixed interrupt with the given kernel.

[0024] In a third aspect, an embodiment of the present application provides a processing node, comprising: one or more clients, which are configured to process applications; multiple links, which are coupled to other nodes among multiple nodes; and control logic; and wherein in response to receiving a broadcast detection from a first node among the multiple nodes, the control logic is configured to: generate a miss response for a second node among the multiple nodes directly connected to the processing node in response to determining that the second node is powered off; and send the miss response to the first node.

[0025] In one embodiment, wherein in response to generating the broadcast probe, the control logic is configured to: generate a miss response for a third node of the plurality of nodes that is directly connected to the processing node in response to determining that the third node is powered down.

[0026] In one embodiment, the control logic is configured to: maintain a sniff filter when servicing memory requests; and in response to determining that each of the one or more clients is idle: invalidate each entry in the sniff filter; and update a system memory connected to the processing node with a dirty cache line received as a result of invalidating the entry in the sniff filter; and power down the plurality of links of the first node and the one or more clients.

[0027] In one embodiment, wherein at least one of the one or more clients includes a plurality of cores, and wherein the processing node is a master node among the plurality of nodes, the master node is configured to store interrupt priorities of the plurality of cores among the plurality of nodes for arbitrating interrupts. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a block diagram of one embodiment of a processing node.

[0030] Figure 2 is a block diagram of one embodiment of a multi-node computing system.

[0031] Figure 3 is a flow chart of one embodiment of a method for performing power management on a multi-node computing system.

[0032] Figure 4 is a flow chart of another embodiment of a method for performing power management on a multi-node computing system.

[0033] Figure 5 is a flow chart of another embodiment of a method for performing power management on a multi-node computing system.

[0034] Figure 6 is a flow chart of another embodiment of a method for performing power management on a multi-node computing system.

[0035] Although the present invention is susceptible to various modifications and alternative forms, specific embodiments are shown in the drawings by way of example and are described in detail herein. However, it should be understood that the drawings and detailed description thereof are not intended to limit the invention to the specific forms disclosed, but on the contrary, the invention is intended to cover all modifications, equivalents and alternatives falling within the scope of the invention as defined by the appended claims. DETAILED DESCRIPTION

[0036] In the following description, many specific details are set forth to provide a thorough understanding of the methods and mechanisms proposed herein. However, it will be appreciated by those of ordinary skill in the art that various embodiments may be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid confusion with the methods described herein. It should be understood that for simplicity and clarity of illustration, the elements shown in the accompanying drawings are not necessarily drawn to scale. For example, the size of some of the elements may be enlarged relative to other elements.

[0037] Various systems, devices, methods and computer-readable media for performing efficient power management on a multi-node computing system are disclosed. In various embodiments, a processing node includes one or more clients for processing applications. Examples of clients located within a node include a general-purpose central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an input / output (I / O) device, and the like. A processing node also includes at least a management controller and a plurality of link interfaces for transmitting messages to other nodes. As used herein, a processing node is also referred to as a node.

[0038] In some embodiments, the computing system is a multi-node system that includes multiple copies of a node, each copy having its own one or more clients and power controller. In various embodiments, the computing system utilizes a shared memory architecture in which a given processor in a given node accesses its local memory with a latency less than accessing the remote memory of another node. In other words, the computing system utilizes a non-uniform memory access (NUMA) architecture, and each of the nodes in the computing system is a NUMA node.

[0039] A power controller of a first node among a plurality of nodes determines when each of its one or more clients is idle. In addition, the power controller determines when there are no remote memory requests attempting to access system memory of the first node, such as a dynamic random access memory (DRAM) of the first node. In one embodiment, when the power controller determines that these conditions are met, the power controller sends an indication of a power-off state specifying the first node to one or more nodes on one or more of its links. The first node then performs a plurality of steps to power off, such as flushing its cache, invalidating each entry of a detection filter, and updating a system memory connected to the first node with a dirty cache line received as a result of invalidating entries in the detection filter. The steps then include disabling a driver for a link interface, disabling a clock for one or more clients and a communication structure for transmitting traffic between the one or more clients and a memory controller, and setting the system memory to perform self-refresh when using the dynamic random access memory (DRAM).

[0040] In one embodiment, in order to increase the amount of time that the first node remains powered off, flushing the cache and invalidating the probe filter as described above are performed. In other embodiments, a second node directly connected to the powered off first node receives a broadcast probe from a third node, and when the second node determines that the first node is powered off, the second node generates a miss response for the powered off first node. The second node does not send a broadcast probe to the first node, which will wake up the first node and cause the first node to transition from a powered off state to an active state. The second node sends a miss response to the third node on behalf of the powered off first node.

[0041] In some embodiments, the computing system includes a master node, the master node is used to store the interrupt priority of multiple processor cores in the computing system, wherein the interrupt priority corresponds to the arbitration interrupt. Therefore, the arbitration interrupt is handled by the processor core with a point-to-point unicast message rather than a broadcast message. Similarly, multiple processor cores in the computing system use a point-to-point unicast message rather than a broadcast message to handle fixed interrupts by supporting the storage of a fixed interrupt identifier (ID) of each processor core in the multiple processor cores. Therefore, the powered-off node is not awakened by a broadcast detection for processing interrupts. These and other embodiments are possible and are foreseeable.

[0042] refer to Figure 1, a generalized block diagram of one embodiment of a processing node 100 is shown. As shown, processing node 100 includes a communication structure 120 located between each of client 110, detection filter 113, memory controller 130, power controller 170, and link interface 180. In some embodiments, the components of processing node 100 are separate dies located on an integrated circuit (IC), such as a system on a chip (SOC). In other embodiments, the components are separate dies located in a system-in-package (SiP) or a multi-chip module (MCM).

[0043] In various embodiments, communication fabric 120 transmits traffic back and forth between client 110 and memory controller 130 and includes interfaces for supporting corresponding communication protocols. In some embodiments, communication fabric 120 includes at least queues for storing requests and responses, selection logic for arbitrating between received requests before sending them across the internal network, logic for constructing and decoding packets, and logic for selecting routes for packets.

[0044] In the illustrated embodiment, client 110 includes a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, and a hub 116. Hub 116 is used to communicate with a multimedia engine 118. CPU 112, GPU 114, and multimedia engine 118 are examples of computing resources that can process applications. Although not shown, in other embodiments, other types of computing resources are included in client 110. In one embodiment, GPU 114 includes a high-speed parallel data micro-architecture with a significant number of parallel execution channels. In one embodiment, micro-architecture uses a single instruction multiple data (SIMD) pipeline for parallel execution channels. Multimedia engine 118 includes a processor for processing audio data and visual data of multimedia applications.

[0045] In some embodiments, each of the one or more processor cores in CPU 112 includes a circuit for executing instructions according to a given selected instruction set architecture (ISA). In various embodiments, each of the processor cores in CPU 112 includes a multithreaded micro-architecture of a superscalar architecture for processing instructions of a given ISA. As used herein, processor cores are also referred to as cores. In one embodiment, CPU 112 includes 8 cores, and each core includes 2 hardware threads for processing 2 software threads. Therefore, in this embodiment, CPU 112 can process 16 threads simultaneously. In other embodiments, any number of threads for simultaneous multithreading are supported by CPU 112.

[0046] Link interface 180 supports the communication between processing node 100 and other processing nodes by transmitting messages on the link. In various embodiments, the message sent between nodes across the link includes an indication of the operating state of one or more nodes, a response to a request, an interruption and other information. In various embodiments, each link is a point-to-point communication channel between two nodes. At the physical level, the link includes one or more channels. In some embodiments, link interface 180, corresponding links and other nodes include communication protocol connections, such as PCIe (Peripheral Component Interconnect High Speed), InfiniBand, RapidIO, HyperTransport, etc. In some embodiments, link interface 180 includes control logic and buffers or queues for communicating with other nodes through interconnection links.

[0047] Although a single memory controller 130 is shown, in other embodiments, another number of memory controllers are used in processing node 100. In various embodiments, memory controller 130 receives memory requests from client 110 via communication fabric 120, schedules the memory requests, and sends the scheduled memory requests to one or more of system memory and main memory. Memory controller 130 also receives responses from system memory and main memory and sends the responses to corresponding request sources in client 110. In various embodiments, system memory is used to fill data from main memory via I / O controller and bus 160 and memory bus 150. A corresponding cache fill line with the request block is transferred from main memory to a corresponding one of the cache memory subsystems in client 110 to complete the original memory request. The cache fill line is placed in one or more levels of cache.

[0048] In some embodiments, the address space of processing node 100 is divided between at least CPU 112, GPU 114 and hub 116 and one or more other components, such as input / output (I / O) peripherals (not shown) and other types of computing resources. A memory map is maintained to determine which addresses are mapped to which component, and therefore to which of CPU 112, GPU 114 and hub 116 a memory request for a particular address should be routed. In one embodiment, system memory is one of a variety of dynamic random access memories (DRAMs), and memory controller 130 supports a corresponding protocol. The protocol determines values ​​for information transmission, such as the number of data transfers per clock cycle, signal voltage levels, signal timing, signal and clock phases, and clock frequencies. In some embodiments, main memory is one of a variety of types of non-volatile random access data secondary storage devices. Examples of main memory are hard disk drives (HDDs) and solid state drives (SSDs).

[0049] In one embodiment, the power controller 170 collects data from the client 110. In some embodiments, the power controller 170 also collects data from the memory controller 130. In some embodiments, the collected data includes a predetermined sampled signal. The switching of the sampled signal indicates the amount of switched capacitance. Examples of selected signals to be sampled include clock gate enable signals, bus driver enable signals, content addressable memory (CAM), CAM word line (WL) driver mismatches, etc. In one embodiment, the power controller 170 collects data to characterize the power consumption in the node 100 during a given sampling interval.

[0050] In some embodiments, on-die current sensors and temperature sensors in the node 100 also send information to the power controller 170. The power controller 170 uses one or more of the sensor information, the count of issued instructions or issued threads, and the sum of weighted sampled signals to estimate the power consumption of the node 100. If the node 100 is operating above (below) a threshold limit, the power controller 170 reduces (or increases) power consumption. In some embodiments, the power controller 170 selects a corresponding power-performance state (P-state) for each of the computing resources in the client 110. The P-state includes at least an operating voltage and an operating clock frequency. In various embodiments, the power controller 170 and the node 100 support N P-states, where N is a non-zero positive integer.

[0051] In various embodiments, processing node 100 is one of the multiple nodes in a multi-node system. In one embodiment, the system is one of the multiple servers. Sometimes, jobs sent to a server in batches are not assigned to each server. A given server may become idle for a significant amount of time. In order to reduce power consumption, multiple nodes within the server may be powered off. In various embodiments, power controller 170 detects when one or more of clients 110 become idle. If each of clients 110 becomes idle, in one embodiment, power controller 170 relays an indication of a power-off state to one or more nodes on one or more of its links via link interface 180. In some embodiments, power controller 170 sends a message to each of CPU 112 and GPU 114 to flush each cache in its cache memory subsystem.

[0052] Additionally, the power controller 170 sends messages to invalidate each entry of the snoop filter 113 and to update the system memory with the dirty cache lines received as a result of invalidating the entries in the snoop filter 113. The power controller 170 then sends messages to disable the driver of the link interface, disable the clock and communication structure 120 of the client 110, the memory controller 130, and set the system memory to perform self-refresh when using dynamic random access memory (DRAM).

[0053] Many directory-based cache coherence protocols include communicating between various caches within a computing system using messages or probes passed from a coherence point such as a memory controller. In one embodiment, the communication structure 120 maintains a probe filter 113. In various embodiments, the probe filter 113 is implemented as a directory with multiple entries. The entries of the probe filter 113 contain information indicating that there is a cache copy of the data. In some embodiments, each entry stores a valid bit, a tag for a cache line, an indication of the cache coherence state of a specified cache line, a node ownership identifier (ID), one or more node IDs of remote nodes where a remote client stores a cache copy of the data in a local system memory, a clean / dirty state, and the like. In some embodiments, the cache coherence states are modified, owned, exclusive, shared, and invalid.

[0054] An entry in the snoop filter 113 tracks each cache line in the processing node 100 that has a copy stored in the cache of another processing node in the computing system. If a line stored in system memory is either cached locally in the processing node 100 or cached remotely in another processing node, an entry is allocated in the snoop filter 113 for the line. The absence of an allocated entry in the snoop filter 113 entries indicates that the particular line is not cached. Rather than broadcasting the snoop unconditionally, the snoop filter 113 is used to determine the particular subsystems (which may contain cached copies of the data) to which the snoop needs to be transmitted in order to cause a particular coherency action.

[0055] In various embodiments, each of the multiple threads being processed by CPU 112 can assert an interrupt and receive the address of an interrupt service routine. In various embodiments, processing node 100 is a node of a multi-node computing system that supports an interrupt redirection protocol for routing interrupts in a multi-processor computing system. This protocol sends interrupts between processors in multiple nodes of a computing system. Based on an interrupt mode or interrupt type, an interrupt is sent from a processor that detects an interrupt to a specific processor in a specific node based on an interrupt identifier (ID). In one example, ID is used to identify a specific processor in a specific node. As used herein, these types of interrupts are referred to as "fixed interrupts". The processor selected for processing fixed interrupts is fixed based on ID. Similarly, based on different interrupt modes or different interrupt types, an interrupt is sent from a processor that detects an interrupt to a processor in a specific node based on arbitration. In one example, an interrupt priority is assigned to each hardware thread or each kernel in a processor, and one or more processors with the lowest priority are selected by arbitration logic to process asserted interrupts. As used herein, these types of interrupts are referred to as "arbitration interrupts". In some embodiments, the protocol used to send interrupts between different processors is the APIC (Advanced Programmable Interrupt Controller) protocol, and the identifier used to identify a core or a hardware thread within a core is an APIC ID.

[0056] In one embodiment, in addition to multiple interrupts asserted by the kernel on the remote CPU, the interrupt controller 111 receives multiple interrupts asserted by multiple threads running on the CPU 112 through the link interface 180. The interrupt controller 111 also routes the interrupt to other kernels in the CPU 112 and other kernels in the remote CPU through the link interface 180 based on a selected interrupt routing protocol such as the APIC protocol. In some instances, the interrupt controller 111 sends the interrupt to other kernels for servicing the interrupt. Other kernels, whether local or remote, serve the interrupt by processing a given interrupt service routine, which is also referred to as an interrupt handler. In some embodiments, the interrupt controller 111 is located outside the CPU 112 and receives interrupts from multiple other sources (such as GPU 114, hub 116, I / O peripherals, and remote computing devices) through the communication structure 120.

[0057] In various embodiments, processing node 100 is a master node in a multi-node system, and the master node is used to store the interrupt priority for arbitration interruption of multiple processor cores in the computing system. In previous examples, CPU 112 has 8 kernels, and each kernel can handle 2 threads simultaneously. Therefore, in one embodiment, CPU 112 has 16 interrupt priorities for arbitration interruption, and an interrupt priority is used for each of 16 hardware threads, and each kernel has 2 interrupt priorities. In other embodiments, the granularity of the interrupt priority for arbitration interruption is located at the kernel level, and each of the multiple hardware threads of a given kernel has the same interrupt priority for arbitration interruption. In some other embodiments, each thread is provided with the same interrupt priority. Other combinations of the interrupt priority distribution of thread are possible and conceivable.

[0058] If the computing system has 4 nodes, each node has a CPU similar to CPU 112, and each core has 2 interrupt priority levels for arbitration interrupts, then in some embodiments, the computing system has 64 interrupt priority levels. However, for arbitration interrupts, in one embodiment, the hardware thread (or core) with the lowest interrupt priority is assigned to service the arbitration interrupt. In some embodiments, the interrupt controller 111 stores 4 interrupt priority levels, one interrupt priority level for arbitration interrupts for each of the 4 nodes. An interrupt priority level for a given node is the lowest interrupt priority level among the 16 interrupt priority levels assigned to the core in the given node.

[0059] When the kernel in the first node other than the master node asserts an arbitration interrupt, the first node sends a message to the interrupt controller 111 of the master node through the link interface 180 and the communication structure 120 point-to-point. In this case, the first node does not use a broadcast message to notify each other node to send one or more interrupt priorities to the first node. In response to the point-to-point communication from the first node, the interrupt controller 111 of the master node determines the lowest interrupt priority among the 4 stored interrupt priorities. In one embodiment, the interrupt controller 111 of the master node also identifies the second node, which includes a kernel with the lowest identified interrupt priority among the 4 stored interrupt priorities. In various embodiments, the interrupt controller 111 of the master node sends the arbitration interrupt to the identified kernel in the second node with a point-to-point message. Therefore, the arbitration interrupt is processed with a point-to-point unicast message rather than a broadcast message. In various embodiments, when the local kernel in the master node, input / output (I / O) device or other device asserts an arbitration interrupt rather than asserting an arbitration interrupt by a remote node (such as the first node in the above example), a similar point-to-point communication is used.

[0060] In various embodiments, software (such as an operating system) reassigns interrupt priorities for arbitration interrupts of cores within the powered-down processing node 100. In one embodiment, the software reassigns the interrupt priority to a maximum value or at least a value greater than the interrupt priority of the active node. Thus, the master node does not select the powered-down node to wake up and service an arbitration interrupt asserted in the computing system.

[0061] When a core in CPU 112 asserts an arbitration interrupt, a message is sent to interrupt controller 111. In response, interrupt controller 111 sends the lowest interrupt priority of the four stored interrupt priorities in the response. The remaining steps are the same as above, such as sending the arbitration interrupt to the identified core in the node identified in the response in unicast communication. Similarly, no broadcast message is sent. Therefore, the powered-off node is not awakened by the broadcast detection for handling arbitration interrupts. In addition to arbitration interrupts, unicast messages are also used for fixed interrupts instead of broadcast messages.

[0062] refer to Figure 2 , a generalized block diagram of an embodiment of a multi-node computing system 200 is shown. In the illustrated embodiment, encapsulation 210 includes nodes 220A and 220B directly connected to each other by link 240. In some embodiments, each of encapsulation 210 and 230 is a multi-chip module (MCM) placed in a socket of a multi-hole socket motherboard in a server. As shown, node 220A includes client 222A-222C, power controller 224 and interrupt controller (IC) 228. Instances of client 222A-222C are CPU, GPU, multimedia engine, I / O peripherals, etc. As shown, at least client 222A includes a detection filter (PF) 223.

[0063] In some embodiments, the interrupt controller 228 is located in one of the clients 222A-222C, rather than being used as a standalone controller. In various embodiments, the power controller 224 has ( Figure 1 For ease of illustration, link interfaces, communication structures, memory interfaces, phase-locked loops (PLLs), or other clock generation circuits are not shown.

[0064] In one embodiment, the links 240-246 are connected using a communication protocol, such as PCIe, InfiniBand, RapidIO, HyperTransport, etc. In some embodiments, in addition to the links 240-246, the computing system 200 also includes other links between the nodes 220A-220D. In one embodiment, these other links are used for data transmission to service the request, while the links 240-246 are used to transmit messages such as for negotiating a system-wide power down.

[0065] In various embodiments, computing system 200 uses a hierarchical topology in which each of nodes 220A-220D is a requester. At the low level or first level of the hierarchy, each node is directly connected to each other node. For example, node 220A is directly connected to node 220B in encapsulation 210 via link 240. Similarly, node 220C is directly connected to node 220D in encapsulation 230 via link 242. At the higher level or second level of the hierarchy, each node is connected to a single node in other hierarchies. For example, node 220A in encapsulation 210 is directly connected to node 220C in encapsulation 230 via link 244. However, node 220A is not directly connected to node 220D in encapsulation 230. Similarly, node 220B in encapsulation 210 is directly connected to node 220D in encapsulation 230 via link 246. However, node 220B is not directly connected to node 220C in encapsulation 230. In other embodiments, a tree or ring topology is used, wherein each intermediate node between the root node and the leaf nodes is directly connected to a single upstream node and directly connected to a single downstream node. In other embodiments, a variety of other topologies are possible and contemplated.

[0066] In one example, if node 220B loses power, then in order to increase the amount of time that node 220B remains powered off, in one embodiment, flushing the cache and invalidating the probe filter 223 is performed. In addition, another node (such as node 220A) directly connected to the powered-off node 220B pretends to respond to a specific broadcast probe on behalf of the powered-off node 220B. In one example, node 220A receives a broadcast probe from node 220C to search for the latest valid copy of the cache line. Since node 220B flushes one or more of its cache memory subsystems before powering off, node 220B cannot have a copy of the requested cache line.

[0067] When node 220A determines that node 220B is powered off, node 220A generates a miss response for powered-off node 220B. In some embodiments, when node 220D determines that node 220B is powered off, node 220D also generates a miss response on behalf of powered-off node 220B. Any one of nodes 220A and 220D does not send a broadcast probe to powered-off node 220B, which will wake up node 220B and cause node 220B to transition to an active state from a powered-off state, just to report a miss response due to its empty cache. One or more representatives of powered-off node 220B in node 220A and node 220D send a miss response to node 220C. One or more of power controller 224 and logic for link interface (not shown) detect conditions and node 220B power-off for broadcast probes of cache lines. Therefore, logic represents powered-off node 220B to fake a miss response.

[0068] In various embodiments, computing system 200 utilizes a shared memory architecture in which a given processor in a given node of nodes 220A-220D accesses its local memory with less latency than accessing remote memory of another node. In other words, computing system 200 utilizes a non-uniform memory access (NUMA) architecture, and each of nodes 220A-220D is a NUMA node. In one embodiment, when servicing a memory request, a communication structure (not shown) in node 220A maintains a probe filter (PF) 223. In various embodiments, probe filter 223 has functionality and stores information related to ( Figure 1 ) detection filter 113 of the same type of information.

[0069] In one embodiment, the entry in the detection filter 223 includes information indicating that one or more of the nodes 220B-220D contain a shared copy of the data block. In response to the command to exclusively access the block, an invalid detection is transmitted to one or more of the nodes 220B-220D. Generally speaking, the node that issues the request in the node 220A-220D is called a "request" node. If the memory address of the request is mapped to the system memory coupled to a node in the node 220A-220D, the node is called a "home" node. If the cache in a node in 220A-220D currently stores the latest copy of the memory block, the node is called an "owner" node. The detection is used to determine whether the cache has a copy of the block and optionally indicates that the cache should make the block become a state. Each node receiving the detection checks its detection filter and responds to the detection. Once all detection responses are received, the command advances to completion.

[0070] When the power controller 224 of the node 220A determines that each of the clients 222A-222C is idle and there are no remote memory requests attempting to access the system memory of the node 220A, in one embodiment, the system-wide power controller 224 sends an indication of the power-down state of the specified node 220A to one or more of the nodes 2202B-220D. The system-wide power controller 224 then sends a message to the clients 222A-222C to flush their caches and invalidate each entry of a probe filter, such as the probe filter 223. The node 220A updates its system memory with the dirty cache lines received as a result of invalidating the entries in the probe filter 223.

[0071] In some embodiments, after performing the above steps, the power controller 224 continues to power off the node 220A. In other embodiments, after performing the above steps, if the given delay has expired, the power controller 224 continues to power off. If the given delay has not expired and the task is received, in one embodiment, the power controller 224 transitions the node 220A from the idle state to the active state, sends an indication of the active state on each of its links 240 and 244, and sends an indication of the processing received task to the client 222A-222C. In some other embodiments, the power controller 224 uses additional information and the delay to determine whether to continue to power off the node 220A or process the received task. Examples of additional information are quality of service (QoS) or other priority levels of tasks, process or software thread identifiers (IDs), hardware IDs of one of the identification clients 222A-222C, etc. If the power controller 224 determines that the node 220A is losing power, then in one embodiment, the power controller 224 disables the driver for the link interface, disables the clocks for the clients 222A-222C, and sets the system memory to perform self-refresh when using DRAM.

[0072] The wake-up condition may occur at any time including shortly after the power controller 224 has powered down the node 220A. The wake-up condition includes one or more of the following: receiving a remote request for a local cache probe from an active node, receiving a remote memory request for local data in the system memory from an active node, receiving an assigned task from an operating system, and the like. The above steps include: flushing the cache before powering down, invalidating the probe filter before powering down, storing a fixed interrupt ID in each node in the nodes 220A-220D, storing the lowest interrupt priority in the master node in the nodes 220A-220D, and responding to a broadcast probe camouflage miss on behalf of the powered-down node seeking a specific cache line. In addition, in addition to maintaining the initial fixed interrupt ID without updating, software such as an operating system helps extend the amount of time a node remains in a powered-down state by scheduling tasks and migrating data to active nodes rather than waking up the powered-down node to process the task.

[0073] In various embodiments, node 220A is a master node in computing system 200, and interrupt controller 228 stores interrupt priorities of multiple processor cores for arbitration interrupts in computing system 200. In one embodiment, the steps for handling arbitration interrupts with unicast messages rather than broadcast messages are the steps previously described for processing node 100. For example, arbitration interrupts are serviced by the processor core selected by arbitration logic based on the lowest interrupt priority. In some embodiments, interrupt controller 228 stores a fixed interrupt identifier (ID) for each of the processor cores in computing system 200. In one embodiment, the fixed interrupt identifier (ID) is an advanced programmable interrupt controller (APIC) ID. In contrast to arbitration interrupts, when a given processor core in computing system 200 asserts a fixed interrupt, the given processor core needs to send the fixed interrupt to a specific processor core with a fixed interrupt ID that matches the asserted fixed interrupt.

[0074] In one embodiment, fixed interrupt ID is assigned in a sequential manner by software such as an operating system. Therefore, the interrupt controller 228 in the node 220A and each interrupt controller in the nodes 220B-220D store the scope of fixed interrupt ID, rather than storing each independent fixed interrupt ID. Using a previous example in which the CPU includes 8 kernels, each kernel includes 2 hardware threads for processing 2 software threads, the CPU has 16 fixed interrupt IDs. In the illustrated embodiment, computing system 200 has 4 nodes, each node has a similar CPU, and therefore there are 64 fixed interrupt IDs. Not that each node in the nodes 220A-220D uses 64 entry tables to store independent fixed interrupt IDs, but in one embodiment, each node in the nodes 220A-220D stores 2 fixed interrupt IDs for each node, or stores a total of 8 fixed interrupt IDs.

[0075] In one embodiment, each node in nodes 220A-220D stores the start fixed interrupt ID and the end fixed interrupt ID in a series of sequential fixed interrupt IDs for each node in computing system 200. For example, if the operating system sequentially assigns fixed interrupt IDs 0 to 15 to node 220A, each node in nodes 220A-220D stores values ​​0 and 15 for node 220A, rather than each of the sixteen fixed interrupt IDs. When a given kernel in node 220C in computing system 200 asserts a fixed interrupt with fixed interrupt ID 9, a relatively small stored search indication node 220A to the fixed interrupt ID of computing system 200 has assigned a processor core to serve the fixed interrupt with ID 9. Then, node 220C sends a unicast message to node 220A to serve the fixed interrupt with the processor core assigned with fixed interrupt ID 9. Broadcast detection is not sent to find that the nodes in nodes 220A-220D have assigned a processor core to serve the fixed interrupt with ID 9.

[0076] Although relatively rare, the operating system can reallocate fixed interrupt ID.In some embodiments, one or more snoops in nodes 220A-220D allocate initial software copies of fixed interrupt ID.In one embodiment, known position (such as memory address range), and check to determine whether any change occurs after performing read operation.In another embodiment, call software process to check initial value and report result.If the update of initial fixed interrupt ID occurs, then in one embodiment, each of nodes 220A-220D invalidates the copy of its fixed interrupt ID.Then, use broadcast message to serve fixed interrupt.In another embodiment, each of nodes 220A-220D updates the copy of its fixed interrupt ID.Then, still use unicast message to serve fixed interrupt.

[0077] Reference now Figure 3 , shows one embodiment of a method 300 for performing power management on a multi-node computing system. For discussion purposes, this embodiment (and Figures 4 to 6 ) in the steps. However, it should be noted that in various embodiments of the described method, one or more of the described elements are performed simultaneously, performed in a different order than shown, or omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or devices described herein is configured to implement method 300.

[0078] In a multi-node system, a fixed interrupt identifier (ID) is initialized (frame 302). The fixed interrupt ID specifies which processor core or which hardware thread in the processor core is assigned to serve the fixed interrupt with a matching fixed interrupt ID. In various embodiments, the operating system initializes the fixed interrupt ID, and in one embodiment, the fixed interrupt ID is initialized in a sequential manner. The fixed interrupt ID is broadcast to the nodes in the computing system (frame 304). For example, the operating system sends an indication of the fixed interrupt ID of the computing system to each of the nodes. Alternatively, one or more nodes snoop a specific memory location to obtain the fixed interrupt ID of the computing system. In some other instances, one or more nodes perform a system call to the operating system to determine the fixed interrupt ID of the computing system. Each node updates a copy of the fixed interrupt ID of each node based on the received fixed interrupt ID (frame 306). Therefore, a broadcast message is not required to process fixed interrupts.

[0079] The plurality of nodes processes the task (block 308). One or more clients within the node execute a computer program or software application. In some embodiments, the plurality of nodes are located within a multi-socket server and a batch job is received by an operating system, which assigns the task to one or more of the plurality of nodes. In various embodiments, the multi-node computing system utilizes a non-uniform memory access (NUMA) architecture, and each of the nodes is a NUMA node.

[0080] If the first node asserts the fixed interrupt ("yes" branch of conditional box 310) and is still using the stored fixed interrupt ID ("yes" branch of conditional box 312), then a second node that services the fixed interrupt is identified based on the stored fixed interrupt ID (box 314). The fixed interrupt is sent to the second node in a unicast message (box 324). No broadcast message is used to service the fixed interrupt. Therefore, if the second node is active, no powered-down node is awakened to service the fixed interrupt.

[0081] If no node asserts a fixed interrupt (the "no" branch of conditional box 310), then the software copy of the initialized fixed interrupt ID is snooped (box 316). As previously described, one or more of the nodes in the computing system snoop the initial software copy of the assigned fixed interrupt ID. In one embodiment, the location (such as a memory address range) is known, and a check is performed after the read operation to determine whether any changes have occurred. In another embodiment, a software process is called to check the initial value and report the result. If it is determined that any update has occurred (the "yes" branch of conditional box 318), the storage of the fixed interrupt ID in multiple nodes is updated or invalidated (box 320). The decision to invalidate or update is a design choice based on the update frequency and the way to store a copy of the fixed interrupt ID. As previously described, the range of fixed interrupt IDs is stored instead of each individual fixed interrupt ID. The storage range reduces the amount of storage used.

[0082] If it is determined that no update has occurred (the "No" branch of conditional box 318), the control flow of method 300 returns to box 308, where multiple nodes process the task. If the first node asserts the fixed interrupt (the "Yes" branch of conditional box 310) and the stored fixed interrupt ID is no longer used (the "No" branch of conditional box 312), a second node servicing the fixed interrupt is identified based on a broadcast message to other nodes in the system (box 322). Then, the fixed interrupt is sent to the second node in a unicast message (box 324).

[0083] Reference now Figure 4, another embodiment of a method 400 for performing power management on a multi-node computing system is shown. Similar to method 300, the multi-node computing system for method 400 utilizes a non-uniform memory access (NUMA) architecture, and each of the nodes is a NUMA node. Interrupt priorities for arbitrating interrupts are initialized in the multi-node system (block 402). In one embodiment, the lowest valued interrupt priority specifies which processor core or which hardware thread within a processor core is assigned to service the arbitrated interrupt.

[0084] The interrupt priority is broadcast to a master node among a plurality of nodes in the computing system (block 404). For example, the operating system sends an indication of the interrupt priority of the computing system to the master node. Alternatively, the master node snoops a specific memory location to obtain the interrupt priority for arbitrating interrupts. In yet other examples, the master node performs a system call to the operating system to determine the interrupt priority of the computing system.

[0085] Multiple nodes process tasks (frame 406). One or more clients in the node execute computer programs or software applications. In some embodiments, multiple nodes are located in a multi-hole socket server and receive batch jobs by an operating system, and the operating system assigns tasks to one or more of the multiple nodes. In various embodiments, a background process is run, which allows the operating system to update the interrupt priority of the kernel. If the interrupt priority of the kernel has been updated (the "yes" branch of conditional frame 408), the updated interrupt priority of the kernel is sent to the master node (frame 410). In some embodiments, the master node only stores the lowest interrupt priority of each node. If the lowest interrupt priority is not updated, then in one embodiment, no update is sent to the master node.

[0086] If no node asserts the arbitration interrupt (the "no" branch of conditional box 412), the control flow of method 400 returns to box 406, where multiple nodes process the task. If the first node asserts the arbitration interrupt (the "yes" branch of conditional box 412), a request is sent to the master node to identify the node with the lowest priority of the kernel (box 414). The request is a single unicast message rather than multiple broadcast messages. Based on the reply from the master node, the second node servicing the arbitration interrupt is identified (box 416).

[0087] When multiple cores have the same interrupt priority, in one embodiment, the master node selects an active node from the multiple nodes. If there are multiple active nodes, the master node uses a round-robin scheme, a distance scheme that selects the node with the least number of hops from the requesting node, or other selection scheme. The arbitration interrupt is sent to the second node in a unicast message (block 418). No broadcast message is used to service the arbitration interrupt. Therefore, if each of the master node and the second node is in an active state, no powered-down node is awakened to service the fixed interrupt.

[0088] Reference now Figure 5 , another embodiment of a method 500 for performing power management on a multi-node computing system is shown. Similar to methods 300 and 400, the multi-node computing system used for method 500 utilizes a non-uniform memory access (NUMA) architecture, and each of the nodes is a NUMA node. The multiple nodes process tasks (block 502). When servicing a memory request, each of the multiple nodes maintains a probe filter (block 504).

[0089] If the given node determines that each of its one or more clients is idle (the "yes" branch of conditional block 506), the given node invalidates each entry in the probe filter (block 508). In some embodiments, before invalidating the entries in the probe filter, the given node also determines that there are no remote memory requests attempting to access its system memory. The given node updates its system memory with dirty cache lines stored in the remote node received as a result of the invalid entries and the corresponding messages or probes sent to the remote node (block 510). The given node is then powered down (block 512). In various embodiments, powering down the given node includes one or more of: disabling a driver for a link interface, disabling a clock and communication structure for a client, and setting the system memory to perform self-refresh when using DRAM.

[0090] Reference now Figure 6 , another embodiment of a method 600 for performing power management on a multi-node computing system is shown. Similar to methods 300-500, the multi-node computing system used for method 600 utilizes a non-uniform memory access (NUMA) architecture, and each of the nodes is a NUMA node. Multiple nodes process a task (block 602). If a first node generates a broadcast probe for a cache line (the "yes" branch of conditional block 604), the broadcast probe is sent to other nodes in the system (block 606). A second node in the computing system receives the broadcast probe (block 608).

[0091] If the second node does not determine that any neighboring nodes are powered down ("No" branch of conditional block 610), the second node sends a response generated for the second node from the second node to the first node (block 612). The second node also relays the broadcast probe to one or more other neighboring nodes based on the connectivity of the nodes used in the selected node topology.

[0092] If the second node determines that the adjacent third node is powered off (the "yes" branch of conditional block 610), the second node abandons sending the broadcast probe to the third node (block 614). The second node generates a miss response for the third node (block 616). The second node sends the miss response from the second node to the first node (block 618). Thus, the second node fakes a miss response to the broadcast probe seeking a particular cache line on behalf of the powered off third node. The third node is not awakened to respond to the miss response. Then, the control flow of method 600 moves to block 612.

[0093] In various embodiments, the previously described methods and / or mechanisms are implemented using program instructions of a software application. The program instructions describe the behavior of the hardware in a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog can be used. The program instructions are stored on a non-transitory computer-readable storage medium. Various types of storage media can be used. The storage medium can be accessed by a computing system during use to provide the computing system with program instructions and accompanying data for program execution. The computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0094] It should be emphasized that the above embodiments are only non-limiting examples of implementations. Once the above disclosure is fully understood, many changes and modifications will become apparent to those skilled in the art. It is intended that the following claims be interpreted as including all such changes and modifications.

Claims

1. A computing system comprising: Multiple processing nodes, each processing node includes: one or more processing units configured to process the application; a plurality of links coupled to other processing nodes of the plurality of processing nodes; and a power controller; and wherein in response to receiving a broadcast probe from a first processing node of the plurality of processing nodes, a second processing node of the plurality of processing nodes is configured to: generating a miss response for a third processing node of the plurality of processing nodes that is directly connected to the second processing node in response to determining that the third processing node is in a powered-down state, wherein: The powered-down state of the third processing node indicates that a cache of the third processing node is empty; and The miss response to the third processing node indicates that the third processing node does not store the cache line requested by the probe; and The miss response is sent to the first processing node.

2. The computing system of claim 1 , wherein in response to generating a broadcast probe, the first of the plurality of processing nodes is configured to: A miss response is generated for a fourth processing node of the plurality of processing nodes that is directly connected to the first processing node in response to determining that the fourth processing node is powered down.

3. The computing system of claim 1 , wherein the first processing node is configured to: maintaining the probe filter when servicing the storage request; and In response to determining that each of the one or more processing units is idle: invalidating each entry in the detection filter; and updating a system memory coupled to the first processing node with a dirty cache line received as a result of invalidating an entry in the snoop filter; and The plurality of links and the one or more processing units of the first processing node are powered down.

4. The computing system of claim 1 , wherein at least one of the one or more processing units comprises a plurality of cores, and wherein a master processing node of the plurality of processing nodes is configured to store interrupt priorities of the plurality of cores of the plurality of processing nodes for arbitrating interrupts.

5. The computing system of claim 4, wherein in response to detecting an arbitration interrupt, the first processing node is configured to: sending a request to the master processing node to identify a given kernel having a lowest priority and a given processing node including the given kernel; and Based on the response from the master processing node identifying the given core and the given processing node, a request is sent to the given processing node to process the arbitration interrupt with the given core.

6. The computing system of claim 4 , wherein in response to determining that the priority of the kernel for arbitrating interrupts is updated in the first processing node, the first processing node is configured to send the updated priority to the master processing node for updating the priority of the kernel for arbitrating interrupts stored on the master processing node.

7. The computing system of claim 1, wherein at least one of the one or more processing units comprises a plurality of cores, and wherein at least the first processing node is configured to store an indication of a fixed interrupt identifier (ID) for each of the plurality of cores.

8. The computing system of claim 7, wherein in response to detecting a stuck-on interrupt, the first processing node is configured to: identifying a given processing node that includes a given core having a fixed interrupt ID that matches the fixed interrupt ID of the detected fixed interrupt; and A request is sent to the given processing node to process the fixed interrupt with the given core.

9. A method for power management, comprising: Applications are processed by multiple processing nodes, each of which includes: one or more processing units configured to process the application; a plurality of links coupled to other processing nodes of the plurality of processing nodes; and power controller; and In response to receiving a broadcast probe from a first processing node of the plurality of processing nodes, a second processing node of the plurality of processing nodes is configured to: generating, by a second processing node of the plurality of processing nodes, a miss response for a third processing node of the plurality of processing nodes that is directly connected to the second processing node in response to determining that the third processing node is in a powered-down state, wherein: The powered-down state of the third processing node indicates that a cache of the third processing node is empty; and The miss response to the third processing node indicates that the third processing node does not store the cache line requested by the probe; and The miss response is sent by the second processing node to the first processing node.

10. The method of claim 9, wherein in response to generating a broadcast probe: A miss response is generated by the first processing node for a fourth processing node of the plurality of processing nodes that is directly connected to the first processing node in response to determining that the fourth processing node is powered down.

11. The method of claim 9, further comprising: maintaining, by said first processing node, a probe filter when servicing a memory request; as well as In response to determining that each of the one or more processing units in the first processing node is idle: invalidating each entry in the detection filter; and updating a system memory coupled to the first processing node with a dirty cache line received as a result of invalidating an entry in the snoop filter; and The plurality of links and the one or more processing units of the first processing node are powered down.

12. The method of claim 9, wherein at least one of the one or more processing units comprises a plurality of cores, and the method further comprises: Interrupt priorities of the plurality of cores in the plurality of processing nodes for arbitrating interrupts are stored by a master processing node in the plurality of processing nodes.

13. The method of claim 12, wherein in response to detecting an arbitration interruption, the method further comprises: sending, by the first processing node, a request to the master processing node to identify a given kernel having a lowest priority and a given processing node including the given kernel; as well as A request is sent, by the first processing node, to the given processing node to process the arbitration interrupt with the given core based on a response from the master processing node identifying the given core and the given processing node.

14. The method of claim 13, wherein in response to determining that a priority of a core for arbitrating interrupts is updated in the first processing node, the method further comprises: The updated priority is sent by the first processing node to the master processing node for updating the priority of a kernel for arbitrating interrupts stored on the master processing node.

15. A processing node, comprising: one or more processing units configured to process the application; a plurality of links coupled to other processing nodes of the plurality of processing nodes; as well as Control logic; and wherein in response to receiving a broadcast probe from a first processing node of the plurality of processing nodes, the control logic is configured to: generating a miss response for a second processing node of the plurality of processing nodes that is directly connected to the processing node in response to determining that the second processing node is in a powered-off state, wherein: The powered-down state of the second processing node indicates that a cache of the second processing node is empty; and The miss response to the second processing node indicates that the second processing node does not store the cache line requested by the probe; and the miss response is sent to the first processing node.