Resetting a multi-node system

Abstract

Each node stores a numerical value (herein called a generation number) that is updated each time the node is reset and restarted from a checkpoint. In most cases, the generation numbers held by the nodes are the same throughout the system, since the nodes in the system participate in the same reset events. However, in some cases, when one of the nodes is reset before the other, the generation numbers between these two nodes will be different. Each data frame transmitted between nodes contains the sending node's generation number, which is checked by the receiving node and is only accepted if the generation number in the frame matches the receiving node's generation number. If the receiving node is reset before the sending node is reset, the generation number in the frame will not match the receiving node's generation number and the frame will not be accepted. Thus, a node that has been reset and restarted is protected from packets related to applications of a previous generation dispatched by nodes that have not yet been reset.

Description

[Technical Field]

[0001] The present disclosure relates to processing systems configured to support applications, and in particular to processing systems that include multiple processing nodes configured to execute instruction sets for portions of the applications. [Background technology]

[0002] In the context of data processing for complex or high-volume applications, a processing unit may be provided to perform the processing of that data. The processing unit may function as a work accelerator, offloading the processing of certain data from a host system. Such a processing unit may have specialized hardware for performing specific types of processing. For example, a processing unit suitable for machine learning may include many processor tiles implemented together on the same chip to achieve massive parallelism supported by the same processor. To support the operation of such a processing unit, additional system-on-chip (SoC) circuitry may be provided along with the processing unit on the same chip. Such SoC circuitry may include, for example, on-chip memory separate from the processor's memory, a port controller for external communication, and boot hardware for booting the processing unit.

[0003] To scale up the processing capacity to support larger applications, multiple processing units can be connected together to form larger systems. In various implementations of such processing systems, each of the chips supporting these processing units are connected together via high-speed links that allow communication between the tile processors of the different processing units.

[0004] During the processing of applications supported across a system of connected chips, various error events may occur on one or more of these chips. Common events include, for example, an overtemperature condition, the occurrence of an uncorrectable memory error, or the failure of a device's external link. Some error events may require a system reset or a restart of the applications running on the system from a checkpoint. Summary of the Invention [Problem to be solved by the invention]

[0005] One problem that can be encountered when performing a reset of a multi-node system is that if one of the nodes is reset and another node is restarted from a previous checkpoint before the reset, the reset node may receive packets from nodes that have not yet been reset. These packets relate to previous generations of applications running on the system. This can disrupt application processing related to new generations of applications that the reset node is currently processing. [Means for solving the problem]

[0006] According to a first aspect, a processing system configured to support an application is provided. The processing system includes a plurality of processing nodes. Each of the plurality of processing nodes includes a processing unit configured to execute a set of instructions for a portion of the application, at least one storage configured to store a copy of a value indicating the number of times a global reset of the processing system has been performed, and a reset circuit. The reset circuit is configured to reset the respective processing node as part of a global reset of the processing system and restore application processing by the processing unit of the respective processing node from a checkpoint. Each of the processing nodes includes circuitry configured to update the copy of the value held in the at least one storage of the processing node as part of the reset of the processing node. The processing unit of a second processing node is configured to execute a set of instructions for transmitting one or more data frames to the first processing node prior to the reset of the second processing node, each of the one or more data frames including a copy of the value held in the at least one storage of the second processing node. The circuitry of the first processing node is configured to, following a reset of the first processing node, cause one or more data frames to be discarded in response to determining a mismatch between copies of the numerical values ​​in the one or more data frames and copies of the numerical values ​​maintained in the at least one storage.

[0007] Each node stores a numerical value (referred to here as a generation number) that is updated each time the node is reset and restarted from a checkpoint. Because nodes in the system participate in the same reset event, in most cases, the generation numbers maintained by the nodes are the same throughout the system. However, in some cases, when one node is reset before another, the generation numbers between these two nodes will differ. Each data frame transmitted between nodes contains the sending node's generation number, which is checked by the receiving node and accepted only if the generation number in the frame matches the receiving node's generation number. If the receiving node is reset before the sending node is reset, the generation number in the frame will not match the receiving node's generation number, and the frame will not be accepted. Thus, a node that has been reset and restarted is protected from packets related to applications of a previous generation dispatched by nodes that have not yet been reset.

[0008] According to a first aspect, a processing system configured to support an application is provided. The processing system includes a plurality of processing nodes. Each of the plurality of processing nodes includes a processing unit configured to execute a set of instructions for a portion of the application, at least one storage configured to store a copy of a numerical value indicating a number of times a global reset of the processing system has been performed, and a reset circuit. The reset circuit is configured to reset the processing node as part of a global reset of the processing system and restore an application state for the portion of the application of the processing node from a checkpoint. Each of the processing nodes includes an update circuit configured to update the copy of the numerical value maintained in the at least one storage of the processing node from a first value to a second value as part of the reset of the processing node. The processing unit of the second processing node is configured to execute a set of instructions for transmitting one or more data frames to the first processing node prior to the reset of the second processing node, each of the one or more data frames including the first value of the numerical value. The circuitry of the first processing node is configured to, following a reset of the first processing node, cause data in the one or more data frames to be discarded in response to determining a mismatch between a first value of the numeric value in the one or more data frames and a second value of the numeric value maintained in at least one storage of the first processing node.

[0009] In some embodiments, for each of the processing nodes, the processing unit is configured to participate in multiple barrier synchronizations as indicated in the instruction set, and for each of the processing nodes, the processing unit is configured, for each barrier synchronization in which it participates, to pass through the respective barrier synchronization in response to at least one further processing unit of the processing system joining the barrier synchronization.

[0010] In some embodiments, for each of the processing nodes, each of a plurality of barrier synchronizations separates a computation phase of the processing unit from an exchange phase of the processing unit.

[0011] In some embodiments, the processing units of the first processing node are configured to stall at the first barrier synchronization following a reset of the first processing node until one of the further processing units that is arranged to also participate in the first barrier synchronization reaches the first barrier synchronization in application processing.

[0012] In some embodiments, one of the further processing units is a processing unit of the second processing node.

[0013] In some embodiments, the first barrier synchronization operates as a barrier between a computation phase and an exchange phase, and the processing unit of the first processing node is configured to execute a set of application instructions to perform computations during the computation phase until the first barrier synchronization is reached, and to exchange data with the second processing node, including data contained in one or more data frames, during the exchange phase following the first barrier synchronization.

[0014] In some embodiments, the circuitry of the first processing node configured to determine the mismatch is part of an interface of the first processing node for connecting with a second processing node.

[0015] In some embodiments, the interface of the first processing node is configured to receive further data frames from the second processing node and to provide data packets encapsulated in the further data frames to the processing unit of the first processing node.

[0016] In some embodiments, the processing unit of the second processing node is configured to execute a set of instructions to cause transmission of one or more further data frames to the first processing node following a reset of the second processing node, each of the one or more further data frames including a second value, and the circuitry of the first processing node is configured to receive the one or more further data frames and, in response to determining a match between the second value in each of the one or more further data frames and a second value of a numerical value maintained in at least one storage of the first processing node, cause acceptance of the data in the one or more data frames.

[0017] In some embodiments, the circuitry of the first processing node is configured to determine the mismatch as part of checking the address fields in each of the data frames.

[0018] In some embodiments, for each of the processing nodes, the update circuitry includes a processor configured to execute computer-readable instructions to perform an update of a copy of the numerical value maintained in at least one storage of the processing node.

[0019] In some embodiments, updating the copy of the numerical value includes incrementing the first value to obtain the second value.

[0020] In some embodiments, the copy of the value forms part of the connection state for the connection between the first processing node and the second processing node.

[0021] In some embodiments, each processing node includes an integrated circuit that includes a processing unit and at least one storage for the processing node.

[0022] In some embodiments, the processing unit of the second processing node is configured to execute a set of instructions to output a first value in one or more data packets of one or more data frames.

[0023] In some embodiments, the processing unit of the second processing node is configured to execute a set of instructions to output data for one or more data frames to an interface circuit, the interface circuit being configured to provide the one or more data frames with a first value to be inserted into each frame.

[0024] According to a second aspect, there is provided a method including: executing, at each of a plurality of processing nodes, a set of instructions for a portion of an application supported by a system including the plurality of processing nodes; performing one of the global resets including storing, at each of the processing nodes, a copy of a numerical value indicative of a number of times a global reset of the processing system has been performed; resetting each of the plurality of processing nodes including updating the copy of the numerical value maintained at each of the processing nodes from a first value to a second value; and restoring an application state for the portion of the application of the processing node from a checkpoint; transmitting one or more data frames from the second processing node to a first processing node prior to resetting the second processing node, the data frames including the first value; and following the reset of the first processing node, causing data in the one or more data frames to be discarded in response to determining a mismatch between the first value in the one or more data frames and the second value maintained at the first processing node.

[0025] According to a third aspect, there is provided a computer program comprising a set of executable instructions for each processing node of a plurality of processing nodes forming a data processing system, wherein each of the executable instructions of the program, when executed by at least one processor of the processing node, causes the processing node to support a portion of a distributed application supported by the plurality of processing nodes, store a copy of a numerical value indicating the number of times a global reset of the processing system has been performed, reset the processing node including updating the copy of the numerical value held in the processing node from a first value to a second value as part of the global reset of the data processing system, and restore application state for the portion of the distributed application of the processing node from a checkpoint. The second set of executable instructions, when executed by at least one processor of the second processing node, is configured to cause the second processing node to transmit one or more data frames including a first value to the first processing node before resetting the second processing node, and the first set of executable instructions, when executed by the at least one processor of the first processing node, is configured to cause the first processing node to discard data in the one or more data frames in response to determining a mismatch between the first value in the one or more data frames and a second value stored by the first processing node following resetting the first processing node.

[0026] According to a fourth aspect, there is provided a non-transitory computer readable medium storing a computer program according to the third aspect.

[0027] To facilitate an understanding of the present disclosure and to show how embodiments may be practiced, reference will now be made, by way of example, to the accompanying drawings, in which: [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 2 is a schematic block diagram of a multi-tile processing unit. [Figure 2]FIG. 2 is a schematic block diagram of one of the tiles that can be implemented in a multi-tile processing unit. [Figure 3] 1 is a first exemplary embodiment of a processing node. [Figure 4] 1 is a second exemplary embodiment of a processing node. [Figure 5A] 1 shows a system of processing nodes connected together to form a spanning tree graph for the exchange of global event state. [Figure 5B] 1 shows a system of processing nodes connected together in a ring topology for the exchange of global event state. [Figure 5C] 1 illustrates the exchange of global event states between a series of processing nodes in the system. [Figure 5D] 1 illustrates how a global event state is propagated to multiple components in each of a series of processing nodes. [Figure 6] 1 illustrates the structure of local and global event vectors in accordance with an exemplary embodiment; [Figure 7] 1 illustrates a bus architecture used for the propagation of event states between components of a processing node. [Figure 8] 1 illustrates an event report dispatched by a component for delivery to a control node of a processing node. [Figure 9] 1 shows a node on the bus for receiving and storing event states broadcast by a control node. [Figure 10] 1 shows the registers and processing circuitry maintained in the control node to enable the control node to store and distribute event states. [Figure 11] Among other things, an example of an interface circuit used to exchange global event state between processing nodes is shown. [Figure 12] 1 shows in more detail the components of the interface circuitry used to transmit and receive global event states. [Figure 13]1 illustrates an example of an exchange block for enabling a tile of a processing node to exchange data plane traffic with a tile of another processing node. [Figure 14] 10 illustrates example registers and circuitry in a switching block that allows the switching block to participate in autonomous event processing in response to event states in a local or global event vector. [Figure 15] An example is given of how event notification hardware is used to notify handler software of the occurrence of an event. [Figure 16] It shows how multiple event handlers in a system can respond to notifications of events occurring in the system. [Figure 17] 10 illustrates how checkpoint data is read from checkpoint memory and returned to the processing unit. [Figure 18a] 1 illustrates a first portion of a method for performing a reset of a multi-node system. [Figure 18b] 10 illustrates a second portion of a method for performing a reset of a multi-node system. [Figure 19] This shows how the generation number used for a connection can be updated in response to a reset event. [Figure 20] 1 illustrates an example of a method for propagating global event states between different nodes of a multi-node system. [Figure 21] An example method is provided for propagating local event states to components so that the components can respond to the local event states and participate in performing relevant actions. DETAILED DESCRIPTION OF THE INVENTION

[0029] Referring to Figure 1, Figure 1 illustrates an exemplary multi-processing unit 2. Multi-tile processing units 2 are described in our prior U.S. patent application Ser. No. 15 / 886,065, the contents of which are incorporated herein by reference. Each of the processing units 2 is formed on a separate integrated circuit (i.e., chip).

[0030] The processing unit 2 includes an array 6 of multiple processor tiles 4 and an interconnect 34 connecting the tiles 4. The processing unit 2 may be implemented singly as one of multiple dies packaged in the same IC package. The interconnect 34 allows the tiles 4 to exchange data with each other, and is therefore sometimes referred to herein as a "switching fabric" 34. Each tile 4 includes respective instances of execution units and memory. For example, by way of illustration, the processing unit 2 may be comprised of hundreds or even thousands of tiles 4. For completeness, it should also be noted that "array," as referred to herein, does not necessarily imply any particular dimensional or physical layout of the tiles 4.

[0031] In an embodiment, each processing unit 2 is part of a chip that also includes one or more external links, allowing the processing unit 2 to be connected to one or more other processing units (e.g., one or more other instances of the same processing unit 2). These external links may include any one or more of: one or more processing-unit-to-host links for connecting the processing unit 2 to a host system, and / or one or more processing-unit-to-processing-unit links for connecting the processing unit 2 with one or more other instances of the processing unit 2 on the same IC package or card or on different cards. The processing unit 2 receives work in the form of application data from the host and processes it.

[0032] Each tile 4 includes processing circuitry and memory. In some example embodiments, the processing circuitry is a multi-threaded processor 10. FIG. 2 illustrates an example tile 4 according to an embodiment of the present disclosure. The tile 4 includes a multi-threaded processor 10 in the form of a barrel-threaded processor 10 and a local memory 11. A barrel-threaded processor 10 is a type of multi-threaded processor 10 in which the execution time of a pipeline is divided into a repeating sequence of interleaved time slots, each of which can be owned by a given thread. This is discussed in more detail below. The memory 11 includes an instruction memory 12 and a data memory 22 (which may be implemented in different addressable memory units or in different regions of the same addressable memory unit). The instruction memory 12 stores machine code executed by the processing unit 10, while the data memory 22 stores both data operated on by the executed code and data output by the executed code (e.g., as a result of such manipulation).

[0033] Memory 12 stores various different threads of the program, each thread containing a respective sequence of instructions for performing a particular task or tasks. Note that an instruction as referred to herein means a machine code instruction (i.e., an instance of one of the basic instructions of the processor's instruction set) consisting of a single opcode and zero or more operands.

[0034] Within processor 10, multiple different threads from instruction memory 12 can be interleaved through a single execution pipeline 13 (although typically only a subset of all threads stored in instruction memory can be interleaved at any given time throughout the program). Multithreaded processor 10 includes multiple context register files 26, each arranged to represent the state (context) of a different thread being executed in parallel, a shared execution pipeline 13 common to the threads being executed in parallel, and a scheduler 24 for scheduling the parallel threads to execute through the shared pipeline in an interleaved manner, preferably in a round-robin manner. Processor 10 is connected to a shared instruction memory 12 common to the multiple threads, and a shared data memory 22 also common to the multiple threads.

[0035] Execution pipeline 13 includes a fetch stage 14, a decode stage 16, and an execute stage 18 that includes execution units capable of performing arithmetic and logical operations, address calculations, load and store operations, and other operations as defined by the instruction set architecture. Each of the context register files 26 includes a set of registers to represent the program state of the respective thread.

[0036] Referring to FIG. 3, FIG. 3 illustrates an example of a processing device 300 including the processing unit 2 discussed above. The processing device 300 constitutes a processing node in a system of processing nodes. The device 300 is implemented on a single integrated circuit (i.e., chip). The device 300 includes multiple components, including the processing unit 2, as well as other system-on-chip (SoC) components that support the operation of the processing unit 2. The device 300 includes a management processor 310, referred to as a management CPU (MCPU) 310. The MCPU 310 executes computer-readable instructions to support the operation of the device 300, for example, by handling error events and setting up connections. The device 300 also includes a temperature sensor 330 for measuring the temperature of the device 300. The device 300 includes a phase-locked loop (PLL) 340 for providing a clock signal for the device 300. The device 300 includes a DRAM module 350 for storing and / or providing data (e.g., checkpoint data) output by application code executing on the processing unit 2 to the processing unit 2. Device 300 includes multiple interface controllers for external communications. These include, for example, Ethernet port controller (EPC) 360. EPC 360 is used to tunnel packets over Ethernet between device 300 and other equivalent devices 300 in a multi-node system. Tunneled packets can be provided to / from tiles 4 of processing unit 2 (for data plane traffic) or to / from various components of device 300 (for control plane traffic). The interface controllers also include PCIe complex 370 for PCIe communications between device 300 and host device 320.

[0037] FIG. 3 illustrates an example of a processing node 300 implemented on a single chip 300. Referring to FIG. 4, FIG. 4 illustrates a processing node 400 including multiple chips 410, 420. The processing node 400 includes a main processing chip 410 that includes a processing unit 2. The processing node 400 also includes an additional chip 420 associated with and connected to the main processing chip 410. The additional chip 420 may be referred to as a fabric chip 420. For simplicity, only one of the four fabric chips 420 of the node 400 is shown in detail. Each of the fabric chips 420 provides access to a DRAM memory 430. The DRAM memory 430 can be read or written by the processing unit 2. One of the fabric chips 420 provides access to a host device 440 via a PCIe complex 450. Each of the fabric chips 420 includes an EPC 460. One of the EPCs 460 is for interfacing with the main processing chip 410. Other EPCs 460 are for interfacing with other processing nodes that are equivalent to processing node 400. Chip 410 also includes EPCs 470. Some of the EPCs 470 are for interfacing with fabric chips 420 that belong to the same node 400. Other EPCs 470 are for interfacing with fabric chips 420 that belong to different nodes 400. In this embodiment, MCPU 310 is moved from main processing chip 410 to fabric chip 420.

[0038] A particular processing node (node ​​300 or node 400) can experience many different events that may indicate an error condition occurring in the processing node. Such events may be, for example, an overheating event detected by temperature sensor 330, an exception event logged by tile 4 while executing its application code, or an uncorrectable error in DRAM module 350, 430. Some error events may be fatal to the application, such as when a link between chips goes down, while others may be non-fatal but still require reporting, such as when a number of correctable errors on a link exceed a threshold. Each event that is reported and aggregated is assigned to an event class. An event class refers to the type of event. For example, an uncorrectable memory error in DRAM module 350, 430 may be one event class, and a security exception may be another. For a processing node 400 that includes multiple chips 410, 420, some events may occur on the main processing chip 410 and some events may occur on one of the fabric chips 420.

[0039] A system of processing nodes is provided, where each processing node is configured to execute a portion of application code for the entire system. Such a system is sometimes referred to as a partition, because a single, isolated application runs across multiple devices in the system. Each processing node in the system stores at least one copy of an event vector, each element of which corresponds to a different event class. Each element indicates whether an event of the corresponding class has occurred anywhere in the system of processing nodes. This system-wide aggregated event vector is referred to herein as a global event vector (GEV). In addition to the GEV, each node also stores its own associated local event vector (referred to herein as a "CEV"). The CEV also includes multiple elements, each of which indicates whether an event of the corresponding event class has occurred on the processing node storing that local event vector.

[0040] Referring to FIG. 6, FIG. 6 shows examples of a GEV and a CEV. Both of these event vectors contain multiple elements, providing a bit mask for each event class indicating whether an event of that event class has occurred. Each element of the CEV indicates whether an event of the respective class has occurred on the processing node storing the CEV. Each element of the GEV indicates whether an event of the respective class has occurred on any processing node in the system. The CEV contains more elements than the GEV, reflecting that only a subset of event classes whose events are captured in the CEV are globally aggregated. In an embodiment, the CEV contains 15 elements, one for each of the 15 event classes. The GEV contains 10 elements, one for each of the 10 event classes, reflecting that events of the first 10 event classes captured in the CEV are globally aggregated. The numbering of the vector elements in FIG. 6 reflects the event class ID associated with each element.

[0041] References herein to the "event state" refer to the state of the CEV and GEV. References herein to the "global event state" refer to the state of the GEV.

[0042] Referring to Figure 5A, Figure 5A illustrates a system 500 of processing nodes 510. Each of the processing nodes 510 may be processing node 300 or processing node 400. Figure 5A illustrates an example of how each of the nodes 510 are connected for aggregation of global event state. These connections are the connections used for transmission of the global event state and, therefore, the connections used to aggregate the global event state across the system 500. It will be understood that in an embodiment, there may be additional connections between the nodes 510 beyond those shown, and that these additional connections are used for the exchange of data plane traffic or other types of control plane traffic (e.g., synchronization messages).

[0043] The connections between nodes 510 used for the exchange of global event state form a spanning tree. A spanning tree allows communication between any nodes 510 in the system 500, but without introducing cycles (or rings) into the graph formed by the interconnected nodes 510. The use of a spanning tree allows messages to be sent between any two nodes 510 in a small number of hops, while maintaining a simple topology with fewer connections than would otherwise be the case.

[0044] Referring to Figure 5B, Figure 5B illustrates an example in which a system 500 of nodes 510 is connected in an alternative topology for the exchange of global event state. In this case, the nodes 510 are connected together in a ring, and global event state is exchanged between the nodes 510 around the ring. A ring topology has the advantage of incorporating redundancy into the system 500. If a link used to send GEV updates goes down, updates can still be sent between any two nodes 510 in the system 500 because there is an alternative path for sending such updates. Thus, aggregation of global event state is possible even in a ring topology.

[0045] In the example of FIGS. 5A and 5B, system 500 is shown to include only 16 processing nodes 510, although in practice system 500 would likely include thousands of processing nodes 510.

[0046] 5C, which illustrates how global event state can be aggregated across different nodes 510 of system 500. A portion 520 of system 500 is shown, which includes exemplary processing nodes 510a-d.

[0047] As shown, each of the system's nodes 510a-d maintains a copy of the GEV and its own copy of the CEV. In reality, each node contains multiple registers, some of which are called Global Event Vector Registers (GEVVRs) and hold a copy of the GEV, and others are called Colossus Event Vector Registers (CEVVRs) and hold a copy of the CEV. However, logically, each of the nodes 510a-d can be thought of as having a single copy of the node's CEV and a single copy of the GEV.

[0048] Each time an event of an event class recorded in the CEV occurs at one of nodes 510a-d, the processing circuitry of node 510a-d updates the CEV maintained at node 510. Specifically, the processing circuitry of that node 510a-d updates the element of the CEV corresponding to the event class of the detected event. If the event class is a class that is propagated throughout nodes 510, in addition to updating the CEV at one of nodes 510a-510d, the circuitry of node 510 also updates the corresponding element of the GEV maintained at node 510. The interface circuitry of processing node 510 then causes update messages (sometimes referred to as GCM messages) containing an indication regarding the update of the GEV to be transmitted to one or more other processing nodes 510. Each update message may include an indication of the entire updated GEV or an element of the GEV to be updated. An update message is sent from one of the nodes 510a-510d to a node 510 that is a neighbor of one of the nodes 510a-510d sending the update (i.e., a node connected to one of the nodes 510a-510d sending the update). A node 510 receiving an update message updates its copy of the GEV according to the update message and then sends one or more further update messages to its own neighbors 510 (nodes 510 other than the neighboring node 510 that provided the update).

[0049] In the example of FIG. 5C, an event occurs at processing node 510a. The event is of a class of events that processing nodes 510a-510d are configured to globally aggregate. As shown, in this example, the event occurs in a fourth event class, with the event being recorded in the CEV and GEV. However, the event class may be a different event class. In response to detecting the event, processing circuitry at node 510a causes a fourth element (associated with the fourth event class) of the CEV at node 510a to be set to indicate that an event of the fourth event class has occurred. Similarly, in response to detecting the event, processing circuitry at node 510a causes a fourth element (associated with the fourth event class) of the GEV at node 510a to be set to indicate that an event of the fourth event class has occurred.

[0050] In response to updating the copy of the GEV at node 510a, circuitry at node 510a causes the issuance of a data packet to node 510b that includes an indication of the update to the GEV. Node 510b receives the data packet, and circuitry at node 510b causes the copy of the GEV 510b maintained at node 510b to be updated in accordance with the indication.

[0051] In response to updating the copy of the GEV at node 510b, circuitry at node 510b causes the issuance of a data packet containing an indication of the GEV update to nodes 510c, 510d, which in turn cause each of the copies of the GEV held at their respective nodes 510c, 510d to be updated according to the indication of the update provided in the data packet.

[0052] This section describes how event states are reported by components of a processing node 510 and propagated to other components of that node 510 .

[0053] When an event is detected by a particular component of a processing node 510, that component forwards an event report to the regulator component of the node 510, which updates its copy of the CEV (known as the master copy) before broadcasting the update to other components that maintain other copies of the CEV. The exchange of these messages between nodes can occur via a control bus architecture.

[0054] Referring to Figure 7, Figure 7 shows an example of a control bus 700. The control bus 700 is implemented on a single chip (i.e., device 300 or device 410). If the control bus 700 is implemented on device 410, device 400 may have components connected to the control bus 700 to transfer packets between the control bus 700 and components on other devices 420 in the processing node 400, allowing event states to be exchanged between the regulator node 740 and components on those devices 420. The control bus 700 shown in Figure 7 is implemented on a single chip 300 / 410.

[0055] Control bus 700 is described in more detail in our previous U.S. patent application Ser. No. 17 / 328,143, which is incorporated herein by reference. Control bus 700 is a ring-shaped data path for carrying single-word control traffic. Control bus 700 is a pipelined data bus over which data packets move from one pipeline stage to the next at a rate determined by clock pulses applied to control bus 700. The ring includes multiple nodes 710, and traffic passes from one node 710 to the next in a flowing direction around the ring. The output port 750 of each node is connected to the input port 760 of the next node 710 in the ring.

[0056] Some nodes (referred to herein as target nodes) have circuitry for receiving requests to read or write configuration settings associated with that node. An example target node 720 is shown connected to bus 700. Such a target node 720 stores configuration settings for its associated components 770, which can be read or written by read / write requests received from control bus 700. The configuration settings (or configuration state) include copies of the CEV and GEV.

[0057] Some nodes (referred to herein as initiator nodes) have request circuitry for issuing read or write requests to bus 700. An example initiator node 730 is shown connected to bus 700. Under the control of a connection module, such an initiator node 730 issues read or write requests to configuration settings associated with target nodes 720 connected to control bus 700. Initiators 730 may be controlled by software or by a fixed-function hardware engine.

[0058] As shown, each initiator node 730 and each target node 720 has an associated component (shown as a connecting block) 770. The initiator nodes 730 and target nodes 720 may be implemented within their associated component 770. Each initiator 730 operates under the control of its associated component 770 and dispatches read or write requests to the control bus 700. Each target node 730 receives and stores configuration state for controlling its associated component 770. This configuration state is output from the target node 720 to its associated component.

[0059] Each initiator node 730 is capable of issuing requests to and receiving completions from a target node 720. Each request is a command to read from storage associated with the target node 720 (e.g., a connected addressable entity or an automatically generated register) or a request to write to such storage associated with the target node 720. In response to receiving such a request, the target node 720 responds by issuing a completion. The completion provides a status update indicating whether the read or write request was successful. For read completions, each completion includes the data read from the target node 720 and returned to the initiator node 730 that issued the read request. For write completions, the write completion includes an indication of whether the write request was successful.

[0060] A further type of transaction supported by control bus 700 is the event report. An event report is a packet dispatched and posted to control bus 700 by an initiator node 730 or a target node 720, containing information related to a detected event. When a hardware unit 770 detects an event, the hardware unit 770 controls its associated initiator node 730 or associated target node 720 to dispatch an event report to control bus 700.

[0061] Referring to Figure 8, Figure 8 shows an event report 800. The event report 800 includes an event class 820 and an event identifier 830, which are provided by the hardware unit 770 that detects the event. The hardware unit 770 provides both of these fields 820, 830 to its associated node 720 / 730. The event report 800 includes an event source field 810, which is prefixed with the node 720 / 730 that received the event class 820 and event identifier 830 of the event. The event source field 810 uniquely identifies the node 720 / 730 among all nodes 720 / 730 that belong to the processing node 510.

[0062] The event class field 820 identifies the event class to which the detected event falls. In an embodiment, the event class identifies one of the 15 event classes discussed above that may be captured in CEV. The event identifier field 830 identifies a unique event that occurred at the component 770 identified by the source field 810 and within the event class 820. The event identifier field 830 also includes further details about the event.

[0063] When an event report is dispatched to the bus 700, it is delivered to a Cbus regulator 740, which may also be referred to as a control node 740 or a control component 740. The event report circulates on the bus 700 until it reaches the regulator 740, where it is consumed and the event is logged.

[0064] Referring to FIG. 10, FIG. 10 illustrates an exemplary embodiment of a regulator node 740. FIG. 10 illustrates components of the regulator 740 for logging event states in response to receiving an event report. The regulator node 740 includes processing circuitry 1010 for performing the operations described herein as being performed by the regulator node 740. The regulator node 740 includes multiple registers 1020 for storing information received in the event reports. These registers 1020 are referred to as event data registers (EDRs) 1020. The regulator node 740 includes an EDR 1020 for each of multiple event classes.

[0065] When an event report is received by regulator 740, regulator 740 causes information from the event report for the event class identified in the event report to be stored in EDR 1020. The information stored in EDR 1020 includes the event identifier 830 from the event report and the event source identifier 810 from the event report. Each EDR 1020 also includes an event flag, which is set to indicate that an event of a class corresponding to the class of the EDR 1020 has occurred. In response to the event report, if the event flag in EDR 1020 corresponding to the class of the event report is not already set, node 740 causes that event flag to be updated to indicate that an event of that event class has occurred.

[0066] The regulator 740 also includes a register 1030 for storing a copy of the CEV for the processing node 510. This register 1030 is referred to as the master CEVVR 1030 because it is the first CEVVR for the node 510 that is updated in response to the detection of an event. When an event report is received by the regulator 740, if the bit in the CEV corresponding to the event class identified in the event report is not already set, the regulator 740 causes that bit in the CEV to be set to indicate that an event of that class has occurred. The CEVs are idempotent, such that if a bit in the CEV for a particular event class is already set, the bit in the CEV remains unchanged when a subsequent event report of the same event class is received. The regulator node 740 controls the access permissions of the CEV so that bits in the CEV are not unset except in response to a reset event.

[0067] The regulator node 740 includes a register 1010 for storing a copy of the GEV. The register 1010 is called the master GEVVR 1010 because it is updated before the GEVVR of a target node 720 in the same processing node 510 is updated. The master GEVVR 1010 may update in response to i) detection of an event in its own node 510 or ii) a packet received at the node 510 indicating an update to the GEV (the update being due to an event occurring at another node 510).

[0068] Regulator node 740 includes a mask register 1050, referred to herein as GEVVRMASK register 1050. GEVVRMASK register 1050 provides, for each event class that has a corresponding element in the GEV, a bit set that indicates whether global aggregation is enabled for that event class. To ensure consistency in how event classes are aggregated across system 500, the value held in GEVVRMASK register 1050 is set to be the same in each of nodes 510 in system 500.

[0069] When an event report is received at regulator 740, if the bit in GEVVRMASK register 1050 corresponding to the event class identified in the event report is set to indicate that global aggregation is enabled, and if the bit set in GEVVR 1010 is not already set to indicate that an event has occurred in that class, regulator 740 causes the bit in the GEV for that class to be set to indicate that an event has occurred in that class. GEVs are idempotent such that if a bit in a GEV for a particular event class is already set, subsequent event reports of the same event class received by regulator 740 do not change that bit in the GEV. Regulator node 740 controls the access permissions of the GEV so that the bit in the GEV indicating the occurrence of an event is not unset except in response to a reset event.

[0070] When one or more of the event flags in the EDR 1020 change state to indicate that an event has occurred, the regulator 740 broadcasts an update message to all target nodes 720 of the processing node 510. The update message is a message sent over the control bus 700 to update the CEVVR of each of the target nodes 720 of the processing node 510. The update message updates each of the node's 510's copies of the CEV, thereby updating the elements of the CEV associated with the class for which the event flag was set to reflect that an event has occurred in that class.

[0071] Additionally, when one or more bits in the master GEVVR 1010 change state, the regulator 740 broadcasts an update message (which may be the same update message used to update copies of the CEV) to all target nodes 720 of the processing node 510. The update message is a message sent over the control bus 700 to update the GEVVR of each of the target nodes 720 of the processing node 510. The update message updates each of the target nodes' 720 copies of the GEV, thereby updating the elements of the GEV associated with the event class whose event flag is set to reflect that an event has occurred in that class.

[0072] Referring to Figure 9, Figure 9 illustrates exemplary target nodes 720. Each target node 720 includes at least one node management register 940. The node management registers 940 may be written via a dedicated node management broadcast write request, which writes a common set of states to all node management registers 940 of the target nodes 720. Such broadcast writes may be performed by the regulator node 740. An example of such a broadcast write is an update message sent to update the copies of the CEV and GEV held at each target node 720.

[0073] In addition to at least one target node management register 910, the target node 720 is associated with additional storage 920 that is part of the component 770 with which the target node 720 is associated. The additional storage 920 may include automatically generated registers and / or storage accessible through an addressable window. This additional storage 920 may include control registers on tile 4 of processing unit 2 that are addressable over an additional bus. This additional storage 920 allows access to off-chip storage, such as host storage accessible over a PCI link. In this case, the additional storage 920 is memory of the host dispatcher, which provides data written to memory to host storage.

[0074] The target node has input and output ports. The input ports receive read and write requests from the control bus 700, which are buffered in a request buffer 960 and then passed to the processing logic 930. The output port sends completions from a completion buffer 970 to the control bus 700. The processing logic 930 can perform functions implemented in hardware or software. The processing logic 930 may include one or more of an ASIC, an FPGA, or at least one processor configured to execute computer-readable instructions stored in at least one memory of the target node 720.

[0075] 9, the node management registers 940 include a CEVVR 980 and a GEVVR 990. The CEV held in the CEVVR 980 is the same for all target nodes 720 in a processing node. Similarly, the GEV held in the GEVVR 990 is the same for all target nodes 720 in all processing nodes in the system. The processing logic 930 outputs all elements of the CEV and GEV from the node management registers 940 to the connectivity component 770, which can then take action or perform its operation in response to the error condition indicated by the event condition.

[0076] When regulator 740 sends an update message for a CEV update, the update message is received by target node 720. Processing logic 930 causes the CEVVR 980 of that target node 720 to be updated according to the update. The CEV in CEVVR 980 is updated to match the CEV held in master CEVVR 1030. Processing logic 930 causes the propagation of this updated CEV state to connection components 770 so that those components can respond to the updated error state. The same update is performed at each of target nodes 720 of processing node 510.

[0077] Similarly, when regulator 740 sends an update message for a GEV update (which may be the same as or different from the one used to update the GEV), this update message is received at target node 720. Processing logic 930 causes the GEVVR 990 of that target node 720 to be updated according to the update. The GEV in GEVVR 990 is updated to match the GEV held in master GEVVR 1010. Processing logic 930 causes the propagation of this updated GEV state to connection component 770 so that that component can respond to the updated error condition. The same update is performed at each of target nodes 720 of processing node 510.

[0078] Among the components 770 that receive updates for their associated GEVs (i.e., GEVs held in target GEVVR 990) are the EPCs (e.g., EPCs 360, 460, 470) of processing nodes 510. When particular EPCs receive updates for their GEVs from the regulator node 740 of the processing node 510 to which they belong, these EPCs are configured to provide an indication in the update message that the GEV state has been updated. The indication of the GEV state is propagated between processing nodes 510 and used to update the GEVs held in those nodes 510. The particular EPCs configured to propagate event state are those used to communicate over the connections shown in Figures 5A and 5B. It is these EPCs that provide the GEV update message shown in Figure 5C.

[0079] Referring to FIG. 11, FIG. 11 illustrates an example of an EPC 1100 according to an embodiment. The EPC 1100 may be one of the EPCs 360, 460, and 470. The EPC 1100 includes a target node 720 and an initiator node 730. The target node 720 enables the EPC 1100 to receive updated global event states received from the regulator node 740. In addition to the interfaces illustrated in FIG. 11, the EPC 1100 also includes an interface for exchanging data plane traffic with the processing unit 2. FIG. 11 illustrates a component within the EPC used for exchanging control plane traffic (including event state updates). This component is referred to herein as a global communication proxy (GCP) 1200. The GCP 1200 can receive Ethernet frames via a receive interface (Rx), which include update messages for updating its copy of the GEV. The GCP 1200 can dispatch update messages itself over the transmit interface (Tx), these update messages being messages for updating GEVs held in other processing nodes 510 of the system. The GCP 1200 causes the update messages to be provided to the frame builder 1120, which then transmits them over the transmit interface in Ethernet frames constructed according to the connection state held in the connection state register 1130.

[0080] Referring to FIG. 12, FIG. 12 illustrates a GCP 1200 and components within the GCP 1200 that are used to propagate event state. The GCP 1200 includes processing circuitry 1220 for performing the operations described herein as being performed by the GCP 1200. In addition to the GCP 1200, FIG. 12 also illustrates a target node 720 and an initiator node 730 that enable the GCP 1200 to exchange event state with a regulator node 740. Here, we describe how the GCP 1200 enables global aggregation of event state. It will be understood that, while a processing node 510 according to an embodiment includes multiple GCPs 1200, only a subset (one or more) of these GCPs 1200 are typically active for purposes of propagating the global event state.

[0081] As will be described, in response to the broadcast of a GEV update message from regulator 740, GEVVR register 990 is updated. This updated GEV state is output from target 720 to GCP 1200. GCP 1200 includes a register 1210, referred to herein as GEVEN register 1210, which contains an indication of whether global event aggregation is enabled in GCP 1200. If register 1210 indicates that global event aggregation is enabled for GCP 1200, GCP 1200 responds to the update of its target GEVVR 990 by dispatching an update packet to another processing node 510.

[0082] Each GCP 1200 includes an additional register 1240, referred to herein as a Global Event GCP External Global Event Vector Register (GEVEXTGEVVR) 1240. GEVEXTGEVVR 1240 stores a copy of the GEV. Circuit 1220 compares the copy of the GEV held in GEVEXTGEVVR register 1240 with the copy of the GEV held in GEVVR register 990 to determine whether to dispatch an update to another node 510 or to a regulator 740 of the same node. If circuit 1220 of GCP 1200 determines that a bit not set in GEVEXTGEVVR 1240 is set in the target GEVVR 990, circuit 1220 of GCP 1200 updates GEVEXTGEVVR register 1240 to match the copy of the GEV held in GEVVR 990. The GCP 1200 also causes an update message to be dispatched to a GCP on another node 510 with which the GCP 1200 is configured to communicate. The update message provides an indication that the GEV has been updated, and causes the receiving GCP to update its own GEVEXTGEVVR to match the GEVEXTGEVVR 1240 of the GCP 1200 that dispatched the update message.

[0083] To provide reliable propagation, the GCP 1200 transmits the update message three times. After transmitting the GEV update message in a frame, the GCP 1200 waits 1000 system clock cycles of its chip 510 before retransmitting the same frame containing the update message. The GCP 1200 then waits another 5000 system clock cycles before transmitting the frame a third time. If the target GEVVR 990 associated with the GCP 1200 changes again before these copies are transmitted, the retransmission sequence is aborted and a new update message is transmitted containing the new state held in the GEVVR 990.

[0084] A GCP 1200 may receive an update message from a GCP on another node 510 that causes it to update its copy of the GEV in its GEVEXTGEVVR 1240. As a result of such an update, the copy of the GEV held in the GEVEXTGEVVR 1240 may not match the copy of the GEV held in the target GEVVR 990 associated with the GCP 1200. In response to determining that a bit is set in the GEVEXTGEVVR 1240 that is not set in the associated GEVVR 990, the GCP 1200 issues a write request to the Cbus regulator 740 on its own node 510 to update the GEVUPDATER register 1060 to match the copy of the GEV currently held in the GEVEXTGEVVR 1240.

[0085] In response to receiving a write to GEVUPDATER 1060, regulator node 740 updates GEVUPDATER 1060 accordingly. In response to updating GEVUPDATER 1060, regulator 740 updates its copy of the GEV held in master GEV register 1010 of processing node 510. The updating of master GEVVR 1010 is performed such that any bit set in GEVUPDATER 1060 to indicate that an event has occurred in a particular class is also set in GEVVR 1010. However, for any bit in GEVUPDATER 1060 that is set to indicate that no event has occurred in that class, regulator 740 does not update GEVVR 1010 to indicate that no event has occurred in that class. In this way, the update policy of regulator 740 ensures that the GEVs held in GEVVR 1010 are idempotent.

[0086] In addition to updating its copy of the GEV held in register 1010, regulator 740 broadcasts an update message on control bus 700 to all target nodes 720 of processing node 510. The update message includes the updated GEV held in register 1010 of regulator 740. Referring to Figure 9, processing logic 930 of each target node 720 receives the update message and updates the GEVVR 990 held in that target node 720 with the updated GEV. A component 770 associated with a target node 720 can access the copy of the GEV held in the GEVVR 990 of that target node 720.

[0087] Some of these components 770 that receive updates to their copies of GEVs via their associated GEVVRs 990 are GCPs that are themselves capable of performing global event aggregation. Each such GCP issues an update message to another node 510 in response to updates to its GEVVR 990. The updated GEV is then propagated to different components of that other node 510 in the same manner as described above.

[0088] Referring to FIG. 5D, FIG. 5D shows in more detail how updated global event states are propagated among nodes 510a-510d of portion 520 of system 500. When component 770a (which may itself be a GCP) encounters an error event, component 770a provides a report about this event to regulator 740a. Regulator 740a updates its master copy of node 510a's CEV and GEV and broadcasts updates to these vectors to each of node 510a's target nodes 720. GCP 1200a detects the update to its copy of the GEV held at its target node 720a and, in response, provides a message including the updated GEV to node 510b. Node 510b includes GCP 1200b, which is configured to receive the update and dispatch the update message to its regulator 740b via its bus 700. In response to receiving the update, regulator 740b updates its copy of the GEV to match the copy of the GEV maintained at node 510a. Regulator 740b also broadcasts the GEV update to target nodes 720 (including target nodes 720b, 720c) of processing node 510b. GCPs 1200c, 1200d associated with target nodes 720b, 1200c respond by dispatching update messages containing updates for GCPs 1200e, 1200f on nodes 510c, 1200d. GCPs 1200e, 1200f propagate these updates to their own regulator nodes 740c, 1200d, which broadcast the updates to components on those nodes 510c, 1200d. In this way, GEV updates generated by component 770 on one node 510a in response to detecting an event are propagated to components on other nodes 510b-1200d in system 500.

[0089] As a result of the propagation of local event state (i.e., CEV) and global event state (i.e., GEV) to components 770, each component 770 in system 500 is provided with access to two different event vectors. Each component 770 can access the device-wide event and error state of its own processing node 510 via its own local copy of that processing node's 510's CEV. Each component 770 can also access the system-wide event and error state via its own local copy of the system's GEV. Based on this state, a component 770 can be configured to perform an action. Such actions, performed based on a component's accessible copy of the CEV or GEV, are referred to herein as autonomous event handling (AEH). Different components 770 in the same processing node 510 can be configured to perform different actions in response to events belonging to the same class. One or more components 770 can be configured with different policies and to perform different actions in response to events belonging to different event classes. Moreover, one or more components 770 may perform different actions in response to events associated with (and reflected in) their own processing node 510 compared to events associated with (and reflected in) the system 500.

[0090] An example of a component that can participate in an AEH is circuitry (referred to herein as an exchange block (XB)) that enables tile 4 of processing unit 2 to exchange data with devices external to the chip 510 on which processing unit 2 is implemented.

[0091] Referring to FIG. 13, FIG. 13 illustrates an example chip 1300 including an XB 1310 and multiple tiles 4. The XB 1310 includes circuitry configured to perform the described operations. The XB 1310 is associated with an external interface 1320 of the chip 1300, through which it is configured to transmit and receive data packets. The XB 1310 receives packets in one format (the "Tlink" format) from one or more tiles 4, converts the packets to another format (the "Elink" format), and then transmits the packets via the external interface 1320. The XB 1310 receives packets in the other format (the "Elink" format), converts the packets to a further format (the "Tlink" format), and then distributes the packets to one or more tiles 4. The XB 1310 is also configured to control transmissions by the tiles 4 by providing flow control messages to pass transmission authorizations between different tiles 4.

[0092] The XB 1310 is configured to engage in certain autonomous error handling (AEH) in response to an indication of an error in the GEV. Referring to FIG. 14, FIG. 14 illustrates the XB 1310 and its associated target node 720. The XB 1310 includes processing circuitry 1400 for performing the operations described herein as being performed by the XB 1310. The XB 1310 includes a mask register 1410 (referred to herein as the AEHGEVVMASKR register 1410). The AEHGEVVMASKR register 1410 stores a set of bits that identify which bits in the GEV will cause an AEH by the XB 1310. If the AEHGEVVMASKR register 1410 stores a bit indicating that an AEH is valid for a particular event class, the XB 1310 will perform an AEH in response to determining that a bit is set in the GEV to indicate that an event of that class has occurred.

[0093] When performing AEH, XB 1310 stops sending packets from tile 4 to interface 1320 and stops sending packets from interface 1320 to tile 4. If XB 1310 receives packets from tile 4, XB 1310 drops those packets. Similarly, if XB 1310 receives packets from interface 1320, XB 1310 drops those packets. XB 1310 thus blocks communication between tile 4 and external devices. As a result, data plane traffic is quickly halted.

[0094] The types of events indicated by the GEV that may cause the XB 1310 to block traffic are an error event that is fatal to the execution of the application (e.g., an unrecoverable memory error) or the downing of one of the external links of the chip 1300. In this case, IO traffic is halted prior to resetting the chip 1300.

[0095] Different components of the system may include different mask registers (such as register 1410) that indicate to each component which events the component is configured to take actions in response to. The mask register for a particular component may include a first set of indications that indicate which of the event classes identified in the CEV the component is configured to take actions in response to. The mask register may also include a second set of indications that indicate which of the event classes identified in the GEV the component is configured to take actions in response to. These registers are defined for each component, and each component can take actions (execute AEHs) in response to different types of events.

[0096] Each event class defined in the CEV (and similarly the GEV) has an associated event handler configured to implement a predetermined response to the occurrence of that event. An event handler is a software module that executes on an event handler device. Actions taken by the processing device are in addition to any AEHs performed by individual components 770 of the processing node 510. For example, in response to the occurrence of an event that represents a fatal application error, the processing entity may cause the application to reset from a checkpoint.

[0097] Different event classes have different event entities. For some event classes, the processing entity may be software running on the MCPU 310. For other event classes, the processing entity may be a host process running on the host 320 / 440. Each of the system-wide event handlers is associated with one or more processing nodes 510 and is configured to take action on the associated processing node(s) 510 in response to event notifications received from the associated processing node(s).

[0098] Referring to Figure 15, Figure 15 illustrates how event notification to an event handler 1510 is performed. The target node 720 receives CEV and GEV updates broadcast by the regulator node 720 after an event occurs. Some of the components 770 associated with the target node 720 include associated event notification hardware 1500. Components that include such event notification hardware 1500 include, for example, the PCIe complex 370 / 450 or the MCPU 310. The handler software 1510 can run on the same device (e.g., the MCPU 310) that includes the event notification hardware 1500, or on a separate device (e.g., the host device).

[0099] Event notification hardware 1500 includes a CEV notification mask register 1520. CEV notification mask register 1520 indicates that, for each event class for which an event has been logged to CEV, an event notification is to be provided to handler software 1510. In response to receiving from target node 720 that an event has occurred in a particular event class, processing circuitry in event notification hardware 1500 checks the corresponding indication in register 1520 for that event class. If register 1520 indicates that an event of that class should be reported to handler software 1510, processing circuitry in event notification hardware 1500 issues an event notification to handler software 1510. The event notification may take the form of an interrupt to handler software 1510.

[0100] The event notification hardware 1500 also includes a GEV notification mask register 1530. The GEV notification mask register 1530 indicates, for each event class for which an event has been logged to the GEV, that an event notification is to be provided to the handler software 1510. In response to receiving from the target node 720 that an event has occurred in a particular event class, the processing circuitry of the event notification hardware 1500 checks the corresponding indication in the register 1530 for that event class. If the register 1530 indicates that an event of that class should be reported to the handler software 1510, the processing circuitry of the event notification hardware 1500 issues an event notification to the handler software 1510. The event notification may take the form of an interrupt to the handler software 1510.

[0101] For each event class reported in the CEV, an event handler is defined to handle events of that class. Similarly, for each event class reported in the GEV, an event handler is defined to handle events of that class. In an embodiment, each processing node 510 is associated with four event handlers: two host processes running on the host device 320 / 440, one process running on the MCPU 310, and one process running on an additional hardware unit for enabling cryptographic functions (called the ICU). Thus, the event handlers can be implemented on separate devices (e.g., the MCPU and the host) or can take the form of separate processes running on the same device (e.g., separate host processes running on a single host).

[0102] In response to receiving an event notification, the handler software 1510 is configured to take appropriate actions. These actions include issuing a read request to read the event data register 1020 of the associated regulator node 740. The event handler 1510 reads the EDR register 1020 of the associated event class. The event handler reads the event flag and event identifier from that EDR register 1020. The read can be issued by the event handler using its associated initiator 730, and details of the event (e.g., event flag and event identifier) ​​are returned to the event handler 1510.

[0103] When a global event is notified to the handler software based on the GEV, the event flag indicates to the event handler 1510 whether the event is associated with the processing node 510 with which the event handler 1510 is associated.

[0104] 16, which illustrates the exchange of messages that occurs after an event notification is issued to an event handler 1510 based on the GEV maintained in each of the processing nodes 510. Although each event handler 1510 is shown as separate from its associated processing node 510, each event handler 1510 can be considered to be part of its associated processing node 510. Additionally, while each event handler 1510 is shown as having a single associated processing node 510, in some embodiments each event handler 1510 is associated with multiple such processing nodes 510.

[0105] As a first step, in response to receiving an event notification, each event handler 1510 issues a read request to read the event data register 1020 associated with the event class identified by the event notification. The event data register 1020 read by each event handler 1510 belongs to the processing node 510 with which the event handler 1510 is associated.

[0106] In response, as a second step, each event handler 1510 receives the state maintained in the EDR 1020 identified by the sent read request. This state includes the event flag and event identifier maintained in that EDR 1020. If the event reflected in the GEV occurred in the processing node 510 with which the event handler 1510 is associated, the event flag is set to a state indicating that the event occurred in that node 510. In the example shown in Figure 16, the event reflected in the GEV occurred in processing node 2, and therefore event handler 2 receives the event flag set to indicate that the event occurred in node 2, and also receives details of the event (e.g., in the form of an event identifier).

[0107] As a third step, Event Handler 2 propagates a message related to the event to the other Event Handlers 1 and 3. The message can be either details of the event in the EDR state received by Event Handler 2, or a message to cause Event Handlers 1 and 3 to reset their associated processing nodes 510. In the example shown, each of the Event Handlers 1510 decides to reset its associated processing node 510. This decision is made based on the EDR state obtained from Processing Node 2. This decision can be made by Event Handler 2, which then propagates the command to the other Event Handlers 1 and 3, or each Event Handler 1510 can make the decision individually based on the EDR state obtained from and distributed by Event Handler 2.

[0108] In this example, as a fourth step, each of the event handlers 1510 is shown issuing a command to reset its associated processing node 510. This reset is accomplished by asserting a signal on the reset pin 1600 on the processing node 510. Circuitry on each processing node 510 causes the reset to occur.

[0109] A reset, sometimes referred to as a software reset, resets the state of several components that are part of each processing node 510. A reset has the effect of clearing the memory of the tile 4 and deleting application data. A reset also has the effect of clearing the state of the encryption hardware used to encrypt communications in and out of the device 510. A reset resets the state of logical links and connections using those links. However, a reset does not bring down any links or change the state of all components of the physical links. Therefore, there is no need to reprogram the EPC 470 after a reset. A reset does not clear the application state of the MCPU 310.

[0110] The effect of a reset is to clear the CEV and GEV held by components of the system so that they no longer indicate that an event has occurred. Clearing the CEV and GEV is performed by a regulator 720 in that processing node 510 that has write permission to the CEVVR and GEVVR.

[0111] Once a particular processing node 510 has reset its state and its tile memory has been cleared, that processing node 510 can restart its application from the application checkpoint. Each processing device 300 / 410 has access to external memory where checkpoint data is periodically stored. Following a read request, a tile 4 of a processing unit can load the checkpoint data from external memory, thereby restarting the application from a previous point.

[0112] Each event handler 1510 is configured to cause its associated processing node 510 to immediately reset, without requiring system-wide synchronization of the reset. One problem that may be encountered when performing a reset without synchronization is that while one of the processing nodes 510 performs its reset and restarts from a previous checkpoint, additional processing nodes 510 that have not yet been reset may continue to run the previous version of the application. In this case, if a processing node 510 that has not yet been reset sends data packets containing application data corresponding to the previous version, these data packets may be received by processing unit 2 that has reset from the checkpoint. In this case, erroneous results may occur in processing by processing unit 2 that has restarted from the checkpoint because processing unit 2 has received data corresponding to the previous application version.

[0113] According to an embodiment, frames transmitted between different processing nodes 510 include a numerical value (called a generation number) that is updated in response to each reset event. Thus, the numerical value indicates the number of times the application has been reset. When data frames are transmitted by one processing node 510 to another processing node 510, the transmitting node 510 includes its own copy of the generation number in each of those frames. The numerical value is checked at the receiving node 510 against the receiving node's own copy of the generation number. If the generation number in the frame does not match the generation number maintained by the receiving node 510, the receiving node 510 discards the frame. In this way, when the receiving node 510 has already been reset and restored from a checkpoint, the receiving node is protected from frames associated with the pre-reset generation of the application that may be dispatched by other nodes in the system that have not yet been reset.

[0114] Referring to FIG. 17, FIG. 17 illustrates an example in which two processing nodes 1700a, 1700b can communicate. Each of nodes 1700a, 1700b may be one of the nodes 510 described above. Typically, a system includes more than two processing nodes, but for simplicity, only two such nodes are shown in FIG. 17. Both processing nodes 1700a, 1700b are configured to be reset and restarted from a checkpoint. The reset of both processing nodes 1700a, 1700b is part of a system-wide reset and restoration from a checkpoint. This reset may be performed in the manner described above with respect to FIG. 16, with the global reset of the system, and thus the individual reset of each node 510, being performed in response to an error event occurring in one of the nodes 510. However, the global resets of nodes 1700a, 1700b may not be synchronized, meaning that one of nodes 1700a, 1700b may proceed to reset and restore from a checkpoint before the other of nodes 1700a, 1700b has gone dormant. In this case, the node 1700a, 1700b that has not yet gone dormant may send a data frame to the node 1700a, 1700b that has restored from a checkpoint. Copying the generation number prevents interruption of application processing on the restored node.

[0115] Referring to Figures 18a and 18b, Figures 18a and 18b illustrate a sequence of events performed by the system of processing nodes 1700a, 1700b shown in Figure 17. The overall method is shown divided into a first portion 1800a (shown in Figure 18a) and a second portion (shown in Figure 18b). At the start of the sequence of events, both processing nodes 1700a, 1700b store identical copies of a generation number. The generation number reflects the number of times a global reset of the system (including a reset across both nodes 1700a, 1700b) has been performed. Because the generation numbers are identical, a frame issued by one of the nodes contains the same generation number as stored in the other of the nodes 1700a, 1700b and is therefore accepted by the other of the nodes 1700a, 1700b, allowing the nodes 1700a, 1700b to communicate.

[0116] At S1810, in response to detecting the error event, reset circuitry 1600 of first processing node 1700a causes first processing node 1700a to reset. As part of this reset, state is cleared from various components of node 1700a. The memory of tile 4 is cleared by circuitry in the hardware modules of node 1700a, which writes zeros to that memory. As part of the reset, a copy of the generation number held in storage of node 1700a is updated. As described, the generation number may be held in storage in interface 1710. The generation number is updated by incrementing it to a new value.

[0117] In S1820, following a reset of processing node 1700a, the portions of the application for which processing node 1700a is responsible are restored from a checkpoint. To accomplish this, a boot loader program is provided from a hardware module of node 1700a to a different tile 4. Tile 4 executes the boot loader program to issue read requests to checkpoint memory 1720. Although checkpoint memory 1720 is shown as a single memory unit, it may include multiple memory units for different nodes 1700a, 1700b. In response to the read request issued by tile 4 of node 1700a, checkpointed application instructions and application data are returned from memory 1720 to those tiles 4. The application data returned to node 1700a represents the state of the portions of the application running on processing unit 2 of that node 1700a that were written out from that processing unit 2 at the last checkpoint.

[0118] At S1830, tile 4 of processing unit 2 of node 1700a continues executing application instructions from the checkpoint. At the same time, second processing node 1700b has not yet been reset, and processing unit 2 of that node 1700b is processing that portion of the application corresponding to the previous generation. Therefore, second processing node 1700b stores the previous generation number, which was also held in storage on first processing node 1700a before the reset at S1810.

[0119] At S1840, tile 4 of processing unit 2 of node 1700b is configured to cause the issuance of one or more data frames to first processing node 1700a. Each of the one or more data frames includes a copy of a previous generation number maintained in storage at node 1700b. Tile 4 causes the issuance of the data frames by issuing a data packet to the EPC of each processing node 1700b, which causes the data packet to be encapsulated in a data frame and transmitted to node 1700a.

[0120] At S1850, first processing node 1700a receives the data frames transmitted at S1840. Circuitry in first processing node 1700a checks the generation number in each data frame and compares it to a copy of the generation number maintained in storage at processing node 1700a. Because these two generation numbers do not match, the circuitry in processing node 1700a discards the frame. As will be described, the checking and discarding of frames can be performed by interface 1710 of first processing node 1700a.

[0121] At S1860, second processing node 1700b is reset. This step performed at S1860 is the same as S1810 described above, but is performed by second processing node 1700b rather than first processing node 1700a.

[0122] At S1870, following a reset of processing node 1700b, the portions of the application for which processing node 1700b is responsible are restored from a checkpoint. To accomplish this, a boot loader program is provided from a hardware module in node 1700b to a different tile 4. Tile 4 executes the boot loader program to issue read requests to checkpoint memory 1720. In response to the read requests issued by tiles 4 of node 1700b, checkpointed application instructions and application data are returned from memory 1720 to those tiles 4. The application data returned to node 1700b represents the state of the portions of the application running on processing unit 2 of that node 1700b that were written out from that processing unit 2 at the last checkpoint.

[0123] At S1880, processing unit 2 of second processing node 1700b continues executing that portion of the application from the checkpoint.

[0124] At S1890, tile 4 of processing unit 2 of node 1700b is configured to cause the issuance of one or more additional data frames to first processing node 1700a. Tile 4 causes these additional data frames to be issued by issuing data packets to the EPC of each processing node 1700b, which causes the data packets to be encapsulated in data frames and transmitted to node 1700a. Unlike the frames transmitted at S1840, each of these additional data frames includes a copy of an updated version of the generation number, which is now maintained in storage at both nodes 1700a, 1700b.

[0125] At S1895, first processing node 1700a receives the data frames sent at S1890. Circuitry in first processing node 1700a checks the generation number in each data frame and compares it to a copy of the generation number maintained in storage at processing node 1700a. Because these two generation numbers match, the circuitry in processing node 1700a accepts the frames. The data from these frames is written to the memory of tile 4.

[0126] The generation number can be inserted at different points in the data frames transmitted between node 1700a and node 1700b. The generation number can be inserted in the packet header (e.g., the Elink packet header) dispatched by tile 4. Alternatively, the generation number can be inserted in the payload of the packet.

[0127] Referring to FIG. 19, FIG. 19 illustrates an exemplary embodiment in which a generation number is included in the MAC address of a frame header. In an embodiment, MCPU 310 is responsible for providing the MAC address to use when transmitting data over a connection. As part of the MAC address, MCPU 310 provides the generation number. MAC addresses for a connection (including the MAC address of interface 1710 and the MAC addresses of other interfaces 1710) are provided by MCPU 310 to and stored in connection status register 1915 and control and status register (CSR) 1916. Specifically, register 1915 stores the MAC of the destination interface 1710 to which a frame is dispatched, and CSR 1916 stores the MAC of the interface 1710 from which the frame is sent. One or more bits in each of these MAC addresses are reserved for inclusion in the data frame as a generation number.

[0128] When tile 4 has data to send, it dispatches this data to interface 1710 in the form of one or more data packets (e.g., Elink packets). Frame builder 1120 encapsulates the data packets into data frames by adding a frame header and a frame tail. In this way, data packets issued by processing unit 2 can be tunneled over Ethernet. Frame builder 1120 inserts the MAC held in connection state register 1915 as the destination MAC of each frame. Frame builder 1120 inserts the MAC held in connection state register 1915 as the source MAC of each frame.

[0129] On the receiving side, a frame check buffer 1908 is provided that receives data frames from other interfaces 1710 belonging to other nodes. The circuitry of interface 1710 shown in Figure 19 checks the destination MAC in the received frames against the MACs held in CSR 1916. If there is a mismatch between these MACs, the circuitry drops the frames. If there is a match between the MACs, the circuitry forwards the data packets encapsulated in those frames to tile 4 of the processing unit 2 identified in the packet header.

[0130] MCPU 310 includes at least one processor that executes instructions to provide the illustrated update function 1920 and address resolution function 1940. The address resolution function is configured to provide a MAC address for a connection, which is provided to interface 1710. Because the MAC addresses are local scope MAC addresses, they may be updated by software rather than being permanently fixed for a particular interface 1710. Address resolution function 1940 inserts into each MAC address the current generation number held in generation number register 1910. The remaining portion of the MAC address may be obtained from storage 1930 or may be determined by function 1940 in another suitable manner.

[0131] MCPU 404 updates the MAC for a connection in response to a reset event. MCPU 404 includes a generation number register 1210 that stores the current generation number provided as part of each MAC address output by address resolution function 1240. Processing circuitry in MCPU 404 supports an update function 1220 that updates the generation number held in generation number register 1210.

[0132] When a reset of the processing node to which MCPU 310 belongs is performed (e.g., the resets occurring at S1810 and S1860), an indication of this reset is delivered to MCPU 310. Update function 1920 responds to this reset event by updating the value of the generation number held in generation number register 1910. Updating the generation number held in generation number register 1910 may include incrementing the current value of the generation number by one. For example, assume that the generation number consists of five bits and that the current value of the generation number held in register 1910 is given by the bit sequence 00001. In response to the reset event, update function 1920 updates the generation number by incrementing its value by one to 00010.

[0133] Because the generation number contains a finite number of bits, updating the generation number involves resetting the generation number to the minimum possible value if the current value of the generation number is equal to the maximum possible value. For example, assume that the generation number consists of five bits and that the current value of the generation number held in register 1210 is given by the bit sequence 11111. In response to a reset event, update function 1220 resets the generation number to the minimum possible value given by 00000. Thus, the generation number is updated by update function 1920 in a wraparound manner.

[0134] In response to the indication of the reset event, the address resolution function 1940 determines updated MAC addresses for the connections in which the device 400 participates. As previously mentioned, one or more bits are reserved in each MAC address to represent a generation number. The address resolution function 1240 sets these bits to the value of the new generation number stored in register 1910 by the update function. The remaining bits of each MAC address are set to the same value as they were before the reset event.

[0135] For each MAC address, the remaining bits of that MAC address can be determined in different ways. In some embodiments, these bits can be maintained in storage 1930 accessible to address resolution function 1240 and concatenated with the new generation number bits to form the complete MAC address. These bits maintained in storage 1930 are not cleared in response to a reset event, but rather persist and remain unchanged after a reset event. In this embodiment, storage 1930 stores the remaining bits (i.e., bits other than the generation number bits) for each of the MAC addresses that address resolution function 1940 is responsible for determining. Address resolution function 1940 provides each of these MACs by combining the remaining bits with the same generation number for each MAC.

[0136] As previously described, the application state of a first processing node 1700a can be restored from a checkpoint before the application state of another processing node (e.g., node 1700b) is similarly restored from a checkpoint. In this case, the first processing node 1700a can continue its application processing while processing node 1700b has not yet been reset and contains application state corresponding to the previous generation of the application. This is made possible by the use of barrier synchronization, which separates the computation phase of participating processing units 2 from the exchange phase of those processing units 2. The barrier synchronization is inserted into the compiled code executing on each processing unit 2. For each barrier synchronization, one or more processing units 2 are configured to participate.

[0137] Barrier synchronization can be used to prevent a first processing unit 2 of node 1700a from executing ahead of a point in its code beyond which it would need to exchange data with a processing unit 2 that has not yet been restored from a checkpoint and restarted. For example, assume that processing unit 2 of node 1700a enters a computation phase, following a checkpoint, in which it performs computations on data and produces results. Processing unit 2 of node 1700a will not exchange data with processing unit 2 of node 1700b until it reaches barrier synchronization, in which both processing units 2 of nodes 1700a and 1700b participate. At this point, processing unit 2 of node 1700a stalls until processing unit 2 of node 1700b also reaches barrier synchronization. Thus, even if processing unit 2 of node 1700a reaches barrier synchronization while node 1700b has not yet reset, processing unit 2 of node 1700a will wait until node 1700b is reset and its processing unit 2 reaches the barrier before proceeding to the exchange phase following barrier synchronization. During the exchange phase, processing unit 2 of node 1700a transmits application data scheduled for transmission during that exchange phase to processing unit 2 of node 1700b.

[0138] The use of barrier synchronization can prevent processing unit 2 of system 500 from sending data to a processing unit 2 of system 500 that has executed earlier and has not yet been reset. The use of barrier synchronization in the context of processing unit 2 is described in more detail in our earlier U.S. patent application Ser. No. 17 / 446,681, which is incorporated herein by reference.

[0139] It has been described how, in response to a reset event, copies of the CEV and GEV are cleared from the processing node (i.e., reset to not indicate the event). Bits in the CEV can also be reset by an event handler for an event class. Resetting a bit in the CEV allows another event in the event class to be detected and processed.

[0140] 10, regulator node 740 includes multiple registers 1040, referred to herein as event handler registration registers 1040. Registers 1040 include an event handler registration register for each of the different event handlers of processing node 510. For example, there is an event handler registration register for each of the two host processors (one for MCPU 310 and one for ICU). Each of event handler registration registers 1040 contains, for each event class captured in the CEV, an indication of whether the respective event handler associated with that register has permission to clear the state of that event class.

[0141] When an event handler 1510 receives an event notification, the event handler 1510 may decide to clear the event state for that event so that, among other things, further occurrences of events of the same event class can be detected. To clear the event state for an event class, the event handler 1510 issues a request to the regulator node 740 via the control bus 700 to clear the EDR for that event class. The request includes the identifier of the associated event handler 1510 and the identifier of the event class. The regulator node 740 checks the permission indication for that event class in the event handler registration register 1040 for that event handler 1510. If the event handler 1510 has permission to clear the event state for that event class, the regulator node 740 updates the EDR for that event class to reset the EDR to its default state. In this state, the EDR indicates that no event has been detected. In addition, the regulator node 740 updates the master CEVVR 1030 to reset the event indication for this event class. The regulator node 740 also updates the copy of the CEV held on the target node 720 of the processing node 510 by broadcasting a write request to reset the event indication.

[0142] Referring to Figure 20, Figure 20 illustrates an exemplary method 2000 for aggregating global event state among processing nodes 510. One or more of these steps may overlap with each other.

[0143] S2010 is executed by each node 510 in system 500. In S2010, each processing node 510 in system 510 stores a copy of the global event vector for processing system 500. Initially, this vector is set to indicate that no events have occurred in system 500. The global event vector is maintained in the master GEVVR 1010 and in the GEVVR 990 of the target node.

[0144] S2020 is executed by each node 510 of the system 500. In S2020, each processing unit 2 of the system 500 executes the instruction set of the distributed application.

[0145] Step S2030 is performed by each node 510 of system 500. In step S2030, each of processing nodes 510 updates each of one or more elements of at least one copy of a global event vector in response to an event of a type associated with the respective element and occurring at one of the processing nodes of the processing system. This step is performed by regulator 740.

[0146] At S2040, a first event of a first type of event is detected on a first processing node 510. This event may be, for example, an event detected by component 770a of node 510a. In response to detecting the first event, the first processing node 510 provides an indication of an update to a first element of a global event vector to one or more other processing nodes 510. The first element is associated with the first type of event.

[0147] S2050 is executed by other nodes 510 (i.e., other than the first node 510) of the system 500 that receive the update provided by the first processing node 510 in S2040. In S2050, each of these other nodes 510, in response to the indication of the update of the first element, updates the corresponding first element in its at least one copy of the global event vector.

[0148] 21, which illustrates an example method 2100 for updating copies of event vectors maintained on a device and using these copies to perform responsive actions. The error event vectors may be CEVs or GEVs as discussed above. The device may be device 300 or device 410.

[0149] At S2110, a copy of the event vector is stored for each of the components 770 of the device 300. Initially, each copy of the event vector may be set to indicate that no events have occurred in any of the event classes captured in the event vector.

[0150] At S2120, events associated with each of the subset of components 770 are detected. The events are reported in respective event reports to the control node 740. The subset of components 770 includes one or more components 770.

[0151] At S2130, the control node 740 receives a first event report from the first component 770, the first event report reporting an event of a first type, the event of the first type being detected by the first component 770.

[0152] At S2140, the control node 740 broadcasts one or more writes to each of the plurality of components 770 to cause them to update their respective copies of the error event vector by setting a first element associated with the first type of event to indicate that the first type of event occurred on the processing device, the copies being maintained in the CEVVR 980 of the target node 720.

[0153] At S2150, in response to setting the first element in its copy of the error event vector, the second component 770 performs an action associated with the first type of event.

[0154] Various functions are described above as being performed by circuitry (e.g., processing logic 930, processing circuitry 1010, processing circuitry 1220, processing circuitry 1400, circuitry 1600). This circuitry may include dedicated hardware (e.g., FPGAs or ASICs) and / or processors configured to execute instructions. Each processing node 510 of system 500 is assigned an instruction set for performing any of the operations performed by a processor configured to execute the instructions.

[0155] The above embodiments are described by way of example only.

Claims

1. 1. A processing system configured to support an application, comprising: a plurality of processing nodes, each of the plurality of processing nodes comprising: a processing unit configured to execute a set of instructions for a portion of the application; at least one storage configured to store a copy of a numerical value indicative of the number of times a global reset of the processing system has been performed; A reset circuit; the reset circuit includes: resetting the processing node as part of a single global reset of the processing system; restoring application state for a portion of the application on the processing node from a checkpoint; configured to run each of the processing nodes includes update circuitry configured to update a copy of a numerical value maintained in at least one storage of the processing node from a first value to a second value as part of a reset of the processing node; a processing unit of the second processing node configured to execute a set of instructions to perform, prior to resetting the second processing node, sending one or more data frames to the first processing node, each of the one or more data frames including the first value; wherein circuitry of the first processing node is configured to, following a reset of the first processing node, cause data in the one or more data frames to be discarded in response to determining a mismatch between the first value in the one or more data frames and the second value maintained in at least one storage of the first processing node.

2. for each of the processing nodes, the processing unit is configured to participate in multiple barrier synchronizations as indicated in an instruction set; 2. The processing system of claim 1, wherein for each of the processing nodes, the processing unit is configured to, for each of the barrier synchronizations in which it participates, pass through the barrier synchronization in response to at least one further processing unit of the processing system joining the barrier synchronization.

3. 3. The processing system of claim 2, wherein for each of the processing nodes, each of the plurality of barrier synchronizations separates an operation phase of the processing unit from an exchange phase of the processing unit.

4. 2. The processing system of claim 1, wherein a processing unit of the first processing node is configured to stall at a first barrier synchronization following a reset of the first processing node until one of the further processing units arranged to participate in the first barrier synchronization reaches the first barrier synchronization in application processing.

5. The processing system of claim 4 , wherein one of the further processing units is a processing unit of the second processing node.

6. the first barrier synchronization acts as a barrier between a computation phase and an exchange phase; a processing unit of the first processing node, performing operations during the operation phase until the first barrier synchronization is reached; subsequent to the first barrier synchronization, during the exchange phase, exchanging data with the second processing node, including data contained in the one or more data frames; 5. The processing system of claim 4, configured to execute an application instruction set that executes:

7. 2. The processing system of claim 1, wherein the circuitry of the first processing node configured to determine the mismatch is part of an interface of the first processing node for connecting with the second processing node.

8. 8. The processing system of claim 7, wherein the interface of the first processing node is configured to receive further data frames from the second processing node and to provide data packets encapsulated in the further data frames to a processing unit of the first processing node.

9. a processing unit of the second processing node configured to execute a set of instructions to perform transmission of one or more further data frames to the first processing node following a reset of the second processing node; the one or more further data frames include the second value; 2. The processing system of claim 1, wherein the circuitry of the first processing node is configured to receive the one or more further data frames and, in response to determining a match between the second value in the one or more further data frames and the second value maintained in at least one storage of the first processing node, cause acceptance of the data in the one or more data frames.

10. 2. The processing system of claim 1, wherein the circuitry of the first processing node is configured to determine the mismatch as part of checking an address field in each of the data frames.

11. 2. The processing system of claim 1, wherein, for each of the processing nodes, the update circuitry includes a processor configured to execute computer-readable instructions to perform an update of a copy of a numerical value maintained in storage of at least one of the processing nodes.

12. 2. The processing system of claim 1, wherein updating the copy of the numerical value comprises incrementing the first value to obtain the second value.

13. 2. The processing system of claim 1, wherein the copy of the value forms part of a connection state for a connection between the first processing node and the second processing node.

14. Each processing node: the processing unit; at least one storage for said processing node; 10. The processing system of claim 1, comprising an integrated circuit comprising:

15. 2. The processing system of claim 1, wherein a processing unit of the second processing node is configured to execute a set of instructions to output the first value in one or more data packets of the one or more data frames.

16. a processing unit of the second processing node configured to execute a set of instructions to output data for the one or more data frames to an interface circuit; 2. The processing system of claim 1, wherein the interface circuitry is configured to provide the one or more frames of data with the first value to be inserted into each frame.

17. Executing, at each of a plurality of processing nodes, a set of instructions for a portion of an application supported by a system including the plurality of processing nodes; storing in each of said processing nodes a copy of a number indicating the number of times a global reset of said system has been performed; resetting the plurality of processing nodes, including updating copies of the numerical values ​​maintained in the processing nodes from a first value to a second value; restoring application state for a portion of the application on the processing node from a checkpoint; performing one of the global resets, including transmitting one or more data frames including the first value from the second processing node to the first processing node before resetting the second processing node; following a reset of the first processing node, in response to determining a mismatch between the first value in the one or more data frames and the second value maintained in the first processing node, causing data in the one or more data frames to be discarded; A method comprising:

18. 1. A computer program comprising a set of executable instructions for each processing node of a plurality of processing nodes forming a data processing system, the set of executable instructions, when executed by at least one processor of a respective processing node, supporting a portion of a distributed application supported by the plurality of processing nodes; storing a copy of a numerical value indicating the number of times a global reset of said data processing system has been performed; as part of a global reset of said data processing system; resetting the processing node, including updating a copy of the numerical value maintained in the processing node from a first value to a second value; restoring application state for the portion of the distributed application on the processing node from a checkpoint; on the processing node; the second set of executable instructions, when executed by at least one processor of a second processing node, is configured to cause the second processing node to transmit one or more data frames including the first value to the first processing node before resetting the second processing node; the first set of executable instructions, when executed by at least one processor of the first processing node, is configured to cause the first processing node to discard data in the one or more data frames in response to determining a mismatch between the first value in the one or more data frames and the second value stored by the first processing node following a reset of the first processing node. Computer program.

Images