Resetting a multi-node system

Abstract

Each node stores a numerical value (herein called a generation number) that is updated each time the node is reset and restarted from a checkpoint. In most cases, the generation numbers held by the nodes are the same throughout the system, since the nodes in the system participate in the same reset events. However, in some cases, when one of the nodes is reset before the other, the generation numbers between these two nodes will be different. Each data frame transmitted between nodes contains the sending node's generation number, which is checked by the receiving node and is only accepted if the generation number in the frame matches the receiving node's generation number. If the receiving node is reset before the sending node is reset, the generation number in the frame will not match the receiving node's generation number and the frame will not be accepted. Thus, a node that has been reset and restarted is protected from packets related to applications of a previous generation dispatched by nodes that have not yet been reset.

Description

[Technical field]

[0001] The present disclosure relates to processing systems configured to support applications, and in particular to processing systems that include multiple processing nodes configured to execute instruction sets for portions of the applications. [Background technology]

[0002] In the context of data processing of complex or high-volume applications, a processing unit may be provided to perform the processing of that data. The processing unit may act as a work accelerator to offload the processing of certain data from the host system. Such a processing unit may have specialized hardware to perform certain types of processing. For example, a processing unit suitable for machine learning may include many processor tiles implemented together on the same chip to achieve massive parallelism supported by the same processor. To support the operation of such a processing unit, additional system-on-chip (SoC) circuitry may be provided along with the processing unit on the same chip. Such SoC circuitry may include, for example, an on-chip memory independent of the processor's memory, a port controller for external communication, and boot hardware for booting the processing unit.

[0003] To scale up the processing capacity to support larger applications, multiple processing units can be connected together to form larger systems. In various implementations of such a processing system, each of the chips supporting these processing units are connected together via high-speed links that allow communication between the tile processors of the different processing units.

[0004] During processing of applications supported across a system of connected chips, various error events may occur on one or more of these chips. Common events include, for example, an over-temperature condition, the occurrence of an uncorrectable memory error, or the failure of a device's external link. Some error events may require a system reset or a restart of the applications running on the system from a checkpoint. Summary of the Invention [Problem to be solved by the invention]

[0005] When performing a reset of a multi-node system, one problem that can be encountered is that if one of the nodes is reset and another node is restarted from a previous checkpoint before the reset, the reset node may receive packets from nodes that have not yet been reset. These packets relate to previous generations of applications running on the system. This can disrupt application processing related to new generations of applications that the reset node is currently processing. [Means for solving the problem]

[0006] According to a first aspect, a processing system configured to support an application is provided. The processing system includes a plurality of processing nodes. Each of the plurality of processing nodes includes a processing unit configured to execute a set of instructions for a portion of the application, at least one storage configured to store a copy of a numerical value indicative of a number of times a global reset of the processing system has been performed, and a reset circuit. The reset circuit is configured to reset the respective processing node as part of a single global reset of the processing system, and to restore application processing by the processing unit of the respective processing node from a checkpoint. Each of the processing nodes includes circuitry configured to update the copy of the numerical value held in the at least one storage of the processing node as part of the reset of the processing node. The processing unit of the second processing node is configured to execute a set of instructions to perform transmission of one or more data frames to the first processing node prior to the reset of the second processing node, each of the one or more data frames including a copy of the numerical value held in the at least one storage of the second processing node. The circuitry of the first processing node is configured to cause one or more data frames to be discarded in response to determining a mismatch between copies of the numerical values ​​in the one or more data frames and copies of the numerical values ​​held in the at least one storage following a reset of the first processing node.

[0007] Each node stores a number (herein called a generation number) that is updated each time the node is reset and restarted from a checkpoint. In most cases, the generation numbers held by the nodes are the same throughout the system, since the nodes in the system participate in the same reset events. However, in some cases, when one of the nodes is reset before the other, the generation numbers between these two nodes will be different. Each data frame transmitted between nodes contains the sending node's generation number, which is checked by the receiving node and is only accepted if the generation number in the frame matches the receiving node's generation number. If the receiving node is reset before the sending node is reset, the generation number in the frame will not match the receiving node's generation number and the frame will not be accepted. Thus, a node that has been reset and restarted is protected from packets related to applications of a previous generation dispatched by nodes that have not yet been reset.

[0008] According to a first aspect, a processing system configured to support an application is provided. The processing system includes a plurality of processing nodes. Each of the plurality of processing nodes includes a processing unit configured to execute a set of instructions for a portion of the application, at least one storage configured to store a copy of a numerical value indicative of a number of times a global reset of the processing system has been performed, and a reset circuit. The reset circuit is configured to reset the processing node as part of a global reset of the processing system, and to restore an application state for the portion of the application of the processing node from a checkpoint. Each of the processing nodes includes an update circuit configured to update the copy of the numerical value held in the at least one storage of the processing node from a first value to a second value as part of the reset of the processing node. The processing unit of the second processing node is configured to execute a set of instructions for transmitting one or more data frames to the first processing node prior to the reset of the second processing node, each of the one or more data frames including a first value of the numerical value. The circuitry of the first processing node is configured to cause data in the one or more data frames to be discarded in response to determining, following a reset of the first processing node, a mismatch between a first value of the numerical value in the one or more data frames and a second value of the numerical value held in at least one storage of the first processing node.

[0009] In some embodiments, for each of the processing nodes, the processing unit is configured to participate in a number of barrier synchronizations as indicated in the instruction set, and for each of the processing nodes, the processing unit is configured, for each barrier synchronization in which it participates, to pass through the respective barrier synchronization in response to at least one further processing unit of the processing system joining the barrier synchronization.

[0010] In some embodiments, for each of the processing nodes, each of a plurality of barrier synchronizations separates a computation phase of the processing unit and an exchange phase of the processing unit.

[0011] In some embodiments, a processing unit of a first processing node is configured to stall at a first barrier synchronization following a reset of the first processing node until one of the further processing units arranged to also participate in the first barrier synchronization reaches the first barrier synchronization in application processing.

[0012] In some embodiments, one of the further processing units is a processing unit of a second processing node.

[0013] In some embodiments, the first barrier synchronization operates as a barrier between a computation phase and an exchange phase, and the processing unit of the first processing node is configured to execute a set of application instructions to perform computations during the computation phase until the first barrier synchronization is reached, and to exchange data with the second processing node, including data contained within the one or more data frames, during the exchange phase, following the first barrier synchronization.

[0014] In some embodiments, the circuitry of the first processing node configured to determine the mismatch is part of an interface of the first processing node for connecting with a second processing node.

[0015] In some embodiments, the interface of the first processing node is configured to receive a further data frame from the second processing node and to provide a data packet encapsulated in the further data frame to a processing unit of the first processing node.

[0016] In some embodiments, the processing unit of the second processing node is configured to execute a set of instructions to cause transmission of one or more further data frames to the first processing node following a reset of the second processing node, each of the one or more further data frames including a second value, and the circuitry of the first processing node is configured to receive the one or more further data frames and, in response to determining a match between the second value in each of the one or more further data frames and a second value of a numerical value held in at least one storage of the first processing node, cause data in the one or more data frames to be accepted.

[0017] In some embodiments, the circuitry of the first processing node is configured to determine the mismatch as part of a check of the address fields in each of the data frames.

[0018] In some embodiments, for each of the processing nodes, the update circuitry includes a processor configured to execute computer-readable instructions to perform an update of a copy of the numerical value held in at least one storage of the processing node.

[0019] In some embodiments, updating the copy of the numerical value includes incrementing the first value to obtain the second value.

[0020] In some embodiments, the copy of the numerical value forms part of a connection state for a connection between the first processing node and the second processing node.

[0021] In some embodiments, each processing node includes an integrated circuit that includes a processing unit and at least one storage for the processing node.

[0022] In some embodiments, a processing unit of the second processing node is configured to execute a set of instructions to output a first value in one or more data packets of one or more data frames.

[0023] In some embodiments, the processing unit of the second processing node is configured to execute a set of instructions to output data for one or more frames of data to an interface circuit, the interface circuit being configured to provide the one or more frames of data with a first value to be inserted into each frame.

[0024] According to a second aspect, a method is provided that includes: executing, at each of a plurality of processing nodes, a set of instructions for a portion of an application supported by a system including the plurality of processing nodes; performing one of the global resets including storing, at each of the processing nodes, a copy of a numerical value indicative of a number of times a global reset of the processing system has been performed; resetting each of the plurality of processing nodes including updating a copy of the numerical value maintained at each of the processing nodes from a first value to a second value; and restoring an application state for the portion of the application of the processing nodes from a checkpoint; transmitting one or more data frames from the second processing node to a first processing node prior to resetting the second processing node, the data frames including the first value; and following the reset of the first processing node, in response to determining a mismatch between the first value in the one or more data frames and the second value maintained at the first processing node, causing data in the one or more data frames to be discarded.

[0025] According to a third aspect, there is provided a computer program comprising a set of executable instructions for each processing node of a plurality of processing nodes forming a data processing system, each of the executable instructions of the program, when executed by at least one processor of the processing node, causes the processing node to: support a portion of a distributed application supported by the plurality of processing nodes; store a copy of a numerical value indicative of the number of times a global reset of the processing system has been performed; reset the processing node as part of the global reset of the data processing system, including updating the copy of the numerical value held in the processing node from a first value to a second value; and restore an application state for the portion of the distributed application of the processing node from a checkpoint. The second set of executable instructions, when executed by at least one processor of the second processing node, is configured to cause the second processing node to transmit one or more data frames including the first value to the first processing node prior to resetting the second processing node, and the first set of executable instructions, when executed by the at least one processor of the first processing node, is configured to cause the first processing node to discard data in the one or more data frames in response to determining a mismatch between the first value in the one or more data frames and a second value stored by the first processing node following resetting the first processing node.

[0026] According to a fourth aspect, there is provided a non-transitory computer readable medium storing a computer program according to the third aspect.

[0027] To aid in understanding the present disclosure and to show how embodiments may be practiced, reference is made by way of example to the accompanying drawings, in which: [Brief description of the drawings]

[0028] [Figure 1] FIG. 2 is a schematic block diagram of a multi-tile processing unit. [Diagram 2]FIG. 2 is a schematic block diagram of one of the tiles that can be implemented in a multi-tile processing unit. [Diagram 3] 1 is a first exemplary embodiment of a processing node. [Figure 4] 1 is a second exemplary embodiment of a processing node. [Figure 5A] 1 illustrates a system of processing nodes connected together to form a spanning tree graph for the exchange of global event states. [Figure 5B] 1 shows a system of processing nodes connected together in a ring topology for the exchange of global event state. [Figure 5C] 1 illustrates the exchange of global event states between a series of processing nodes of the system. [Figure 5D] 1 illustrates how a global event state is propagated to multiple components in each of a series of processing nodes. [Figure 6] 1 illustrates a structure of a local event vector and a global event vector in accordance with an exemplary embodiment; [Figure 7] 2 illustrates a bus architecture used for the propagation of event states between components of a processing node. [Figure 8] 1 illustrates an event report dispatched by a component for delivery to a control node of a processing node. [Figure 9] 1 shows a node on the bus for receiving and storing event states broadcast by a control node. [Figure 10] 1 shows registers and processing circuitry maintained in a control node to enable the control node to store and distribute event state. [Figure 11] Among other things, an example of an interface circuit used to exchange global event state between processing nodes is shown. [Figure 12] 2 illustrates in more detail the components of the interface circuitry used to transmit and receive global event states. [Figure 13]1 illustrates an example of a switching block for enabling a tile of a processing node to switch data plane traffic with a tile of another processing node. [Figure 14] 13 illustrates example registers and circuitry in a switching block that allows the switching block to participate in autonomous event processing in response to event states in a local or global event vector. [Figure 15] An example is given of how event notification hardware is used to notify handler software of the occurrence of an event. [Figure 16] It illustrates how multiple event handlers in a system can respond to notification of events occurring in the system. [Figure 17] 1 illustrates how checkpoint data is read from checkpoint memory and returned to a processing unit. [Figure 18a] 1 illustrates a first portion of a method for performing a reset of a multi-node system. [Figure 18b] 4 illustrates a second portion of a method for performing a reset of a multi-node system. [Figure 19] Illustrates how the generation number used for a connection can be updated in response to a reset event. [Figure 20] 1 illustrates an example of a method for propagating global event states between different nodes of a multi-node system. [Figure 21] An example method is provided for propagating local event states to components so that the components can participate in performing associated actions in response to the local event state. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Referring to Figure 1, Figure 1 illustrates an exemplary multi-processing unit 2. The multi-tile processing unit 2 is described in our earlier U.S. patent application Ser. No. 15 / 886,065, the contents of which are incorporated herein by reference. Each of the processing units 2 is formed on a separate integrated circuit (i.e., chip).

[0030] The processing unit 2 includes an array 6 of multiple processor tiles 4 and an interconnect 34 connecting the tiles 4. The processing unit 2 may be implemented as one of multiple dies packaged in the same IC package. The interconnect 34 allows the tiles 4 to exchange data with each other, and is therefore sometimes referred to herein as a "switching fabric" 34. Each tile 4 includes respective instances of execution units and memory. For example, by way of illustration, the processing unit 2 may be made up of hundreds or even thousands of tiles 4. For completeness, it should also be noted that "array" as referred to herein does not necessarily imply any particular dimensional value or physical layout of the tiles 4.

[0031] In an embodiment, each processing unit 2 is part of a chip that also includes one or more external links, by which the processing unit 2 can be connected to one or more other processing units (e.g., one or more other instances of the same processing unit 2). These external links may include any one or more of one or more processing-unit-to-host links for connecting the processing unit 2 to a host system, and / or one or more processing-unit-to-processing-unit links for connecting the processing unit 2 together with one or more other instances of the processing unit 2 on the same IC package or card or on different cards. The processing unit 2 receives work in the form of application data from the host and processes it.

[0032] Each of the tiles 4 includes processing circuitry and memory. In some exemplary embodiments, the processing circuitry is a multi-threaded processor 10. FIG. 2 illustrates an example of a tile 4 according to an embodiment of the present disclosure. The tiles 4 include a multi-threaded processor 10 in the form of a barrel-threaded processor 10 and a local memory 11. The barrel-threaded processor 10 is a type of multi-threaded processor 10 in which the execution time of a pipeline is divided into a repeating sequence of interleaved time slots, each time slot may be owned by a given thread, as will be discussed in more detail below. The memory 11 includes an instruction memory 12 and a data memory 22 (which may be implemented in different addressable memory units or in different regions of the same addressable memory unit). The instruction memory 12 stores machine code that is executed by the processing unit 10, while the data memory 22 stores both data that is operated on by the executed code and data that is output by the executed code (e.g., as a result of such manipulation).

[0033] Memory 12 stores various different threads of the program, each thread including a respective sequence of instructions for performing a particular task or tasks. Note that an instruction as referred to herein means a machine code instruction (i.e., an instance of one of the basic instructions of the processor's instruction set) consisting of a single opcode and zero or more operands.

[0034] Within the processor 10, multiple different threads from an instruction memory 12 can be interleaved through a single execution pipeline 13 (although typically only a subset of all threads stored in the instruction memory can be interleaved at any given time throughout the program). The multithreaded processor 10 includes multiple context register files 26 arranged to each represent a state (context) of a different thread being executed in parallel, a shared execution pipeline 13 common to the threads being executed in parallel, and a scheduler 24 for scheduling the parallel threads to execute through the shared pipeline in an interleaved manner, preferably in a round-robin manner. The processor 10 is connected to a shared instruction memory 12 common to the multiple threads, and a shared data memory 22 also common to the multiple threads.

[0035] Execution pipeline 13 includes a fetch stage 14, a decode stage 16, and an execute stage 18 that includes execution units capable of performing arithmetic and logical operations, address calculations, load and store operations, and other operations as defined by the instruction set architecture. Each of the context register files 26 includes a set of registers for representing the program state of the respective thread.

[0036] Referring to FIG. 3, FIG. 3 illustrates an example of a processing device 300 including the processing unit 2 discussed above. The processing device 300 constitutes a processing node in a system of processing nodes. The device 300 is implemented in a single integrated circuit (i.e., chip). The device 300 includes multiple components including the processing unit 2, and also includes other system-on-chip (SoC) components that support the operation of the processing unit 2. The device 300 includes a management processor 310, referred to as a management CPU (MCPU) 310. The MCPU 310 executes computer-readable instructions to support the operation of the device 300, for example, by handling error events and setting up connections. The device 300 also includes a temperature sensor 330 for measuring the temperature of the device 300. The device 300 includes a phase-locked loop (PLL) 340 for providing a clock signal for the device 300. The device 300 includes a DRAM module 350 for storing and / or providing data (e.g., checkpoint data) output by application code executing on the processing unit 2 to the processing unit 2. The device 300 includes multiple interface controllers for external communications. These include, for example, an Ethernet port controller (EPC) 360. The EPC 360 is used to tunnel packets over Ethernet between the device 300 and other equivalent devices 300 in a multi-node system. The tunneled packets can be provided to / from tiles 4 of the processing unit 2 (for data plane traffic) or to / from various components of the device 300 (for control plane traffic). The interface controllers also include a PCIe complex 370 for PCIe communications between the device 300 and a host device 320.

[0037] FIG. 3 illustrates an example of a processing node 300 implemented on a single chip 300. Referring to FIG. 4, FIG. 4 illustrates a processing node 400 including multiple chips 410, 420. The processing node 400 includes a main processing chip 410 that includes a processing unit 2. The processing node 400 also includes an additional chip 420 associated with and connected to the main processing chip 410. The additional chip 420 may also be referred to as a fabric chip 420. For simplicity, only one of the four fabric chips 420 of the node 400 is shown in detail. Each of the fabric chips 420 provides access to a DRAM memory 430. The DRAM memory 430 may be read or written by the processing unit 2. One of the fabric chips 420 provides access to a host device 440 via a PCIe complex 450. Each of the fabric chips 420 includes an EPC 460. One of the EPCs 460 is for interfacing with the main processing chip 410. Other EPCs 460 are for interfacing with other processing nodes that are equivalent to processing node 400. Chip 410 also includes EPCs 470. Some of the EPCs 470 are for interfacing with fabric chips 420 that belong to the same node 400. Other EPCs 470 are for interfacing with fabric chips 420 that belong to different nodes 400. In this embodiment, MCPU 310 is moved from main processing chip 410 to fabric chip 420.

[0038] A number of different events may occur in a particular processing node (node ​​300 or node 400) that may indicate an error condition has occurred in the processing node. Such events may be, for example, an overheating event detected by a temperature sensor 330, an exception event logged by a tile 4 while executing its application code, or an uncorrectable error in a DRAM module 350, 430. Some error events may be fatal to the application, such as when a link between chips goes down, while others may not be fatal but still require reporting, such as when a number of correctable errors on a link exceed a threshold. Each event that is reported and aggregated is assigned to an event class. An event class refers to the type of event. For example, an uncorrectable memory error in a DRAM module 350, 430 may be one event class, and a security exception may be another event class. For a processing node 400 that includes multiple chips 410, 420, some events may occur on the main processing chip 410 and some events may occur on one of the fabric chips 420.

[0039] A system of processing nodes is provided, where each processing node is configured to execute a portion of application code for the entire system. Such a system is sometimes referred to as a partition, since a single isolated application executes across multiple devices in the system. Each processing node in the system stores at least one copy of an event vector, where each element of the event vector corresponds to a different event class. Each element indicates whether an event of the corresponding class has occurred anywhere in the system of processing nodes. This event vector, aggregated across the entire system, is referred to herein as a global event vector (GEV). In addition to the GEV, each node also stores its own associated local event vector (referred to herein as a "CEV"). The CEV also includes multiple elements, each of which indicates whether an event of the corresponding event class has occurred on the processing node that stores its local event vector.

[0040] Referring to FIG. 6, FIG. 6 shows examples of GEV and CEV. Both of these event vectors contain multiple elements, providing a bitmask for each event class indicating whether an event of that event class has occurred. Each element of the CEV indicates whether an event of the respective class has occurred on the processing node storing the CEV. Each element of the GEV indicates whether an event of the respective class has occurred on any processing node of the system. The CEV contains more elements than the GEV, reflecting that only a subset of the event classes whose events are captured in the CEV are globally aggregated. In an embodiment, the CEV contains 15 elements, one for each of the 15 event classes. The GEV contains 10 elements, one for each of the 10 event classes, reflecting that events of the first 10 event classes captured in the CEV are globally aggregated. The numbering of the vector elements in FIG. 6 reflects the event class ID associated with each element.

[0041] References herein to the "event state" refer to the state of the CEV and GEV, and references herein to the "global event state" refer to the state of the GEV.

[0042] 5A, which illustrates a system 500 of processing nodes 510. Each of the processing nodes 510 may be processing node 300 or processing node 400. In FIG. 5A, an example is shown of how each of the nodes 510 are connected for aggregation of global event state. These connections are the connections used for transmission of the global event state and thus the connections used to aggregate the global event state across the system 500. It will be understood that in an embodiment, there may be additional connections between the nodes 510 other than those shown, and these additional connections are used for the exchange of data plane traffic or other types of control plane traffic (e.g., synchronization messages).

[0043] The connections between nodes 510 used for the exchange of global event state form a spanning tree. A spanning tree allows communication between any nodes 510 in the system 500, but without introducing cycles (or rings) in the graph formed by the interconnected nodes 510. The use of a spanning tree allows messages to be sent between any two nodes 510 in a small number of hops, while maintaining a simple topology with fewer connections than would otherwise be the case.

[0044] Referring to Figure 5B, Figure 5B shows an example where a system 500 of nodes 510 is connected in an alternative topology for the exchange of global event state. In this case, the nodes 510 are connected together in a ring and the global event state is exchanged between the nodes 510 around the ring. The ring topology has the advantage that redundancy is built into the system 500. If a link used to send GEV updates goes down, updates can still be sent between any two nodes 510 in the system 500 since there is an alternative path for sending such updates. Thus, aggregation of global event state is possible even in a ring topology.

[0045] In the example of FIGS. 5A and 5B, system 500 is shown to include only 16 processing nodes 510, although in reality system 500 would likely include thousands of processing nodes 510.

[0046] 5C, which illustrates how global event state can be aggregated across different nodes 510 of the system 500. A portion 520 of the system 500 is shown, which includes exemplary processing nodes 510a-d.

[0047] As shown, each of the system's nodes 510a-d maintains a copy of the GEV and a copy of its own CEV. In reality, each node includes multiple registers, some of which are called Global Event Vector Registers (GEVVRs) and hold a copy of the GEV, and some of which are called Colossus Event Vector Registers (CEVVRs) and hold a copy of the CEV. However, logically, each of the nodes 510a-d can be thought of as having a single copy of the node's CEV and a single copy of the GEV.

[0048] Each time an event of an event class recorded in the CEV occurs at one of the nodes 510a-d, the processing circuitry of the node 510a-d updates the CEV held at the node 510. Specifically, the processing circuitry of that node 510a-d updates the element of the CEV that corresponds to the event class of the detected event. If the event class is a class that is propagated throughout the nodes 510, in addition to updating the CEV at one of the nodes 510a-d, the circuitry of the node 510 also updates the corresponding element of the GEV held at the node 510. The interface circuitry of the processing node 510 then causes update messages (sometimes referred to as GCM messages) to be sent to one or more other processing nodes 510 that include an indication regarding the update of the GEV. Each of the update messages may include an indication of the entire updated GEV or an element of the GEV to be updated. An update message is sent from one of the nodes 510a-510d to a node 510 that is a neighbour of one of the nodes 510a-510d sending the update (i.e. a node connected to one of the nodes 510a-510d sending the update). A node 510 receiving an update message updates its copy of the GEV according to the update message and then sends one or more further update messages to its own neighbours 510 (nodes 510 other than the neighbouring node 510 that provided the update).

[0049] In the example of FIG. 5C, an event occurs at processing node 510a. The event is of a class of events that processing nodes 510a-510d are configured to globally aggregate. As shown, in this example, the event occurs in a fourth event class where the event is recorded in the CEV and the GEV. However, the event class could be a different event class. In response to detecting the event, processing circuitry at node 510a causes a fourth element (associated with the fourth event class) of the CEV of node 510a to be set to indicate that an event of the fourth event class has occurred. Similarly, in response to detecting the event, processing circuitry at node 510a causes a fourth element (associated with the fourth event class) of the GEV of node 510a to be set to indicate that an event of the fourth event class has occurred.

[0050] In response to updating the copy of the GEV at node 510a, circuitry at node 510a causes the issuance of a data packet to node 510b that includes an indication of the update to the GEV. Node 510b receives the data packet and the circuitry at node 510b causes the copy of the GEV 510b held at node 510b to be updated in accordance with the indication.

[0051] In response to updating the copy of the GEV at node 510b, circuitry at node 510b causes the issuance of a data packet including an indication of the update to the GEV to nodes 510c, 510d, which in turn cause each of their copies of the GEV held at their respective nodes 510 to be updated according to the indication of the update provided in the data packet.

[0052] This section describes how event states are reported by components of a processing node 510 and propagated to other components of that node 510 .

[0053] When an event is detected by a particular component of a processing node 510, that component forwards an event report to the regulator component of the node 510, which updates its copy of the CEV (known as the master copy) before broadcasting the update to other components that maintain other copies of the CEV. The exchange of these messages between nodes may occur via a control bus architecture.

[0054] Referring to Figure 7, Figure 7 shows an example of a control bus 700. The control bus 700 is implemented on a single chip (i.e., device 300 or device 410). If the control bus 700 is implemented on device 410, the device 400 may have components connected to the control bus 700 to transfer packets between the control bus 700 and components on other devices 420 belonging to the processing node 400, allowing event states to be exchanged between the regulator node 740 and components on these devices 420. The control bus 700 shown in Figure 7 is implemented on a single chip 300 / 410.

[0055] The control bus 700 is described in more detail in our previous U.S. patent application Ser. No. 17 / 328,143, which is incorporated herein by reference. The control bus 700 is a ring-shaped data path for carrying single word control traffic. The control bus 700 is a pipelined data bus over which data packets move from one stage of the pipeline to the next at a rate determined by clock pulses applied to the control bus 700. The ring includes multiple nodes 710, and traffic passes from one node 710 to the next node 710 in a flowing direction around the ring. The output port 750 of each node is connected to the input port 760 of the next node 710 in the ring.

[0056] Some nodes, referred to herein as target nodes, have circuitry for receiving requests to read or write configuration settings associated with that node. An example target node 720 is shown as being connected to the bus 700. Such a target node 720 stores configuration settings for its associated components 770, which can be read or written by read / write requests received from the control bus 700. The configuration settings (or configuration state) include copies of the CEV and GEV.

[0057] Some nodes, referred to herein as initiator nodes, have request circuitry for issuing read or write requests to the bus 700. An example initiator node 730 is shown connected to the bus 700. Under the control of a connection module, such an initiator node 730 issues read or write requests to configuration settings associated with target nodes 720 connected to the control bus 700. The initiator 730 may be controlled by software or may be controlled by a fixed function hardware engine.

[0058] As shown, each of the initiator nodes 730 and each of the target nodes 720 has an associated component (shown as a connecting block) 770. The initiator nodes 730 and the target nodes 720 may be implemented within their associated components 770. Each of the initiators 730 operates under the control of its associated component 770 and dispatches read or write requests to the control bus 700. Each of the target nodes 730 receives and stores configuration state for controlling its associated component 770. This configuration state is output from the target nodes 720 to the associated component.

[0059] Each of the initiator nodes 730 can issue requests to and receive completions from the target node 720. Each request is a command to read from storage associated with the target node 720 (e.g., a connected addressable entity or an automatically generated register) or a request to write to such storage associated with the target node 720. In response to receiving such a request, the target node 720 responds by issuing a completion. The completions provide a status update indicating whether the read or write request was successful. In the case of read completions, each completion includes the data read from the target node 720 and returned to the initiator node 730 that issued the read request. In the case of write completions, the write completions include an indication of whether the write request was successful.

[0060] A further type of transaction supported by the control bus 700 is an event report. An event report is a packet dispatched and posted to the control bus 700 by an initiator node 730 or a target node 720 that contains information related to a detected event. When a hardware unit 770 detects an event, the hardware unit 770 controls its associated initiator node 730 or associated target node 720 to dispatch an event report to the control bus 700.

[0061] Referring to Figure 8, Figure 8 shows an event report 800. The event report 800 includes an event class 820 and an event identifier 830, which are provided by the hardware unit 770 that detects the event. The hardware unit 770 provides both of these fields 820, 830 to its associated node 720 / 730. The event report 800 includes an event source field 810, which is prefixed with the node 720 / 730 that received the event's event class 820 and event identifier 830. The event source field 810 uniquely identifies the node 720 / 730 among all nodes 720 / 730 that belong to the processing node 510.

[0062] The event class field 820 identifies the event class to which the detected event falls. In an embodiment, the event class identifies one of the 15 event classes discussed above that may be captured in CEV. The event identifier field 830 identifies a unique event that occurred at the component 770 identified by the source field 810 and within the event class 820. The event identifier field 830 also includes further details about the event.

[0063] When an event report is dispatched onto the bus 700, it is delivered to a Cbus regulator 740, which may also be referred to as a control node 740 or a control component 740. The event report circulates on the bus 700 until it reaches the regulator 740, where it is consumed and the event is logged.

[0064] Referring to Figure 10, Figure 10 illustrates an exemplary embodiment of a regulator node 740. Figure 10 illustrates components of the regulator 740 for logging event states in response to receiving an event report. The regulator node 740 includes processing circuitry 1010 for performing operations described herein as being performed by the regulator node 740. The regulator node 740 includes a number of registers 1020 for storing information received in the event report. These registers 1020 are referred to as event data registers (EDRs) 1020. The regulator node 740 includes an EDR 1020 for each of a number of event classes.

[0065] When an event report is received at regulator 740, regulator 740 causes information from the event report for the event class identified in the event report to be stored in EDR 1020. The information stored in EDR 1020 includes the event identifier 830 from the event report and the event source identifier 810 from the event report. Each EDR 1020 also includes an event flag, which is set to indicate that an event of a class corresponding to the class of the EDR 1020 has occurred. In response to the event report, if the event flag of the EDR 1020 corresponding to the class of the event report is not already set, node 740 causes the event flag to be updated to indicate that an event of that event class has occurred.

[0066] The regulator 740 also includes a register 1030 for storing a copy of the CEV for the processing node 510. This register 1030 is referred to as the master CEVVR 1030 because it is the first CEVVR of the node 510 that is updated in response to the detection of an event. When an event report is received at the regulator 740, if the bit in the CEV corresponding to the event class identified in the event report is not already set, the regulator 740 causes that bit in the CEV to be set to indicate that an event of that class has occurred. The CEVs exhibit idempotence such that if a bit in the CEV for a particular event class is already set, then the bit in the CEV is not changed when a subsequent event report of the same event class is received. The regulator node 740 controls the access permissions of the CEVs such that bits in the CEV are not unset except in response to a reset event.

[0067] The regulator node 740 includes a register 1010 for storing a copy of the GEV. The register 1010 is called the master GEVVR 1010 because it is updated before the GEVVR of a target node 720 of the same processing node 510 is updated. The master GEVVR 1010 may be updated in response to i) detection of an event in its own node 510 or ii) a packet received at the node 510 indicating an update to the GEV (the update is due to an event occurring at another node 510).

[0068] Regulator node 740 includes a mask register 1050, referred to herein as GEVVRMASK register 1050. GEVVRMASK register 1050 provides, for each event class that has a corresponding element in the GEV, a set of bits that indicates whether global aggregation is enabled for that event class. To ensure consistency in how event classes are aggregated across system 500, the value held in GEVVRMASK register 1050 is set to be the same in each of nodes 510 of system 500.

[0069] When an event report is received at regulator 740, if the bit in GEVVRMASK register 1050 corresponding to the event class identified in the event report is set to indicate that global aggregation is enabled, and if the bit set in GEVVR 1010 is not already set to indicate that an event has occurred in that class, regulator 740 causes the bit in the GEV of that class to be set to indicate that an event has occurred in that class. GEVs are idempotent such that if a bit in a GEV for a particular event class is already set, subsequent event reports of the same event class received at regulator 740 do not change that bit in the GEV. Regulator node 740 controls the access permissions of the GEVs such that the bit in the GEV indicating the occurrence of an event is not unset except in response to a reset event.

[0070] When one or more of the event flags in the EDR 1020 change state to indicate that an event has occurred, the regulator 740 broadcasts an update message to all target nodes 720 of the processing node 510. The update message is a message sent over the control bus 700 to update the CEVVR of each of the target nodes 720 of the processing node 510. The update message updates each of the copies of the CEV in the node 510, thereby updating the elements of the CEV associated with the class for which the event flag was set to reflect that an event has occurred in that class.

[0071] In addition, when one or more bits in the master GEVVR 1010 change state, the regulator 740 broadcasts an update message (which may be the same update message used to update the copy of the CEV) to all target nodes 720 of the processing node 510. The update message is a message sent over the control bus 700 to update the GEVVR of each of the target nodes 720 of the processing node 510. The update message updates each of the target nodes' 720 copies of the GEV, thereby updating the elements of the GEV associated with the event class whose event flag is set to reflect that an event has occurred in that class.

[0072] Referring to Figure 9, Figure 9 illustrates an exemplary target node 720. Each target node 720 includes at least one node management register 940. The node management registers 940 may be written via a dedicated node management broadcast write request, which writes a common set of states to all node management registers 940 of the target nodes 720. Such broadcast writes may be performed by the regulator node 740. An example of such a broadcast write is an update message sent to update the copies of the CEV and GEV held at each target node 720.

[0073] In addition to at least one target node management register 910, the target node 720 is associated with additional storage 920 that is part of the component 770 with which the target node 720 is associated. The additional storage 920 may include automatically generated registers and / or storage that can be accessed via an addressable window. This additional storage 920 may include control registers on a tile 4 of the processing unit 2 that are addressable on a further bus. This additional storage 920 allows access to off-chip storage, such as host storage accessible over a PCI link. In this case, the additional storage 920 is the memory of the host dispatcher that provides data written to the memory to the host storage.

[0074] The target node has input and output ports. The input ports receive read and write requests from the control bus 700, which are buffered in a request buffer 960 and then passed to the processing logic 930. The output ports send completions from a completion buffer 970 to the control bus 700. The processing logic 930 may perform functions implemented in hardware or software. The processing logic 930 may include one or more of an ASIC, an FPGA, or at least one processor configured to execute computer readable instructions stored in at least one memory of the target node 720.

[0075] 9, the node management registers 940 include a CEVVR 980 and a GEVVR 990. The CEV held in the CEVVR 980 is the same for all target nodes 720 in a processing node. Similarly, the GEV held in the GEVVR 990 is the same for all target nodes 720 in all processing nodes in the system. The processing logic 930 causes all of the CEV and GEV elements to be output from the node management registers 940 to the connectivity component 770. This enables the connectivity component 770 to take action or perform its operation in response to the error condition indicated by the event condition.

[0076] When regulator 740 sends an update message for a CEV update, the update message is received at target node 720. Processing logic 930 causes the CEVVR 980 of that target node 720 to be updated accordingly. The CEV in CEVVR 980 is updated to match the CEV held in master CEVVR 1030. Processing logic 930 causes the propagation of this updated CEV state to connection component 770 so that the component can react to the updated error condition. The same update is performed at each of target nodes 720 in processing node 510.

[0077] Similarly, when regulator 740 sends an update message for a GEV update (which may be the same as or different from the one used to update the GEV), the update message is received at target node 720. Processing logic 930 causes the GEVVR 990 of that target node 720 to be updated according to the update. The GEV in the GEVVR 990 is updated to match the GEV held in the master GEVVR 1010. Processing logic 930 causes the propagation of this updated GEV state to connection component 770 so that the component can react to the updated error condition. The same update is performed at each of target nodes 720 in processing node 510.

[0078] Among the components 770 that receive updates for their associated GEVs (i.e., GEVs held in the target GEVVR 990) are the EPCs (e.g., EPCs 360, 460, 470) of the processing nodes 510. When certain EPCs receive updates for their GEVs from the regulator node 740 of the processing node 510 to which they belong, these EPCs are configured to provide an indication that the GEV state has been updated in the update message. The indication of the GEV state is propagated between the processing nodes 510 and used to update the GEVs held in those nodes 510. The certain EPCs that are configured to propagate the event state are those that are used to communicate over the connections shown in Figures 5A and 5B. It is these EPCs that provide the GEV update message shown in Figure 5C.

[0079] Referring to FIG. 11, FIG. 11 illustrates an example of an EPC 1100 according to an embodiment. The EPC 1100 may be one of the EPCs 360, 460, 470. The EPC 1100 includes a target node 720 and an initiator node 730. The target node 720 enables the EPC 1100 to receive updated global event states received from the regulator node 740. In addition to the interfaces illustrated in FIG. 11, the EPC 1100 also includes an interface for exchanging data plane traffic with the processing unit 2. FIG. 11 illustrates a component in the EPC that is used for exchanging control plane traffic (including event state updates). This component is referred to herein as a global communication proxy (GCP) 1200. The GCP 1200 may receive Ethernet frames via a receive interface (Rx), which include update messages for updating its copy of the GEV. The GCP 1200 can dispatch update messages itself over the transmit interface (Tx), these update messages being messages for updating GEVs held in other processing nodes 510 of the system. The GCP 1200 causes the update messages to be provided to the frame builder 1120, which then transmits them over the transmit interface in Ethernet frames constructed according to the connection state held in the connection state register 1130.

[0080] Referring to FIG. 12, FIG. 12 illustrates a GCP 1200 and components within the GCP 1200 that are used to propagate event states. The GCP 1200 includes a processing circuit 1220 for performing the operations described herein as being performed by the GCP 1200. In addition to the GCP 1200, FIG. 12 also illustrates a target node 720 and an initiator node 730 for enabling the GCP 1200 to exchange event states with a regulator node 740. Here, we describe how the GCP 1200 enables global aggregation of event states. It will be understood that while a processing node 510 according to an embodiment includes multiple GCPs 1200, only a subset (one or more) of these GCPs 1200 are typically active for purposes of propagating a global event state.

[0081] As will be described, in response to the broadcast of a GEV update message from regulator 740, the GEVVR register 990 is updated. This updated GEV state is output from target 720 to GCP 1200. GCP 1200 includes a register 1210, referred to herein as GEVEN register 1210. Register 1210 includes an indication of whether global event aggregation is enabled in GCP 1200. If register 1210 indicates that global event aggregation is enabled for execution in GCP 1200, GCP 1200 responds to the update of its target GEVVR 990 by dispatching an update packet to another processing node 510.

[0082] Each GCP 1200 includes an additional register 1240 referred to herein as a Global Event GCP External Global Event Vector Register (GEVEXTGEVVR) 1240. The GEVEXTGEVVR 1240 stores a copy of the GEV. The circuit 1220 compares the copy of the GEV held in the GEVEXTGEVVR register 1240 with the copy of the GEV held in the GEVVR register 990 to determine whether to dispatch an update to another node 510 or to a regulator 740 of the same node. If the circuit 1220 of the GCP 1200 determines that a bit that is not set in the GEVEXTGEVVR 1240 is set in the target GEVVR 990, the circuit 1220 of the GCP 1200 updates the GEVEXTGEVVR register 1240 to match the copy of the GEV held in the GEVVR 990. The GCP 1200 also causes the dispatch of an update message to a GCP on another node 510 with which the GCP 1200 is configured to communicate. The update message provides an indication that the GEV has been updated, and causes the receiving GCP to update its own GEVEXTGEVVR to match the GEVEXTGEVVR 1240 of the GCP 1200 that dispatched the update message.

[0083] To provide reliable propagation, the GCP 1200 transmits the update message three times. After transmitting the GEV update message in a frame, the GCP 1200 waits for 1000 system clock cycles of its chip 510 to elapse before retransmitting the same frame containing the update message. The GCP 1200 then waits another 5000 system clock cycles before transmitting the frame a third time. If the target GEVVR 990 associated with the GCP 1200 is changed again before these copies have been transmitted, the retransmission sequence is aborted and a new update message is transmitted containing the new state held in the GEVVR 990.

[0084] A GCP 1200 may receive an update message from a GCP on another node 510 that causes it to update its copy of the GEV in its GEVEXTGEVVR 1240. As a result of such an update, the copy of the GEV held in the GEVEXTGEVVR 1240 may not match the copy of the GEV held in the target GEVVR 990 associated with the GCP 1200. In response to determining that a bit is set in the GEVEXTGEVVR 1240 that is not set in the associated GEVVR 990, the GCP 1200 issues a write request to the Cbus regulator 740 on its own node 510 to update the GEVUPDATER register 1060 to match the copy of the GEV currently held in the GEVEXTGEVVR 1240.

[0085] In response to receiving a write to GEVUPDATER 1060, regulator node 740 updates GEVUPDATER 1060 accordingly. In response to updating GEVUPDATER 1060, regulator 740 updates its copy of the GEV held in master GEV register 1010 of processing node 510. The updating of master GEVVR 1010 is performed such that any bit set in GEVUPDATER 1060 to indicate that an event has occurred in a particular class is also set in GEVVR 1010. However, for any bit in GEVUPDATER 1060 that is set to indicate that an event has not occurred in that class, regulator 740 does not update GEVVR 1010 to indicate that no event has occurred in that class. In this way, the update policy of regulator 740 ensures that the GEVs held in GEVVR 1010 are idempotent.

[0086] In addition to updating its copy of the GEV held in register 1010, regulator 740 broadcasts an update message on control bus 700 to all target nodes 720 of processing node 510. The update message includes the updated GEV held in register 1010 of regulator 740. Referring to Figure 9, processing logic 930 of each target node 720 receives the update message and updates the GEVVR 990 held in that target node 720 with the updated GEV. A component 770 associated with a target node 720 can access the copy of the GEV held in the GEVVR 990 of that target node 720.

[0087] Some of these components 770 that receive updates to their copies of GEVs via their associated GEVVRs 990 are themselves GCPs capable of performing global event aggregation. Each such GCP responds to updates to its GEVVR 990 by issuing an update message to another node 510. The updated GEV is then propagated to the different components of that other node 510 in the same manner as described above.

[0088] Referring to FIG. 5D, FIG. 5D shows in more detail how updated global event states are propagated among nodes 510a-510d of portion 520 of system 500. When component 770a (which may itself be a GCP) encounters an error event, component 770a provides a report about this event to regulator 740a. Regulator 740a updates its master copies of node 510a's CEVs and GEVs and broadcasts updates to these vectors to each of node 510a's target nodes 720. GCP 1200a detects updates to its copy of the GEVs held at its target node 720a and in response provides a message to node 510b that includes the updated GEVs. Node 510b includes GCP 1200b that is configured to receive the updates and dispatch the update message to its regulator 740b via its bus 700. In response to receiving the update, regulator 740b updates its copy of the GEV to match the copy of the GEV held at node 510a. Regulator 740b also broadcasts the GEV update to target nodes 720 (including target nodes 720b, 720c) of processing node 510b. GCPs 1200c, d associated with target nodes 720b, c respond by dispatching update messages that include updates for GCPs 1200e, f on nodes 510c, d. GCPs 1200e, f propagate these updates to their own regulator nodes 740c, d, which in turn broadcast the updates to components on those nodes 510c, d. In this manner, updates to the GEV generated by component 770 on one node 510a in response to detecting an event are propagated to components on other nodes 510b-d of system 500.

[0089] As a result of the propagation of local event state (i.e., CEV) and global event state (i.e., GEV) to components 770, each component 770 of the system 500 is provided with access to two different event vectors. Each component 770 can access the device-wide event and error state of its own processing node 510 via its own local copy of the CEV of that processing node 510. Each component 770 can also access the system-wide event and error state via its own local copy of the system's GEV. Based on this state, the component 770 can be configured to perform an action. Such actions, performed based on the copy of the CEV or GEV that the component has access to, are referred to herein as autonomous event handling (AEH). Different components 770 of the same processing node 510 can be configured to perform different actions in response to events belonging to the same class. One or more components 770 can be configured with different policies and / or configured to perform different actions in response to events belonging to different event classes. Moreover, one or more components 770 may perform different actions in response to events associated with (and reflected in) their own processing node 510 as compared to events associated with (and reflected in) the system 500.

[0090] An example of a component that can participate in an AEH is circuitry (referred to herein as an exchange block (XB)) that enables a tile 4 of a processing unit 2 to exchange data with devices external to the chip 510 on which the processing unit 2 is implemented.

[0091] Referring to FIG. 13, FIG. 13 illustrates an example of a chip 1300 including an XB 1310 and multiple tiles 4. The XB 1310 includes circuitry configured to perform the described operations. The XB 1310 is associated with an external interface 1320 of the chip 1300 configured to transmit and receive data packets therethrough. The XB 1310 receives packets in one format ("Tlink" format) from one or more tiles 4, converts the packets to another format ("Elink" format), and then transmits the packets through the external interface 1320. The XB 1310 receives packets in the other format ("Elink" format), converts the packets to a further format ("Tlink" format), and then distributes the packets to one or more tiles 4. The XB 1310 is also configured to control transmission by the tiles 4 by providing flow control messages to pass transmission authorization between different tiles 4.

[0092] XB 1310 is configured to engage in certain autonomous error handling (AEH) in response to an indication of an error in the GEV. Referring to FIG. 14, FIG. 14 illustrates XB 1310 and its associated target node 720. XB 1310 includes processing circuitry 1400 for performing operations described herein as being performed by XB 1310. XB 1310 includes a mask register 1410 (referred to herein as AEHGEVVMASKR register 1410). AEHGEVVMASKR register 1410 stores a set of bits that identify which bits in the GEV will cause an AEH by XB 1310. If AEHGEVVMASKR register 1410 stores a bit indicating that AEH is valid for a particular event class, then XB 1310 performs an AEH in response to determining that a bit is set in the GEV to indicate that an event of that class has occurred.

[0093] When performing AEH, XB 1310 stops transmitting packets from tile 4 to interface 1320 and stops transmitting packets from interface 1320 to tile 4. If XB 1310 receives packets from tile 4, XB 1310 drops those packets. Similarly, if XB 1310 receives packets from interface 1320, XB 1310 drops those packets. XB 1310 thus blocks communication between tile 4 and external devices. As a result, data plane traffic is quickly halted.

[0094] The types of events indicated by the GEV that may cause the XB 1310 to block traffic are an error event that is fatal to the execution of the application (e.g., an unrecoverable memory error) or the downing of one of the external links of the chip 1300. In this case, IO traffic is halted prior to resetting the chip 1300.

[0095] Different components of the system may include different mask registers (such as register 1410) that indicate to each component which events the respective component is configured to take actions in response to. The mask register for a particular component may include a first set of indications that indicate which of the event classes identified in the CEV the component is configured to take actions in response to. The mask register may also include a second set of indications that indicate which of the event classes identified in the GEV the component is configured to take actions in response to. These registers are defined for each component, and each component may take actions (perform AEH) in response to different types of events.

[0096] Each event class defined in CEV (as well as GEV) has an associated event handler configured to implement a predetermined response to the occurrence of that event. An event handler is a software module that executes on an event handler device. The actions taken by the processing device are in addition to any AEHs performed by the individual components 770 of the processing node 510. For example, in response to the occurrence of an event that represents a fatal application error, the processing entity may cause the application to be reset from a checkpoint.

[0097] Different event classes have different event entities. For some event classes, the processing entity may be software running on the MCPU 310. For other event classes, the processing entity may be a host process running on the host 320 / 440. Each of the system-wide event handlers is associated with one or more processing nodes 510 and is configured to take action on the associated processing node or nodes 510 in response to event notifications received from the associated processing node or nodes 510.

[0098] Referring to Figure 15, Figure 15 illustrates how event notification to an event handler 1510 is performed. The target node 720 receives updates of the CEV and GEV broadcast by the regulator node 720 after the occurrence of an event. Some of the components 770 associated with the target node 720 include associated event notification hardware 1500. Components include such event notification hardware 1500, including, for example, the PCIe complex 370 / 450 or the MCPU 310. The handler software 1510 can run on the same device (e.g., MCPU 310) that includes the event notification hardware 1500, or on a separate device (e.g., the host device).

[0099] The event notification hardware 1500 includes a CEV notification mask register 1520, which indicates that for each of the event classes for which an event has been logged to CEV, an event notification is provided to the handler software 1510. In response to receiving from the target node 720 that an event has occurred in a particular event class, the processing circuitry of the event notification hardware 1500 checks the corresponding indication in the register 1520 for that event class. If the register 1520 indicates that an event of that class should be reported to the handler software 1510, the processing circuitry of the event notification hardware 1500 issues an event notification to the handler software 1510. The event notification may take the form of an interrupt to the handler software 1510.

[0100] The event notification hardware 1500 also includes a GEV notification mask register 1530, which indicates that for each of the event classes for which an event has been logged to the GEV, an event notification is provided to the handler software 1510. In response to receiving from the target node 720 that an event has occurred in a particular event class, the processing circuitry of the event notification hardware 1500 checks the corresponding indication in the register 1530 for that event class. If the register 1530 indicates that an event of that class should be reported to the handler software 1510, the processing circuitry of the event notification hardware 1500 issues an event notification to the handler software 1510. The event notification may take the form of an interrupt to the handler software 1510.

[0101] For each event class reported in the CEV, an event handler is defined to handle events of that class. Similarly, for each event class reported in the GEV, an event handler is defined to handle events of that class. In an embodiment, each processing node 510 is associated with four event handlers: two host processes running on the host device 320 / 440, one process running on the MCPU 310, and one process running on an additional hardware unit for enabling cryptographic functions (called the ICU). Thus, the event handlers can be implemented on separate devices (e.g., the MCPU and the host) or can take the form of separate processes running on the same device (e.g., separate host processes running on a single host).

[0102] In response to receiving an event notification, the handler software 1510 is configured to take appropriate actions. These actions include issuing a read request to read the event data register 1020 of the associated regulator node 740. The event handler 1510 reads the EDR register 1020 of the associated event class. The event handler reads the event flag and event identifier from that EDR register 1020. A read can be issued by the event handler using its associated initiator 730, and details of the event (e.g., the event flag and event identifier) ​​are returned to the event handler 1510.

[0103] When a global event is notified to the handler software based on the GEV, the event flag indicates to the event handler 1510 whether the event is associated with the processing node 510 with which the event handler 1510 is associated.

[0104] 16, which illustrates an exchange of messages that occurs after an event notification is issued to an event handler 1510 based on the GEV maintained in each of the processing nodes 510. Although each event handler 1510 is shown as separate from its associated processing node 510, each event handler 1510 can be considered to be part of its associated processing node 510. In addition, while each event handler 1510 is shown as having a single associated processing node 510, in some embodiments each event handler 1510 is associated with multiple such processing nodes 510.

[0105] As a first step, in response to receiving an event notification, each event handler 1510 issues a read request to read the event data register 1020 associated with the event class identified by the event notification. The event data register 1020 read by each event handler 1510 belongs to the processing node 510 with which the event handler 1510 is associated.

[0106] In response, as a second step, each event handler 1510 receives the state held in the EDR 1020 identified by the sent read request. This state includes the event flag and the event identifier held in that EDR 1020. If the event reflected in the GEV occurred in a processing node 510 with which the event handler 1510 is associated, then the event flag is set to a state indicating that the event occurred in that node 510. In the example shown in Figure 16, the event reflected in the GEV occurred in processing node 2, and therefore event handler 2 receives the event flag set to indicate that the event occurred in node 2, and also receives details of the event (e.g. in the form of an event identifier).

[0107] As a third step, event handler 2 propagates a message related to the event to the other event handlers 1 and 3. The message can be either details of the event in the EDR state received by event handler 2 or a message to make event handlers 1 and 3 reset their associated processing nodes 510. In the example shown, each of the event handlers 1510 decides to reset its associated processing node 510. This decision is made based on the EDR state obtained from processing node 2. This decision can be made by event handler 2, which then propagates the command to the other event handlers 1 and 3, or each event handler 1510 can make it individually based on the EDR state obtained from and distributed by event handler 2.

[0108] In this example, as a fourth step, each of the event handlers 1510 is shown issuing a command to reset its associated processing node 510. This reset is accomplished by asserting a signal on a reset pin 1600 on the processing node 510. Circuitry on each processing node 510 causes the reset to occur.

[0109] A reset, sometimes referred to as a software reset, resets the state of several components that are part of each processing node 510. A reset has the effect of clearing the memory of the tiles 4 and deleting application data. A reset also has the effect of clearing the state of the encryption hardware used to encrypt communications in and out of the device 510. A reset resets the state of logical links and connections using those links. However, a reset does not take down any links, nor does it change the state of all of the components of the physical links. Therefore, there is no need to reprogram the EPC 470 after a reset. A reset does not clear the application state of the MCPU 310.

[0110] The effect of a reset is to clear the CEVs and GEVs held in components of the system so that they no longer indicate that an event has occurred. Clearing the CEVs and GEVs is performed by a regulator 720 in that processing node 510 that has write permission to the CEVVR and GEVVR.

[0111] Once a particular processing node 510 has reset its state and the tile memory has been cleared, that processing node 510 can restart its application from the application checkpoint. Each processing device 300 / 410 has access to external memory where checkpoint data is periodically stored. Following a read request, a tile 4 of a processing unit can load the checkpoint data from the external memory, thereby restarting the application from a previous point.

[0112] Each of the event handlers 1510 is configured to cause its associated processing node 510 to be reset immediately, without requiring synchronization of the reset across the system. One problem that may be encountered when performing a reset without synchronization is that while one of the processing nodes 510 performs its reset and restarts from a previous checkpoint, further processing nodes 510 that have not yet been reset may continue to run the previous version of the application. In this case, if the processing nodes 510 that have not yet been reset send data packets containing application data corresponding to the previous version, these data packets may be received by the processing unit 2 that has reset from the checkpoint. In this case, erroneous results may occur in the processing by the processing unit 2 that has restarted from the checkpoint, since the processing unit 2 has received data corresponding to the previous application version.

[0113] According to an embodiment, frames transmitted between different processing nodes 510 include a numerical value (called a generation number) that is updated in response to each reset event. Thus, the numerical value indicates the number of times the application has been reset. When data frames are transmitted by one processing node 510 to another processing node 510, the transmitting node 510 includes its own copy of the generation number in each of those frames. The numerical value is checked at the receiving node 510 against the receiving node's own copy of the generation number. If the generation number in the frame does not match the generation number held by the receiving node 510, the receiving node 510 discards the frame. In this way, when the receiving node 510 has already been reset and restored from a checkpoint, the receiving node is protected from frames related to the generation of the application before the reset that may be dispatched by other nodes in the system that have not yet been reset.

[0114] Referring to FIG. 17, FIG. 17 illustrates an example in which two processing nodes 1700a, 1700b can communicate. Each of the nodes 1700a, 1700b can be one of the nodes 510 described above. Typically, a system includes more than two processing nodes, but for simplicity, only two such nodes are shown in FIG. 17. Both processing nodes 1700a, 1700b are configured to be reset and restarted from a checkpoint. The reset of both processing nodes 1700a, 1700b is part of a system-wide reset and restoration from a checkpoint. This reset can be performed in the manner described above with respect to FIG. 16, with the global reset of the system, and thus each node 510 individually, being performed in response to an error event occurring in one of the nodes 510. However, the global reset of nodes 1700a, 1700b may not be synchronized, meaning that one of nodes 1700a, 1700b may proceed to reset and restore from a checkpoint before the other of nodes 1700a, 1700b has gone quiescent. In this case, the node 1700a, 1700b that has not yet gone quiescent may send a data frame to the one of nodes 1700a, 1700b that has restored from a checkpoint. The copy of the generation number prevents interruption of application processing on the restored node.

[0115] Referring to Figures 18a and 18b, Figures 18a and 18b show a sequence of events performed by the system of processing nodes 1700a, 1700b shown in Figure 17. The overall method is shown divided into a first part 1800a (shown in Figure 18a) and a second part (shown in Figure 18b). At the start of the sequence of events, both processing nodes 1700a, 1700b store identical copies of a generation number. The generation number reflects the number of times a global reset of the system (including a reset across both nodes 1700a, 1700b) has been performed. Because the generation numbers are the same, a frame issued by one of the nodes contains the same generation number as that stored in the other of the nodes 1700a, 1700b. Thus, the nodes 1700a, 1700b can communicate because it is accepted by the other of the nodes 1700a, 1700b.

[0116] In S1810, in response to detecting the error event, the reset circuitry 1600 of the first processing node 1700a causes the first processing node 1700a to reset. As part of this reset, state is cleared from various components of the node 1700a. The memory of tile 4 is cleared by the circuitry of the hardware modules of the node 1700a, which writes zeros to that memory. As part of the reset, a copy of the generation number held in storage of the node 1700a is updated. As described, the generation number may be held in storage of the interface 1710. The generation number is updated by incrementing to a new value.

[0117] In S1820, following a reset of processing node 1700a, the portions of the application for which processing node 1700a is responsible are restored from the checkpoint. To accomplish this, a boot loader program is provided from a hardware module of node 1700a to the different tiles 4. Tile 4 executes the boot loader program to issue read requests to checkpoint memory 1720. Checkpoint memory 1720 is shown as a single memory unit, but may include multiple memory units for different nodes 1700a, 1700b. In response to the read requests issued by tiles 4 of node 1700a, the application instructions and application data of the checkpoint are returned from memory 1720 to those tiles 4. The application data returned to node 1700a represents the state of the portions of the application running on processing unit 2 of that node 1700a that were written out of that processing unit 2 at the last checkpoint.

[0118] At S1830, tile 4 of processing unit 2 of node 1700a continues executing application instructions from the checkpoint. At the same time, the second processing node 1700b has not yet been reset, and processing unit 2 of that node 1700b is still processing that portion of the application that corresponds to the previous generation. Thus, the second processing node 1700b stores the previous generation number, which was also held in the storage of the first processing node 1700a prior to the reset at S1810.

[0119] At S1840, tile 4 of processing unit 2 of node 1700b is configured to cause issuance of one or more data frames to first processing node 1700a, each of which includes a copy of a previous generation number held in storage at node 1700b. Tile 4 causes the issuance of the data frames by issuing a data packet to the EPC of each processing node 1700b, which causes the data packet to be encapsulated in a data frame and transmitted to node 1700a.

[0120] At S1850, first processing node 1700a receives the data frames sent at S1840. Circuitry in first processing node 1700a checks the generation number in each data frame and compares it to a copy of the generation number held in storage at processing node 1700a. Because these two generation numbers do not match, circuitry in processing node 1700a discards the frame. As will be described, the checking and discarding of frames may be performed by interface 1710 of first processing node 1700a.

[0121] At S1860, the second processing node 1700b is reset. This step performed at S1860 is the same as S1810 described above, but is performed by the second processing node 1700b rather than the first processing node 1700a.

[0122] In S1870, following a reset of processing node 1700b, the portions of the application for which processing node 1700b is responsible are restored from the checkpoint. To accomplish this, a boot loader program is provided from a hardware module of node 1700b to the different tiles 4. The tiles 4 execute the boot loader program to issue read requests to checkpoint memory 1720. In response to the read requests issued by the tiles 4 of node 1700b, the checkpointed application instructions and application data are returned from memory 1720 to those tiles 4. The application data returned to node 1700b represents the state of the portions of the application running on processing unit 2 of that node 1700b that were written out of that processing unit 2 at the last checkpoint.

[0123] In S1880, processing unit 2 of the second processing node 1700b continues executing that portion of the application from the checkpoint.

[0124] At S1890, tile 4 of processing unit 2 of node 1700b is configured to cause issuance of one or more additional data frames to the first processing node 1700a. Tile 4 causes these additional data frames to be issued by issuing data packets to the EPC of each processing node 1700b, which causes the data packets to be encapsulated in data frames and transmitted to node 1700a. Unlike the frames transmitted at S1840, each of these additional one or more data frames includes a copy of an updated version of the generation number, which is now held in the storage of both nodes 1700a, 1700b.

[0125] At S1895, first processing node 1700a receives the data frames sent at S1890. Circuitry in first processing node 1700a checks the generation number in each data frame and compares it to a copy of the generation number held in storage at processing node 1700a. These two generation numbers match, so the circuitry in processing node 1700a accepts the frames. Data from these frames is written to the memory of tile 4.

[0126] The generation number can be inserted at different points in the data frames transmitted between node 1700a and node 1700b. The generation number can be inserted in the packet header (e.g., the Elink packet header) dispatched by tile 4. Alternatively, the generation number can be inserted in the payload of the packet.

[0127] Referring to FIG. 19, FIG. 19 illustrates an exemplary embodiment in which a generation number is included in the MAC address of the frame header. In an embodiment, MCPU 310 is responsible for providing the MAC address for use in transmitting data over the connection. As part of the MAC address, MCPU 310 provides the generation number. The MAC addresses for the connection (including the MAC address of the interface 1710 and the MAC addresses of the other interfaces 1710) are provided by MCPU 310 to and stored in connection status register 1915 and control and status register (CSR) 1916. Specifically, register 1915 stores the MAC of the destination interface 1710 to which the frame is to be dispatched, and CSR 1916 stores the MAC of the interface 1710 from which the frame is to be sent. In each of these MAC addresses, one or more bits are reserved for inclusion in the data frame as a generation number.

[0128] When tile 4 has data to send, it dispatches this data to interface 1710 in the form of one or more data packets (e.g., Elink packets). Frame builder 1120 encapsulates the data packets into data frames by adding a frame header and a frame tail. In this way, data packets issued by processing unit 2 can be tunneled over Ethernet. Frame builder 1120 inserts the MAC held in connection state register 1915 as the destination MAC of each frame. Frame builder 1120 inserts the MAC held in connection state register 1915 as the source MAC of each frame.

[0129] On the receiving side, a frame check buffer 1908 is provided which receives data frames from other interfaces 1710 belonging to other nodes. The circuitry of interface 1710 shown in Figure 19 checks the destination MAC in the received frames against the MACs held in CSR 1916. If there is a mismatch between these MACs, the circuitry causes the frames to be dropped. If there is a match between the MACs, the circuitry causes the data packets encapsulated in those frames to be forwarded to the tile 4 of the processing unit 2 identified in the packet headers.

[0130] MCPU 310 includes at least one processor that executes instructions to provide the illustrated update function 1920 and address resolution function 1940. The address resolution function is configured to provide a MAC address for a connection, which is provided to interface 1710. Because the MAC addresses are local scope MAC addresses, they may be updated by software rather than being permanently fixed for a particular interface 1710. Address resolution function 1940 inserts into each MAC address the current generation number held in generation number registers 1910. The remaining portion of the MAC address may be obtained from storage 1930 or may be determined by function 1940 in another suitable manner.

[0131] MCPU 404 updates the MAC for a connection in response to a reset event. MCPU 404 includes a generation number register 1210 that stores the current generation number provided as part of each MAC address output by address resolution function 1240. MCPU 404 processing circuitry supports an update function 1220 that updates the generation number held in generation number register 1210.

[0132] When a reset of the processing node to which MCPU 310 belongs is performed (e.g., the resets occurring at S1810 and S1860), an indication of this reset is delivered to MCPU 310. Update function 1920 responds to this reset event by updating the value of the generation number held in generation number register 1910. Updating the generation number held in generation number register 1910 may include incrementing the current value of the generation number by one. For example, assume that the generation number consists of five bits and that the current value of the generation number held in register 1910 is given by the bit sequence 00001. In response to the reset event, update function 1920 updates the generation number by increasing its value by one to 00010.

[0133] Because the generation number comprises a finite number of bits, updating the generation number involves resetting the generation number to the minimum possible value if the current value of the generation number is equal to the maximum possible value. For example, assume that the generation number consists of five bits and that the current value of the generation number held in register 1210 is given by the bit sequence 11111. In response to a reset event, update function 1220 resets the generation number to the minimum possible value given by 00000. Thus, the generation number is updated by update function 1920 in a wraparound manner.

[0134] In response to the indication of the reset event, the address resolution function 1940 determines updated MAC addresses for the connections in which the device 400 participates. As previously described, in each MAC address, one or more bits are reserved to represent a generation number. The address resolution function 1240 sets these bits to the value of the new generation number stored in register 1910 by the update function. The remaining bits of each MAC address are set to the same value as they were before the reset event.

[0135] For each MAC address, the remaining bits of the MAC address can be determined in different ways. In some embodiments, these bits can be maintained in storage 1930 accessible to the address resolution function 1240 and concatenated with the bits of a new generation number to form a complete MAC address. These bits maintained in storage 1930 are not cleared in response to a reset event, but rather persist and do not change after a reset event. In this embodiment, storage 1930 stores the remaining bits (i.e., bits other than the generation number bits) for each of the MAC addresses that the address resolution function 1940 is responsible for determining. The address resolution function 1940 provides each of these MACs by combining the remaining bits with the same generation number for the respective MAC.

[0136] As mentioned above, the application state of the first processing node 1700a can be restored from a checkpoint before the application state of the other processing node (e.g., node 1700b) is similarly restored from the checkpoint. In this case, the first processing node 1700a can continue its application processing while processing node 1700b has not yet been reset and contains an application state corresponding to a previous generation of the application. This is made possible by using barrier synchronizations that separate the computation phases of the participating processing units 2 from the exchange phases of those processing units 2. The barrier synchronizations are inserted into the compiled code that runs on each processing unit 2. For each barrier synchronization, one or more processing units 2 are configured to participate.

[0137] The use of barrier synchronization can prevent a first processing unit 2 of node 1700a from executing beyond a point in its code beyond which it would need to exchange data with a processing unit 2 that has not yet been restored from a checkpoint and restarted. For example, assume that processing unit 2 of node 1700a enters a computation phase following a checkpoint, performing computations on data to produce results. Processing unit 2 of node 1700a will not exchange data with processing unit 2 of node 1700b until it reaches a barrier synchronization in which both processing units 2 of nodes 1700a and 1700b participate. At this point, processing unit 2 of node 1700a stalls until processing unit 2 of node 1700b reaches barrier synchronization as well. Thus, even if processing unit 2 of node 1700a reaches barrier synchronization while node 1700b has not yet been reset, processing unit 2 of node 1700a will still wait until node 1700b is reset and its processing unit 2 reaches the barrier before proceeding to the exchange phase following the barrier synchronization. During the exchange phase, processing unit 2 of node 1700a transmits to processing unit 2 of node 1700b application data that is scheduled for transmission during that exchange phase.

[0138] The use of barrier synchronization can prevent processing unit 2 of system 500 from sending data to a processing unit 2 of system 500 that has previously executed and has not yet been reset. The use of barrier synchronization in the context of processing unit 2 is described in more detail in our earlier U.S. patent application Ser. No. 17 / 446,681, which is incorporated herein by reference.

[0139] It has been described how in response to a reset event, copies of the CEV and GEV are cleared from a processing node (i.e., reset to not indicate the event). A bit in the CEV can also be reset by an event handler for an event class. Resetting a bit in the CEV allows another event in the event class to be detected and processed.

[0140] 10, regulator node 740 includes a number of registers 1040, referred to herein as event handler registration registers 1040. Registers 1040 include an event handler registration register for each of the different event handlers of processing node 510. For example, there is an event handler registration register for each of the two host processors (one for MCPU 310 and one for the ICU). Each of the event handler registration registers 1040 contains, for each event class captured in the CEV, an indication of whether the respective event handler associated with that register has permission to clear the state of that event class.

[0141] When the event handler 1510 receives an event notification, it may decide to clear the event state of that event so that, among other things, further occurrences of events of the same event class can be detected. To clear the event state of an event class, the event handler 1510 issues a request to the regulator node 740 via the control bus 700 to clear the EDR of that event class. The request includes an identifier of the associated event handler 1510 and an identifier of the event class. The regulator node 740 checks the permission indication for that event class in the event handler registration register 1040 for that event handler 1510. If the event handler 1510 has permission to clear the event state of that event class, the regulator node 740 updates the EDR of that event class to reset the EDR to its default state. In this state, the EDR indicates that no event has been detected. In addition, the regulator node 740 updates the master CEVVR 1030 to reset the event indication for this event class. The regulator node 740 also updates the copy of the CEV held on the target node 720 of the processing node 510 by broadcasting a write request to reset the event indication.

[0142] 20, which illustrates an example method 2000 for aggregating global event state among processing nodes 510. One or more of these steps may overlap with each other.

[0143] S2010 is executed by each node 510 of the system 500. In S2010, each processing node 510 of the system 510 stores a copy of the global event vector for the processing system 500. Initially, this vector is set to indicate that no events have occurred in the system 500. The global event vector is maintained in the master GEVVR 1010 and in the GEVVR 990 of the target node.

[0144] S2020 is executed by each node 510 of the system 500. In S2020, each processing unit 2 of the system 500 executes an instruction set of the distributed application.

[0145] S2030 is performed by each node 510 of system 500. In S2030, each of the processing nodes 510 updates each of one or more elements of at least one copy of a global event vector in response to an event of a type associated with the respective element and of a type that has occurred at one of the processing nodes of the processing system. This step is performed by regulator 740.

[0146] In S2040, a first event of a first type of event is detected on the first processing node 510. The event may be, for example, an event detected by component 770a of node 510a. In response to detecting the first event, the first processing node 510 provides an indication of an update of a first element of a global event vector to one or more other processing nodes 510. The first element is associated with the first type of event.

[0147] S2050 is executed by other nodes 510 of the system 500 (i.e., other than the first node 510) that receive the update provided by the first processing node 510 in S2040. In S2050, each of these other nodes 510, in response to the indication of the update of the first element, updates a corresponding first element in its at least one copy of the global event vector.

[0148] 21, which illustrates an example method 2100 for updating copies of event vectors maintained on a device and using these copies to perform responsive actions. The error event vectors may be CEVs or GEVs as discussed above. The device may be device 300 or device 410.

[0149] At S2110, a copy of the event vector is stored for each of the components 770 of the device 300. Initially, each copy of the event vector may be set to indicate that no events have occurred in any of the event classes captured in the event vector.

[0150] At S2120, in each of the subset of components 770, an event associated with that component 770 is detected. The events are reported in respective event reports to the control node 740. The subset of components 770 may include one or more components 770.

[0151] At S2130, the control node 740 receives a first event report from the first component 770, the first event report reporting a first type of event, which has been detected by the first component 770.

[0152] At S2140, the control node 740 broadcasts one or more writes to each of the plurality of components 770 to cause them to update their respective copies of the error event vector, which are maintained in the CEVVR 980 of the target node 720, by setting a first element associated with the first type of event to indicate that the first type of event has occurred on the processing device.

[0153] At S2150, in response to setting the first element in its copy of the error event vector, an action associated with the first type of event is performed at the second component 770.

[0154] Various functions are described above as being performed by circuitry (e.g., processing logic 930, processing circuitry 1010, processing circuitry 1220, processing circuitry 1400, circuitry 1600). This circuitry may include dedicated hardware (e.g., FPGAs or ASICs) and / or processors configured to execute instructions. Each processing node 510 of system 500 is assigned an instruction set for performing any of the operations performed by a processor configured to execute the instructions.

[0155] The above embodiments have been described by way of example only.

Claims

1. 1. A processing system configured to support an application, comprising: a plurality of processing nodes, each of the plurality of processing nodes comprising: a processing unit configured to execute a set of instructions for a portion of the application; at least one storage configured to store a copy of a numerical value indicative of the number of times a global reset of the processing system has been performed; A reset circuit; The reset circuit includes: resetting the processing node as part of a single global reset of a processing system; restoring application state for a portion of the application on the processing node from a checkpoint; configured to run each of the processing nodes includes an update circuit configured to update, as part of a reset of the processing node, a copy of a numerical value maintained in at least one storage of the processing node from a first value to a second value; a processing unit of a second processing node configured to execute a set of instructions to perform, prior to a reset of said second processing node, transmission of one or more data frames to the first processing node, each of said one or more data frames including said first value; a first processing node configured to, in response to determining a mismatch between the first value in the one or more data frames and the second value held in at least one storage of the first processing node, cause data in the one or more data frames to be discarded following a reset of the first processing node;

2. For each of the processing nodes, the processing unit is configured to participate in a plurality of barrier synchronizations as indicated in an instruction set; 2. The processing system of claim 1, wherein for each of the processing nodes, the processing unit is configured to, for each of the barrier synchronizations in which it participates, pass through the barrier synchronization in response to at least one further processing unit of the processing system joining the barrier synchronization.

3. 3. The processing system of claim 2, wherein for each of said processing nodes, each of said plurality of barrier synchronizations separates a computation phase of said processing unit from an exchange phase of said processing unit.

4. 4. The processing system of claim 1, wherein a processing unit of the first processing node is configured to stall at the first barrier synchronization following a reset of the first processing node until one of the further processing units arranged to participate in the first barrier synchronization reaches the first barrier synchronization in application processing.

5. The processing system of claim 4 , wherein one of the further processing units is a processing unit of the second processing node.

6. The first barrier synchronization acts as a barrier between a computation phase and an exchange phase; The processing unit of the first processing node comprises: performing operations during the computation phase until the first barrier synchronization is reached; subsequent to the first barrier synchronization, during the exchange phase, exchanging data with the second processing node, including data contained within the one or more data frames; 6. A processing system as claimed in claim 4 or 5, configured to execute an application instruction set which executes:

7. 7. The processing system of claim 1, wherein the circuitry of the first processing node configured to determine the mismatch is part of an interface of the first processing node for connecting with the second processing node.

8. 8. The processing system of claim 7, wherein the interface of the first processing node is configured to receive further data frames from the second processing node and to provide data packets encapsulated in the further data frames to a processing unit of the first processing node.

9. a processing unit of the second processing node configured to execute a set of instructions to perform transmission of one or more further data frames to the first processing node following a reset of the second processing node; said one or more further frames of data comprising said second value; 9. A processing system according to any one of claims 1 to 8, wherein the circuitry of the first processing node is configured to receive the one or more further data frames and, in response to determining a match between the second value in the one or more further data frames and the second value held in at least one storage of the first processing node, cause data in the one or more data frames to be accepted.

10. A processing system according to any preceding claim, wherein the circuitry of the first processing node is configured to determine the mismatch as part of a check of an address field in each of the data frames.

11. A processing system according to any one of claims 1 to 10, wherein, for each of the processing nodes, the update circuitry includes a processor configured to execute computer readable instructions to perform an update of a copy of a numerical value held in storage of at least one of the processing nodes.

12. The processing system of any one of claims 1 to 11, wherein updating the copy of the numerical value comprises incrementing the first value to obtain the second value.

13. A processing system according to any preceding claim, wherein the copy of the numerical value forms part of a connection state for a connection between the first processing node and the second processing node.

14. Each processing node: The processing unit; at least one storage for said processing node; 14. The processing system of claim 1, comprising an integrated circuit comprising:

15. 15. The processing system of claim 1, wherein a processing unit of the second processing node is configured to execute a set of instructions to output the first value in one or more data packets of the one or more data frames.

16. a processing unit of the second processing node configured to execute a set of instructions to output data for the one or more data frames to an interface circuit; A processing system according to any preceding claim, wherein the interface circuit is configured to provide the one or more frames of data with the first value to be inserted in each frame.

17. executing, at each of a plurality of processing nodes, a set of instructions for a portion of an application supported by a system including the plurality of processing nodes; storing in each of said processing nodes a copy of a numerical value indicative of the number of times a global reset of said processing system has been performed; resetting the plurality of processing nodes, including updating copies of the numerical values ​​maintained in the processing nodes from a first value to a second value; restoring application state for a portion of the application on the processing node from a checkpoint; performing one of the global resets including: transmitting one or more data frames including the first value from the second processing node to the first processing node prior to resetting the second processing node; in response to determining a mismatch between the first value in the one or more data frames and the second value maintained in the first processing node following a reset of the first processing node, causing data in the one or more data frames to be discarded; The method includes:

18. 1. A computer program comprising: a set of executable instructions for each processing node of a plurality of processing nodes forming a data processing system, each of said executable instructions, when executed by at least one processor of a respective processing node, supporting a portion of a distributed application supported by the plurality of processing nodes; storing a copy of a numerical value indicative of the number of times a global reset of the processing system has been performed; as part of a global reset of said data processing system; resetting the processing node, including updating a copy of the numerical value held in the processing node from a first value to a second value; restoring application state for the portion of the distributed application on the processing node from a checkpoint; on the processing node; the second set of executable instructions, when executed by at least one processor of a second processing node, is configured to cause the second processing node to transmit one or more data frames including the first value to the first processing node before resetting the second processing node; the first set of executable instructions, when executed by at least one processor of the first processing node, is configured to cause the first processing node to discard data in the one or more data frames in response to determining a mismatch between the first value in the one or more data frames and the second value stored by the first processing node following a reset of the first processing node. Computer program.

Images