Resetting a multi-node system

Abstract

Each node stores a number referred to herein as an occurrence number, and this number is updated whenever each node undergoes a reset and restart from a checkpoint. In most cases, since the nodes of the system participate in the same reset event, the occurrence number held by each node will be the same across the system. However, in some cases, when one node is reset before another node, the occurrence numbers between these two nodes will differ. Data frames sent between nodes each contain the occurrence number of the sending node, which is checked by the receiver, and are accepted only if the occurrence number within the frames matches the occurrence number of the receiver node. If the receiver node is reset before the sending node is reset, the occurrence number within the frames will not match the occurrence number of the receiver node, and the frames will not be accepted. Thus, the reset and restarted node is protected against packets related to the earlier occurrence of the application dispatched by the node that has not yet been reset.

Description

Technology Field

[0001] The present disclosure relates to a processing system configured to support an application, and in particular to a processing system comprising a plurality of processing nodes, wherein each of the plurality of processing nodes comprises a processing unit configured to execute a set of instructions for a part of the application. Background Technology

[0002] In the context of processing data for complex or high-volume applications, a processing unit may be provided to perform such data processing. This processing unit may function as a work accelerator to which the processing of specific data is offloaded from the host system. This processing unit may have specialized hardware for performing specific types of processing. For example, a processing unit configured for machine learning may include a large number of processor tiles implemented together on the same chip to provide a high level of parallelism supported by the same processors. To support the operation of this processing unit, additional System on Chip (SoC) circuitry may be provided on the same chip along with the processing unit. This SoC circuitry may include, for example, on-chip memory independent of the processors' memory, port controllers for external communication, and boot hardware for booting the processing units.

[0003] To expand the processing volume to enable support for larger applications, multiple processing units can be connected together to form a larger system. In various embodiments of such a processing system, each of the chips supporting these processing units is connected together via high-speed links that enable communication between the tile processors of different processing units.

[0004] During the processing of applications supported across a system of connected chips, various error events may occur on one or more of these chips. Common events include, for example, an overtemperature condition, the occurrence of an uncorrectable memory error, and a failure of the device's external link. Some of the error events may require a system reset and a restart of the system's application execution from a checkpoint.

[0005] One problem that may be encountered when performing a reset on a multi-node system is that if one of the nodes is reset and restarted from an earlier checkpoint before another node undergoes a reset, the node that has undergone the reset may receive packets from nodes that have not yet been reset, and these packets relate to earlier occurrences of applications running on the system. This can interfere with application processing related to new occurrences of the applications currently being processed by the node that has undergone the reset.

[0006] According to a first embodiment, a processing system configured to support an application is provided, wherein the processing system comprises a plurality of processing nodes, each comprising: a processing unit configured to execute a set of instructions for a part of the application; and at least one storage configured to store a copy of a number indicating the number of times a global reset of the processing system has been performed;And a reset circuit is included, wherein the reset circuit is configured to cause each processing node to be reset as part of one of the global resets for the processing system and to restore application processing by the processing unit of each processing node from a checkpoint, wherein each processing node includes a circuit configured to update a copy of a number held in at least one storage of each processing node as part of the reset of each processing node, wherein the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to cause one or more data frames to be sent to the first processing node among the processing nodes prior to the reset of the second processing node among the processing nodes, each of the one or more data frames includes a copy of a number held in at least one storage of the second processing node among the processing nodes, wherein the circuit of the first processing node among the processing nodes, following the reset of the first processing node among the processing nodes, and the copy of a number in one or more data frames and the at least one storage held It is configured to cause one or more data frames to be discarded in response to determining a mismatch between copies of numbers.

[0007] Each node stores a number referred to herein as the generation number, and this number is updated whenever each node undergoes a reset and restart from a checkpoint. In most cases, since the nodes of the system participate in the same reset event, the generation number held by each node will be the same across the system. However, in some cases, when one node is reset before another node, the generation numbers between these two nodes will differ. Data frames sent between nodes each contain the generation number of the sending node, which is checked by the receiver, and are accepted only if the generation number within the frames matches the generation number of the receiver node. If the receiver node is reset before the sending node is reset, the generation number within the frames will not match the generation number of the receiver node, and the frames will not be accepted. Thus, the reset and restarted node is protected against packets related to the earlier generation of the application dispatched by the node that has not yet been reset.

[0008] According to a first embodiment, a processing system configured to support an application is provided, wherein the processing system comprises a plurality of processing nodes, each comprising: a processing unit configured to execute a set of instructions for a part of the application; and at least one storage configured to store a copy of a number indicating the number of times a global reset of the processing system has been performed; And a reset circuit is included, wherein the reset circuit is configured to cause each processing node to be reset as part of a global reset of one of the global resets for the processing system and to restore the application state of a part of the application for each processing node from a checkpoint, wherein each processing node includes an update circuit configured to update a copy of a number held in at least one storage of each processing node from a first value to a second value as part of the reset of each processing node, wherein the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to cause one or more data frames to be sent to the first processing node among the processing nodes prior to the reset of the second processing node among the processing nodes, each of the one or more data frames includes a first value of a number, wherein the circuit of the first processing node among the processing nodes is configured to, following the reset of the first processing node among the processing nodes, a discrepancy between the first value of a number in one or more data frames and the second value of a number held in at least one storage of the first processing node among the processing nodes In response to the decision, it is configured to perform the task of discarding data within one or more data frames.

[0009] In some embodiments, for each of the processing nodes, each processing unit is configured to participate in a plurality of barrier synchronizations indicated in a set of instructions, wherein for each of the processing nodes, each processing unit is configured to pass each barrier synchronization in response to at least one additional processing unit among the processing units in the processing system also participating in each barrier synchronization.

[0010] In some embodiments, for each of the processing nodes, each of the plurality of barrier synchronizations separates the compute phase of each processing unit from the exchange phase of each processing unit.

[0011] In some embodiments, the processing unit of the first processing node among the processing nodes is configured to stall until an additional processing unit among the processing units also configured to participate in the first barrier synchronization in the application processing reaches the first barrier synchronization in the first barrier synchronization, following the reset of the first processing node among the processing nodes.

[0012] In some embodiments, the additional processing unit among the processing units is the processing unit of the second processing node among the processing nodes.

[0013] In some embodiments, the first barrier synchronization acts as a barrier between the computation phase and the exchange phase, wherein the processing unit of the first processing node among the processing nodes is configured to execute a set of application instructions to perform computations during the computation phase until the first barrier synchronization is reached, and to exchange data with the second processing node among the processing nodes during the exchange phase following the first barrier synchronization, and the data exchanged includes data contained within one or more data frames.

[0014] In some embodiments, the circuit of the first processing node among the processing nodes configured to determine the discrepancy is part of the interface of the first processing node among the processing nodes for interfacing with the second processing node among the processing nodes.

[0015] In some embodiments, the interface of the first processing node among the processing nodes is configured to receive additional data frames from the second processing node among the processing nodes and to provide data packets encapsulated within the additional data frames to the processing unit of the first processing node among the processing nodes.

[0016] In some embodiments, the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to cause one or more additional data frames to be sent to the first processing node among the processing nodes following a reset of the second processing node among the processing nodes, each of the one or more additional data frames includes a second value of a number, wherein the circuit of the first processing node among the processing nodes is configured to receive one or more additional data frames and to accept data in one or more data frames in response to determining a match between the second value of a number in each of the one or more additional data frames and the second value of a number held in at least one storage of the first processing node among the processing nodes.

[0017] In some embodiments, the circuit of the first processing node among the processing nodes is configured to determine the discrepancy as part of checking the address field within each of the data frames.

[0018] In some embodiments, for each of the processing nodes, each update circuit includes a processor configured to execute computer-readable instructions to perform an update of a copy of a number held in at least one storage of each processing node.

[0019] In some embodiments, updating a copy of a number includes incrementing a first value to obtain a second value.

[0020] In some embodiments, a copy of the number forms part of the connection state for the connection between the first processing node among the processing nodes and the second processing node among the processing nodes.

[0021] In some embodiments, each processing node comprises an integrated circuit including a processing unit; and at least one storage for each processing node.

[0022] In some embodiments, the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to output a first value of a number in one or more data packets of one or more data frames.

[0023] In some embodiments, a processing unit of a second processing node among the processing nodes is configured to execute a set of instructions to output data for one or more data frames to an interface circuit, wherein the interface circuit is configured to provide a first value of a number inserted into each frame to one or more data frames.

[0024] According to a second embodiment, a method is provided, the method comprising: executing a set of instructions for a portion of an application supported by a system comprising a plurality of processing nodes on each of a plurality of processing nodes; storing on each of the processing nodes a copy of a number indicating the number of times a global reset of a processing system has been performed; performing one of the global resets, wherein performing one of the global resets comprises resetting each of the plurality of processing nodes, wherein resetting each of the plurality of processing nodes comprises updating copies of the number held on each of the processing nodes from a first value to a second value, and restoring the application state for a portion of the application for each processing node from a checkpoint; and, prior to resetting a second processing node among the processing nodes, sending one or more data frames from a second processing node among the processing nodes to a first processing node among the processing nodes, wherein each of the one or more data frames includes a first value of a number; And, following the resetting of the first processing node among the processing nodes, and in response to determining a discrepancy between a first value of a number in one or more data frames and a second value of a number held by the first processing node among the processing nodes, data in one or more data frames is discarded.

[0025] According to a third embodiment, a computer program is provided comprising a set of executable instructions for each of a plurality of processing nodes forming a data processing system, wherein each set of executable instructions, when executed by at least one processor of each processing node, each processing node supports a portion of a distributed application, wherein the distributed application is supported by a plurality of processing nodes; and storing a copy of a number indicating the number of times a global reset of the processing system has been performed;And as part of a global reset for a data processing system, resetting each processing node, wherein resetting each processing node comprises updating a copy of a number held on each processing node from a first value to a second value, and restoring the application state of a part of a distributed application for each processing node from a checkpoint, wherein a second set of executable instructions is configured such that, when executed by at least one processor of a second processing node among the processing nodes, the second processing node among the processing nodes sends one or more data frames to a first processing node among the processing nodes prior to the reset of the second processing node among the processing nodes, each of the one or more data frames includes a first value of a number, wherein a first set of executable instructions is configured such that, when executed by at least one processor of a first processing node among the processing nodes, the first processing node among the processing nodes, following the reset of the first processing node among the processing nodes, the first value of a number in one or more data frames and processing It is configured to discard data within one or more data frames in response to determining a discrepancy between a second value of a number stored by a first processing node among the nodes.

[0026] According to the fourth embodiment, a non-transient computer-readable medium for storing a computer program according to the third embodiment is provided. Brief explanation of the drawing

[0027] To aid in understanding the present disclosure and to show how embodiments can be practiced, the accompanying drawings are referenced by way of example, and in the accompanying drawings, FIG. 1 is a schematic block diagram of a multi-tile processing unit; FIG. 2 is a schematic block diagram of one of the tiles that can be implemented within a multi-tile processing unit; FIG. 3 is a first exemplary embodiment of a processing node; FIG. 4 is a second exemplary embodiment of a processing node; FIG. 5a illustrates a system of processing nodes connected together to form a spanning tree graph for the exchange of global event states; FIG. 5b illustrates a system of processing nodes connected together in a ring topology for the exchange of global event states; FIG. 5c illustrates the exchange of global event states between a set of processing nodes within a system; FIG. 5d illustrates how a global event state is propagated from each set of processing nodes to multiple components; FIG. 6 illustrates the structure of a local event vector and a global event vector according to an exemplary embodiment; FIG. 7 illustrates a bus architecture used for propagating event states among components of a processing node; FIG. 8 illustrates an event report dispatched to a component for transmission to the control node of a processing node; FIG. 9 illustrates a node on a bus for receiving and storing event states broadcast by a control node; FIG. 10 illustrates registers and processing circuits held within a control node that enable the control node to store and distribute event states; FIG. 11 illustrates an example of an interface circuit used to exchange global event states between processing nodes, among other things; FIG. 12 illustrates in more detail the components of an interface circuit used to send and receive global event states; FIG. 13 illustrates an example of an exchange block for enabling tiles of a processing node to exchange data plane traffic with tiles of another processing node; FIG. 14 illustrates an example of registers and circuitry within a switching block to enable the switching block to engage in autonomous event handling in response to an event state in a local event vector or a global event vector; FIG. 15 illustrates an example of how event notification hardware is used to notify handler software of the occurrence of an event; FIG. 16 illustrates how multiple event handlers within a system can respond to notification of an event occurring in the system; FIG. 17 illustrates how checkpoint data is read from checkpoint memory and returned to a processing unit; FIG. 18a illustrates a first part of a method for performing a reset of a multi-node system; FIG. 18b illustrates a second part of a method for performing a reset of a multi-node system; FIG. 19 illustrates how the occurrence number used for the connection can be updated in response to a reset event; FIG. 20 illustrates an example of a method for propagating a global event state among different nodes of a multi-node system; and FIG. 21 illustrates an example of a method for propagating local event states to components so that these components can respond to them and participate in performing related actions. Specific details for implementing the invention

[0028] Reference is made to FIG. 1, which illustrates an exemplary multiple-processing unit (2). A multiple-tile processing unit (2) is described in our earlier U.S. application No. 15 / 886065, the contents of which are incorporated herein by reference. Each of the processing units (2) is formed on a separate integrated circuit (i.e., a chip).

[0029] The processing unit (2) comprises an array (6) of multiple processor tiles (4) and an interconnect (34) connecting the tiles (4). The processing unit (2) may be implemented as a single die among multiple dies packaged within the same IC package. The interconnect (34) may also be referred to herein as an "exchange fabric" (34) because it enables the tiles (4) to exchange data with one another. Each tile (4) includes an instance of an execution unit and a memory. For example, the processing unit (2) may consist of approximately several hundred tiles (4), or even more than a thousand tiles (4). For complete illustrative purposes, it should also be noted that the term "array" as used herein does not necessarily imply any specific number of dimensions or physical layout of the tiles (4).

[0030] In the embodiments, each processing unit (2) is a part of a chip that also includes one or more external links, and these external links enable the processing unit (2) to be connected to one or more other processing units (e.g., to be connected to one or more other instances of the same processing unit (2)). These external links may include any one or more processing unit-to-host links for connecting the processing unit (2) to a host system, and / or one or more processing unit-to-processing unit links for connecting one or more other instances of the processing unit (2) together on the same IC package or card or on different cards. The processing unit (2) receives work from the host in the form of application data that it processes.

[0031] Each of the tiles (4) includes a processing circuit and memory. In some exemplary embodiments, the processing circuit is a multi-threaded processor (10). FIG. 2 illustrates an example of a tile (4) according to embodiments of the present disclosure. The tile (4) includes a multi-threaded processor (10) in the form of a barrel-threaded processor (10) and local memory (11). The barrel-threaded processor (10) is a type of multi-threaded processor (10) in which the execution time of the pipeline is divided into a repeating sequence of interleaved time slots, each of which may be owned by a given thread. This will be discussed in more detail later. Memory (11) includes instruction memory (12) and data memory (22) (these may be implemented in different addressable memory units or in different regions of the same addressable memory unit). Instruction memory (12) stores machine code to be executed by the processing unit (10), while data memory (22) stores both data on which operations are performed by the executed code and data output by the executed code (as a result of these operations).

[0032] Memory (12) stores various different threads of a program, each thread containing a specific task or a sequence of instructions for performing tasks. It should be noted that instructions as referred to herein mean machine code instructions, that is, instances of one of the basic instructions of a processor's instruction set consisting of a single opcode and zero or more operands.

[0033] In a processor (10), multiple different threads from instruction memory (12) may be interleaved through a single execution pipeline (13) (however, typically only a subset of all threads stored in instruction memory may be interleaved at any given point in the entire program). A multi-threading processor (10) comprises multiple context register files (26), each of which is configured to represent the state (context) of a different thread among the threads to be executed concurrently; a shared execution pipeline (13) common to the threads to be executed concurrently; and a scheduler (24) for scheduling threads to be executed concurrently for execution through the shared pipeline in an interleaved manner, preferably in a round-robin manner. The processor (10) is connected to a shared instruction memory (12) that is common to multiple threads, and is also connected to a shared data memory (22) that is common to multiple threads.

[0034] The execution pipeline (13) includes a fetch stage (14), a decode stage (16), and an execution stage (18), wherein the execution stage (18) includes an execution unit capable of performing arithmetic and logical operations, address calculations, load and store operations, and other operations as defined by an instruction set architecture. Each of the context register files (26) includes a respective set of registers for representing the program state of each thread.

[0035] Reference is made to FIG. 3, which illustrates an example of a processing device (300) including the processing unit (2) discussed above. The processing device (300) constitutes a processing node within a system of such processing nodes. The device (300) is implemented as a single integrated circuit, i.e., a chip. The device (300) includes a plurality of components, and these plurality of components include the processing unit (2) and also include other System on Chip (SoC) components that support the operation of the processing unit (2). The device (300) includes a management processor (310) referred to as a Management CPU (MCPU) (310). The MCPU (310) executes computer-readable commands to support the operation of the device (300), for example, by handling error events and establishing connections. The device (300) also includes a temperature sensor (330) for measuring the temperature of the device (300). The device (300) includes a Phase Locked Loop (PLL) (340) for providing a clock signal for the device (300). The device (300) includes a DRAM module (350) for storing data (e.g., checkpoint data) to be output by and / or provided to the processing unit (2) by application code executed on the processing unit (2). The device (300) includes a plurality of interface controllers for external communication. These include, for example, an Ethernet Port Controller (EPC) (360). The EPCs (360) are used to tunnel packets over Ethernet between the device (300) and other equivalent devices (300) within a multi-node system.Tunneled packets may be provided to tiles (4) of the processing unit (2) (in the case of data plane traffic) and may be provided from tiles (4) of the processing unit (2), or may be provided to various components of the device (300) and may be provided from various components of the device (300) (in the case of control plane traffic). Interface controllers also include a PCIe complex (370) for PCIe communication between the device (300) and the host device (320).

[0036] FIG. 3 illustrates an example of a processing node (300) implemented on a single chip (300). FIG. 4 is referenced, which illustrates a processing node (400) comprising a plurality of chips (410, 420). The processing node (400) comprises a main processing chip (410) comprising a processing unit (2). The processing node (400) also comprises additional chips (420) associated with and connected to the main processing chip (410). The additional chips (420) may be referred to as fabric chips (420). For brevity, only one of the four fabric chips (420) of the node (400) is shown in detail. Each of the fabric chips (420) provides access to a DRAM memory (430) that can be read or written by the processing unit (2). One of the fabric chips (420) provides access to the host device (440) via a PCIe complex (450). Each of the fabric chips (420) includes EPCs (460), one of which is for interfacing with the main processing chip (410), and the other of which is for interfacing with other processing nodes equivalent to the processing node (400). The chip (410) includes EPCs (470), some of which are for interfacing with the fabric chips (420) belonging to the same node (400), and some of which are for interfacing with the fabric chips (420) belonging to different nodes (400). In these embodiments, the MCPU (310) is moved from the main processing chip (410) to the fabric chip (420).

[0037] At a specific processing node (node ​​(300) or node (400)), a number of different events may occur that indicate error conditions occurring at the processing node. These events may be, for example, an overtemperature event detected by a temperature sensor (330), an exception event recorded by a tile (4) while executing application code, or the occurrence of an uncorrectable error within a DRAM module (350, 430). Some of the error events may be fatal to the application (e.g., when a link between chips goes down), while others may not be fatal but may still require reporting (e.g., when a number of correctable errors exceeding a threshold occur on the link). The events for which reporting and aggregation are performed are each assigned to event classes indicating the type of event. For example, an unmodifiable memory error within a DRAM module (350, 430) may be one event class, while a security exception may be another event class. In the case of a processing node (400) comprising multiple chips (410, 420), some of the events may occur on the main processing chip (410), while others may occur on one of the fabric chips (420).

[0038] A system of processing nodes is provided, wherein each processing node is configured to execute a portion of application code for the entire system. This system may be referred to as a partition because a single isolated application is executed across the devices of the system. Each processing node within the system stores at least one copy of an event vector, wherein each element of the event vector corresponds to a different event class. Each element indicates whether an event of that corresponding class occurred at any location within the system of processing nodes. This event vector aggregated across the system is referred to herein as a Global Event Vector (GEV). In addition to the GEV, each node also stores its own associated local event vector (referred to herein as 'CEV'). The CEV also includes a plurality of elements, wherein each of these elements indicates whether an event of that corresponding event class occurred on the processing node storing the local event vector.

[0039] Reference is made to FIG. 6, which illustrates examples of GEV and CEV. Both event vectors contain multiple elements and provide a bitmask, which indicates whether, for each event class, an event of that event class has occurred. Each element of the CEV indicates whether an event of each class has occurred on the processing node storing the CEV. Each element of the GEV indicates whether an event of each class has occurred on any of the processing nodes of the system. The CEV contains more elements than the GEV, reflecting that when events are captured within the CEV, only a subset of the corresponding event classes is globally aggregated. In the embodiments, the CEV contains 15 elements (one for each of the 15 event classes). The GEV contains 10 elements, one for each of the 10 event classes, reflecting that events from the first 10 event classes captured within the CEV are globally aggregated. In Fig. 6, the numbering of vector elements reflects the event class ID associated with each element.

[0040] In this specification, 'event state' is referred to. This indicates the state of the CEV and GEV. 'Global event state' is also referred to, which indicates the state of the GEV.

[0041] Reference is made to FIG. 5a, which illustrates a system (500) of processing nodes (510). Each of the processing nodes (510) may be a processing node (300) or a processing node (400). In FIG. 5a, an example is shown of how each of the nodes (510) is connected for the aggregation of global event states. These connections are used for the transmission of global event states and are therefore used for the aggregation of global event states across the system (500). In the embodiments, it will be understood that additional connections other than those shown may exist between the nodes (510), where these additional connections are used for the exchange of data plane traffic or other types of control plane traffic (e.g., sync messages).

[0042] The connections between the nodes (510) used for the exchange of global event states form a spanning tree. The spanning tree enables communication between any two nodes (510) within the system (500), but without any cycles (or rings) in the graph formed by the interconnected nodes (510). The use of the spanning tree enables messages to be transmitted between any two nodes (510) in a small number of hops, while maintaining a simple topology with fewer connections than in other cases.

[0043] Reference is made to FIG. 5b, which illustrates an example in which a system (500) of nodes (510) is connected in an alternative topology for the exchange of global event states. In this case, the nodes (510) are connected together in a ring, and global event states are exchanged between the nodes (510) around the ring. This ring topology has the advantage of establishing redundancy within the system (500). If the link used to send updates to the GEV becomes inoperable, the updates can still be sent between any two nodes (510) within the system (500) because an alternative path exists for sending these updates. Thus, aggregation of global event states is still possible with the ring topology.

[0044] Although the system (500) in the examples of FIG. 5a and FIG. 5b is shown to include only 16 processing nodes (510), in reality, the system (500) is likely to include thousands of processing nodes (510).

[0045] Reference is made to FIG. 5c, which illustrates how global event states can be aggregated among different nodes (510) of the system (500). A portion (520) of the system (500) is shown, wherein this portion (520) includes exemplary processing nodes (510a to 510d).

[0046] As shown, each of the nodes (510a to 510d) of the system maintains a copy of the GEV and a copy of its own CEV. In practice, each node includes multiple registers, some of which are referred to as Global Event Vector Registers (GEVVRs) and hold one of the copies of the GEV, and some of which are referred to as Colossus Event Vector Registers (CEVVRs) and hold one of the copies of the CEV. However, logically, it can be considered that a single copy of the node's CEV and a single copy of the GEV exist on each of the nodes (510a to 510d).

[0047] Whenever an event of an event class recorded in the CEV occurs on one of the nodes (510a to 510d), the processing circuit of the node (510a to 510d) updates the CEV held on the node (510). Specifically, the processing circuit of the node (510a to 510d) updates the elements of the CEV corresponding to the event class of the detected event. If the event class is a class that propagates across the nodes (510), in addition to updating the CEV on one of the nodes (510a to 510d), the circuit of the node (510) also updates the corresponding elements of the GEV held on the node (510). Then, the interface circuit of the processing node (510) causes an update message (which may be referred to as a GCM message) containing an indication of the update to the GEV to be sent to one or more other processing nodes (510). Each of these update messages may include the entire updated GEV or may include an indication of the elements of the GEV to be updated. The update message is sent from one of the nodes (510a to 510d) to a neighboring node (i.e., a node connected to that node) (510) of the nodes (510a to 510d) sending the update. The node (510) receiving the update message updates a copy of its GEV according to the update message and then sends one or more additional update messages to its neighboring nodes (510) (nodes different from the neighboring node (510) that provided the update).

[0048] In the example of FIG. 5c, an event occurred on the processing node (510a). This event is an event of a class of events configured to allow the processing nodes (510a through 510d) to aggregate globally. As shown, in this example, the event occurred in a fourth event class where events are recorded in the CEV and GEV. However, the event class may be a different event class. In response to the detection of the event, the processing circuit of the node (510a) causes the fourth element of the CEV for the node (510a) (an element associated with the fourth event class) to be set to indicate that an event of the fourth event class has occurred. Similarly, in response to the detection of the event, the processing circuit of the node (510a) causes the fourth element of the GEV for the node (510a) (an element associated with the fourth event class) to be set to indicate that an event of the fourth event class has occurred.

[0049] In response to an update to a copy of the GEV on node (510a), the circuit of node (510a) causes a data packet to be issued to node (510b), wherein this data packet contains an indication of an update to the GEV. Node (510b) receives the data packet, and the circuit of node (510b) causes a copy of the GEV (510b) held on node (510b) to be updated according to the indication.

[0050] In response to an update to a copy of the GEV on node (510b), the circuit of node (510b) causes data packets to be issued to nodes (510c, 150d), each of which contains an indication of an update to the GEV. The circuit of each of these nodes (510c, 150d) causes each copy of the GEV held on each node (510) to be updated according to the indicated update provided in the data packets.

[0051] It will now be explained how the event state is reported by a component within the processing node (510) and propagated to other components within the node (510).

[0052] When an event is detected by a specific component of a processing node (510), that component transmits the event report to a regulator component of the node (510), and the regulator component updates a copy of the CEV (known as the master copy) but before broadcasting the update to other components that maintain other copies of the CEV. The exchange of these messages between nodes can occur through a control bus architecture.

[0053] Reference is made to FIG. 7, which illustrates an example of a control bus (700). The control bus (700) is implemented within a single chip, that is, within a device (300) or a device (400). When the control bus (700) is implemented within a device (410), components attached to the control bus (700) to transmit packets between the control bus (700) and components on other devices (420) belonging to the processing node (400) may be provided within the device (400), thereby enabling the exchange of event states between the components on these devices (420) and the controller node (740). The control bus (700) illustrated in FIG. 7 is implemented on a single chip (300 / 410).

[0054] The control bus (700) is described in more detail in our earlier application No. 17 / 328143, which is incorporated herein by reference. The control bus (700) is a data path for carrying single-word control traffic in a ring. The control bus (700) is a pipelined data bus through which data packets move from stage to stage within the pipeline at a rate determined by a clock pulse applied to the control bus (700). The ring comprises a plurality of nodes (710), where traffic proceeds from one node (710) to the next node (710) in the direction of flow surrounding the ring. The output port (750) of each node is connected to the input port (760) of the next node (710) within the ring.

[0055] Some of the nodes (referred to herein as target nodes) have circuitry for receiving requests to read or write configuration settings associated with the nodes. An example of a target node (720) attached to a bus (700) is shown. This target node (720) stores configuration settings for its associated component (770), wherein these configuration settings can be read or written by read / write requests received from the control bus (700). The configuration settings (or configuration states) include copies of the CEV and GEV.

[0056] Some of the nodes (referred to herein as initiator nodes) have request circuits for issuing read or write requests onto the bus (700). An example of an initiator node (730) attached to the bus (700) is shown. Under the control of the attached module, this initiator node (730) issues read or write requests for configuration settings related to a target node (720) attached to the control bus (700). The initiators (730) may be controlled by software in some cases and by fixed function hardware engines in other cases.

[0057] As shown, each of the initiator nodes (730) and each of the target nodes (720) has an associated component (770) (shown as an attached block). The initiator nodes (730) and the target nodes (720) may be implemented within their associated component (770). Each of the initiators (730) operates to dispatch read or write requests onto the control bus (700) under the control of the associated component (770). Each of the target nodes (730) receives and stores a configuration state to control the associated component (770). This configuration state is output from the target node (720) to the associated component.

[0058] Each of the initiator nodes (730) may issue requests to the target nodes (720) and receive completions from the target nodes (720). Each request is a command to perform a read from a storage associated with the target node (720) (e.g., an attached addressable entity or an auto-generated register) or a request to perform a write to such storage associated with the target node (720). In response to the receipt of such requests, the target node (720) responds by issuing a completion. The completion provides a status update indicating whether the read or write request was successful or not. In the case of read completions, each completion contains the data read from the target node (720) and is returned to the initiator node (730) that issued the read request. In the case of write completions, the write completion includes an indication of whether the write request was successful or not.

[0059] Another type of transaction supported by the control bus (700) is an event report. Event reports are packets posted to the control bus (700), which are dispatched onto the control bus (700) by the initiator node (730) or the target node (720), and contain information regarding the detected event. When the hardware unit (770) detects an event, the hardware unit (770) controls the relevant initiator node (730) or the relevant target node (720) to cause the event report to be dispatched onto the control bus (700).

[0060] Reference is made to FIG. 8, which illustrates an event report (800). The event report (800) includes an event class (820) and an event identifier (830), which are supplied by a hardware unit (770) that detected the event. The hardware unit (770) supplies both of these fields (820, 830) to the relevant nodes (720 / 730). The event report (800) includes an event origin field (810), which is prefixed by the nodes (720 / 730) that received the event class (820) and event identifier (830) for the event. The event origin field (810) uniquely identifies the nodes (720 / 730) among all nodes (720 / 730) belonging to the processing node (510).

[0061] The event class field (820) identifies the event class to which the detected event belongs. In embodiments, the event class identifies one of the 15 previously discussed event classes that can be captured within the CEV. The event identifier field (830) identifies a unique event occurring within the event class (820) as well as occurring in the component (770) identified by the source field (810). The event identifier field (830) also includes additional details of the event.

[0062] When an event report is dispatched onto the bus (700), the event report is delivered to the Cbus regulator (740). The Cbus regulator (740) may also be referred to as a control node (740) or a control component (740). The event report circulates on the bus (700) until it reaches the regulator (740), where the event report is consumed and the event is logged.

[0063] Reference is made to FIG. 10, which illustrates an exemplary embodiment of a controller node (740). FIG. 10 illustrates components of a controller (740) for logging event status in response to the reception of event reports. The controller node (740) includes a processing circuit (1010) for performing operations described herein as being performed by the controller node (740). The controller node (740) includes a plurality of registers (1020) for storing information received within event reports. These registers (1020) are referred to as Event Data Registers (EDRs) (1020). The controller node (740) includes an EDR (1020) for each of a plurality of event classes.

[0064] When an event report is received by the controller (740), the controller (740) causes information from the event report to be stored in the EDR (1020) for the event class identified within the event report. The information stored in the EDR (1020) includes an event identifier (830) from the event report and an event source identifier (810) from the event report. Each EDR (1020) also includes an event flag, which is set to indicate that an event of a class corresponding to the class of the EDR (1020) has occurred. In response to the event report, if the event flag for the EDR (1020) corresponding to the class of the event report has not already been set, the node (740) causes the event flag to be updated to indicate that an event of that event class has occurred.

[0065] The controller (740) also includes a register (1030) for storing a copy of the CEV for the processing node (510). This register (1030) is referred to as the master CEVVR (1030) because it is the first CEVVR among the CEVVRs in the node (510) to be updated in response to the detection of an event. When an event report is received by the controller (740), if the bit of the CEV corresponding to the event class identified in the event report has not already been set, the controller (740) sets the corresponding bit of the CEV to indicate that an event of that class has occurred. The CEV exhibits idempotency, and accordingly, if the bit of the CEV for a specific event class has already been set, the corresponding bit of the CEV remains unchanged when a subsequent event report of the same event class is received. The controller node (740) controls access permissions for the CEV, and accordingly, the bits of the CEV cannot be unset except in response to a reset event.

[0066] The regulator node (740) includes a register (1010) for storing a copy of the GEV. The register (1010) is referred to as the master GEVVR (1010) because it is updated before the updates of the GEVVRs of the target nodes (720) within the same processing node (510). The master GEVVR (1010) may be updated in response to i) the detection of an event within its own node (510), or ii) the receipt of a packet from the node (510) indicating an update to the GEV (such an update is due to an event occurring on another node (510).

[0067] The controller node (740) includes a mask register (1050), referred to herein as the GEVVRMASK register (1050). The GEVVRMASK register (1050) provides a set of bits, wherein the bits indicate whether global aggregation for each event class having a corresponding element within the GEV is enabled. To ensure consistency in the aggregation of event classes across the system (500), the values ​​held in the GEVVRMASK register (1050) are set to be the same for each of the nodes (510) within the system (500).

[0068] When an event report is received by the controller (740), if the bit in the GEVVRMASK register (1050) corresponding to the event class identified in the event report is set to indicate that global aggregation is enabled, and if the bit set in the GEVVR (1010) is not already set to indicate that an event occurred in that class, the controller (740) causes the bit in the GEV for that class to be set to indicate that an event of that class has occurred. The GEV is idempotent, and accordingly, if the bit in the GEV for a specific event class is already set, the corresponding bit in the GEV remains unchanged when a subsequent event report of the same event class is received by the controller (740). The controller node (740) controls access permissions for the GEV, so that the bit in the GEV indicating the occurrence of an event cannot be deset except in response to a reset event.

[0069] When one or more of the event flags in the EDRs (1020) change their state to indicate that an event has occurred, the controller (740) broadcasts an update message to all target nodes (720) in the processing node (510). The update message is a message sent via the control bus (700) to update the CEVVRs of each of the target nodes (720) in the processing node (510). The update message causes each copy of the CEV in the node (510) to be updated, and accordingly, the element of the CEV associated with the class for which the event flag was set is updated to reflect that the event occurred in that class.

[0070] Additionally, when one or more bits in the master GEVVR (1010) change state, the controller (740) broadcasts an update message (which may be the same update message used to update copies of the CEV) to all target nodes (720) in the processing node (510). The update message is a message sent via the control bus (700) to update the GEVVRs of each of the target nodes (720) in the processing node (510). The update message causes each copy of the GEV in the target nodes (720) to be updated, and accordingly, the element of the GEV associated with the corresponding event class for which the event flag is set is updated to reflect that the event occurred in that class.

[0071] Reference is made to FIG. 9, which illustrates an exemplary target node (720). Each target node (720) includes at least one node management register (940). The node management register (940) may be written via a dedicated node management broadcast write request that writes a common set of states to all node management registers (940) within the target nodes (720). These broadcast writes may be performed by a controller node (740). Examples of these broadcast writes are update messages sent to update copies of CEV and GEV held within each target node (720).

[0072] In addition to at least one target node management register (910), the target node (720) is associated with an additional storage (920) which is part of the component (770) to which the target node (720) is associated. The additional storage (920) may include auto-generated registers and / or storage accessible through an addressable window. This additional storage (920) may include control registers on tiles (4) of a processing unit (2) that are addressable through an additional bus. This additional storage (920) may enable access to off-chip storage, such as host storage accessible through PCI links. In this case, the additional storage (920) is the memory of a host dispatcher that provides data written to memory to the host storage.

[0073] The target node has an input port and an output port. The input port receives read and write requests from the control bus (700), where these requests are buffered in a request buffer (960) and then passed to the processing logic (930). The output port sends completions from the completion buffer (970) onto the control bus (700). The processing logic (930) can perform functions implemented in hardware or software. The processing logic (930) may include one or more of an ASIC, FPGA, or at least one processor configured to execute computer-readable instructions stored in at least one memory of the target node (720).

[0074] As shown in FIG. 9, the node management registers (940) include CEVVR (980) and GEVVR (990). The CEV held in the CEVVR (980) is the same in all target nodes (720) within the processing node. Similarly, the GEV held in the GEVVR (990) is the same in all target nodes (720) within all processing nodes of the system. Processing logic (930) causes all elements of the CEV and GEV to be output from the node management registers (940) to the attached component (770). These enable the attached component (770) to take action or perform operations according to error states indicated by event states.

[0075] When the controller (740) sends an update message to update the CEV, this update message is received at the target node (720). Processing logic (930) causes the CEVVR (980) of the target node (720) to be updated according to the update. The CEV in the CEVVR (980) is updated to match the CEV held in the master CEVVR (1030). Processing logic (980) causes this updated CEV state to be propagated to the attached component (770), so that the component can take action regarding the updated error state. The same updates are performed at each of the target nodes (720) within the processing node (510).

[0076] Likewise, when the controller (740) sends an update message to update the GEV (which may be the same or different as the one used to update the GEV), this update message is received at the target node (720). Processing logic (930) causes the GEVVR (990) of the target node (720) to be updated according to the update. The GEV in the GEVVR (990) is updated to match the GEV held in the master GEVVR (1010). Processing logic (980) causes this updated GEV state to be propagated to the attached component (770), so that the component can take action regarding the updated error state. The same updates are performed at each of the target nodes (720) within the processing node (510).

[0077] Among the components (770) that receive updates regarding their associated GEVs (i.e., those held within the target GEVVR (990)) are the EPCs of the processing node (510) (e.g., EPCs (360, 460, 470)). When specific EPCs receive updates regarding their GEVs from the controller node (740) of the processing node (510) to which they belong, these EPCs are configured to provide an indication of the updated GEV status in the update message. The indication of the GEV status is propagated between the processing nodes (510) and is used to update the GEVs held within these nodes (510). Specific EPCs configured to propagate the event status are those used to communicate through the connections shown in FIGS. 5a and 5b. It is these EPCs that provide the GEV update messages shown in FIG. 5c.

[0078] Reference is made to FIG. 11, which illustrates an example of an EPC (1100) according to embodiments. The EPC (1100) may be one of the EPCs (360, 460, 470). The EPC (1100) includes a target node (720) and an initiator node (730). The target node (720) enables the updated global event status received from the controller node (740) to be provided to the EPC (1100). In addition to the interfaces shown in FIG. 11, the EPC (1100) also includes interfaces for exchanging data plane traffic with the processing unit (2). FIG. 11 shows a component within the EPC used for exchanging control plane traffic (including updates to event status). This component is referred to herein as a Global Communications Proxy (GCP) (1200). GCP (1200) can receive Ethernet frames through a receiving interface (Rx), wherein these Ethernet frames contain update messages for updating a copy of the GEV. GCP (1200) can dispatch update messages itself via a transmitting interface (Tx), wherein these update messages are messages for updating the GEV held in other processing nodes (510) within the system. GCP (1200) causes the update messages to be provided to a frame establisher (1120), and the frame establisher (1120) causes these update messages to be sent through the transmitting interface in Ethernet frames configured according to the connection status held in a connection status register (1130).

[0079] Reference is made to FIG. 12, which illustrates a GCP (1200) and components within the GCP (1200) used to propagate event states. The GCP (1200) includes a processing circuit (1220) for performing operations described herein as performed by the GCP (1200). In addition to the GCP (1200), FIG. 12 also illustrates target and initiator nodes (720, 730) for enabling the GCP (1200) to exchange event states with a controller node (740). What is described herein is how the GCP (1200) enables global aggregation of event states to be performed. Although the processing node (510) according to the embodiments includes a plurality of GCPs (1200), it will be understood that only a subset of these GCPs (1200) (one or more of these GCPs (1200)) will typically be active for the purpose of propagating global event states.

[0080] As described, the GEVVR register (990) is updated in response to a GEV update message broadcast from the controller (740). This updated state of the GEV is output from the target (720) to the GCP (1200). The GCP (1200) includes a register (1210), referred herein as the GEVEN register (1210). The register (1210) includes an indication regarding whether the GCP (1200) is enabled for global event aggregation. If the GCP (1200) is enabled to perform global event aggregation by the indication in the register (1210), the GCP (1200) will respond to updates to its target GEVVR (990) by dispatching update packets to another processing node (510).

[0081] Each GCP (1200) includes an additional register (1240) referred herein as the Global Event GCP External Global Event Vector Register (GEVEXTGEVVR) (1240). GEVEXTGEVVR (1240) stores a copy of the GEV. The circuit (1220) compares the copy of the GEV held in the GEVEXTGEVVR register (1240) with the copy of the GEV held in the GEVVR register (990) to determine whether updates should be dispatched to another node (510) or to the controller (740) of the same node. If the circuit (1220) of the GCP (1200) determines that a bit is set in the target GEVVR (990) and not set in the GEVEXTGEVVR (1240), the circuit (1220) of the GCP (1200) updates the GEVEXTGEVVR register (1240) to match the copy of the GEV held in the GEVVR (990). The GCP (1200) also causes the update message to be dispatched to a GCP on another node (510) configured to communicate with the GPC (1200). The update message provides an indication of the updated GEV and causes the receiving GCP to update its own GEVEXTGEVVR to match the GEVEXTGEVVR (1240) of the GCP (1200) that dispatched the update message.

[0082] To provide reliable propagation, GCP (1200) sends update messages three times. When GEV update messages are dispatched in a frame, GCP (1200) waits for 1,000 system clock cycles of its chip (510) to elapse before resending the same frame containing the update messages. Then, GCP (1200) waits for another 5,000 system clock cycles and then sends the frame at the third time. If the GCP (1200)’s associated target GEVVR (990) changes again before these copies are sent, the preceding resend sequence is aborted, and a new update message containing the new state held within the GEVVR (990) is sent.

[0083] GCP (1200) may receive update messages from another GCP on a different node (510) to update the copy of GEV within its GEVEXTGEVVR (1240). As a result of these updates, the copy of GEV held within GEVEXTGEVVR (1240) may not match the copy of GEV held within the target GEVVR (990) associated with GCP (1200). In response to determining that a bit is set in GEVEXTGEVVR (1240) and not set in the associated GEVVR (990), GCP (1200) issues a write request to the Cbus controller (740) on its own node (510), the write request being to update the GEVUPDATER register (1060) to match the copy of GEV currently held within GEVEXTGEVVR (1240).

[0084] In response to the reception of a write to the GEVUPDATER (1060), the regulator node (740) updates the GEVUPDATER (1060) accordingly. In response to the update to the GEVUPDATER (1060), the regulator (740) updates the copy of the GEV held in the master GEV register (1010) for the processing node (510). The update to the master GEVVR (1010) is performed so that any bit set in the GEVUPDATER (1060) to indicate that an event occurred in a specific class is also set in the GEVVR (1010). However, for any bits in the GEVUPDATER (1060) set to indicate that no event occurred in that class, the regulator (740) will not update the GEVVR (1010) to indicate that no event occurred in that class. In this way, the update policies of the controller (740) ensure that the GEV held in the GEVVR (1010) is idempote.

[0085] In addition to updating the copy of the GEV held in the register (1010), the controller (740) broadcasts an update message to all target nodes (720) of the processing node (510) on the control bus (700). The update message contains the updated GEV held in the register (1010) of the controller (740). Referring to FIG. 9, the processing logic (930) of each target node (720) receives the update message and updates the GEVVR (990) held in the target node (720) with the updated GEV. The component (770) associated with the target node (720) has access to the copy of the GEV held in the GEVVR (990) of its target node (720).

[0086] Some of these components (770) that receive updates to their copies of the GEV through their associated GEVVR (990) are GCPs that are enabled to perform global event aggregation. Each of these GCPs publishes an update message to another node (510) in response to an update to its GEVVR (990). Then, the updated GEV is propagated to different components of this other node (510) in the same manner as previously described.

[0087] Reference is made to FIG. 5d, which further illustrates in detail how an updated global event state is propagated among the nodes (510a to 510d) of a part (520) of the system (500). When a component (770a) (which may itself be a GCP) encounters an error event, the component (770a) provides a report of this event to the controller (740a). The controller (740a) updates its master copy of the CEV and GEV for the node (510a) and broadcasts the updates of these vectors to each of the target nodes (720) within the node (510a). The GCP (1200a) detects an update to the copy of the GEV held within its target node (720a) and, in response, provides a message containing the updated GEV to the node (510b). Node (510b) includes a GCP (1200b) configured to receive an update and dispatch the update message to the regulator (740b) of the node (510b) via the bus (700) of the node (510b). In response to receiving the update, the regulator (740b) updates its own copy of the GEV to match the copies of the GEV held on the node (510a). The regulator (740b) also broadcasts the update of the GEV to the target nodes (720) of the processing node (510b) (including target nodes (720b, 720c). The GCPs (1200c, 1200d) associated with the target nodes (720b, 720c) respond by dispatching update messages containing updates to the GCPs (1200e, 1200f) on the nodes (510c, 510d). The GCPs (1200e, 1200f) propagate these updates to their own controller nodes (740c, 740d), and these controller nodes (740c, 740d) broadcast the updates to the components on these nodes (510c, 510d).In this way, updates to the GEV generated in response to an event detected by a component (770) on one node (510a) are propagated to components of other nodes (510b to 510d) within the system (500).

[0088] The result of propagating local event states (i.e., CEV) and global event states (i.e., GEV) to components (770) is to provide each component (770) within the system (500) with access to two different event vectors. Each component (770) has access to device-wide event and error states for its own processing node (510) through its own local copy of the CEV for its processing node (510). Each component (770) also has access to system-wide event and error states through its own local copy of the GEV for the system. Based on these states, the components (770) may be configured to perform actions. These actions taken based on copies of the CEV or GEV accessible to the components are referred to herein as Autonomous Event Handling (AEH). Different components (770) within the same processing node (510) may be configured to perform different actions in response to events belonging to the same class. One or more components (770) may be configured to have different policies and may be configured to perform different actions in response to events belonging to different event classes. Furthermore, one or more components (770) may perform different actions in response to events related to their own node (510) (and reflected in their CEV) compared to events related to the system (500) (and reflected in the GEV).

[0089] An example of a component that can participate in AEH is a circuit (referred to herein as an eXchange Block (XB)) that enables the tiles (4) of the processing unit (2) to exchange data with devices outside the chip (510) on which the processing unit (2) is implemented.

[0090] Reference is made to FIG. 13, which illustrates an example of a chip (1300) comprising an XB (1310) and a plurality of tiles (4). The XB (1310) comprises a circuit configured to perform the described operations. The XB (1310) is associated with an external interface (1320) of the chip (1300), through which the XB (1310) is configured to send and receive data packets. The XB (1310) receives packets in one format ("TLink" format) from one or more tiles (4) and converts these packets into another format ("Elink" format) before allowing these packets to be transmitted through the external interface (1320). The XB (1310) receives packets in this other format ("Elink" format) and converts these packets into another format ("TLink" format) before delivering these packets to one or more of the tiles (4). XB (1310) is also configured to control sending by tiles (4) by providing flow control messages to convey permission for sending between different tiles (4).

[0091] XB (1310) is configured to engage in specified Autonomous Error Handling (AEH) in response to an indication of an error within the GEV. Reference is made to FIG. 14, which illustrates XB (1310) and its associated target node (720). XB (1310) includes a processing circuit (1400) for performing operations described herein as being performed by XB (1310). XB (1310) includes a mask register (1410) (referred herein as the AEHGEVVMASKR register (1410)). The AEHGEVVMASKR register (1410) stores a set of bits that identify which bits of the GEV will cause AEH by XB (1310). XB (1310) performs AEH in response to determining that a bit is set in GEV to indicate that an event of a specific event class has occurred, which occurs when the AEHGEVVMASKR register (1410) stores a bit indicating that AEH is enabled for that class.

[0092] In performing AEH, XB (1310) stops sending packets from the tiles (4) to the interface (1320) and stops sending packets from the interface (1320) to the tiles (4). If XB (1310) receives packets from the tiles (4), XB (1310) causes these packets to be dropped. Likewise, if XB (1310) receives packets from the interface (1320), XB (1310) causes these packets to be dropped. In this way, XB (1310) blocks communication between the tiles (4) and external devices. The result is that data plane traffic is quickly quiesced.

[0093] The type of event displayed in the GEV that can cause the XBs (1310) to block traffic is a fatal error event in the execution of the application, such as an unrecoverable memory error or one of the external links of the chip (1300) becoming inoperable. In this case, IO traffic is stopped prior to the resetting of the chip (1300).

[0094] Different components within the system may include different mask registers, such as a register (1410), which indicate, for each component, which of the events each component is configured to perform actions in response to. A mask register within a specific component may include a first set of indications indicating which of the event classes identified in the CEV the component is configured to take action in response to. A mask register may also include a second set of indications indicating which of the event classes identified in the GEV the component is configured to take action in response to. These registers are defined per component, so that each component is enabled to perform action in response to different types of events (i.e., to perform AEH).

[0095] Each of the event classes defined in the CEV (and further defined in the GEV) has an associated event handler configured to implement a given response to the occurrence of the event. The event handler is a software module running on an event handler device. The action taken by the handling device is in addition to any AEH performed by the individual components (770) of the processing node (510). For example, in response to the occurrence of an event indicating a fatal application error, the handling entity may cause the application to be reset from a checkpoint.

[0096] The handling entity is different for different event classes. For some event classes, the handling entity may be software running on the MCPU (310). For other event classes, the handling entity may be a host process running on the host (320 / 440). Each event handler in the entire system is associated with one or more processing nodes (510) and is configured to take action against one or more processing nodes (510) associated with it in response to an event notification received from one or more processing nodes associated with it among the processing nodes (510).

[0097] Reference is made to FIG. 15, which illustrates how event notification to an event handler (1510) can be performed. A target node (720) receives updates for CEV and GEV broadcast by a regulator node (720) following the occurrence of an event. Some of the components (770) associated with the target node (720) include associated event notification hardware (1500). Components including such event notification hardware (1500) include, for example, a PCIe complex (370 / 450) or an MCPU (310). Handler software (1510) may be executed on the same device (e.g., MCPU (310)) that includes the event notification hardware (1500), or on a separate device (e.g., a host device).

[0098] The event notification hardware (1500) includes a CEV notification mask register (1520). The CEV notification mask register (1520) indicates that event notifications are provided to the handler software (1510) for each of the corresponding event classes when events are logged in the CEV. In response to receiving from the target node (720) that an event has occurred in a specific event class, the processing circuit of the event notification hardware (1500) checks for a corresponding indication in the register (1520) for the corresponding event class. If the register (1520) indicates that events in that class are to be reported to the handler software (1510), the processing circuit of the event notification hardware (1500) issues an event notification to the handler software (1510). The event notification may take the form of an interrupt to the handler software (1510).

[0099] The event notification hardware (1500) also includes a GEV notification mask register (1530). The GEV notification mask register (1530) indicates that event notifications are provided to the handler software (1510) for each of the corresponding event classes when events are logged in the GEV. In response to receiving from the target node (720) that an event has occurred in a specific event class, the processing circuit of the event notification hardware (1500) checks for a corresponding indication in the register (1530) for the corresponding event class. If the register (1530) indicates that events in that class are to be reported to the handler software (1510), the processing circuit of the event notification hardware (1500) issues an event notification to the handler software (1510). The event notification may take the form of an interrupt to the handler software (1510).

[0100] For each event class reported in the CEV, one event handler is defined to handle the events of that class. Similarly, for each event class reported in the GEV, one event handler is defined to handle the events of that class. In embodiments, each processing node (510) is associated with four event handlers: two host processes running on the host device (320 / 440), a process running on the MCPU (310), and a process running on an additional hardware unit to enable cryptographic functions (referred to as ICU). Thus, the event handlers may be implemented on separate devices (e.g., MCPU and host) or may take the form of separate processes running on the same device (e.g., separate host processes running on a single host).

[0101] In response to the receipt of an event notification, the handler software (1510) is configured to take appropriate action. These actions include issuing a read request to read the event data registers (1020) within the associated controller node (740). The event handler (1510) reads the EDR register (1020) for the associated event class. The event handler reads the event flag and event identifier from the corresponding EDR register (1020). This read may be issued by the event handler using the associated initiator (730) for the event handler, whereby event details (e.g., event flag and event identifier) ​​are returned to the event handler (1510).

[0102] When a global event is notified to the handler software based on the GEV, the event flag indicates to the event handler (1510) whether the event is associated with the processing node (510) to which the event handler (1510) is associated.

[0103] Reference is made to FIG. 16, which illustrates the exchange of messages performed following the issuance of event notifications to an event handler (1510) based on the GEV held within each of the processing nodes (510). Although each event handler (1510) may appear to be separate from its associated processing node (510), each event handler (1510) may be considered to be part of its associated processing node (510). Additionally, although each event handler (1510) may appear to have a single associated processing node (510), in some embodiments, each of the event handlers (1510) is associated with a plurality of such processing nodes (510).

[0104] As a first step, each event handler (1510) issues a read request to read an event data register (1020) associated with the event class identified by the event notice in response to receiving an event notice. The event data register (1020) read by each event handler (1510) belongs to the processing node (510) associated with the event handler (1510).

[0105] In the second step, each event handler (1510) receives, in response, a state held within the EDR (1020) identified by the read request it sent. This state includes an event flag and an event identifier held within the EDR (1020). If the event reflected in the GEV occurred on the processing node (510) to which the event handler (1510) is associated, the event flag will be set in the state to indicate that the event occurred on that node (510). In the example shown in FIG. 16, the event reflected in the GEV occurred on processing node 2, and thus event handler 2 receives an event flag set to indicate that the event occurred on node 2, and additionally receives details about the event (e.g., in the form of an event identifier).

[0106] In the third step, Event Handler 2 propagates messages related to the event to other Event Handlers 1 and 3. The messages may be event details received by Event Handler 2 in the EDR state, or messages to cause Event Handler 1 and Event Handler 3 to reset their associated processing nodes (510). In the illustrated example, each of the Event Handlers (1510) decides to reset its associated processing node (510). This decision is made based on the EDR state obtained from Processing Node 2. This decision may be made by Event Handler 2, and then Event Handler 2 propagates the command to other Event Handlers 1 and 3, or this decision may be made individually by each Event Handler (1510) based on the EDR state obtained and distributed by Event Handler 2.

[0107] In this example, and as a fourth step, it is shown that each of the event handlers (1510) issues a command to reset their respective processing node (510). This reset is achieved by asserting a signal to the reset pin (1600) on the processing node (510). The circuitry on each processing node (510) enables the reset to be performed.

[0108] A reset, which may be referred to as a software reset, resets the state of multiple components that are part of each processing node (510). The reset has the effect of wiping the memory of the tiles (4) to remove application data. The reset also has the effect of wiping the state of the cryptographic hardware used to encrypt communications within the device (510) and communications outside the device (510). The reset causes the state of local links and connections using these links to be reset. However, the reset does not render the links inoperable, and all states for configuring the physical links are not modified. Therefore, there is no need to reprogram the EPCs (470) after the reset. The reset does not wipe the application state of the MCPU (310).

[0109] The reset has the effect of clearing the CEVs and GEVs held within the components of the system, so that they do not indicate that any events have occurred. Clearing the CEVs and GEVs is performed by a controller (720), wherein the controller (720) has write permissions for the CEVVRs and GEVVRs within its processing node (510).

[0110] When a specific processing node (510) resets its state and the tile memory is wiped, the processing node (510) can restart its application from an application checkpoint. Each processing device (300 / 410) has access to external memory where checkpoint data is periodically stored. The tiles (4) of the processing unit load checkpoint data from external memory following a read request, which allows the application to be restarted from an earlier point.

[0111] Each of the event handlers (1510) is configured to allow its associated processing node (510) to be reset immediately without requiring synchronization of the reset across the system. One problem that may be encountered when performing a reset without synchronization is that while one of the processing nodes (510) may perform its reset and restart from an earlier checkpoint, another processing node (510) that has not yet undergone a reset continues to run an earlier version of the application. In this case, if the processing node (510) that has not yet undergone a reset sends data packets containing application data corresponding to the earlier version, these data packets may be received by the processing unit (2) that has been reset from the checkpoint. In this case, errors may occur in the processing by the processing unit (2) that has been restarted from the checkpoint, because the processing unit (2) is receiving data corresponding to the earlier version of the application.

[0112] According to embodiments, frames sent between different processing nodes (510) contain a number (referred to as an occurrence number), which is updated in response to each reset event. Thus, this number indicates how many times the application has been reset. When data frames are sent by one processing node (510) to another processing node, the sending node (510) includes its own copy of the occurrence number in each of these frames. This number is checked against the recipient's own copy of the occurrence number at the recipient node (510). If the occurrence number in the frames does not match the occurrence number held by the recipient node (510), the recipient node (510) discards the frames. In this way, if the recipient node (510) has already been reset and restored from the checkpoint, the recipient node is protected from frames related to the application's pre-reset occurrences that may be dispatched by other nodes in the system that have not yet undergone a reset.

[0113] Reference is made to FIG. 17, which illustrates an example in which two processing nodes (1700a, 1700b) can communicate. Each of the nodes (1700a, 1700b) may be one of the nodes (510) described previously. Typically, the system will include more than two processing nodes, but for the sake of brevity, only two of these nodes are shown in FIG. 17. Both processing nodes (1700a, 1700b) are configured to be reset and restarted from a checkpoint. The reset of both processing nodes (1700a, 1700b) is part of the reset and restoration of the entire system from a checkpoint. This reset can be performed in the manner described previously in relation to FIG. 16, where the global reset of the system, and accordingly the individual reset of each node (510), is performed in response to an error event occurring on one of the nodes (510). However, the global reset of the nodes (1700a, 1700b) may not be synchronized, which means that one of the nodes (1700a, 1700b) can proceed with the reset and restoration from the checkpoint before the other of the nodes (1700a, 1700b) is stopped. In this case, the node (1700a, 1700b) that is not yet stopped can send data frames to one of the nodes (1700a, 1700b) that has been restored from the checkpoint. Copies of the occurrence numbers provide protection against disruption to application processing on the restored node.

[0114] Reference is made to FIGS. 18a and FIGS. 18b, which illustrate a sequence of events performed by a system of processing nodes (1700a, 1700b) shown in FIG. 17. The entire method is shown divided into a first part (1800a) (shown in FIG. 18a) and a second part (shown in FIG. 18b). At the beginning of the sequence of events, both processing nodes (1700a, 1700b) store identical copies of the occurrence number. The occurrence number reflects the number of times a global reset (including a reset across both nodes (1700a, 1700b)) has been performed on the system. Because the occurrence numbers are the same, the nodes (1700a, 1700b) can communicate, because frames issued by one node will contain the same occurrence numbers as the occurrence numbers stored on the other node (1700a, 1700b) and thus will be accepted by the other node (1700a, 1700b).

[0115] In S1810, in response to a detected error event, the reset circuit (1600) of the first processing node (1700a) causes the first processing node (1700a) to be reset. As part of this reset, the state of the various components of the node (1700a) is cleared. The memory of the tiles (4) is wiped by the circuit of the hardware module of the node (1700a), which causes zeros to be written to the memory. As part of the reset, a copy of the occurrence number held in the storage within the node (1700a) is updated. As will be described, the occurrence number may be held in the storage within the interface (1710). The occurrence number is updated by incrementing it with a new value.

[0116] In S1820, following the resetting of the processing node (1700a), the part of the application for which the processing node (1700a) is responsible is restored from the checkpoint. To achieve this, a bootloader program is provided from the hardware module of the node (1700a) to different tiles among the tiles (4). The tiles (4) execute the bootloader program to issue read requests to the checkpoint memory (1720). Although the checkpoint memory (1720) is shown as a single memory unit, it may include multiple memory units for different nodes (1700a, 1700b). In response to the read requests issued by the tiles (4) of the node (1700a), application commands and application data for the checkpoint are returned from the memory (1720) to these tiles (4). The application data returned to the node (1700a) represents the state of a part of the application running on the processing unit (2) of the node (1700a), which was written from the processing unit (2) at the last checkpoint.

[0117] At S1830, the tiles (4) of the processing unit (2) of node (1700a) continue to execute application commands from the checkpoint. At the same time, the second processing node (1700b) has not yet undergone a reset, and the processing unit (2) of the node (1700b) is performing processing of its own part of the application corresponding to the earlier occurrence. Thus, the second processing node (1700b) stores the earlier occurrence number, which is also held in the storage of the first processing node (1700a) prior to the reset at S1810.

[0118] In S1840, the tiles (4) of the processing unit (2) of the node (1700b) are configured to cause one or more data frames to be published to the first processing node (1700a). Each of the one or more data frames contains a copy of an earlier occurrence number held in the storage of the node (1700b). The tiles (4) cause the data frames to be published, which is accomplished by publishing data packets to the EPC of each processing node (1700b), which causes the data packets to be encapsulated within the data frames and sent to the node (1700a).

[0119] At S1850, the first processing node (1700a) receives data frames sent from S1840. The circuit of the first processing node (1700a) checks the occurrence number within each of the data frames and compares it with a copy of the occurrence number held in the storage of the processing node (1700a). If these two occurrence numbers do not match, the circuit of the processing node (1700a) discards the frames. As will be described, the checking and discarding of frames can be performed by the interface (1710) of the first processing node (1700a).

[0120] In S1860, the second processing node (1700b) is reset. This step performed in S1860 is the same as S1810 described earlier, but is performed by the second processing node (1700b) instead of the first processing node (1700a).

[0121] In S1870, following the resetting of the processing node (1700b), the part of the application for which the processing node (1700b) is responsible is restored from the checkpoint. To achieve this, a bootloader program is provided from the hardware module of the node (1700b) to different tiles among the tiles (4). The tiles (4) execute the bootloader program to issue read requests to the checkpoint memory (1720). In response to the read requests issued by the tiles (4) of the node (1700b), application commands and application data for the checkpoint are returned from the memory (1720) to these tiles (4). The application data returned to the node (1700b) represents the state of the part of the application running on the processing unit (2) of the node (1700b), which was written from the processing unit (2) at the last checkpoint.

[0122] In S1880, the processing unit (2) of the second processing node (1700b) continues the execution of its own part of the application from the checkpoint.

[0123] In S1890, the tiles (4) of the processing unit (2) of the node (1700b) are configured to cause one or more data frames to be published to the first processing node (1700a). The tiles (4) cause these additional data frames to be published by publishing data packets to the EPC of each processing node (1700b), which causes the data packets to be encapsulated within data frames and sent to the node (1700a). Unlike the frames sent in S1840, each of these additional one or more data frames contains a copy of an updated version of the occurrence number, which is currently held in the storage of both nodes (1700a, 1700b).

[0124] At S1895, the first processing node (1700a) receives data frames sent from S1890. The circuit of the first processing node (1700a) checks the occurrence number within each of the data frames and compares it with a copy of the occurrence number held in the storage of the processing node (1700a). If these two occurrence numbers match, the circuit of the processing node (1700a) accepts the frames. Data from these frames is written to the tile (4) memory.

[0125] The occurrence number can be inserted at different points within data frames sent between nodes (1700a, 1700b). The occurrence number can be inserted within a packet header (e.g., an Elink packet header) dispatched by the tile (4). Alternatively, the occurrence number can be inserted within the payload of the packet.

[0126] Reference is made to FIG. 19, which illustrates an exemplary embodiment in which a generation number is included within the MAC addresses of frame headers. In the embodiment, the MCPU (310) is responsible for providing MAC addresses for use in sending data over a connection. As part of the MAC addresses, the MCPU (310) provides a generation number. MAC addresses for a connection (including the MAC address of interface (1710) and the MAC address of another interface (1710)) are provided by the MCPU (310) to the connection status register (1915) and the control and status register (CSR) (1916) for storage in these registers (1915, 1916). Specifically, the register (1915) stores the MAC of the destination interface (1710) to which the frames are dispatched, while the CSR (1916) stores the MAC of the interface (1710) sending the frames. In each of these MAC addresses, one or more bits are reserved for inclusion as occurrence numbers within the data frames.

[0127] When the tile (4) has data to send, the tile (4) dispatches this data to the interface (1710) in the form of one or more data packets (e.g., Elink packets). The frame builder (1120) encapsulates the data packets into data frames by adding frame headers and a frame tail. In this way, the data packets issued by the processing unit (2) can be tunneled through Ethernet. The frame builder (1120) inserts the MAC held in the connection status register (1915) as the destination MAC for each frame. The frame builder (1120) inserts the MAC held in the connection status register (1915) as the source MAC for each frame.

[0128] On the receiving side, a frame check buffer (1908) is provided to receive data frames from another interface (1710) belonging to another node. The circuit of the interface (1710) shown in FIG. 19 checks the destination MAC in the received frames against the MAC held in the CSR (1916). If there is a discrepancy between these MACs, the circuit causes the frames to be dropped. If there is a match between the MACs, the circuit causes the data packets encapsulated in these frames to be delivered to the tiles (4) of the processing unit (2) identified in the packet header.

[0129] The MCPU (310) includes at least one processor that executes instructions to provide a visible update function (1920) and an address resolution function (1940). The address resolution function is configured to provide MAC addresses for a connection, and these MAC addresses are provided to the interface (1710). The MAC addresses are local range MAC addresses and, accordingly, are not always fixed for a specific interface (1710) but can be updated by software. The address resolution function (1940) inserts the current generation number held in the generation number register (1910) into each MAC address. The remaining parts of the MAC address can be obtained from the storage (1930) or determined by the function (1940) in another suitable way.

[0130] The MCPU (404) updates the MACs for connection in response to a reset event. The MCPU (404) includes a generation number register (1210) that stores the current generation number, where the current generation number is provided as part of each MAC address output by the address resolution function (1240). The processing circuit of the MCPU (404) supports an update function (1220) that updates the generation number held in the generation number register (1210).

[0131] When a reset of the processing node to which the MCPU (310) belongs (e.g., resets occurring in S1810 and S1860) is performed, indication of such reset is transmitted to the MCPU (310). The update function (1920) responds to this reset event by updating the value of the occurrence number held in the occurrence number register (1910). The update of the occurrence number held in the occurrence number register (1910) may include incrementing the current value of the occurrence number by 1. For example, assume that the occurrence number consists of 5 bits and the current value of the occurrence number held in the register (1910) is given by the bit sequence 00001. In response to the reset event, the update function (1920) updates the occurrence number by incrementing the value by 1 to become 00010.

[0132] Since the occurrence number contains a finite number of bits, when the current value of the occurrence is equal to the maximum possible value, updating the occurrence number involves resetting the occurrence number to the lowest possible value. For example, assume that the occurrence number consists of 5 bits and the current value of the occurrence number held in the register (1210) is given by the bit sequence 11111. In response to the reset event, the update function (1220) resets the occurrence number to the lowest possible value given by 00000. Thus, the occurrence number is updated in a wraparound manner by the update function (1920).

[0133] In response to the indication of a reset event, the address resolution function (1940) determines updated MAC addresses for the connections joined by the device (400). As mentioned, one or more bits are reserved to indicate the occurrence number within each MAC address. The address resolution function (1240) sets these bits to the value of the new occurrence number stored in the register (1910) by the update function. The remaining bits within each MAC address are set to the same values ​​as before the reset event.

[0134] For each MAC address, the remaining bits of the corresponding MAC address may be determined in different ways. In some embodiments, these bits may be held in a storage (1930) accessible to the address resolution function (1240) and may be concatenated with the bits of a new occurrence number to form the entire MAC address. These bits held in the storage (1930) are not wiped in response to a reset event but persist and are not changed following a reset event. In this embodiment, the storage (1930) stores the remaining bits for each MAC address for which the address resolution function (1940) is responsible for determining (i.e., such bits that are different from the bits of the occurrence number). The address resolution function (1940) provides each of these MACs by combining the same occurrence number with the remaining bits for each MAC.

[0135] As mentioned, the application state for the first processing node (1700a) can be restored from a checkpoint, before the application state of other processing nodes (e.g., node (1700b)) among the processing nodes is also restored. In this case, the first processing node (1700a) can continue its application processing while the processing node (1700b) has not yet been reset and contains an application state corresponding to a previous occurrence of the application. This can be made possible by using barrier synchronizations, where the barrier synchronizations separate the computation phase for the participating processing units (2) from the exchange phase for these processing units (2). The barrier synchronizations are inserted into the compiled code executed on each of the processing units (2). For each of the barrier synchronizations, one or more processing units (2) are configured to participate.

[0136] The use of barrier synchronizations prevents the first processing unit (2) of node (1700a) from executing before a point in its code (such a point where, beyond this point, it becomes necessary to exchange data with the processing unit (2) that has not yet undergone restoration and restart from the checkpoint). For example, assume that the processing unit (2) of node (1700a) enters a computation phase after the checkpoint, performing calculations on the data to produce results. The processing unit (2) of node (1700a) does not exchange data with the processing unit (2) of node (1700b) until it reaches a barrier synchronization in which both processing units (2) of nodes (1700a, 1700b) participate. When this barrier synchronization is reached, the processing unit (2) of node (1700a) stops until the processing unit of node (1700b) also reaches a barrier synchronization. In this way, even if the processing unit (2) of node (1700a) reaches barrier synchronization while node (1700b) has not yet undergone reset, the processing unit (2) of node (1700a) will wait until node (1700b) is reset and its processing unit (2) reaches the barrier before continuing to move to the switching phase following barrier synchronization. During the switching phase, the processing unit (2) of node (1700a) sends any application data scheduled to be sent during this switching phase to the processing unit (2) of node (1700b).

[0137] The use of barrier synchronizations prevents the processing units (2) of the system (500) from sending data to any processing units (2) of the system (500) that have not yet undergone resetting. The use of barrier synchronizations in the context of the processing units (2) is described in more detail in our earlier U.S. application No. 17 / 446681, which is incorporated herein by reference.

[0138] It was explained how copies of CEV and GEV are cleared from the processing nodes in response to a reset event (i.e., reset to indicate that there are no events). The bits of the CEV can also be reset by the event handler for an event class. Resetting the bits of the CEV allows another event within the event class to be detected and handled.

[0139] Referring again to FIG. 10, the controller node (740) includes a plurality of registers (1040), referred herein as event handler registration registers (1040). The registers (1040) include event handler registration registers for each of the different event handlers for the processing node (510). For example, there may be event handler registration registers for each of the two host processors (one for the MCPU (310) and one for the ICU). Each of the event handler registration registers (1040) includes an indication of whether or not the respective event handler associated with the register has permission to clear the state for each event class captured within the CEV.

[0140] When the event handler (1510) receives an event notification, the event handler (1510) may decide to clear the event status for the event among other things, thereby enabling any additional occurrences of events within the same event class to be detected. To clear the event status for the event class, the event handler (1510) issues a request to the regulator node (740) via the control bus (700), wherein the request is a request to clear the EDR for the event class. The request includes the identifier of the relevant event handler (1510) and the identifier of the event class. The regulator node (740) examines the permission indication for the event class within the event handler registration register (1040) for the event handler (1510). If the event handler (1510) has permission to clear the event status for the corresponding event class, the regulator node (740) updates the EDR for the corresponding event class to reset the EDR to the default state. In this state, the EDR indicates that no events were detected. Additionally, the regulator node (740) updates the master GEVVR (1030) to reset the event indication for this event class. The regulator node (740) also broadcasts a write request to update copies of the CEV held within the target nodes (720) of the processing node (510) to reset the event indication in the same manner.

[0141] Reference is made to FIG. 20, which illustrates an exemplary method (2000) for aggregating global event states among processing nodes (510). One or more of these steps may overlap each other.

[0142] S2010 is performed by each node (510) within the system (500). In S2010, each processing node (510) within the system (500) stores a copy of the global event vector for the processing system (500). Initially, this vector is set to indicate that no events have occurred within the system (500). The global event vector is held in the master GEVVR (1010) and in the GEVVRs (990) of the target nodes.

[0143] S2020 is performed by each node (510) within the system (500). In S2020, each processing unit (2) within the system (500) executes a set of commands for a distributed application.

[0144] S2030 is performed by each node (510) within the system (500). In S2030, each processing node (510) updates each of one or more elements of at least one copy of the global event vector in response to an event of the type associated with each element, and also in response to an event of the type occurring on one of the processing nodes within the processing system. This step is performed by the controller (740).

[0145] In S2040, on the first processing node among the processing nodes (510), a first event of the first type of event types is detected. This event may be, for example, an event detected by the component (770a) of the node (510a). In response to the detection of the first event, the first processing node among the processing nodes (510) provides one or more other processing nodes among the processing nodes (510) with an indication of an update to the first element among the elements of the global event vector. The first element among the elements is associated with the first type among the types of events.

[0146] S2050 is performed by other nodes in the system (500) (i.e., nodes other than the first node among the nodes (510)) that receive an update provided by the first processing node among the processing nodes (510) in S2040. In S2050, each of these other nodes (510) updates the corresponding first element among the elements in at least one copy of the global event vector in response to the indication of an update to the first element among the elements.

[0147] Reference is made to FIG. 21, which illustrates an exemplary method (2100) for updating copies of event vectors held on a device and using these copies to perform responsive actions. The error event vector may be the CEV or GEV discussed above. The device may be device (300) or device (410).

[0148] In S2110, a copy of the event vector for each of the components (770) within the device (300) is stored. Initially, each copy of the event vector may be configured to indicate that no events have occurred within any of the event classes captured within the event vector.

[0149] In S2120, an event associated with a component (770) is detected in each subset of components (770). The event is reported to the control node (740) in each event report. A subset of components (770) includes one or more components (770).

[0150] In S2130, the control node (740) receives a first event report among event reports from a first component among the components (770), wherein the first event report among the event reports reports a first type of event, wherein the corresponding first type of event was detected by the first component among the components (770).

[0151] In S2140, the control node (740) broadcasts one or more writes to each of the multiple components (770) so that each copy of the error event vector is updated by setting a first element among the elements associated with the first type of event to indicate that the first type of event has occurred on the processing device. These copies are held in the CEVVRs (980) of the target nodes (720).

[0152] In S2150, in the second component among the components (770), an action related to the first type of event is performed in response to the setting of the first element among the elements in its own copy of the error event vector.

[0153] Various functions are described above as being performed by circuits, such as processing logic (930), processing circuit (1010), processing circuit (1220), processing circuit (1400), and circuit (1600). These circuits may include dedicated hardware, such as FPGAs or ASICs configured to execute instructions, and / or processors. Each processing node (510) within the system (500) is assigned a set of instructions to perform any of the operations performed by processors configured to execute instructions.

[0154] The preceding embodiments are described merely as examples.

Claims

Claim 1 A processing system configured to support an application, wherein the processing system comprises a plurality of processing nodes, and each of the plurality of processing nodes comprises: a processing unit configured to execute a set of instructions for a part of the application; and at least one storage configured to store a copy of a number indicating the number of times a global reset of the processing system has been performed; And a reset circuit is included, wherein the reset circuit is configured to cause each processing node to be reset as part of one of the global resets for the processing system and to restore the application state of the part of the application for each processing node from a checkpoint, and each of the processing nodes includes an update circuit configured to update a copy of the number held in the at least one storage of each processing node from a first value to a second value as part of the reset of each processing node, and the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to cause one or more data frames to be sent to the first processing node among the processing nodes before the reset of the second processing node among the processing nodes, each of the one or more data frames includes the first value of the number, and the circuit of the first processing node among the processing nodes, following the reset of the first processing node among the processing nodes,A processing system configured to discard data within the one or more data frames in response to determining a mismatch between a first value of the number within the one or more data frames and a second value of the number held in at least one storage of the first processing node among the processing nodes, wherein the processing unit of the first processing node among the processing nodes is configured to stall until an additional processing unit among the processing units, which is also configured to participate in the first barrier synchronization in the first barrier synchronization following the resetting of the first processing node among the processing nodes, reaches the first barrier synchronization in application processing. Claim 2 A processing system according to claim 1, wherein for each of the processing nodes, each processing unit is configured to participate in a plurality of barrier synchronizations indicated in a set of instructions, and for each of the processing nodes, each processing unit is configured to pass each barrier synchronization in response to at least one additional processing unit among the processing units in the processing system also participating in each barrier synchronization. Claim 3 A processing system according to paragraph 2, wherein for each of the processing nodes, each of the plurality of barrier synchronizations separates the compute phase of each processing unit from the exchange phase of each processing unit. Claim 4 delete Claim 5 A processing system according to claim 1, characterized in that the additional processing unit among the processing units is the processing unit of the second processing node among the processing nodes. Claim 6 A processing system according to claim 1 or 5, wherein the first barrier synchronization operates as a barrier between a computation phase and an exchange phase, and the processing unit of the first processing node among the processing nodes is configured to execute a set of application instructions to perform computations during the computation phase until the first barrier synchronization is reached, and to exchange data with the second processing node among the processing nodes during the exchange phase following the first barrier synchronization, and wherein the exchanged data includes data contained within the one or more data frames. Claim 7 A processing system according to claim 1, wherein the circuit of the first processing node among the processing nodes configured to determine the discrepancy is a part of the interface of the first processing node among the processing nodes for interfacing with the second processing node among the processing nodes. Claim 8 A processing system according to claim 7, wherein the interface of the first processing node among the processing nodes is configured to receive additional data frames from the second processing node among the processing nodes and to provide data packets encapsulated within the additional data frames to the processing unit of the first processing node among the processing nodes. Claim 9 A processing system according to claim 1, wherein the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to cause one or more additional data frames to be sent to the first processing node among the processing nodes following a reset of the second processing node among the processing nodes, each of the one or more additional data frames includes the second value of the number, and the circuit of the first processing node among the processing nodes is configured to receive the one or more additional data frames and to accept data within the one or more data frames in response to determining a match between the second value of the number in each of the one or more additional data frames and the second value of the number held in at least one storage of the first processing node among the processing nodes. Claim 10 A processing system according to claim 1, characterized in that the circuit of the first processing node among the processing nodes is configured to determine the discrepancy as part of checking the address field within each of the data frames. Claim 11 A processing system according to claim 1, wherein for each of the processing nodes, each update circuit comprises a processor configured to execute computer-readable instructions to perform the update of the copy of the number held in the at least one storage of each processing node. Claim 12 A processing system according to claim 1, characterized in that updating the copy of the number includes incrementing the first value to obtain the second value. Claim 13 A processing system according to claim 1, characterized in that the copy of the number forms a part of the connection state for the connection between the first processing node among the processing nodes and the second processing node among the processing nodes. Claim 14 A processing system according to claim 1, wherein each processing node comprises an integrated circuit including the processing unit; and the at least one storage for each processing node. Claim 15 A processing system according to claim 1, wherein the processing unit of the second processing node among the processing nodes is configured to execute a set of instructions to output the first value of the number within one or more data packets of the one or more data frames. Claim 16 A processing system according to claim 1, wherein the processing unit of the second processing node among the processing nodes is configured to execute a set of commands to output data for the one or more data frames to an interface circuit, and the interface circuit is configured to provide the first value of the number inserted into each frame to the one or more data frames. Claim 17 As a method, the method comprises: executing a set of instructions for a portion of an application supported by a system comprising the plurality of processing nodes on each of the plurality of processing nodes; storing on each of the processing nodes a copy of a number indicating the number of times a global reset of the processing system has been performed; performing one of the global resets, wherein performing one of the global resets comprises resetting each of the plurality of processing nodes, wherein resetting each of the plurality of processing nodes comprises updating the copies of the number held on each of the processing nodes from a first value to a second value, and restoring the application state for the portion of the application for each processing node from a checkpoint; sending one or more data frames from the second processing node among the processing nodes to the first processing node among the processing nodes before resetting the second processing node among the processing nodes, wherein each of the one or more data frames contains the first value of the number; and subsequently, after resetting the first processing node among the processing nodes, and the one or more data A method characterized by comprising: discarding data within one or more data frames in response to determining a discrepancy between the first value of the number within the frames and the second value of the number held at the first processing node among the processing nodes; and, following the resetting of the first processing node among the processing nodes, stopping an additional processing unit among the processing units also configured to participate in the first barrier synchronization in the application processing until the first barrier synchronization is reached. Claim 18 A non-transient computer-readable medium for storing a computer program comprising a set of executable instructions for each of a plurality of processing nodes forming a data processing system, wherein each of the sets of executable instructions, when executed by at least one processor of each processing node, each processing node supports a portion of a distributed application, wherein the distributed application is supported by the plurality of processing nodes; and storing a copy of a number indicating the number of times a global reset of the processing system has been performed;And as part of a global reset for the data processing system, resetting each processing node, wherein resetting each processing node comprises updating the copy of the number held on each processing node from a first value to a second value, and restoring the application state of the part of the distributed application for each processing node from a checkpoint, and a second set of executable instructions is configured such that when executed by at least one processor of the second processing node among the processing nodes, the second processing node among the processing nodes sends one or more data frames to the first processing node among the processing nodes before the resetting of the second processing node among the processing nodes, each of the one or more data frames includes the first value of the number, and a first set of executable instructions is configured such that when executed by at least one processor of the first processing node among the processing nodes, the first processing node among the processing nodes A non-transient computer-readable medium characterized by being configured to discard data within one or more data frames in response to determining a discrepancy between the first value of the number within the one or more data frames and the second value of the number stored by the first processing node among the processing nodes, following the resetting of the first processing node among the processing nodes; and to cause an additional processing unit among the processing units also configured to participate in the first barrier synchronization to stop until the first barrier synchronization is reached in the application processing, following the resetting of the first processing node among the processing nodes.

Images