On-chip router

Abstract

A router for routing data in a computing chip having multiple processing elements is disclosed, the router having a packet processing pipeline, a drop packet buffer, and one or more circuits configured to determine that a data packet in the packet processing pipeline should be dropped; move the to-be-dropped data packet from the packet processing pipeline to the drop packet buffer; and re-insert the dropped data packet from the drop packet buffer into the packet processing pipeline for reprocessing.

Description

[Technical Field]

[0001] This specification relates to a router for routing data in a computing chip having multiple processing elements and a corresponding method for processing data packets by the router. [Background technology]

[0002] Many of today's more powerful computers, including those used in demanding scientific applications such as weather forecasting, drug discovery, neural simulation, and self-driving vehicles, use large numbers of processors connected to each other through some kind of inter-processor network. It is crucial for these machines that their communication networks do not deadlock as a result of circular dependencies between packets of information traveling through the network. A deadlock is a situation in which packet A cannot proceed because its path is blocked by packet B, and packet B cannot proceed because its path is blocked by packet A. More than two packets may be involved in such circular dependencies, but the principle is the same. Summary of the Invention

[0003] To avoid the risk of circular dependencies resulting in deadlock, it is common to design such networks so that circular dependencies never arise. This can be achieved through the use of multiple network layers, or the use of virtual channels, or other known techniques. While such techniques are effective in eliminating any possibility of circular dependencies, they incur significant additional cost in terms of the hardware required to support the additional communication functionality. Therefore, it is desirable to provide an improved means of preventing deadlock in inter-processor networks.

[0004] According to a first aspect, there is provided a router for routing data in a computing chip having a plurality of processing elements, the router having a packet processing pipeline, a drop packet buffer, and one or more circuits configured to: determine that a data packet in the packet processing pipeline should be dropped; move the data packet to be dropped from the packet processing pipeline to the drop packet buffer; and re-insert the dropped data packet from the drop packet buffer into the packet processing pipeline for reprocessing.

[0005] As mentioned above, conventional methods attempt to design inter-processor networks so that circular dependencies that could cause deadlocks never occur. However, such techniques incur significant additional costs in terms of the hardware required to support the required additional communication functionality. The present invention enables even simple networks to be deadlock-free by allowing potentially blocking packets to be temporarily removed (“dropped”) from the network fabric and held in a buffer, thereby eliminating any circular dependencies. The packets can then be reinserted into the network and delivered normally. Using a hardware mechanism for packet dropping and reinsertion allows the network to maintain good throughput, even when congested, thereby ensuring fast and reliable packet delivery over a low-cost network fabric.

[0006] In some other prior art methods, the process for dropping data packets is implemented in software. For example, in one prior art method, a processing element on a chip is designated as a monitor processor. If it is determined that a packet should be dropped, an interrupt signal is sent to the monitor processor. The monitor processor can then copy the dropped packet and resend the copied packet. However, this process is slow considering the time required for the monitor processor to service the interrupt. The latency incurred to copy and resend the packet on the network further increases the time required to process dropped packets. The present invention provides a hardware-based packet discard and reinsertion mechanism as part of the router. Thus, the time required to process dropped packets is significantly reduced, increasing router throughput during congestion. Because packet discard and reinsertion is performed entirely within the router, there is no need to send dropped packets to the monitor processor and then resend them back to the router, and this process is also more power efficient. Furthermore, because a monitor processor is not required, the chip's processing elements can be fully specialized to perform their respective tasks.

[0007] It will be understood that a router may route data packets between processing elements on the same chip, and that chips may be connected such that data packets may be routed between processing elements on different chips. It will also be understood that a router includes a hardware-based packet discard and reinsertion mechanism that includes a drop packet buffer and one or more circuits.

[0008] The one or more circuits may include a packet reinsertion module including a drop packet buffer, the packet reinsertion module configured to reinsert dropped data packets from the drop packet buffer into the packet processing pipeline for reprocessing.

[0009] The one or more circuits may be configured to determine that a data packet in the packet processing pipeline should be dropped based on a signal indicating that the destination of the data packet is unable to receive the data packet. It will be understood that the destination need not necessarily be the final destination of the data packet; rather, the destination may include an intermediate node in an on-chip data network or may be an output port of a router. The data packet may be dropped to avoid blocking the packet processing pipeline.

[0010] The packet processing pipeline may further include an out-of-order packet buffer, and the one or more circuits may be configured to move data packets to be dropped from the out-of-order packet buffer to a drop packet buffer. The one or more circuits may be further configured to select a data packet for processing by the packet processing pipeline; determine one or more destinations for the data packet; determine that one of the one or more destinations of the data packet is unable to receive the data packet; and move the data packet to the out-of-order packet buffer. The out-of-order packet buffer may allow stuck data packets, i.e., data packets that have not yet been transmitted, to be queued so that the packet processing pipeline can continue processing additional packets without having to wait for the stuck data packet to be dropped before continuing. If a data packet cannot be sent, it may be stored in an out-of-order packet buffer until such time as the data packet can be sent to its destination, or it may be removed from the out-of-order packet buffer and dropped.

[0011] The out-of-order packet buffer may be a pipeline queue. If a new data packet is stuck, an existing stuck data packet in the out-of-order packet buffer may be moved further in the queue to accommodate the new stuck data packet. The data packet to be dropped may be the oldest data packet in the out-of-order packet buffer.

[0012] The one or more circuits may be configured to determine that a data packet in the packet processing pipeline should be dropped based on the expiration of a timer. For example, if an out-of-order packet buffer is used, the timer may be started when the data packet reaches the end slot of the buffer. This can ensure that the packet processing pipeline is not unduly delayed, indicating that the buffer is full and the data packet is to be dropped. If an out-of-order packet buffer is not used, the timer may be started when it is determined that the data packet cannot be sent. The timer provides a minimum latency period so that packets are not dropped unnecessarily.

[0013] The one or more circuits may be configured to reinsert dropped data packets from the drop packet buffer into the packet processing pipeline for reprocessing based on a timer. For example, the timer may be started when the packet is moved to the drop packet buffer and the drop packet buffer is no longer empty. The timer may be reset when the dropped packet is removed from the drop packet buffer and reinserted into the packet processing pipeline. The timer may be restarted if the drop packet buffer is not empty. The timer thereby enforces a minimum time interval between reinsertion of dropped packets. The minimum time interval can be dynamically set based on the current load of the drop packet buffer and / or the current state of the router and / or the current state of the on-chip network.

[0014] The one or more circuits may be configured to reinsert dropped data packets from the drop packet buffer into the packet processing pipeline for reprocessing based on the load of the packet processing pipeline. For example, a reinsertion request may be sent to the packet processing pipeline, and the reinsertion is performed if the request is granted. The granting of the request may be based on the load of the packet processing pipeline, and / or the presence of a free slot at the beginning of the pipeline, and / or the absence of new data packets arriving for processing, and / or on a periodic or alternating basis with respect to newly arriving data packets, and / or other conditions deemed appropriate by those skilled in the art. The one or more circuits may include a reinsertion request unit as part of the packet reinsertion module for processing the reinsertion request, and the reinsertion request unit may be connected to a reinsertion timer that indicates when a reinsertion request should be generated.

[0015] The router may be configured to perform software-based packet offload processing to remove data packets stored in a drop packet buffer for reprocessing outside the router. For example, if the drop packet buffer is full or approaching full, a software-based mechanism may be used to remove packets from the drop packet buffer for storage elsewhere on the chip, such as in local memory associated with a processing element. The removed packets may then be retransmitted over the on-chip network. The router may include a register in a general register file from which to read and initiate a software-based packet offload operation. The general register file may be configured as an intermediate storage location for storing packets to be dropped before copying them from the router to other locations on the chip.

[0016] The packet processing pipeline may be configured to process multicast data packets. The router may include multiple packet processing pipelines. Each packet processing pipeline may be configured to process a particular type of data packet, e.g., multicast packets or system type packets. A drop packet buffer may be configured to service all of the packet processing pipelines. One or more circuits may include an arbiter for selecting between drop packet requests of different packet processing pipelines. Alternatively, there may be multiple drop packet buffers, each serving one or more of the multiple packet processing pipelines.

[0017] The drop packet buffer may be configured to store a plurality of dropped data packets. The drop packet buffer may include one or more static RAM (SRAM) modules. The drop packet buffer may further include a packet disassembly buffer for copying dropped packets to the drop packet buffer and a packet reassembly buffer for removing dropped packets from the drop packet buffer.

[0018] The computing chip may be a neuromorphic chip. A neuromorphic chip is a specialized chip for performing biological neural simulations. In biological neural networks, processing is achieved by the firing of electrochemical impulses between neurons in the network. Information is contained in the rate and timing of these impulse firings. This can be simulated using one or more neuromorphic chips, whereby the processing elements of the neuromorphic chip are configured to simulate the activity of individual neurons. Data packets representing the impulses are sent between neurons (processing elements) at appropriate times and speeds. Therefore, given the potentially large-scale nature of the simulations and the desire for biological real-time simulations, effective management of data traffic on and between chips is critical.

[0019] Furthermore, data packets representing neural communications may be sent as multicast packets, taking into account the one-to-many nature of neural connections. To simplify multicast packet delivery, an all-or-nothing policy may be used, such that a packet must be deliverable to all destinations or it is not sent at all. Such a policy may result in additional latency before the data packet is sent. Furthermore, multicasting packets may require generating multiple copies of the same packet, which may introduce delays within routers and may also affect router throughput.

[0020] Furthermore, components on a neuromorphic chip may operate at different speeds, and limitations on transmission rates from the router to various components may be necessary to avoid overwhelming these components. For example, a processing element may not be able to process packets as quickly as the router can transmit them. Thus, there may be limitations on how quickly data packets can be transmitted from the router to the same processing element. In another example, neuromorphic chips may be connected together to increase available processing power. Packets may be transmitted to processing elements on different chips via inter-chip links. The inter-chip links may operate slower than the router, and therefore limitations may exist on transmitting packets from the router via the inter-chip links, which may further affect the router's throughput.

[0021] Network congestion and the inability to maintain sufficient router throughput can also result in significant performance degradation for on-chip networks. For example, congestion at a particular node can prevent packets from being sent out of a router, stalling the packet processing pipeline. This can cause further congestion as packets queue up at nodes in the network to the point where no more packets can be sent, considering that blocked nodes may depend on other blocked nodes for routing and a deadlock situation may occur. Congestion at a node can occur if the node cannot process packets as quickly as they are being received or if the node crashes. In another example, network congestion can delay the timely transmission of system / configuration-type packets, slowing down or even crashing the system.

[0022] The present invention is particularly suited for use with neuromorphic chips, given the provision of a hardware mechanism for packet dropping and reinsertion, as described above, that is capable of maintaining good throughput even in the presence of network congestion. It will be appreciated that the above description is also applicable more generally to computing chips such as those used in high performance computing applications (HPC).

[0023] According to another aspect, there is provided a method for routing data in a computing chip having a plurality of processing elements, the router having a packet processing pipeline, a drop packet buffer, and one or more circuits, the method including: determining, by the one or more circuits, that a data packet in the packet processing pipeline should be dropped; moving, by the one or more circuits, the data packet to be dropped from the packet processing pipeline to the drop packet buffer; and re-inserting, by the one or more circuits, the dropped data packet from the drop packet buffer into the packet processing pipeline for reprocessing.

[0024] The method may correspond to operation of one or more circuits of the first aspect. The step of determining that the data packet in the packet processing pipeline should be dropped may be based on a signal indicating that a destination of the data packet is unable to receive the data packet.

[0025] The method may further comprise the step of moving the data packets to be dropped from the out-of-order packet buffer to a drop packet buffer.

[0026] The method may further include selecting a data packet for processing by the packet processing pipeline; determining one or more destinations for the data packet; determining that a destination of the one or more destinations of the data packet is unable to receive the data packet; and moving the data packet to an out-of-order packet buffer.

[0027] The data packet to be dropped may be the oldest data packet in the out-of-order packet buffer.

[0028] The step of determining that a data packet should be dropped in the packet processing pipeline may be based on the expiration of a timer.

[0029] Reinserting the dropped data packets from the drop packet buffer into the packet processing pipeline for reprocessing may be based on a load on the packet processing pipeline.

[0030] The method may further include removing the data packets stored in the drop packet buffer in response to a software-based packet offload process.

[0031] The packet processing pipeline may be configured to process multicast data packets.

[0032] The drop packet buffer may be configured to store multiple dropped data packets.

[0033] The drop packet buffer may include one or more SRAM modules.

[0034] The computing chip may be a neuromorphic chip.

[0035] It will be understood that embodiments may be combined, and that features described in the context of one embodiment may be combined with other embodiments. [Brief explanation of the drawings]

[0036] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] Figure 1 is a schematic diagram of the neuromorphic chip. [Figure 2] Figure 2 is a schematic diagram of the network topography on the neuromorphic chip. [Figure 3] Figure 3 is a schematic diagram of the packet processing pipeline of the neuromorphic-on-chip router. [Figure 3A] FIG. 3A is a schematic diagram of a packet reinsertion module. [Figure 3B] Figure 3B is a schematic diagram of the drop packet buffer. [Figure 4] 10 is a flowchart illustrating a process performed by a router. [Figure 5] FIG. 5 is a flow chart showing in more detail the processing performed by the router. [Figure 6] FIG. 6 is a flowchart illustrating an exemplary process for determining whether to drop a data packet. [Figure 7] FIG. 7 is a flowchart illustrating an alternative example process for determining whether to drop a data packet. [Figure 8] FIG. 8 is a flowchart showing the process for handling dropped data packets. DETAILED DESCRIPTION OF THE INVENTION

[0037] Referring now to Figure 1, there is shown a schematic diagram of a neuromorphic chip 100. The neuromorphic chip 100 comprises a plurality of processing elements 101. While Figure 1 shows a chip with eight processing elements, it will be understood that there may be a greater (or lesser) number of processing elements on the chip 100. For example, one exemplary neuromorphic chip has 152 processing elements.

[0038] Each processing element may include a processor core, local memory, and modules for sending and receiving data. Data may be communicated between processing elements 101 of chip 100 and / or between processing elements of different chips. Chip 100 may include multiple inter-chip links 102 in this regard. External devices may also be connected to chip 100 via external I / O interface 103.

[0039] Chip 100 additionally includes a primary router 105. Primary router 105 is configured to direct designated types of on-chip data traffic, such as multicast data packets and all inter-chip data traffic. Data is communicated via communication links on chip 100 that connect different components of chip 100 and form a network-on-chip (NoC). If data is to be communicated with components on other chips, primary router 105 can communicate with primary routers on other chips via inter-chip links 102.

[0040] The primary router 105 includes a packet processing pipeline configured to receive data packets sent to the primary router 105 for processing, determine a destination for the data packets, determine an appropriate routing path for the data packets, and transmit the data packets to the determined destination via the determined routing path.

[0041] When a data packet is ready to be sent down the pipeline to its destination, it may not be possible to send the data packet at that time due to factors such as network congestion. For example, neuromorphic chip 100 may be used to perform neural simulations. Each processing element of chip 100 may be used to simulate a large number of neurons; for example, one processing element may simulate 1,000 neurons. Therefore, congestion may occur at router 105 if communications are directed to a large number of neurons on the same processing element, or if a large amount of communications is directed to a small number of neurons on a processing element. Furthermore, a single neuron is typically connected to many other neurons. This is reflected in the use of multicasting for communication between neurons; i.e., one data packet sent from a processing element to router 105 results in the generation of many data packets sent from router 105, thereby increasing network traffic. Furthermore, to simplify the multicast packet delivery mechanism, the router may operate an all-or-nothing policy, whereby a multicast packet is only transmitted from router 105 if all destinations are available to receive the multicast packet. This policy avoids the need for mechanisms to track multicast packet delivery and formulate potentially complex retransmission policies, thereby reducing chip storage and power requirements. However, an all-or-nothing policy increases the probability that packets will be unable to be transmitted and therefore stall at the end of the packet processing pipeline. This can cause the packet processing pipeline to be blocked until the stalled packet can be transmitted.

[0042] The router 105 provides a hardware-based mechanism for handling stalled packets and maintaining throughput of the router 105. In this regard, the router 105 further includes one or more circuits within the router 105 configured to determine that a data packet in the packet processing pipeline should be dropped, and a drop packet buffer. The one or more circuits are further configured to move the data packet to be dropped from the packet processing pipeline to the drop packet buffer. The one or more circuits are additionally configured to reinsert the dropped data packet from the drop packet buffer into the packet processing pipeline for reprocessing. Each of these operations is described in further detail below with reference to FIG. 3, which illustrates an exemplary packet processing pipeline including a drop packet buffer 306.

[0043] In some prior art approaches, the process for handling stuck packets is implemented in software. For example, in one prior art approach, a processing element on chip 100 is designated as a monitor processor. If a packet is to be dropped, an interrupt signal is sent to the monitor processor. The monitor processor can then copy the dropped packet and resend the copied packet. However, this process is slow given the time required for the monitor processor to service the interrupt. The latency incurred to copy and resend the packet on the network further increases the time required to process the dropped packet. By providing a hardware-based packet discard and reinsertion mechanism as part of router 105, the time required to process dropped packets is significantly reduced, thereby increasing the throughput of router 105 under congestion conditions. This process is also more power efficient because packet discard and reinsertion is performed entirely within router 105 and does not require dropped packets to be sent to the monitor processor or resent to the router. Furthermore, because a monitor processor is not required, the chip's processing elements can be fully specialized to perform their respective tasks. An analysis of the throughput of the router 105 is provided in more detail below.

[0044] Referring now to Figure 2, a schematic diagram of an exemplary network topology for an NoC is shown. In Figure 2, the NoC is configured as a 2D mesh network. This type of topology provides a more scalable solution compared to other network topologies, such as a star topology, as the number of processing elements on a chip increases.

[0045] The processing elements on the chip are arranged in groups of four as Quad Processing Elements (QPEs) 201. Each QPE is associated with a Network-on-Chip (NoC) router 202 configured to direct traffic to the QPE's 201's corresponding processing element 101 and forward data packets to the appropriate neighboring NoC router 202 in the mesh network. However, the routing of data packets is controlled by the primary router 105. Thus, all data packets sent from a processing element are sent to the primary router 105, possibly via several NoC routers 202, which then determines the data packet's one or more destinations and the optimal route to the one or more destinations.

[0046] It will be appreciated that other configurations of processing elements and network topologies may be used as deemed appropriate by those skilled in the art. It will also be appreciated that Figure 2 shows a partial view of an NoC with components omitted for clarity.

[0047] The operation of router 105 will now be described in more detail. Figure 3 is a schematic diagram of the flow of data packets through router 105. While Figure 3 shows one packet processing pipeline, it will be appreciated that router 105 may include multiple packet processing pipelines. In particular, each packet processing pipeline may be responsible for processing a particular type of data packet, e.g., multicast packets or system type packets.

[0048] 3, data packets for processing by router 105 may be received via parallel data streams 301. Because router 105 is configured to process data packets for all processing elements 101, router 105 may include multiple input ports, each configured to correspond to a particular region of chip 100 or one or more inter-chip links 102. Thus, each input port may be configured to provide an input data packet stream to router 105.

[0049] Arbiter 302 is configured to select a data packet from input data stream 301 for processing. For example, arbiter 302 may be configured to perform the selection in a round robin manner, selecting a data packet in turn from each input data stream 301. However, it will be appreciated that alternative selections may be used as would be appreciated by one skilled in the art to be suitable.

[0050] The routing engine 303 is configured to process selected data packets to determine one or more destinations for the data packets and appropriate routing paths through the NoC to the one or more destinations. The one or more destinations may be determined based on information included in the data packets. For example, multicast data packets may be source routed. That is, the data packets may include an identifier of the source of the data packets, such as a neuron identifier. The router 105 may further include an association memory 308 configured to map neuron identifiers according to the connectivity of neurons in a running neural simulation. The routing engine 303 can access the association memory 308 based on the neuron identifier to determine the output connection of the corresponding neuron and which processing element and neuron the data packet is intended for. Alternatively, the data packets may include information indicating the destination of the data packets. For example, system / configuration type data packets may use destination routing, while application (neural simulation) type data packets may use source routing. Although Figure 3 shows association memory 308 as separate from routing engine 303, association memory 308 may be part of routing engine 303.

[0051] Once processing of the data packet by the routing engine 303 is complete, the data packet can be transmitted to one or more determined destinations. The router 105 may further include multiple output ports 304 for transmitting the data packet. Like the multiple input ports, each output port may be configured to correspond to a particular region of the chip 100 or to one or more inter-chip links 102. However, as described above, the data packet may not currently be transmitted. For example, if the destination of the data packet is not currently able to receive the data packet. It will be appreciated that a destination may not necessarily be the final destination of the data packet; rather, the destination may include any point along the data packet's transmission path, beginning at the end of the packet processing pipeline. For example, transmission of the data packet from the router 105 may be via a particular output port 304. However, the particular output port 304 may be unavailable, and therefore the data packet cannot be transmitted at the current time. In another example, an intermediate node in the network, such as the NoC router 202, may be unavailable, thereby preventing the data packet from being transmitted. To avoid blocking the packet processing pipeline, it is possible to drop data packets that cannot be sent.

[0052] A data packet may be dropped based on a signal indicating that the destination of the data packet is unable to receive the data packet. For example, output port 304 of router 105 may provide a signal indicating that it is currently unavailable to process the data packet. Similarly, other nodes on the NoC may provide signals indicating their availability.

[0053] Instead of immediately dropping the data packet, one or more circuits of router 105, such as routing engine 303, may be configured to wait a specified period of time before determining that the data packet should be dropped. If, during this waiting period, the blocked destination becomes available, the data packet may be transmitted as normal without being dropped. Otherwise, if the waiting period expires without the blocked destination becoming available, the data packet may be dropped.

[0054] This waiting period may be implemented using a timer 309. The waiting period may be set based on the type of data packet, with different types of data packets having different waiting times. The waiting period may also be dynamically configurable based on the current state of the router 105 and the NoC.

[0055] The router 105 may also include an out-of-order packet buffer 305 that allows it to queue stuck data packets so that the packet processing pipeline can continue processing further packets without having to wait for the stuck data packet to be dropped before continuing. In this case, if a data packet cannot be sent, it should either be moved to the out-of-order packet buffer 305 or removed from the out-of-order buffer 305 to be dropped, at which point the data packet can be sent to its destination.

[0056] For example, out-of-order packet buffer 305 may be implemented as a pipeline queue. That is, a new stalled data packet may be inserted at the start of the queue, pushing an existing stalled data packet in out-of-order packet buffer 305 toward the end of the queue along with the next slot in the queue. If at any time a destination that prevents a particular packet from being transmitted becomes available, that packet may be removed from out-of-order packet buffer 305 and transmitted to its destination via the associated output port 304. Thus, in this regard, data packets may be transmitted out of order compared to the order of processing by the packet processing pipeline. If multiple packet destinations become available for transmission simultaneously, the oldest data packet, i.e., the data packet furthest toward the end of the queue, may be selected from out-of-order packet buffer 305 for transmission and removal.

[0057] A timer 310 may be triggered when the stuck data packet reaches an end slot of the out-of-order packet buffer 305. If the stuck data packet remains in an end slot of the out-of-order packet buffer 305 upon expiration of the timer 310, i.e., if the stuck data packet still cannot be transmitted, it may be determined that the stuck data packet in the end slot should be dropped.

[0058] Out-of-order packet buffer 305 may be configured such that a newly stalled data packet is only inserted into out-of-order packet buffer 305 if an entry slot into out-of-order packet buffer 305 is available. The newly stalled data packet can trigger a signal configured to push existing packets forward in out-of-order packet buffer 305 to make space for the newly stalled data packet. If out-of-order packet buffer 305 is full, this may not be possible, and the new stalled data packet must wait at the output of routing engine 303, which may temporarily stall the packet processing pipeline.

[0059] In an alternative example, the out-of-order packet buffer 305 may include multiple out-of-order slots. A newly stuck packet may be stored in an available slot, triggering a timer associated with that particular slot. If the stuck packet cannot be transmitted when the timer expires, it may determine that the stuck packet should be dropped. Thus, in this example, the slots in the out-of-order packet buffer 305 are not linked. However, in the previous example, stuck packets may be given a longer time to still be transmitted before being dropped because they are only dropped when pushed to the end slot of the out-of-order packet buffer 305, which initiates a timeout. How quickly stuck packets are dropped may depend on the rate at which processed data packets become stuck.

[0060] If it is determined that a data packet should be dropped, one or more circuits of the router 105 are configured to move the data packet from the packet processing pipeline (e.g., from the routing engine 303 or from the out-of-order packet buffer 305, if in use) to a drop packet buffer 306. The drop packet buffer 306 may be part of a packet reinsertion module 307 of the router 105. The dropped packet can then be reinserted from the drop packet buffer 306 into the packet processing pipeline.

[0061] It will be appreciated that if multiple packet processing pipelines are present, the drop packet buffer 306 may service all of the packet processing pipelines. For example, the arbiter may be configured to select one drop packet request from the multiple packet processing pipelines for processing by the packet reinsertion module 307 if more than one drop packet request is outstanding. The selection of the drop packet request to process may follow any arbitration scheme as recognized by those skilled in the art as appropriate. For example, a round-robin scheme may be used, or the drop packet requests may be prioritized according to the status of the individual packet processing pipelines or on a first-come, first-served basis. As an alternative to a single drop packet buffer 306, there may be multiple drop packet buffers 306, each serving one or more of the multiple packet processing pipelines.

[0062] As briefly mentioned above, each of the packet processing pipelines may be configured to process different types of data packets, and each packet processing pipeline may have its own routing engine configured to process a particular packet type in a particular manner. For example, as described above, packets representing neural communications in a neural simulation may be multicast and routed through a source. A packet processing pipeline and routing engine may be dedicated to processing multicast packets.

[0063] Another packet processing pipeline and routing engine may be configured to process "core-to-core" type packets. These packets primarily provide communication between processing elements for system management and control purposes and may be destination routed. Further packet processing pipelines and routing engines may be configured to process "nearest-neighbor" type packets. These packets are intended for system initialization and debug purposes and provide communication between monitor processors on adjacent chips.

[0064] Referring now to FIG. 3A, an exemplary packet reinsertion module 307 is shown in more detail. The packet reinsertion module 307 may be configured to process drop packet storage requests, if in use, from either the routing engine 303 or the out-of-order packet buffer 305, either drop packet reinsertion requests or software-based packet offload requests. The router 105 may be configured to perform software-based packet offload processing to remove data packets stored in the drop packet buffer for reprocessing of the data packets outside the router. For example, if the drop packet buffer is full or approaching full, a software-based mechanism may be used to remove packets from the drop packet buffer for storage elsewhere on-chip, as described in more detail below. Referring again to FIG. 3A, it will be appreciated that the packet reinsertion module 307 may process other types of requests not listed above.

[0065] The arbiter 311 may be configured to select one of the requests to process when there are multiple pending requests. In one example, the selection is prioritized in the following order: drop packet storage requests, software-based offload requests, and finally, drop packet reinsertion requests. Alternatively, a different arbitration scheme may be used as recognized by those skilled in the art.

[0066] When a drop packet storage request is selected for processing, the packet to be dropped may be copied to the drop packet buffer 306, as described above. The packet reinsertion module 307 may further include a standby register buffer 312. The packet reinsertion module 307 may be configured to store dropped packets in the standby register buffer 312 in the event of a failure in the drop packet buffer 306. The standby register buffer 312 may be configured to store a single dropped packet. The packet reinsertion module 307 may be configured to not overwrite dropped packets stored in the standby register buffer 312 before the dropped packets are processed. Dropped packets in the standby register buffer 312 may be handled using a software-based mechanism. For example, an interrupt may be sent to a monitor processor, which then transfers the dropped packets from the standby register buffer 312 to another memory location for retransmission. This process is similar to software-based packet offloading, which is described in more detail below.

[0067] The reinsertion of a dropped packet stored in drop packet buffer 306 into the packet processing pipeline may be timer-based. For example, a reinsertion timer 313 may be started when a packet is stored in drop packet buffer 306, causing drop packet buffer 306 to become non-empty. When reinsertion timer 313 expires, a request signal from reinsertion request unit 314 may be sent to the packet processing pipeline requesting permission to reinsert the packet. If the pipeline can handle the reinsertion of the packet, the request may be granted. For example, a reinsertion request may be granted if there are free slots at the beginning of the pipeline, and / or if there are no new data packets arriving from input data stream 301, and / or in a periodic or alternating fashion with newly arriving data packets, and / or under other conditions deemed appropriate by those skilled in the art.

[0068] If the reinsertion request is granted, the dropped packet in the drop packet buffer 306 may be reinserted into the packet processing pipeline and removed from the drop packet buffer 306. In one example, the reinsertion request unit 314 may be configured to receive a signal indicating grant of the reinsertion request and then send a request to the arbiter 311 of the packet reinsertion module 307 to perform the packet reinsertion. If the request is selected by the arbiter 311 for processing, the dropped packet may be removed from the drop packet buffer and reinserted into the packet processing pipeline. The reinserted packet may then be reprocessed by the packet processing pipeline in the same manner as described above. It is possible to provide the reinserted packet with its original destination information so that it does not need to be re-determined upon reprocessing. However, doing so may incur undesirably large overhead, which may not be as preferable as simply re-determining the packet's destination. When a dropped packet is removed from the drop packet buffer 306, the reinsertion timer 313 may be reset. If the drop packet buffer 306 contains more dropped packets, the reinsertion timer may be started again, and the reinsertion process may be repeated until the drop packet buffer 306 is cleared. The reinsertion timer 313 thereby enforces a minimum time interval between reinsertions of dropped packets. The actual period between reinsertions may be variable because a packet may not be reinserted until the corresponding reinsertion request is granted. The minimum time interval may be dynamically set based on the current load of the drop packet buffer 306.

[0069] 3B, an exemplary implementation of the drop packet buffer 306 will be described. The drop packet buffer 306 may include one or more SRAM modules and may be configured to store multiple dropped data packets. More specifically, the dropped packets may be stored in an SRAM buffer 318. For example, the SRAM buffer 318 may be implemented using a single-port SRAM macro with data packets ranging in size from 64 bits to 192 bits.

[0070] Packets 317 stored in SRAM buffer 318 may first be written to packet disassembly buffer 319. Packet disassembly buffer 319 may be implemented, for example, as six parallel 32-bit registers to allow the largest possible packet size to be written to packet disassembly buffer 319 in one cycle. The packets may then be transferred to SRAM buffer 318, which may take several cycles depending on the width of the input to SRAM buffer 318. Timers 309 and 310, which control when packets are dropped from the packet processing pipeline, may be reset when the packet is copied to packet disassembly buffer 319. Thus, the transfer of a packet to SRAM buffer 318 may occur during a packet discard timeout period. If the timeout period expires before the transfer of the packet to SRAM buffer 318 is complete, another packet cannot be dropped until the transfer of the current packet to SRAM buffer 318 is complete. If the destination of the other packet becomes available during this additional waiting period, the packet can be sent from router 105 to that destination rather than being a dropped packet.

[0071] To facilitate the removal of packets from SRAM buffer 318, drop packet buffer 306 may further include a packet reassembly buffer 320 similar to packet disassembly buffer 319. However, packet reassembly buffer 320 may include one less register because the final portion of the packet may be transferred directly from SRAM buffer 318 in parallel with the remaining portion of packet data already transferred to packet reassembly buffer 320, which reassembles the dropped packet 321. Writing to and from packet disassembly and reassembly buffers 319, 320 and SRAM buffer 318 may be controlled according to finite state machine (FSM) 322.

[0072] In the event that the drop packet buffer 306 is likely to overflow, a backup software mechanism may be used to transfer packets in the drop packet buffer 306 to memory elsewhere on the chip for retransmission. This may include sending an interrupt signal to a monitor processor if the drop packet buffer 306 is full and / or approaching overflow. For example, referring again to FIG. 3A , the packet reinsertion module 307 may further include a set of status registers 315 that provide monitoring or diagnostic data regarding the packet reinsertion module 307. This may include an SRAM word counter that indicates the current number of words stored in the SRAM buffer 318. A “full-point” value may be stored in the general register file 316 of the router 105. The general register file 316 contains various control and configuration settings for the router 105 that are stored in registers of the general register file 316.

[0073] A processing element on the chip may be designated as a monitor processor. The monitor processor may periodically poll status register 315 or may generate a "nearing full" interrupt to alert the monitor processor if the SRAM word counter exceeds a "full point" value. The monitor processor can then take appropriate action, such as adjusting configurable timer settings using general register file 316 or forwarding packets from drop packet buffer 306. If there is not enough available storage space in drop packet buffer 306 to store newly dropped packets, an overflow interrupt may be sent to the monitor processor, which may initiate the removal of packets from drop packet buffer 306 via software-based instructions.

[0074] In one example, to transfer a packet from the drop packet buffer 306, the monitor processor can initiate a read from an “offload header” register in the general register file 316. Reading the “offload / header” register can trigger a software packet offload request to be generated and sent to the packet reinsertion module 307. In response to the request, the packet reinsertion module 307 can be configured to transfer the packet from the drop packet buffer 306 to a register in the general register file 316 along with the packet's header information, which is returned to the monitor processor in response to the “offload header” register read operation. The monitor processor can then interpret the header information and initiate a read of the remaining packet data held in the general register file 316 and transfer of the packet to another memory location external to the router 105 for retransmission. For example, the packet may be transferred to a local memory of the monitor processor.

[0075] In addition to an SRAM word counter for recording the current capacity usage of the SRAM buffer 318, the status register 315 may also include a register that records a count of the total number of reinserted packets (through normal hardware-based mechanisms), a word water-level counter that records the maximum number of words stored in the SRAM buffer 318 observed during a specified period, an equivalent packet water-level counter for the maximum number of non-word packets, and a recurrent dropped packet counter that records the number of packets that have been dropped more than once. The number of recurrently dropped packets can be monitored by setting a bit in the packet's header indicating that the packet has been dropped. Such monitoring data may be useful for diagnostic and configuration purposes.

[0076] Referring now to Figure 4, a process for routing data in a neuromorphic chip, particularly a process for handling data packets that cannot be transmitted, is shown. It will be appreciated that the process may be performed by router 105 of Figures 1, 2, and 3. Figure 4 depicts a high-level process, while Figures 5 through 8 provide more detailed examples.

[0077] 4, in step S401, one or more circuits of router 105 determine that a data packet in the packet processing pipeline should be dropped. In step S402, the data packet to be dropped is moved from the packet processing pipeline to drop packet buffer 306 using one or more circuits. In step S403, one or more circuits reinsert the dropped data packet from drop packet buffer 306 into the packet processing pipeline for reprocessing.

[0078] 5, exemplary processing of a data packet in a packet processing pipeline is described in more detail. In step S501, a data packet is selected for processing. The data packet may be selected from multiple input data streams 301, as described above. If only a single data stream is present, the data packet may be the next data packet in the input queue.

[0079] In step S502, one or more destinations for the data packet are determined. This may be performed by the routing engine 303, and as described above, the one or more destinations may be determined based on information contained within the data packet, such as a source identifier and a destination identifier. The identifiers may be used to search the association memory 308 to determine the one or more destinations and determine the appropriate routing path.

[0080] In step S503, it is determined whether the data packet can be transmitted to one or more destinations. As mentioned above, the determination may be based on a signal indicating that the destination of the data packet is unable to receive the data packet.

[0081] If the data packet can be transmitted, processing continues to step S504, in which case the data packet is transmitted to one or more destinations. As described above, the data packet may be transmitted via one or more output ports 304. If the data packet cannot be transmitted, processing continues to step S505, in which case it is determined whether the data packet should be dropped.

[0082] Two exemplary methods for determining whether to drop a data packet are shown in FIGS. 6 and 7, respectively. Referring first to FIG. 6, in step S601, timer 309 is started from zero. This may occur in response to a determination in step S505 of FIG. 5 that the data packet cannot be transmitted. Steps S602 and S603 form a loop in which processing waits until the data packet is available for transmission or timer 309 expires. If the data packet is available for transmission, processing returns to step S504, in which case the data packet is transmitted. If timer 309 has expired, processing continues to step S604, in which case it is first determined whether drop packet buffer 306 is available, and if so, the data packet is moved to drop packet buffer 306 in step S605. Otherwise, if drop packet buffer 306 is not available, processing returns to step S602, in which case it is determined whether the data packet is available for transmission. Exemplary handling of dropped packets is described in further detail below with reference to FIG.

[0083] 7, a second exemplary method for determining whether to drop a data packet will now be described. In step S701, the data packet is moved to out-of-order packet buffer 305. This may be in response to a determination in step S505 that the data packet cannot be transmitted. As described above, existing data packets in out-of-order packet buffer 305 may be pushed toward the end of out-of-order packet buffer 305 to accommodate the new data packet.

[0084] In step S702, it is determined whether the end slot of out-of-order packet buffer 305 is occupied. If the end slot is not occupied, the data packet in out-of-order packet buffer 305 remains in a waiting state until the data packet can be transmitted (steps S703 and S704) or until the data packet reaches the end slot of out-of-order packet buffer 305.

[0085] If the end slot of the out-of-order packet buffer 305 is occupied, the timer 310 is started from zero in step S705, which provides the final time interval in which a packet can be transmitted before it is determined that the data packet should be dropped, as indicated by the loop in steps S706 and S707. If a data packet becomes available for transmission during this period, processing continues to step S704, in which case the data packet is transmitted. Otherwise, if the timer 310 has expired, it is first determined in step S708 whether the drop packet buffer 306 is available. If so, it is determined that the data packet in the end slot of the out-of-order packet buffer 305 should be dropped, and in step S709 the data packet is moved to the drop packet buffer 306. An exemplary process for a dropped packet is described in detail below with reference to FIG. 8. If the drop packet buffer 306 is not available, processing returns to step S706, where it is determined whether the data packet in the end slot of the out-of-order packet buffer 305 can be transmitted. It will be appreciated that if any packet in the out-of-order packet buffer 305 becomes available for transmission during the processing of FIG. 7, that data packet may be transmitted at any time. Existing packets in the out-of-order packet buffer 305 may be moved forward to fill slots left behind by transmitted packets. However, to reduce power consumption, packets may only be moved to fill empty slots if there is a new packet to insert into the out-of-order packet buffer 305.

[0086] Referring now to FIG. 8, an exemplary process for handling dropped packets will be described. It will be appreciated that the process may be implemented by the packet reinsertion module 307 of the router 105. The process begins at step S801 when the dropped packet buffer 306 is no longer empty. In response, at step S802, the reinsertion timer 313 is started, and the process waits for a specified time interval before continuing to step S803. At step S803, a packet reinsertion request is sent, for example, from the reinsertion request unit 314. At step S804, a signal indicating that the packet reinsertion request has been granted is received. As described above, the packet reinsertion request is sent to the packet processing pipeline and may be granted if the packet processing pipeline is able to process the reinserted data packet.

[0087] Processing then continues to step S805, where the data packet from the drop packet buffer 306 is reinserted into the packet processing pipeline. As described above, the reinserted data packet may be the oldest data packet in the drop packet buffer 306. In step S806, the reinsertion timer 313 is reset. If the drop packet buffer is still not empty, processing may resume at step S801.

[0088] A comparative analysis of the throughput of exemplary embodiments of the router will now be discussed. Reference numerals from Figures 1 through 3B are shown in parentheses for clarity. The exemplary router (105) has a clock speed of 400 MHz and includes a drop packet buffer (306) implemented as a 2K x 32-bit SRAM macro (SRAM buffer 318). It has a capacity to store 1K data packets without payload and up to 333 packets with a maximum 128-bit payload. The exemplary router (105) also includes six output ports (304) corresponding to 152 processing elements (101) and seven inter-chip links (102). The 152 processing elements (101) are implemented using ARM Cortex M4F processors arranged as 38 quad-core processing elements (201). The processors have a clock speed of 300 MHz. The on-chip network is implemented using a 2D mesh topology.

[0089] A comparison was made with the SpiNNaker 1 neuromorphic chip (University of Manchester, UK). The SpiNNaker 1 chip router has a clock speed of 133 MHz and utilizes software-based packet reinsertion as described above. The SpiNNaker 1 chip has 18 ARM968 processor cores. Given the smaller number of processing elements, the on-chip network can be implemented as a star topology. Additionally, there are six inter-chip links.

[0090] Turning now to a comparative analysis, the throughput of the SpiNNaker 1 router without packet reinsertion is compared to the exemplary router (105). Assume that the processing element (101) requires on the order of 30 cycles to process a packet from the network. However, this can be significantly worse depending on the processor load. Note that the processing element (101) of the exemplary embodiment implements packet DMA and therefore has a higher receive rate than the processing elements of the SpiNNaker 1 chip. It will be understood that the worst-case throughput is determined based on the slowest component of all possible communication paths.

[0091] Under conditions where all packets are dropped, the throughput of the SpiNNaker-1 router is likely limited by the ability of the software packet reinsertion mechanism to keep up with the rate at which packets are being dropped (i.e., the time required for the software to copy and process each dropped packet). This is determined to be a rate of 1 packet every 240 router clock cycles, which, at a router clock frequency of 133 MHz, results in a throughput of 0.56 M packets / s.

[0092] Due to its hardware mechanisms, the exemplary router can drop packets at a rate of one every four clock cycles, and its 400 MHz clock results in a packet throughput of 100 Mpackets / s, 180 times faster than the SpiNNaker-1 router. At equivalent clock frequencies, an improvement factor of 60 is observed.

[0093] Of course, there are several other factors that can limit the throughput of a router, including the link bandwidth between chips and the ability of the processor receiving the stream of packets to keep up with that stream, but the above estimates show that in SpiNNaker-1, the software packet reinsertion mechanism would typically become a throughput-limiting factor in situations where congestion causes packets to be dropped, but in the example router, the hardware mechanism ensures that such a case is highly unlikely.

[0094] While the above embodiments describe the invention in the context of a neuromorphic chip, it will be understood that the invention is not limited to neuromorphic chips and may be applied to any type of computing chip, particularly "many-core" chips of the type used for high performance computing (HPC). While particular embodiments of the present invention have been described above, it will be appreciated that various modifications can be made to the described embodiments without departing from the spirit and scope of the invention. That is, the described embodiments are to be considered in all respects as illustrative and not restrictive. In particular, where particular forms are described with respect to particular processes, it will be appreciated that such processes may be carried out in any suitable form arranged to provide suitable output data.

Claims

1. 1. A router for routing data in a computing chip having a plurality of processing elements, comprising: a packet processing pipeline; a hardware-based packet drop and reinsertion mechanism having a drop packet buffer and one or more circuits; the one or more circuits comprising: determining that a data packet in the packet processing pipeline should be dropped due to transmission of the data packet to one or more destinations not being possible at the time transmission of the data packet is attempted; moving the to-be-dropped data packet from the packet processing pipeline to the dropped packet buffer; and 10. A router configured to re-insert dropped data packets from the dropped packet buffer into the packet processing pipeline for re-processing, if the pipeline is capable of handling packet re-insertion.

2. A router as described in claim 1, wherein the one or more circuits are configured to determine that a data packet in the packet processing pipeline should be dropped due to the destination of the data packet being unable to receive the data packet.

3. A router as described in claim 1 or 2, wherein the one or more circuits are configured to determine that a data packet in the packet processing pipeline should be dropped if none of the destinations of the data packet are able to receive the data packet.

4. 4. The router of claim 1, wherein the one or more circuits are configured to determine that a data packet in the packet processing pipeline should be dropped based on a signal indicating that a destination of the data packet is unable to receive the data packet.

5. 5. The router of claim 1, wherein the packet processing pipeline further comprises an out-of-order packet buffer, and wherein the one or more circuits are configured to move the to-be-dropped data packets from the out-of-order packet buffer to the drop packet buffer.

6. 6. The router of claim 5, wherein the one or more circuits further comprise: selecting a data packet for processing by the packet processing pipeline; determining one or more destinations for the data packets; determining that a destination of the one or more destinations of the data packet is unable to receive the data packet; and A router configured to move the data packet to the out-of-order packet buffer.

7. 7. The router according to claim 5, wherein the data packet to be dropped is the oldest data packet in the out-of-order packet buffer.

8. 8. A router according to claim 1, wherein the one or more circuits are configured to determine that a data packet in the packet processing pipeline should be dropped based on the expiration of a timer, the timer being started in response to determining that data packets cannot be sent to all of their destinations, and wherein the data packet is dropped when the timer expires and the data packet still cannot be sent.

9. 9. The router of claim 1, wherein the one or more circuits are configured to re-insert the dropped data packet from the dropped packet buffer into the packet processing pipeline for re-processing based on a timer.

10. 10. The router of claim 9, wherein the one or more circuits are configured to reinsert the dropped data packets from the dropped packet buffer into the packet processing pipeline for reprocessing based on a load of the packet processing pipeline, and the one or more circuits are configured to send a reinsertion request to the packet processing pipeline and, if the request is granted, reinsert the dropped packet.

11. 11. The router of claim 1, wherein the router is configured to perform software-based packet offload processing to remove data packets stored in the dropped packet buffer for reprocessing of the data packets outside the router.

12. 12. A router according to any preceding claim, wherein the packet processing pipeline is configured to process multicast data packets.

13. 13. The router of claim 1, wherein the dropped packet buffer is configured to store a plurality of dropped data packets.

14. 14. The router of claim 1, wherein the drop packet buffer comprises one or more SRAM modules.

15. 15. The router of claim 1, wherein the computing chip is a neuromorphic chip.

16. 1. A method for routing data in a computing chip having a plurality of processing elements, wherein the router has a packet processing pipeline, a drop packet buffer, and one or more circuits, the method comprising: the one or more circuits determining that a data packet in the packet processing pipeline should be dropped due to transmission of the data packet to one or more destinations not being possible at the time transmission of the data packet is attempted; the one or more circuits moving the to-be-dropped data packet from the packet processing pipeline to the drop packet buffer; and the one or more circuits reinserting dropped data packets from the dropped packet buffer into the packet processing pipeline for reprocessing if the pipeline is capable of handling packet reinsertion; A method comprising:

17. 17. The method of claim 16, wherein determining that a data packet in the packet processing pipeline should be dropped is based on a signal indicating that a destination of the data packet is unable to receive the data packet.

18. 18. The method of claim 16 or 17, further comprising the step of moving the to-be-dropped data packets from an out-of-order packet buffer to the drop packet buffer.

19. 20. The method of claim 18, further comprising: selecting a data packet for processing by the packet processing pipeline; determining one or more destinations for said data packets; determining that a destination of the one or more destinations of the data packet is unable to receive the data packet; and moving the data packet to the out-of-order packet buffer; A method having the following.

20. 20. The method of claim 18 or 19, wherein the data packet to be dropped is the oldest data packet in the out-of-order packet buffer.

21. 21. The method of claim 16, wherein the step of determining that a data packet in the packet processing pipeline should be dropped is based on the expiry of a timer, the timer being started in response to determining that data packets cannot be sent to all of their destinations, and wherein the data packet is dropped when the timer expires and the data packet still cannot be sent.

22. 22. The method of any one of claims 16 to 21, wherein the step of re-inserting dropped data packets from the dropped packet buffer into the packet processing pipeline for re-processing is based on a load of the packet processing pipeline, the method further comprising the step of sending a re-insertion request to the packet processing pipeline and, if the request is granted, re-inserting the dropped packet.

23. 23. The method of any one of claims 16 to 22, further comprising the step of removing data packets stored in the drop packet buffer in response to a software-based packet offload process.

24. 24. The method of any one of claims 16 to 23, wherein the packet processing pipeline is configured to process multicast data packets.

25. 25. The method of any one of claims 16 to 24, wherein the dropped packet buffer is configured to store a plurality of dropped data packets.

26. 26. The method of any one of claims 16 to 25, wherein the drop packet buffer comprises one or more SRAM modules.

27. 26. The method of any one of claims 16 to 25, wherein the computing chip is a neuromorphic chip.

Images