Mitigating communication latency for on-chip networks

Abstract

The present application relates to a system and method for reducing delays in an array of computing nodes. In some embodiments, a method for routing data may include outputting a first bypass signal and a second bypass signal from a first computing node of the array of computing nodes, where the first bypass signal indicates routing packet data through the second computing node and the second bypass signal indicates turning the packet data at a third computing node. The packet may be routed through the second node based on the first bypass signal in a single clock cycle, and the packet may be routed from the second computing node to the third computing node in a single clock cycle. The second computing node receives the first bypass signal by a faster route than it receives the packet data.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit of U.S. Provisional Application No. 63 / 235,018, filed on August 19, 2021, entitled "COMMUNICATION LATENCY MITIGATION FOR ON - CHIP NETWORKS", the disclosure of which is hereby incorporated by reference in its entirety for all purposes.

[0002] This disclosure relates to electronic assemblies and communication within electronic assemblies.

Background Art

[0003] High - performance computing systems are important for many applications. However, conventional computing system designs encounter significant communication latency in on - chip networks, which can lead to performance degradation.

Summary of the Invention

[0004] The technological innovations described in the claims each have several aspects, and no single one of them alone bears the desirable attributes. Without limiting the claims, some of the prominent features of this disclosure are briefly described here.

[0005] In some embodiments, the techniques described herein relate to a method for routing packets in a computing system, the method comprising: outputting a first bypass signal and a second bypass signal from a first computing node of an array of computing nodes, the first bypass signal indicating that the packet should be routed through a second computing node of the array of computing nodes, and the second bypass signal indicating that the packet should be turned at a third computing node of the array of computing nodes; routing the packet through the second computing node based on the first bypass signal from the first computing node, the packet being routed from the first computing node through the second computing node in a single clock cycle, the second computing node receiving the first bypass signal via a route faster than the second computing node receiving the packet; and turning the packet in the third computing node based on the second bypass signal, the packet being received by the third computing node from the second computing node.

[0006] In some embodiments, the techniques described herein relate to a method by which a third computing node receives a third bypass signal based on a second bypass signal via a route faster than the third computing node can receive the packets.

[0007] In some embodiments, the techniques described herein relate to a method for routing packets to a third computing node in two clock cycles.

[0008] In some embodiments, the techniques described herein relate to a method by which a packet comprises a header portion and a data portion, and the header portion is routed one cycle before the data portion.

[0009] In some embodiments, the techniques described herein relate to a method for routing a packet through a second computing node, comprising the steps of routing the header portion in a first clock cycle and routing the data portion in a second clock cycle.

[0010] In some embodiments, the technique described herein relates to a method for routing a packet through a second computing node, comprising the steps of: storing a first bypass signal in a state element of the second computing node; routing a header from the first computing node to the second computing node at least partially based on the first bypass signal; and routing a data portion from the first computing node to the second computing node, at least partially based on the first bypass signal, after routing the header from the first computing node to the second computing node.

[0011] In some embodiments, the technique described herein relates to a method wherein a packet comprises a plurality of subpackets, each subpacket comprising a header and a data portion, and the step of routing the packet through a second computing node comprises the steps of routing the plurality of subpackets from a first computing node to a second computing node, and comparing at least a portion of each header of each of the plurality of subpackets.

[0012] In some embodiments, the techniques described herein further include the steps of determining, based on the comparison, that a header mismatch exists, and providing an error signal in response to the determination.

[0013] In some embodiments, the techniques described herein relate to a method for routing packets through a second computing node, further based on one or more other packets waiting to leave the second computing node, and the available capacity of the packet destination queue.

[0014] In some embodiments, the technique described herein further includes the step of outputting a third bypass signal from a second computing node, the third bypass signal indicating that another packet is to be routed to a fourth computing node in the array of computing nodes.

[0015] In some embodiments, the technology described herein relates to a method in which, when a first bypass signal indicates that the packet can bypass the second computing node, the step of routing a packet from a first computing node to a second computing node includes the step of routing the packet over a connection on which the packet cannot be turned at the second computing node.

[0016] In some embodiments, the technology described herein relates to a computing system comprising a first computing node and a second computing node, wherein the first and second computing nodes are included in a computing node array, and the first computing node is configured to route a bypass signal on a first route to the second computing node and to route packet data to the second computing node on a second route, wherein the first route is faster than the second route and the bypass signal indicates whether the packet data will be turned at the second computing node.

[0017] In some embodiments, the technology described herein further comprises a computing system including a third computing node, wherein the first, second, and third computing nodes are located in the same row or column of a computing node array, and the first computing node is configured to output a second bypass signal indicating whether to allow packet data to be turned to the third computing node.

[0018] In some embodiments, the technology described herein relates to a computing system configured such that a third computing node turns packets and outputs them in two clock cycles.

[0019] In some embodiments, the technology described herein relates to a computing system in which a packet comprises a header and a data portion, and a second computing node is configured to route the header to the third computing node in at least one clock cycle before routing the data portion to the third computing node.

[0020] In some embodiments, the technology described herein relates to a computing system in which a packet comprises a plurality of subpackets, each subpacket comprising a header and a data portion, and a second computing node is configured to compare at least a portion of the headers of each subpacket.

[0021] In some embodiments, the technology described herein relates to a computing system configured to route packets through a second computing node in a path between a first computing node and a third computing node in a single clock cycle.

[0022] In some aspects, the technology described herein relates to a computing system configured to perform neural network training.

[0023] In some aspects, the technology described herein relates to a computing system in which a system on a wafer includes an array of computing nodes.

[0024] In some aspects, the technology described herein relates to a computing system configured to determine a first route based at least in part on one of several other packets waiting to exit a second computing node or the available capacity of a packet destination queue.

Brief Description of the Drawings

[0025] This disclosure is described herein with reference to the drawings of certain embodiments that are intended to illustrate, but not to limit, the disclosure. It is to be understood that the accompanying drawings, which are incorporated herein and form a part of this specification, are for the purpose of illustrating the concepts disclosed herein and may not be to scale.

[0026] [Figure 1] FIG. is a diagram showing an exemplary array of computing nodes.

[0027] [Figure 2] FIG. is an example of a schematic diagram of a computing node and packet routing according to some embodiments.

[0028] [Figure 3] FIG. is an example of packet routing according to some embodiments.

[0029] [Figure 4]This is a diagram of a computing node having bypass routing according to several embodiments.

[0030] [Figure 5] This figure shows an example of routing that uses bypass and bypass-next signals to bypass computing nodes within an array, according to several embodiments.

[0031] [Figure 6] This figure shows an example of packet routing using several embodiments.

[0032] [Figure 7] This is an illustrative diagram of subpacket processing and parity checking according to several embodiments. [Modes for carrying out the invention]

[0033] The following descriptions of specific embodiments present various descriptions of those specific embodiments. However, the technological innovations described herein can be embodied in numerous different ways, for example, as defined and encompassed by the claims. While this description refers to drawings, similar reference numbers may indicate identical or functionally similar elements. It will be understood that the elements shown in the drawings are not necessarily drawn to scale. Furthermore, it will be understood that a particular embodiment may include more elements and / or subsets of elements shown in the drawings than those shown. Additionally, some embodiments may incorporate any suitable combination of features from two or more drawings.

[0034] Figure 1 shows an exemplary array of computing nodes that may be used in high-performance computing systems and / or other settings where high computing density is desired. As shown in Figure 1, array 100 can include multiple computing nodes 102 arranged in a grid or other pattern. The computing nodes 102 may be arranged in rows and columns. Any appropriate number of computing nodes 102 can be included in array 100. For example, a computing node array may include about 100 computing nodes 102 in a particular application. Array 100 may include routing lines 104 that can be used to enable communication between the computing nodes 102 of array 100. Array 100 may be implemented on a single integrated circuit die. A computing node 102 can be any appropriate circuit configured to provide one or more of the following functions: computing, storage, control, communication, or monitoring. A computing node 102 can be included in a central processing unit (CPU), graphics processing unit, application-specific integrated circuit (ASIC), system-on-a-chip (SOC), or other die.

[0035] The computing nodes 102 of array 100 can interface with each other to implement distributed computing functions. In some embodiments, each computing node of array 100 can perform computing operations that include one or more of the following: computation, storage, routing decisions, and external communication. In some embodiments, each computing node of a plurality of computing nodes 102 can be an instance of the same design. However, in some embodiments, the array can include two or more types of nodes with different capabilities, such as different routing capabilities, different computing capabilities (e.g., not computing capabilities), different memory amounts (e.g., static random access memory (SRAM)), and different sensors (e.g., temperature, voltage, etc.). For specific applications, array 100 may be implemented on a system on a wafer.

[0036] For example, in a multi-computing node network as shown in Figure 1, communication delays between computing nodes can significantly impact system performance. Computing nodes may reside on a common die, and therefore, embodiments of this disclosure can achieve relatively low communication delays for on-die communication. Embodiments described herein can facilitate communication between computing nodes, enabling data packets to travel across the on-chip network with a single-cycle delay per computing node. For example, the maximum size of a computing node can be selected or determined so that packets can travel across computing nodes in a single clock cycle. In a typical network, each die can operate at frequencies of approximately 2 gigahertz (GHz), e.g., 1 GHz, 1.5 GHz, 2 GHz, 2.5 GHz, 3 GHz, or any frequency between these frequencies, or more depending on the specific die. A typical computing node size is approximately 1 mm. 2 , about 1cm 2For example, at a frequency of approximately 2 GHz, packets can travel from one computing node to the next in less than 0.5 nanoseconds to complete their journey in a single cycle.

[0037] As a packet travels across computing nodes, network routing decisions can be made regarding whether to route the packet straight, turn it around, or whether the packet has reached its destination. If a system waits for a packet to reach a computing node before making a routing decision about the packet's route from that node, the system may not be able to achieve both packet reception and routing decision within a single cycle. More specifically, forwarding a packet using a single cycle and determining where to route it next to reach its destination may be difficult to achieve in a single cycle without reducing the computing node size below the desired size. Therefore, such a method is inefficient and can result in significant packet communication delays.

[0038] Embodiments of this disclosure can address inefficiencies caused by packet routing. In some applications, the width, height, or both of the on-chip network can be selected based at least in part on the time it takes for a packet to travel over an average global wire, which can route signals between compute nodes. In some embodiments, the system can include several wider and / or thicker wires that can be used to carry critical signals. For example, wider or thicker wires can carry valid bits, fields indicating which virtual channel the packet is traveling on, etc. In some embodiments, larger spaces can be provided between wider or thicker wires to reduce coupling between wires. Thicker or wider wires can, in some cases, transfer information faster than normal wires. However, only a limited number of such wires may be available. Such wires may occupy a considerably larger space than normal wires, for example, about three, four, or five normal wires' worth of space. In some embodiments, when a packet enters a compute node array, a processing routine can perform a lookup in a routing table to determine which compute node row and compute node column the packet should enter. Wider or thicker wires may be located in higher metal layers than narrower wires. For packets moving within a die, row / column identifier fields, etc., can be used directly, without a routing table, to determine where the packet will turn. In some embodiments, a processing routine can determine whether the packet will terminate at a different computing node after the turn or continue off the edge of the die.

[0039] In some embodiments, for each individual computing node, a processing routine can determine (e.g., decode) whether a packet should turn at a computing node two network hops away. For example, if a packet is moving horizontally and should turn at column 15, the system can be configured to determine this turn when the packet is at the computing node in column 13. This determination can be used to generate a bypass qualified signal. The bypass qualified signal can be communicated over a faster route (e.g., a thicker and / or wider wire) so that the decoded bypass qualified determination and the forwarding of the packet across computing nodes can be performed in a single clock cycle. For example, a processing routine can perform a bypass qualified determination at each computing node, and as a result, the determination can be made in a timely manner to allow the packet to turn at the correct location.

[0040] In some embodiments, bypass-eligible signals may be carried on wider or thicker wires as the packet leaves adjacent computing nodes. For example, continuing to refer to the above example, bypass-eligible signals may be carried on wider or thicker wires as the packet leaves computing node 14. Thus, control signals may arrive in front of the packet at column 15 and be used to steer the packet's data.

[0041] In some embodiments, a packet may have two indicators related to bypassing compute nodes (e.g., whether to route through a compute node without turning). A “Bypass” (BYP) signal may indicate whether the packet is permitted to bypass the next compute node, and a “Bypass Next” (BYP_NEXT) signal may indicate whether the packet is permitted to bypass a compute node two hops away. When the packet reaches the next compute node, the BYP_NEXT value can become a new BYP value, and the new BYP_NEXT value can be determined. By determining whether to route by bypassing the next two compute nodes (e.g., whether the packet turns at the next compute node or at a compute node after the next compute node), sufficient time is provided to determine the route and send the packet, while reducing wasted cycles. In principle, different numbers of operations can be used. For example, the bypass signal may be determined at 3 hops away, 4 hops away, etc. In some embodiments, control signals may be carried on a high-speed wire while data travels on a normal low-speed wire. The high-speed wire for routing such control signals may be implemented in a higher metal layer than the low-speed wire for routing packet data. For example, semiconductor devices manufactured according to modern processes may contain multiple metal layers, e.g., 10, 15, or several other layers. Lower metal layers can typically be narrower and thinner than higher metal layers to accommodate higher densities and typically carry signals over relatively short ranges. Higher layers in the stack generally have thicker / wider wires to support global communications as well as efficient distribution of power and / or clock signals. In some embodiments, the top one or two layers can be used to carry bypass signals, and the next one or two layers can be used to carry the majority of packets between nodes.

[0042] The number of operations to be determined in advance can be based, at least in part, on the speed of the high-speed wire compared to a normal wire, the number of available high-speed wires, and so on. For example, determining more hops in advance can allow more time to perform calculations. Thus, for example, packets may be routed adaptively based on congestion rather than statically based on the destination node address. However, determining in advance to bypass one or more nodes may place further demands on the high-speed wire and may limit its capacity.

[0043] Figure 2 is an illustrative schematic diagram of computing nodes and packet routing according to several embodiments. As shown in Figure 2, a packet may be routed from computing node N-2 to computing node N, passing through N-1. Each computing node N, N-1, N-2 may include state elements (e.g., flip-flops) 202A-202F that can be used to store routing information, packet information, or both. Each computing node N, N-1, N-2 may include one or more multiplexers 201A-201F, which can be used to instruct a packet to forward or to turn a packet based on the routing information. By routing a packet forward, the packet can continue along a row or column of the array of computing nodes. Turning a packet involves propagating the packet in a direction orthogonal to the direction in which the packet is received by the computing node (e.g., a packet may be received by a route along a row of the array and output on a route along a column of the array). As shown in Figure 2, packets can move from left to right and / or from top to bottom. However, movement from right to left and / or from bottom to top can be enabled by additional state machines, multiplexers, etc. Figure 2 shows state elements 202A to 202F that come before each multiplexer 201A to 201F, but it will be understood that other configurations are possible according to the principles and advantages disclosed herein, as shown in Figure 4, for example. Thus, in some embodiments, state elements can take data after the multiplexers 201A to 201F.

[0044] Each computing node N-2, N-1, and N can receive and / or generate a bypass signal BYP. The bypass signal BYP indicates whether to continue routing the packet forward along a row or column. The bypass logics 205A, 205B, and 205C of the computing nodes can determine whether to route the packet forward, at least in part, based on the bypass signal BYP. If the bypass logics 205A, 205B, or 205C determine to route the packet forward, they can select the packet by asserting the selection signal of their respective multiplexers 201A, 201B, or 201C. This allows the packet to propagate along the same row or column from which it was received by the computing node. If the bypass logics 205A, 205B, or 205C determine to turn the packet back, the packet may be stored by their respective state elements 202D, 202E, or 202F. The packets can then be selected by asserting selection signals to the respective multiplexers 201D, 201E, and 201F in subsequent clock cycles, allowing the packets to propagate out of the computing node along a route perpendicular to the route in which the computing node received them.

[0045] Figure 3 shows an example of packet routing according to several embodiments. Packet data may be associated with the values ​​BYP and BYP_NEXT at computing node 301A. BYP can determine whether the packet data can be bypassed at 301B, and BYP_NEXT can indicate whether the packet data can be bypassed at computing node 301C. Computing node 301B can assign the value of BYP_NEXT to BYP, setting a new BYP_NEXT value that indicates whether the packet data can be bypassed at computing node 301C. Similarly, at computing node 301C, BYP can take the value of BYP_NEXT, setting a new BYP_NEXT value that indicates whether the packet data can be bypassed at computing node 301D. In some embodiments, the BYP and / or BYP_NEXT values ​​can be provided to the multiplexer to determine whether bypassing is permitted (e.g., whether the packet has a turn at a computing node one or two hops away). The bypass logic of the computing nodes can generate and / or process the BYP and BYP_NEXT signals. The Bypass (BYP) and Bypass Next (BYP_NEXT) signals can be active high signals. Alternatively, one or both of these signals can be logically inverted and processed accordingly.

[0046] Figure 4 is a schematic diagram of a computing node using bypass in several embodiments. As shown in Figure 4, computing nodes N-2, N-1, and N may have multiplexers 401A-401C used to determine whether to route signals horizontally, and may have state elements 402A-402C that can be used to store routing information (e.g., bypass signals) and / or other information. Bypass (BYP), bypass next (BYP_NEXT), header, and other signals may be provided to the multiplexer 401A in computing node N-2.

[0047] The BYP_NEXT value of compute node N-2 can be the BYP value of compute node N-1. The BYP_NEXT value of compute node N-1 can be determined, for example, by comparing the current compute node (e.g., N-2) with the compute node to which the packet will turn (e.g., N). If the turning compute node (e.g., N) is two hops away from the current compute node (e.g., N-2), the BYP_NEXT value of compute node N-1 can be set to a value (e.g., 0) indicating that the packet will turn at compute node N. Otherwise, the BYP_NEXT of node N-1 can be set to a value (e.g., 1) indicating that the packet will be routed forward without turning at compute node N. Therefore, for example, if an incoming packet to compute node N-2 should turn at compute node N, both BYP_NEXT and BYP can initially be set to values ​​indicating that node N-2 and node N-1 may be bypassed. After calculating node N-2, BYP can take the previous value of BYP_NEXT for compute node N-1 (e.g., 1), indicating that node N-1 can be bypassed. A new BYP_NEXT can be calculated and set to 0 in the current example of a packet turning at compute node N. When the packet is at node N-1, the BYP value can be set to the previous value of BYP_NEXT (e.g., 0), indicating that the packet will turn at compute node N and that the packet cannot bypass compute node N.

[0048] Figure 5 shows an example of routing using bypass (BYP) and bypass next (BYP_NEXT) values ​​in several embodiments. The BYP and BYP_NEXT signals are active-high signals in Figure 5. As shown in Figure 5, computing node 501a can receive values ​​BYP=1 and BYP_NEXT=1, indicating that bypass is permitted for computing nodes 501a and 501b (e.g., the packet does not turn at either 501a or 501b). After computing node 501a, BYP=1 (i.e., the value before BYP_NEXT) and BYP_NEXT=0, indicating that bypass is permitted for computing node 501b but not for 501c. That is, the packet associated with the bypass signal turns at computing node 501c, and therefore 501c should not be bypassed. Rather, the bypass signal can turn and pass to computing node 501d. In some embodiments, packet data may follow one computing node (e.g., one hop) behind the packet header. Bypass signals may be routed along with the packet header and stored in state elements for use with the packet data.

[0049] Figure 6 shows an example of packet routing according to several embodiments. As shown in Figure 6, control signals can be generated and used to route both the header and data portions of a packet through a computing node. In some embodiments, the header and data portions of a packet can be routed separately using a configuration such as that shown in Figure 6. For example, the header can be routed in one cycle, and the control signal can be progressively deployed to route the data after one cycle of the header. The control signal can be a bypass signal. The bypass signal can be used to route the header in one cycle, and can be stored by a state element so that the bypass signal can be used in the next cycle to route the packet data.

[0050] As shown in Figure 6, the header can be steered using the header circuit 604 with the control logic 602. The control logic 602 can generate a bypass signal and / or one or more control signals to be stored and used in the next cycle (e.g., the cycle immediately following the header being steered) for routing the data using the data circuit 606. For example, a state element 605 can store the bypass signal and / or other control signals. In some embodiments, the header circuit 604 and / or the data circuit 606 may include one or more buffers for storing control signals, packet bits, and / or other information for steering the packet.

[0051] In some embodiments, the system may have a bypass control mechanism that can prioritize packets that are eligible to bypass a particular computing node, while still allowing other traffic to leave the computing node. In some embodiments, whether a packet bypasses a computing node may depend on more than whether the packet is eligible to bypass (e.g., whether BYP is yes). For example, bypassing may depend on the number of waiting packets (e.g., packets preceding an incoming packet that were not bypassed or terminated in the previous cycle, packets waiting to turn at the computing node, etc.), whether the queue is full or nearly full, etc. For example, if the destination or intermediate queue to which a packet is routed is full or nearly full, there may be little or no benefit in speeding up the packet, and resources may instead be used to route other packets.

[0052] In some embodiments, a packet may have a smaller header portion (e.g., about 20 bits) and a larger data portion (e.g., about 200 bits, about 400 bits, about 800 bits, etc.). The header portion may be used to control the path of the packet through the network. In some embodiments, the header can travel through the network using the mechanisms described herein, and the rest of the packet (e.g., data) may follow one cycle after the header. In some embodiments, the same signals used to control the header can be stored in a state element such as a flip-flop, which can then be deployed in the next cycle to control the rest of the packet.

[0053] In some embodiments, packets can be larger than other packets. For example, a packet can be a data packet carrying a relatively large amount of data. In some embodiments, the processing routine can divide a packet into multiple smaller packets, e.g., 2 packets, 3 packets, 4 packets, etc. The processing routine can duplicate the header and send a portion of the data (e.g., half for 2 packets, a quarter for 4 packets, etc.) with each copy of the header. In some embodiments, the processing routine may be configured so that the header travels through the system in a lock step, and thus the expansion into data can always occur within a single clock cycle. In some embodiments, the processing routine can perform a parity check to verify that all copies of the data remain in the lock step.

[0054] Figure 7 is an exemplary diagram of parity checking according to several embodiments. A packet may be divided into four subpackets 706A-706D, each having headers 708A-708D. At least a portion of each of the headers 708A-708D may be identical to one another. Each of the four subpackets 706A-706D can carry a subset of the data of the larger packet. The packet can travel from the first computing node 702 to the second computing node 704. After the four packets reach the second computing node 704, the system may be configured to check the headers 712A-721D in comparator 714B. In some embodiments, the headers 708A-708C may be checked in comparator 714A after arriving from the previous computing node and / or before leaving computing node 702 and traveling to computing node 704. If the transmission is error-free, at least portions of subpackets 710A-710D must be identical to subpackets 706A-706D, and at least portions of headers 712A-712D must be identical to headers 708A-708D. Furthermore, at least portions of each of headers 712A-712D must all be identical to each other. In addition to splitting large packets for transmission, in some embodiments, even smaller packets can be split, and check 714B can function as an integrity check that helps ensure packets are correctly transported across the network.

[0055] If there are unexpected differences in headers 712A-712D, this may indicate a problem in the transmission of packets from the first computing node 702 to the second computing node 704. In some embodiments, the system may be configured to provide an error signal, restart, and / or perform other actions. In some embodiments, if there is an unexpected mismatch in headers 712A-712D, the system may be configured to adjust one or more operating parameters. For example, the system may reduce the operating frequency and increase the operating voltage.

[0056] In some embodiments, when a packet turns, it may take an extra clock cycle to turn the packet. This can happen, for example, because the flip-flops and logic in the horizontal and vertical parts of the network are often not in the same (or adjacent) physical locations on the die. Therefore, in some embodiments, turning a packet may take two clock cycles, but the packet may be routed straight through the compute node in a single clock cycle. Thus, it may be advantageous to minimize the number of turns required to route a packet from source to destination.

[0057] The systems and methods described herein can be used in a variety of processing systems for high-performance computing and / or computationally intensive applications, such as neural network processing, neural network training, machine learning, and artificial intelligence. In some applications, the systems and methods described herein can be used to generate data for autopilot systems for vehicles (e.g., automobiles), other autonomous vehicle functions, and / or advanced driver-assistance systems (ADAS) functions.

[0058] In the aforementioned specification, systems and processes are described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made without departing from the broader spirit and scope of the embodiments disclosed herein. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.

[0059] In fact, while the systems and processes are disclosed in the context of specific embodiments and examples, it will be understood by those skilled in the art that various embodiments of the systems and processes extend beyond the specifically disclosed embodiments to other alternative embodiments and / or uses of the systems and processes, as well as obvious modifications and equivalents thereof. Furthermore, while several variations of the embodiments of the systems and processes are shown and described in detail, other modifications within the scope of this disclosure will be readily apparent to those skilled in the art based on this disclosure. Also, various combinations or partial combinations of specific features and aspects of the embodiments may be made and may still be included within the scope of this disclosure. It should be understood that various features and aspects of the disclosed embodiments may be combined with or substituted for each other to form various aspects of the disclosed embodiments of the systems and processes. The methods disclosed herein do not need to be performed in the order enumerated. Accordingly, it is intended that the scope of the systems and processes disclosed herein should not be limited by the specific embodiments described above.

[0060] Each of the systems and methods disclosed herein has several innovative aspects, and it will be understood that not just one of them is solely responsible for or required for the desired attributes disclosed herein. The various features and processes described above may be used independently of each other or in various combinations. All possible combinations and partial combinations are intended to fall within the scope of this disclosure.

[0061] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, features are described above as acting in a particular combination and may be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from the combination, and the claimed combination may be directed towards a partial combination or a variation of a partial combination. A single feature or group of features is not required or essential in every embodiment.

[0062] It will also be understood that the conditional language used herein, in particular "can," "could," "might," "may," and "for example," is generally intended to convey that a particular embodiment includes certain features, elements, and / or steps, but other embodiments do not, unless otherwise specified or understood in the context in which they are used. Therefore, such conditional language is not generally intended to mean that features, elements, and / or steps are required in some way in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included in or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," and "having" are synonymous and are used comprehensively and in an open-ended manner, without precluding additional elements, features, actions, operations, etc. Furthermore, the term "or" is used in an inclusive sense (rather than an exclusive sense), and for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a," "an," and "the" used in this application and the attached claims should be interpreted as meaning "one or more" or "at least one" unless otherwise specified. Similarly, while operations are shown in a particular order in the drawings, it should be recognized that, in order to achieve the desired result, such operations do not need to be performed in the particular order shown or in a sequential order, or that all shown operations may be performed. Furthermore, drawings may schematically illustrate one or more exemplary processes in the form of flowcharts. However, other operations not illustrated may be incorporated into the exemplary methods and processes schematically shown.For example, one or more additional operations may be performed before, after, simultaneously with, or in between any of the illustrated operations. Furthermore, the operations may be rearranged or reordered in other embodiments. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Further embodiments are within the scope of the following claims. In some cases, the operations described in the claims may be performed in a different order and still achieve the desired results.

[0063] Furthermore, the methods and apparatus described herein may be subject to various modifications and alternative forms, specific examples of which are shown in the drawings and described in detail herein. However, it should be understood that embodiments should not be limited to any particular form or method disclosed, but rather, embodiments should encompass all modifications, equivalents, and alternatives that fall within the spirit and scope of the various embodiments described and the accompanying claims. Furthermore, any particular features, aspects, methods, characteristics, properties, qualities, attributes, elements, etc., disclosed herein relating to an implementation or embodiment may be used in all other implementations or embodiments described herein. The methods disclosed herein do not need to be performed in the order listed. The methods disclosed herein may include specific actions performed by a practitioner. However, the methods may also include any third-party instructions for those actions, explicitly or implicitly. The scope disclosed herein also encompasses any and all overlaps, partial scopes, and combinations thereof. Language such as “up to,” “at least,” “greater than,” “less than,” and “between” includes the numbers listed. Numbers preceded by terms such as "about" or "approximately" should include the listed number and be interpreted on a case-by-case basis (e.g., as accurately as reasonably possible under the circumstances, such as ±5%, ±10%, ±15%). For example, "about 3.5 mm" includes "3.5 mm". Phrases preceded by terms such as "substantially" should include the listed phrase and be interpreted on a case-by-case basis (e.g., as accurately as reasonably possible under the circumstances). For example, "substantially constant" includes "constant". Unless otherwise specified, all measurements are taken under standard conditions, including temperature and pressure.

[0064] Where used herein, the phrase “at least one of” in relation to a list of items refers to any combination of those items, including a single component. For example, “at least one of A, B, or C” is intended to encompass the following: A, B, C, A and B, A and C, B and C, and A, B and C. Pacts such as “at least one of X, Y, and Z” are understood separately in contexts where they are commonly used to convey that an item, term, etc., may be at least one of X, Y, or Z, unless otherwise specified. Thus, such pacts are not generally intended to mean that a particular embodiment requires at least one of X, at least one of Y, and at least one of Z to exist, respectively. Headings provided herein, where present, are for convenience only and do not necessarily affect the scope or meaning of the apparatus and methods disclosed herein.

[0065] Accordingly, the claims are not intended to be limited to the embodiments shown herein, but should be given the broadest scope that is consistent with the disclosures, principles, and novel features disclosed herein.

Claims

1. A method for routing packets in a computing system, A step of outputting a first bypass signal and a second bypass signal from a first computing node of an array of computing nodes, wherein the first bypass signal indicates routing the packet through a second computing node of the array of computing nodes, and the second bypass signal indicates turning the packet at a third computing node of the array of computing nodes. A step of routing the packet through the second computing node based on the first bypass signal from the first computing node, wherein the packet is routed from the first computing node through the second computing node in a single clock cycle, and the second computing node receives the first bypass signal via a route faster than the second computing node receives the packet. A method comprising the step of turning the packet in the third computing node based on the second bypass signal, wherein the packet is received by the third computing node from the second computing node.

2. The method according to claim 1, wherein the third computing node receives a third bypass signal based on the second bypass signal via a route faster than the third computing node receives the packet.

3. The method according to claim 1, wherein the packet is routed to the third computing node in two clock cycles.

4. The method according to claim 1, wherein the packet includes a header portion and a data portion, and the header portion is routed one cycle before the data portion.

5. The step of routing the packets through the second computing node is: The steps include routing the header section in the first clock cycle, The method according to claim 4, further comprising the step of routing the data section in a second clock cycle.

6. The step of routing the packets through the second computing node is: The steps include storing the first bypass signal in the state element of the second computing node, The steps include routing the header from the first computing node to the second computing node based at least partially on the first bypass signal, The method according to claim 4, further comprising the steps of routing the header from the first computing node to the second computing node, and then routing the data portion from the first computing node to the second computing node based at least in part on the first bypass signal.

7. The step of routing the packet through the second computing node is as follows: The steps include routing the plurality of subpackets from the first computing node to the second computing node, The method according to claim 1, further comprising the step of comparing at least a portion of each header of each of the plurality of subpackets.

8. A step of determining that a header mismatch exists based on the comparison step described above, The method according to claim 7, further comprising the step of providing an error signal in response to the determination.

9. The method according to claim 1, wherein the step of routing the packet through the second computing node is further based on one or more other packets waiting to leave the second computing node and the available capacity of the destination queue for the packet.

10. The method according to claim 1, further comprising the step of outputting a third bypass signal from the second computing node, the third bypass signal indicating that another packet is to be routed to a fourth computing node in the array of computing nodes.

11. The method according to claim 1, wherein if the first bypass signal indicates that the packet can bypass the second computing node, the step of routing the packet from the first computing node to the second computing node includes the step of routing the packet over a connection on which the packet cannot be turned at the second computing node.

12. A computing system including a first computing node and a second computing node, The first and second computing nodes are included in the computing node array, A computing system in which the first computing node is configured to route a bypass signal on a first route to the second computing node and to route packet data to the second computing node on a second route, and the first route is faster than the second route, and the bypass signal causes the packet data to turn at the second computing node.

13. The computing system according to claim 12, further comprising a third computing node, wherein the first, second, and third computing nodes are located in the same row or column of the computing node array, and the first computing node is configured to output a second bypass signal indicating whether the packet data is to be turned over by the third computing node.

14. The computing system according to claim 13, wherein the third computing node is configured to turn the packet and output the packet in two clock cycles.

15. The computing system according to claim 13, wherein the packet includes a header and a data portion, and the second computing node is configured to route the header to the third computing node in at least one clock cycle before routing the data portion to the third computing node.

16. The computing system according to claim 13, wherein the packet includes a plurality of subpackets, each subpacket includes a header and a data portion, and the second computing node is configured to compare at least a portion of the headers of each subpacket.

17. The computing system according to claim 13, wherein the computing system is configured to route the packets through the second computing node in a path between the first computing node and the third computing node in a single clock cycle.

18. The computing system according to claim 12, wherein the computing system is configured to perform neural network training.

19. The computing system according to claim 12, wherein the system on the wafer comprises the computing node array.

20. The computing system according to claim 12, wherein the computing system is configured to determine the first route at least in part on one of several other packets waiting to leave the second computing node, or on the available capacity of the destination queue for the packets.

Images