Inter-chip latency characteristics in multi-chip systems

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for determining, for each pair of adjacent chips in a plurality of chips connected in a serial loop arrangement of a semiconductor device, a corresponding loop latency for a round-trip data transmission between the pair of chips. Identifying a maximum loop latency from the loop latencies. Determining a loop latency for a data transmission originating from a chip in the plurality of chips to be sent around the serial loop arrangement and returned to the chip. Comparing half the maximum loop latency to one-Nth of the loop latency, where N is the number of chips in the plurality of chips, and storing the larger value as an inter-chip latency of the semiconductor device, the inter-chip latency representing an operating characteristic of the semiconductor device.

Description

Technical Field

[0001] The present disclosure relates to timing synchronization and data transfer in a multi-chip system. Background Art

[0002] Electronic devices can be made up of multiple different chips that need to communicate data between them in order for the electronic device to operate. Data communication between chips can be non-deterministic. For example, data communication between chips is subject to variable latency between the time of transmission at one chip and the time of reception at another chip. That is, the time it takes for data to propagate from one chip to another is not constant, but is subject to many different sources of transmission time variation. Summary of the invention

[0003] Generally speaking, the innovative aspects of the subject matter described in this specification can be embodied in an inter-chip latency characterization method, comprising the following acts: for each pair of chips among a plurality of chips of a semiconductor device, determining a corresponding loop latency of round-trip data transmission between the pair of chips around a transmission path through the plurality of chips;

[0004] identifying a maximum loop waiting time from the loop waiting times;

[0005] determining a full path latency for a data transmission originating from a chip among the plurality of chips, the data transmission to be sent around the path and back to the chip;

[0006] Half of the maximum loop latency is compared to one-Nth of the full path latency, where N is the number of chips in the transmission path of the chip, and the larger value is stored as the inter-chip latency of the semiconductor device, wherein the inter-chip latency represents an operating characteristic of the semiconductor device.

[0007] In a second general aspect, the innovative features of the subject matter described in this specification can be embodied in a method for characterizing inter-chip latency, comprising the following actions: for each pair of adjacent chips in a plurality of chips connected in a serial loop arrangement of a semiconductor device, determining a corresponding loop latency for round-trip data transmission between the pair of chips. The action includes identifying a maximum loop latency from the loop latencies. The action includes determining a loop latency for a data transmission originating from a chip in the plurality of chips, the data transmission to be sent around the serial loop arrangement and returned to the chip. The action includes comparing half of the maximum loop latency with one-Nth of the loop latency, where N is the number of chips in the plurality of chips, and storing the larger value as the inter-chip latency of the semiconductor device, the inter-chip latency representing an operational characteristic of the semiconductor device. Other embodiments of the aspect include corresponding systems, apparatus, and computer programs configured to perform the actions of the method encoded on a computer storage device.

[0008] These and other implementations can each optionally include one or more of the following features.

[0009] In some embodiments, determining the loop latency of the round-trip data transmission between the pair of chips includes the following actions: sending first timestamped data from the first chip of the pair of chips to the second chip of the pair of chips; determining a first relative one-way latency between the pair of chips based on the first timestamped data; sending second timestamped data from the second chip to the first chip; determining a second relative one-way latency between the pair of chips based on the second timestamped data, and determining the loop latency of the round-trip data transmission between the pair of chips based on the first relative one-way latency and the second relative one-way latency. In some embodiments, the first timestamped data indicates a local counter time of the first chip when the first timestamped data is sent. In some embodiments, determining the first relative one-way latency between the pair of chips includes: when the second chip receives the first timestamped data, calculating a difference between the time indicated in the timestamped data and a local counter time of the second chip. In some embodiments, determining the loop latency of the round-trip data transmission between the pair of chips includes calculating a difference between the first relative one-way latency and the second relative one-way latency.

[0010] In some implementations, one or more of the plurality of chips is an application specific integrated circuit (ASIC) chip configured to perform neural network operations.

[0011] In a third general aspect, the innovative features of the subject matter described in this specification can be embodied in a method for inter-chip timing synchronization, comprising the following actions: for each pair of chips in a plurality of chips of a semiconductor device, determining a first one-way wait time for transmission from a first chip in the pair of chips to a second chip in the pair of chips, and determining a second one-way wait time for transmission from the second chip in the pair of chips to the first chip in the pair of chips. The actions include receiving the first one-way wait time and the second one-way wait time of each pair of chips at a semiconductor device driver. The actions include determining, by the semiconductor device driver, a loop wait time between each pair of chips based on the first one-way wait time and the second one-way wait time of each pair of chips. The actions include: adjusting, by the semiconductor device driver, for at least one pair of chips, a local counter of the second chip in the at least one pair of chips based on the characteristic inter-chip wait time of the semiconductor device and the first one-way wait time of the at least one pair of chips. Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs configured to perform the actions of the method encoded on a computer storage device.

[0012] These and other implementations can each optionally include one or more of the following features.

[0013] In some implementations, the actions include determining, by a semiconductor device driver, that each loop latency is less than or equal to a characteristic inter-chip latency of the semiconductor device.

[0014] In some embodiments, adjusting the local counter of the second chip in the at least one pair of chips includes increasing the value of the local counter by an adjustment value. In some embodiments, the adjustment value is equal to a characteristic inter-chip latency of the semiconductor device plus a first one-way latency of a transmission from the first chip in the pair to the second chip in the pair.

[0015] In some implementations, determining the loop latency between each pair of chips includes, for each pair of chips, calculating a difference between a first relative one-way latency associated with the pair of chips and a second relative one-way latency associated with the pair of chips.

[0016] In some embodiments, determining a first one-way latency of a transmission from a first chip in the pair to a second chip in the pair includes sending first time-stamped data from the first chip to the second chip, and determining a first relative one-way latency between the pair of chips based on the first time-stamped data. In some embodiments, the first time-stamped data indicates a local counter time of the first chip when the first time-stamped data is sent. In some embodiments, determining the first relative one-way latency between the pair of chips includes: when the second chip receives the first time-stamped data, calculating a difference between the time indicated in the time-stamped data and the local counter time of the second chip.

[0017] In some implementations, one or more of the plurality of chips is an application specific integrated circuit (ASIC) chip configured to perform neural network operations.

[0018] In a fourth general aspect, the innovative aspects of the subject matter described in this specification can be embodied in a method for sending data between chips, comprising an action of sending data from a first chip in a series loop arrangement of chips of a semiconductor device to an adjacent second chip at a first time. The action includes storing the data in a buffer at the second chip. The action includes releasing the data from the buffer at a second time, wherein the interval between the first time and the second time is based on a characteristic inter-chip latency of the series loop arrangement of the chips. The action includes sending the data from the second chip to a third chip, the third chip being adjacent to the second chip in the series loop arrangement of the chips. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs configured to perform the actions of the method encoded on a computer storage device.

[0019] These and other implementations can each optionally include one or more of the following features.

[0020] In some implementations, the characteristic inter-chip latency represents a maximum expected one-way data transmission latency between two chips in a series ring arrangement of chips.

[0021] In some implementations, the second time is a pre-scheduled time of an operation schedule of the second chip.

[0022] In some implementations, the actions include transferring data from a buffer of the second chip to a communication interface of the second chip coupled to the third chip along an internal bypass path.

[0023] In some implementations, one or more of the first, second, and third chips are application specific integrated circuit (ASIC) chips configured to perform neural network operations.

[0024] Various embodiments provide one or more of the following advantages. For example, in some embodiments, the processes described herein minimize the variation in potential data arrival times for inter-chip communications. Reducing the variation in data communications can allow for the use of smaller receive data buffers in the chips of the system. In some embodiments, the processes described herein enable data transfer operations between chips to be deterministic. For example, an embodiment can cause a program compiler to use a constant (e.g., deterministic) wait time when calculating a local counter time for a receiving chip to access data sent from an adjacent chip to the receiving chip at a particular time from an input buffer.

[0025] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram illustrating an example multi-chip system according to an embodiment of the present disclosure.

[0027] Figure 2 is a flow chart of an example process for characterizing maximum latency in a multi-chip system according to an embodiment of the present disclosure.

[0028] Figure 3A-3C A series of block diagrams illustrating loop latency measurement between two chips according to an embodiment of the present disclosure are shown.

[0029] Figure 4 is a flow chart of an example process for synchronizing local counters of chips in a multi-chip system according to an embodiment of the present disclosure.

[0030] Figure 5 is a flow chart of an example process for transferring data between chips in a multi-chip system according to an implementation of the present disclosure.

[0031] Fig. 6A and 6B Shows the description Figure 5 A series of block diagrams of the data transmission process.

[0032] Figure 7 It shows that it can be Figure 1 Schematic diagram of an example of a dedicated logic chip used in a multi-chip system. DETAILED DESCRIPTION

[0033] In general, the present disclosure relates to inter-chip time synchronization and data transmission in a multi-chip system. More specifically, the present disclosure provides chip operation processes that improve the predictability of data transmission between chips (and, in some examples, around a series loop topology of chips). The present disclosure provides example processes for synchronizing local counters of chips in a system and performing data transmission in a manner that accounts for the inherent variability of data arrival times of inter-chip data transmission, so that data reception times are more, and in some cases, completely deterministic.

[0034] Referring first to inter-chip time synchronization, time synchronization includes two aspects. The first aspect is the characterization of the inter-chip latency of data transfer between various chip pairs on the processing system. This process provides the operating characteristics of the board (e.g., maximum inter-chip latency), which is used as a constant for synchronizing local chip counters each time the board is booted. The second aspect is synchronizing local chip counters when the board is booted (e.g., "boot time synchronization").

[0035] More specifically, the characterization process must be completed for each redesign of the board. For example, the maximum inter-chip latency is typically dependent on the physical properties of the chip layout on the board. The characterization process includes measuring the "round-trip" loop latency of transmissions between chip pairs (e.g., adjacent chip pairs) on the board that will engage in direct communication with each other. In addition, in embodiments that include chips connected in a series loop arrangement, the characterization process can also include measuring the round-trip transmission latency around the entire loop. The data collected from these measurements can be used to determine the maximum inter-chip latency that will be experienced between any two chips.

[0036] Each time the board is booted, reset, or both, a boot time synchronization is performed to synchronize the local counters of the chips. Although each chip is clocked by a local clock that is synchronized with the local clocks of the other chips (e.g., each chip's clock has the same frequency and phase), the chips operate using local counters to time individual chip operations, and when the board is booted or the chip comes out of reset, the individual counters will typically be at different count values. Therefore, boot time synchronization is used to approximately synchronize the local count values ​​of the chips.

[0037] The boot time synchronization process includes measuring the one-way latency of transmissions between pairs of chips on the board. The board driver determines a local counter adjustment for one of the chips in the pair based on one of the maximum inter-chip latency that has been characterized for the board and the one-way latency between the chips in each pair. For example, the driver can adjust the local counter of one of the chips in the pair by increasing the counter value by the sum of the maximum inter-chip latency and one of the one-way latency between the chips. In some embodiments, the boot time process includes adjusting the round trip latency between one or more chip pairs, for example by adjusting a FIFO buffer of one of the chips.

[0038] In some embodiments, the semiconductor chip can be an application specific integrated circuit (ASIC) designed to perform machine learning operations. An ASIC is an integrated circuit (IC) customized for a specific purpose. For example, an ASIC can be designed to perform operations of a machine learning model, including, for example, identifying an object in an image as part of a deep neural network, machine translation, speech recognition, or other machine learning algorithm. For example, when used as an accelerator for a neural network, the ASIC can receive inputs to the neural network and calculate neural network inferences of the inputs. The data input to a neural network layer (e.g., an input to a neural network or an output of another layer of a neural network) can be referred to as an activation input. Inferences can be calculated based on various sets of weight inputs associated with a layer of a neural network. For example, some or all of the layers can receive a set of activation inputs, and the activation inputs are processed based on a set of weight inputs of the layer to generate an output. In addition, neural network operations can be performed by a system of ASICs according to explicit operation scheduling. Therefore, deterministic and synchronous data transmission between ASIC chips can improve the reliability of neural network operations and simplify debugging operations.

[0039] Figure 1 1 is a schematic diagram showing an example multi-chip system 100. Multi-chip system 100 can be an integrated circuit network configured to perform machine learning operations. For example, multi-chip system 100 can be configured to implement a neural network architecture. Multi-chip system includes multiple semiconductor chips 102. Chip 102 can be a general-purpose integrated circuit chip or a dedicated integrated circuit chip. For example, one or more of chips 102 can be an ASIC, a field programmable gate array (FPGA), a graphics processing unit (GPU), or any other suitable integrated circuit chip. Clock 106 is coupled to each chip 102 to provide a synchronous timing signal. For example, clock 106 can include a crystal oscillator that provides a common timing signal (e.g., a 1 GHz clock signal) to each chip 102.

[0040] The multi-chip system 100 also includes a system driver 104. The system driver 104 can be, for example, an external computing system, such as a laptop computer, a desktop computer, or a server system. The system driver 104 can be used to perform or manage the chip synchronization process described herein or a portion thereof. For example, the system driver 104 can be configured to program the chip, manage the boot operation of the multi-chip system 100, debug the chip, or a combination thereof. The system driver can be coupled to the chip 102 via a communication link. The system driver 104 can be coupled to the chip 102 through a configuration status register (e.g., a low-speed interface for programming and debugging the chip).

[0041] In the illustrated example, the multi-chip system 100 includes eight ASIC chips 102 and one FPGA chip 102 arranged in a serial loop topology. More specifically, each chip 102 communicates data with two adjacent chips; one on each side, so that data is transmitted from the chip around the loop to the adjacent chip. The chips 102 and their data communication links form a closed loop. In addition, the multi-chip system 100 includes two data paths between each pair of chips; a clockwise data path 108 and a counterclockwise data path 110.

[0042] In some embodiments, each ASIC chip (P0-P7) can be configured to implement a layer of a neural network. Input activation data can be received by the FPGA chip 104 and sent to P0. For example, P0 can be configured to implement the input layer of the neural network. P0 will perform calculations on the activation data to generate layer output data, which will be sent to P1. P1 can be configured to implement the first hidden layer of the neural network, and will perform calculations on the output from P0, and then send its output to the next neural network layer implemented by P2. The process can continue around the loop through each of the ASIC chips 102 through the extensions processed by each layer of the neural network. Such a process can rely on the precise timing of data transmission between adjacent chips (and around the entire loop) so that the neural network operates reliably and accurately. Therefore, synchronization of data transmission between each ASIC can be important to ensure proper operational coordination between chips.

[0043] Operations within a single chip in a synchronous system are synchronous and deterministic, meaning that there is no variation in the timing of such internal operations. However, for inter-chip operations such as data transfer, there is inherent and non-deterministic variability in the timing of the operations, even in a synchronous system. One source of timing variability is the properties of the physical link between two adjacent chips, which can introduce variations of, for example, about 0-3 clock cycles in the latency of data transfer between adjacent chips. A second, larger source of timing variability is the lack of synchronization between internal chip operations and the forward error correction scheme implemented by the multi-chip system. In a forward error correction scheme, error correction data is added to the data transfer between chips, but the added error correction data is not necessarily synchronized with the data transfer. The introduction of asynchronous data into the data transfer can introduce variations of, for example, up to 16 clock cycles in the latency of data transfer between adjacent chips.

[0044] When data is sent from one chip to another non-adjacent chip (e.g., from P0 to P7), the variation in latency of each inter-chip transfer (e.g., from P0 to P1, from P1 to P2, etc.) is accumulated into the cumulative latency at the destination chip (P7). Taking only the variation due to forward error correction as an example, the latency of a single inter-chip transfer (e.g., from P0 to P1) has a variation of ±16 clock cycles. However, some operations may require sending data from one chip 102 to another non-adjacent chip 102, for example, from chip P0 to chip P3, or even around a loop from the first chip P0 to the last chip P7. As discussed in more detail below, in order to send data from one chip to another non-adjacent chip (e.g., P0 to P7), it is possible to use a bypass operation to transmit the data through each of the intermediate chips (e.g., through chips P1 to P6). However, the latency variation between chips will be accumulated across 8 chips, where the total variation in latency around the loop is close to ±128 clock cycles. The process described below improves the predictability of data transfers between chips and, in some examples, allows inter-chip data transfers to be performed in a deterministic manner.

[0045] Figure 2 A flow chart of an example process 200 for characterizing the maximum latency in a multi-chip system 100 is depicted. Figure 1 , Figure 2 and Figure 3A-3B Process 200 is described. In some implementations, process 200 or portions thereof are performed or controlled by system driver 104. In some examples, process 200 or portions thereof are performed by individual chips 102 of multi-chip system 100. Characterization process 200 is used to determine characteristic inter-chip latency of a multi-chip system design, such as maximum inter-chip latency (L max ). For example, process 200 may be performed for an initial chip layout and / or a new system topology.

[0046] The first step of process 200 includes determining the loop latency between each pair of chips in multi-chip system 100 (step 202). Figure 1As shown, the depicted multi-chip system 100 has ten separate inter-chip communication loops (single-chip loops 112, system loops 114) with independently measurable wait times. There are nine single-chip loops 112 between adjacent chips 102, and there is a system loop 114 around the entire loop. In a multi-chip system, absolute wait time values ​​may only be measurable in these single-chip loops 112, system loops 114, because there is no common time reference available. That is, although each of the single-chip loops 112, system loops 114 is driven by a common clock 106, the local counters on each single-chip loop 112, system loop 114 are not necessarily synchronized to the same count value. In other words, the "local time" on each single-chip loop 112, system loop 114 can be different. As described in more detail below, measuring loop wait times rather than individual one-way wait times between chips can be used to take into account the differences between local counters on each chip.

[0047] The nine single-chip loops 112 going first counterclockwise and then clockwise have the same latency as the loop going first clockwise because the loop latencies are simply the sum of the latencies in each direction. Likewise, the counterclockwise system loop 114 has the same latency as the sum of all nine single-chip loops 112 minus the latency of the clockwise system loop 114. Measuring the latency differences in different directions around the loop between the two chips does not provide more information because these differences can be derived from the nine single-chip loops 112 and the single system loop 114.

[0048] Figure 3A-3C A series of block diagrams illustrating loop latency measurement between adjacent chips 102 are shown. Figure 3A-3C A simplified block diagram of two adjacent chips 102: Chip A and Chip B is depicted. Each chip 102 includes a controller 304 that controls the local operation of the chip, a local counter 306, and a communication interface 308. For clarity of explanation, the communication interface 308 is represented from a transmitter interface (Tx) to a receiver interface (Rx). The communication interface 308 includes a first-in-first-out (FIFO) buffer.

[0049] To measure the loop latency, each chip initializes its local counter 306, for example, by booting chip 102. As discussed above, each chip's local counter 306 represents its local time. In some embodiments, chips 102 perform their respective operations (e.g., calculations, reading data from input buffers, and sending data to other chips) at pre-arranged counter times. For process 200, counters 306 do not need to be synchronized in any way. For example, in Figure 3AIn the example shown, the local counter 306 of chip A is initialized to time 0, and the local counter 306 of chip B is initialized at time 150, so the local counters of chip A and chip B are out of sync by 150 clock cycles. The boot synchronization process discussed below is used to synchronize the local counters 306 in chip 102. It should be noted that for the purpose of explanation, Figure 3A-3C (as well as Fig. 6A and Figure 6B ) is simplified.

[0050] refer to Figure 3B and Figure 3C , in order to measure the round-trip latency between chip A and chip B, chip A and chip B perform a series of timestamped data transfers, first from chip A to chip B, and then from chip B to chip A. For example, first chip A sends timestamped data 309 to chip B to measure the relative one-way latency of a transfer in a first direction (e.g., from chip A to chip B on the clockwise data path 108). Chip A sends data 309 to chip B, the data 309 including a timestamp with a local counter time (e.g., 10) of chip A when the data 309 is sent. For clarity of explanation, Figure 3B Only one data transmission is shown being sent to chip A. In practice, for example, chip A can send a series of data transmissions at different points in a 512-cycle physical coding sublayer (PCS) cycle, each of which is timestamped with chip A's local counter time at the time of transmission. Chip B receives data 309 and records its own local counter time (e.g., 180). The difference between chip A's local time when data 309 was sent (e.g., 10) and chip B's local time when data 309 was received (e.g., 180) is equal to the relative one-way latency from chip A to chip B. For example, if Figure 3B The relative one-way latency shown is 170 clock cycles.

[0051] like Figure 3C As shown, chip B performs the same process to measure the relative one-way latency of transmission in the second direction (e.g., from chip B to chip A on the counterclockwise data path 110). Chip B sends data 310 to chip A, and the data 310 includes a timestamp with the local counter time of chip B when the data 310 is sent (e.g., 200). For clarity of explanation, Figure 3COnly one data transmission is shown being sent to chip A. In practice, for example, chip B can send a series of data transmissions at different points in a 512-cycle PCS cycle, each of which is timestamped with chip B's local counter time at the time of transmission. Chip A receives data 310 and records its own local counter time (e.g., 60). The difference between chip B's local time when data 310 was sent (e.g., 200) and chip A's local time when data 310 was received (e.g., 60) is equal to the relative one-way relative latency in the direction from chip B to chip A. For example, if Figure 3C The relative one-way latency shown is -140 clock cycles. It should be noted that due to local counter differences between two adjacent chips 102, the relative one-way latency can be negative.

[0052] Once a series of data transmissions are performed, each chip 102 (e.g., chip A and B) calculates the relative one-way latency in one direction based on the timestamp value included in the data (e.g., data 309 and data 310) and its own local counter time when the data was received. Each chip 102 can then identify the maximum relative one-way latency it measured and send the maximum relative one-way latency to the system driver 104 to calculate the respective maximum loop latency. In some embodiments, each chip 102 sends the timestamp data from each transmission in the series of transmissions to the system driver 104 along with its own associated local counter value when each transmission was received. The system driver 104 then calculates the relative one-way latency in each direction for each pair of chips, identifies the maximum one-way latency in each direction, and calculates the respective maximum loop latency.

[0053] The relative one-way latency value by itself is meaningless because the local counters on each chip 102 will be in an unknown state. However, when the two relative one-way latencies between a given pair of chips 102 are summed (e.g., the relative one-way latency from chip A to chip B and the relative one-way latency from B back to A), the local counter differences cancel out, leaving only the absolute latency around the loop between chip A and chip B.

[0054] max(R b -S a )=L ab +C ba ;

[0055] max(R a -S b )=L ba –C ba ;as well as

[0056] Linter-chip_loop_max =max(R a -S b )+max(R b -S a )=L ab +C ba +L ba -C ba =L ab

[0057] +L ba .

[0058] R a , R b represents the local counter time when the time-stamped data is received on chip A or chip B, respectively (for example, in this example R a is 60 and R b is 180). a , S b represents the counter time when data is sent by chip A or chip B, respectively (for example, in this example, S a is 10 and S b is 200). ba is the difference in counter time between the local counter time of chip B and the local counter time of chip A: Cba=Cb-Ca (this is not directly observable) (e.g., in this example C ba is 150). ab is the maximum jitter absolute latency from chip A to chip B (this is not directly observable). L ba is the maximum jitter absolute latency from chip B to chip A (this is not directly observable). max(R b -S a) represents the maximum relative one-way waiting time from chip A to chip B. max(Rb-Sa) is the difference between the local counter time of chip B when receiving data from chip A and the local counter time of chip A when sending data. This is also equal to the actual waiting time (Lab) in the direction from chip A to chip B plus the difference between the counter of chip B and the counter of chip A (Cba). Max(RA-SB) represents the maximum relative one-way waiting time from chip B to chip A. Max(RA-SB) is the difference between the local counter time of chip A when receiving data from chip B and the local counter time of chip B when sending data. This is also equal to the actual waiting time (Lba) in the direction from chip B to chip A minus the difference between the counter of chip B and the counter of chip A (Cba). This relationship can also be restated as max(Ra-Sb)=Lba+Cab, where Cab is the counter value of chip A minus the counter value of chip B, for example, the opposite of Cba. Simply put, the offset between the local counters on the two chips creates an "additive" latency for transfers in one direction and a "subtractive" latency for transfers in the opposite direction. inter-chip_loop_max represents the maximum loop latency of a given single-chip loop 112 between two chips.

[0059] After running several measurements of a single chip to adjacent single chip loops 112, the system driver 104 identifies the maximum loop latency among all chip pairs (step 204). For example, the system driver 104 can compare the maximum measured loop latency from each of the single chip loops 112 between each pair of chips to identify the maximum chip-to-chip loop latency (L loop_max ).

[0060] One of the chip 102 or the system driver 104 determines the loop latency of the data transmission around the entire system loop 114 (step 206). For example, using Figure 3A-3C The technique described above is similar to the technique used to measure and calculate the latency around the entire loop, except that the time stamped data is transmitted around the entire loop and received at the same chip 102 that sent the data. The maximum transmission time measured around the system loop 114 will be the maximum full loop latency (L ring_max ). Therefore, local counter differences are not a concern.

[0061] The system driver 104 determines the characteristic chip-to-chip latency (L max)(Step 208). For example, the system driver 104 can compare half of the maximum chip-to-chip loop latency with one-Nth of the maximum full loop latency, where N is the total number of chips 102 in the multi-chip system 100, to estimate the maximum one-way latency in the multi-chip system 100. The larger of the two values ​​is the characteristic chip-to-chip latency (L max ). The system driver 104 can store the characteristic inter-chip wait time for future operations. For example, as discussed below, the characteristic inter-chip wait time will be a constant used in other operations such as boot time synchronization and data transfer. In some embodiments, the compiler also uses the characteristic inter-chip wait time to generate an operation schedule for each chip 102 to execute a specific software application, such as a specific machine learning algorithm. For example, the characteristic inter-chip wait time represents the maximum time it takes to transfer data from one chip to an adjacent chip. The compiler can use the characteristic inter-chip wait time to schedule a receiving chip to read data from an input FIFO buffer after an adjacent chip has sent data, and ensure that all data has arrived before the scheduled read time.

[0062] In some embodiments, L max can be increased by design factors to account for any changes that may not have been measured during the characterization process. For example, the measured L max The maximum possible variation in latency of data transmission between adjacent chips may not be considered. Therefore, in some embodiments, L can be increased max To ensure that the actual inter-chip waiting time experienced by the multi-chip system 100 will not exceed L max The value of .

[0063] Figure 4 is a flow chart of an example process 400 for synchronizing local counters of chips in multi-chip system 100. Figure 1 , Figure 3A-3B and Figure 4 Process 400 is described. In some embodiments, process 400 or portions thereof are performed or controlled by system driver 104. In some examples, process 400 or portions thereof are performed by individual chips 102 of multi-chip system 100. Synchronization process 400 is used to synchronize local counters 306 of chips 102 in multi-chip system 100. Process 400 can be performed when multi-chip system 100 is booted, and is therefore referred to as a "boot synchronization" process. However, process 400 can also be performed at other times, for example, when the multi-chip system is reset.

[0064] For each pair of chips, a first relative one-way latency for data transmission from a first chip in the pair (e.g., chip A) to a second chip in the pair (e.g., chip B) is determined (step 402a), and a second relative one-way latency for data transmission from the second chip in the pair (e.g., chip B) to the first chip in the pair (e.g., chip A) is determined (step 402b). For example, the relative one-way latency on the clockwise data path 108 between the two chips can be determined, and then the relative one-way latency on the counterclockwise data path 110 between the two chips can be determined. The above reference can be used, for example, to Figure 3A-3C The chip 102 measures the first and second relative one-way latencies using the techniques described above. The chip 102 sends the measured relative one-way latencies back to the system driver 104. In some embodiments, the system driver 104 controls each chip 102 to perform the relative one-way latency measurement. In some embodiments, each chip 102 includes software (e.g., firmware) that controls each chip 102 to perform the relative one-way latency measurement when the system is booted or reset.

[0065] The system driver 104 determines the loop latency between each pair of chips (step 404). For example, the system driver 104 can determine the loop latency between a pair of chips based on the respective relative one-way latencies measured between the pair of chips. For example, the system driver 104 can use the equation L loop =(R a -S b )+(R b -S a ) to calculate the loop latency between a given pair of chips. The system driver 104 can repeat the calculation for each single-chip loop 112 between each pair of chips in the multi-chip system 100.

[0066] Optionally, the system driver 104 confirms that each loop latency is less than or equal to the characteristic inter-chip latency (L max )(Step 406). For example, the system driver 104 can compare the calculated loop latency for each pair of chips with the stored value of the characteristic inter-chip latency. In some embodiments, if any of the calculated loop latencies is greater than the characteristic inter-chip latency, the system driver 104 can re-perform the loop latency measurement. For example, the system driver 104 can cause steps 402 and 404 to be re-performed. In some embodiments, if any of the calculated loop latencies is greater than the characteristic inter-chip latency, the system driver 104 can generate an error signal.

[0067] The system driver 104 performs the following operations based on the characteristic chip-to-chip latency (Lmax ) adjusts the local counter of one or more chips to synchronize the chip 102 (step 408). For example, referring to Figure 1 , one chip 102 of the multi-chip system 100 can be selected as a reference chip. For example, the counter value of the reference chip will be used as a basis for adjusting the local counters 306 of other chips 102 in the multi-chip system 100 to synchronize the chips 102. In this example, the FPGA chip will be used as the reference chip. The system driver 104 adjusts the local counters in pairs starting from the reference chip. The system driver 104 is based on L max The local counter time of one chip in each pair of adjacent chips is adjusted based on the one-way latency between the chips and the measured one-way latency between the chips. For example, starting from FPGA and P0, the system driver 104 adjusts the local counter time of one chip in each pair of adjacent chips based on L max and the measured one-way latency of data transmission from the FPGA to P0 (e.g., the measured one-way latency of data transmission along the clockwise data path 108) to adjust the local counter time of P0. After the local counter in P0 is adjusted, the system driver 104 adjusts the local counter of chip P1. For example, the system driver 104 adjusts the local counter of chip P1 based on L max The local counter time of P1 is adjusted by the measured one-way latency of the data transmission from P0 to P1. The system driver 104 repeats this process to adjust the local counter of each chip 102 around the loop until all chips have been synchronized. However, the local counter of the FPGA (e.g., the reference chip) is not adjusted.

[0068] More specifically, use Figure 3A and Figure 3B In the example shown in FIG. 1 , the system driver 104 is based on L max The system driver 104 can adjust the local counter time of one chip in a pair of chips by one of the relative one-way latency between the two chips and the measured one. max The chip's local counter 306 is adjusted by subtracting the measured relative one-way latency from one chip in the pair to the chip whose local counter is being adjusted. That is, the new counter value (T new ) can be determined by the following formula: T new =T old +L max –(R b -S a ), where T old is the original counter value, and (R b -S a ) represents the relative one-way latency measured by the chip whose counter is being adjusted. For example, Figure 3BIn the example, the relative one-way latency from chip A to chip B is measured to be 170. Assume that L max is 30, then the adjustment to the counter of chip B will be L max –(R b -S a ) or 30-170=-140. Therefore, the system driver 104 increases the local counter of chip B by -140 counts (eg, decreases the local counter by 140). Using the simplest case as an example (eg, Figure 3A ), the system driver 104 adjusts the local counter of chip B from 150 to 10. Although the adjusted value of the local counter of chip B is different from the value of the local counter of chip A (e.g., 0), for the purpose of this disclosure, the two chips can be considered synchronized. For example, the synchronization process does not necessarily force the local counters of the two chips to be equal, but rather synchronizes the data transmission latency between each pair of chips across the multi-chip system so that the maximum relative one-way latency between each pair of chips is less than or equal to L max .

[0069] In some embodiments, the FIFO buffer at the Rx communication interface 308 of the chip 102 can also be adjusted. For example, the system driver 104 can adjust the perceived latency between inter-chip links by increasing or decreasing the receive buffer size (e.g., adding or removing latency in 4ns increments) until the single-chip loop 112 has a range of [2L max -3,2L max ]. Therefore, the whole system loop 114 will then have a loop latency of [NL max -3,NL max], where N is the number of chips in the loop. Typically, only latency will need to be added, but it may be necessary to remove latency from some of the counter-clockwise data paths 110, for example, where all of the two-chip loops are within their limits, but the clockwise system loop 114 requires more latency. In this case, the system driver 104 can remove some latency on some of the counter-clockwise data paths 110 (e.g., by reducing one or more of the receive buffers of the chips connected to the counter-clockwise data paths 110 and adding the same amount of latency to the clockwise links (e.g., by increasing the appropriate receive data buffers)), thereby retaining the latency on each single-chip loop 112 while increasing the latency on the clockwise system loop 114. In some embodiments, the latency can be adjusted by increasing or decreasing the appropriate transmitter FIFO buffer size instead of or in addition to adjusting the receiver side FIFO buffer.

[0070] refer to Figure 1 , even after the chips 102 have been synchronized, one or more other remaining variations in chip-to-chip and data transmission may need to be addressed. For example, latency variations due to forward error correction (FEC) operations may need to be addressed. Such variations affect transmissions from one chip to another non-adjacent chip more than transmissions between adjacent chips. For example, to send data from one chip to another non-adjacent chip (e.g., around a loop from the first chip P0 to the last chip P7), the data can be transmitted through each of the intermediate chips (e.g., through chips P1 to P6) using a bypass operation. The latency of each data path between adjacent chips has a constant and variable component. The exact value of the variable component may be difficult or impossible to determine. When data is sent from one chip (e.g., P0) to another non-adjacent chip (e.g., P7) in a bypass operation, for example, the cumulative variation in latency is the sum of the latency variations of each link on which the data is sent. At the destination chip (e.g., P7), the cumulative variation in latency may be significant, resulting in a large amount of variability in the arrival time of the transmitted data at the destination chip (e.g., P7). In a synchronous system, this variability in arrival time would necessitate heavy buffering of the data at the destination. To eliminate the variability in arrival time at the destination chip (e.g., P7) caused by the cumulative variation in latency, a delay can be imposed on the data transmission from each chip 102. Introducing a delay at each chip 102 eliminates the effect of latency variation, and the arrival time of the data at the destination chip becomes deterministic and compatible with a synchronous system.

[0071] In some embodiments, the delay is built into the operation of each chip by the program compiler. For example, the program compiler uses Lmax to generate program instructions as explicit scheduling operations for each chip. Figure 5 , Fig. 6A and Figure 6B In more detail, each chip can be pre-scheduled to resend data received from a neighboring chip at a maximum inter-chip waiting time (e.g., Lmax) after sending the data to the chip. For example, the operation of chip P0 can be pre-scheduled to send data to chip P1 at local counter time t. Then, chip P1 will be pre-scheduled to resend data to chip P2 at local counter time t+Lmax.

[0072] One source of timing variability is the properties of the physical link between two adjacent chips (e.g., PCS jitter), which can introduce variations in the latency of data transmission between adjacent chips 102. This source of variability is addressed by the system characterization and synchronization processes (200 and 400) described above. However, a second source of timing variability is the lack of synchronization between internal chip operations and the forward error correction scheme implemented by the multi-chip system 100. In the forward error correction scheme, error correction data is added to the data transmission between chips 102, but the added error correction data is not necessarily synchronized with the data transmission. Introducing asynchronous data into the data transmission can introduce variations in the latency of the data transmission between adjacent chips, for example, up to 16 clock cycles.

[0073] When data is sent from one chip 102 to another non-adjacent chip 102 (e.g., from P0 to P7), the variation in latency of each inter-chip transfer (e.g., from P0 to P1, from P1 to P2, etc.) is accumulated into the cumulative latency at the destination chip (P7). Taking the variation due to forward error correction as an example only, the latency of a single inter-chip transfer (e.g., from P0 to P1) has a variation of ±16 clock cycles. However, some operations may require sending data from one chip 102 to another non-adjacent chip 102, for example, from chip P0 to chip P3, or even around a loop from the first chip P0 to the last chip P7. As discussed in more detail below, in order to send data from one chip to another non-adjacent chip (e.g., from P0 to P7), it is possible to use a bypass operation to send the data through each of the intermediate chips (e.g., through chips P1 to P6). However, the latency variation between chips will be accumulated across 8 chips, with the total variation in latency around the loop approaching ±128 clock cycles. To make this large variability of arrival times at the destination chip compatible with a synchronous system, extensive data buffering can be implemented at the receiver interface (eg, RX communication interface 308 ) by increasing the receive FIFO buffer size on each chip 102 .

[0074] However, additional buffering can be avoided by preventing the accumulation of latency variations throughout the multi-chip data transfer process. To achieve this, a small amount of delay can be introduced into the data transfer operation at each chip 102 so that the latency of data transmission between each pair of adjacent chips 102 is fixed rather than variable. Specifically, the maximum inter-chip latency (L max During data transmission, when data is received at chip 102 in bypass operation, the data is stored in a receive buffer (such as a FIFO buffer) instead of being immediately sent to the next chip. The data is received only after a maximum inter-chip latency (e.g., L) has elapsed since the data transmission was initiated at the previous chip 102. max ) before the data is released from the buffer. When controlling the timing of each bypass operation in the data transfer process, the exact amount of time for the entire data transfer process will then be a known value, meaning that there is no variation in the arrival time of the data perceived at the destination chip 102.

[0075] Figure 5 is a flow chart of an example process 500 for transferring data between chips in multi-chip system 100. Figure 1 , Figure 5 and Figure 6A-6B Process 500 is described. Fig. 6A and 6B A series of block diagrams illustrating a data transmission process 500 are shown. The block diagrams are similar to Figures 3A-3C , only the internal bypass data paths 602 and 604 are labeled. For example, chip 102 can include bypass data paths in both directions that allow chip 102 to route data directly to the next chip in the loop topology.

[0076] Process 500 or portions thereof are performed by individual chips 102 of multi-chip system 100. Data transfer process 500 is used to reduce the variability of data arrival times at destination chips 102 in multi-chip system 100 so as to make data communication between chips 102 more deterministic. In addition, data transfer process 500 can reduce the data input buffer size required on each chip 102. Process 500 also allows a sequence of operations performed by each chip 102 in multi-chip system 100 to be pre-scheduled and executed at pre-scheduled local counter times.

[0077] like Fig. 6A and Figure 6BAs shown, data 606 is sent from a first chip (e.g., chip A) to a second chip (e.g., chip B) (step 502). For example, data 606 is bypass data intended for another chip 102 in the multi-chip system 100, not for chip B. Chip B receives data 606 and stores data 606 in a buffer (step 504). For example, chip B stores data 606 in a FIFO buffer. Chip B stores the data until a maximum inter-chip latency has elapsed from when chip A sent the data. In the illustrated example, chip B receives data 606 at local counter time 32, and the maximum inter-chip latency is assumed to be L max = 30 counter cycles. Therefore, chip B does not send data 606 to the next chip (e.g., chip C) in multi-chip system 100 until, for example, local counter time 40 (e.g., 40 = local counter time (10) of chip A when data 606 is sent to chip B plus maximum inter-chip latency (30)). This represents an exemplary latency of eight counter cycles.

[0078] After the maximum inter-chip latency (eg, L) has elapsed since the first chip (eg, chip A) sent data 606. max ), the second chip (eg, chip B) releases the stored data from the buffer (step 506), and sends the released data 608 ( Figure 6B ) to a third chip (e.g., chip C) (step 508). For example, after 30 cycles of chip B's counter have elapsed from the time chip A sent data 606 to chip B, chip B can release data 606 from its FIFO buffer, pass the data along internal bypass path 602, and store the data (at Figure 6B 608) is sent to chip C.

[0079] In some embodiments, chip operations are explicitly scheduled with predetermined counter values. Thus, for example, the delay time for storing bypass data in a given chip buffer is taken into account in the scheduling operation. For example, referring to the example described above, the scheduling operation instruction of chip A will instruct chip A to send data 606 to chip B at chip A's local counter time 10. The scheduling operation instruction of chip B will instruct chip B to release data 606 from its input buffer and resend the data to chip C at chip B's local counter time 40. Therefore, chip B does not need to internally calculate the delay time for resending data 606.

[0080] Figure 7 It is shown that it can be used as Figure 11 is a schematic diagram of an example of a dedicated logic chip (e.g., ASIC 700) of one of the chips 102 (e.g., ASIC chips P0-P7) in a multi-chip system 100. ASIC 700 includes a plurality of tiles 702, wherein one or more of tiles 702 include dedicated circuits configured to perform operations (e.g., multiplication and addition operations). In particular, each tile 702 can include a computational array of cells (e.g., similar to Figure 1 24), wherein each unit is configured to perform a mathematical operation (see, e.g. Figure 4 ). In some embodiments, tiles 702 are arranged in a grid pattern, where tiles 702 are arranged along a first dimension 701 (e.g., rows) and along a second dimension 703 (e.g., columns). Figure 7 In the example shown, the tile 702 is divided into four different sections (710a, 710b, 710c, 710d), each section containing 288 tiles arranged in a grid of 18 tiles down by 16 tiles. In some embodiments, Figure 7 The illustrated ASIC 700 may be understood as comprising a single systolic array of cells subdivided / arranged into separate tiles, where each tile comprises a subset / subarray of cells, local memory and bus lines (see e.g. Figure 4 ).

[0081] ASIC 700 also includes a vector processing unit 704. The vector processing unit 704 includes a circuit configured to receive an output from the tile 702 and calculate a vector calculation output value based on the output received from the tile 702. For example, in some embodiments, the vector processing unit 704 includes a circuit configured to perform an accumulation operation on the output received from the tile 702 (e.g., a multiplication circuit, an adder circuit, a shifter and / or a memory). Alternatively or additionally, the vector processing unit 704 includes a circuit configured to apply a nonlinear function to the output of the tile 702. Alternatively or additionally, the vector processing unit 704 generates a normalized value, a pooled value, or both. The vector calculation output of the vector processing unit can be stored in one or more tiles. For example, the vector calculation output can be stored in a memory uniquely associated with the tile 702. Alternatively or additionally, the vector calculation output of the vector processing unit 704 can be transmitted to a circuit outside the ASIC 700, for example, as an output of the calculation.

[0082] In some embodiments, the vector processing unit 704 is segmented such that each segment includes circuitry configured to receive outputs from a corresponding tile set 702 and compute vector computation outputs based on the received outputs. Figure 7In the example shown, the vector processing unit 704 includes two rows spanning along the first dimension 701, each row including 32 segments 706 arranged in 32 columns. Each segment 706 includes circuitry (e.g., multiplication circuitry, adder circuitry, shifters, and / or memory) configured to perform vector calculations as explained herein based on outputs (e.g., accumulated sums) from the tiles 702 of the corresponding column. The vector processing unit 704 can be located in the middle of the grid of tiles 702, such as Figure 7 Other arrangements of the vector processing unit 704 are also possible.

[0083] ASIC 700 also includes a communication interface 708 (e.g., interfaces 7010a, 7010b). Communication interface 708 includes one or more groups of serializer / deserializer (SerDes) interfaces and general purpose input / output (GPIO) interfaces. The SerDes interface is configured to receive input data of ASIC 700 and output data from ASIC 700 to external circuits. For example, the SerDes interface can be configured to send and receive data at a rate of 32 Gbps, 56 Gbps, or any suitable data rate through a set of SerDes interfaces included in communication interface 708. For example, ASIC 700 can run a boot program when it is turned on. The GPIO interface can be used to load instructions (e.g., operation scheduling) onto ASIC 700 and communicate with system driver 104 to perform a boot synchronization process (e.g., process 400).

[0084] ASIC 700 also includes a plurality of controllable bus lines (see e.g. Figure 4 ), which is configured to transfer data between the communication interface 708, the vector processing unit 704, and the plurality of tiles 702. The controllable bus lines include, for example, wires extending along both a first dimension 701 (e.g., rows) of the grid and a second dimension 703 (e.g., columns) of the grid. A first subset of the controllable bus lines extending along the first dimension 701 can be configured to be connected in a first direction (e.g., to Figure 7 A second subset of controllable bus lines extending along the first dimension 701 can be configured to transmit data in a second direction (e.g., to the right of the bus line). Figure 7 The first subset of controllable bus lines extending along the second dimension 703 can be configured to transmit data in a third direction (e.g., to the left of Figure 7 A second subset of controllable bus lines extending along the second dimension 703 can be configured to transmit data in a fourth direction (e.g., to the top of Figure 7 data is transmitted on the bottom of the

[0085] Each controllable bus line includes a plurality of transmitter elements, such as triggers, which are used to transmit data along the line according to a clock signal. Transmitting data through the controllable bus line can include shifting data from a first transmitter element of the controllable bus line to a second adjacent transmitter element of the controllable bus line at each clock cycle. In some embodiments, data is transmitted through the controllable bus at a rising edge or a falling edge of a clock cycle. For example, data present on a first transmitter element (e.g., a trigger) of a controllable bus line in a first clock cycle can be transmitted to a second transmitter element (e.g., a trigger) of the controllable bus line in a second clock cycle. In some embodiments, the transmitter elements can be periodically spaced apart from each other at a fixed distance. For example, in some cases, each controllable bus line includes a plurality of transmitter elements, wherein each transmitter element is positioned within or near a corresponding tile 702.

[0086] In order to minimize the latency associated with the internal operation of the ASIC chip 700, the tiles 702 and the vector processing unit 704 can be positioned to reduce the distance that data travels between the various components. In a particular embodiment, both the tiles 702 and the communication interface 708 can be divided into multiple sections, wherein both the tile sections and the communication interface sections are arranged so that the maximum distance that data travels between the tiles and the communication interface is reduced. For example, in some embodiments, a first group of tiles 702 can be arranged in a first section on a first side of the communication interface 708, and a second group of tiles 702 can be arranged in a second section on a second side of the communication interface. As a result, the distance from the communication interface to the farthest tile can be halved compared to a configuration in which all tiles 702 are arranged in a single section on one side of the communication interface.

[0087] Alternatively, the tiles may be arranged in a different number of segments, such as four segments. Figure 7In the example shown, a plurality of tiles 702 of the ASIC 700 are arranged in a plurality of sections 710a, 710b, 710c, 710d. Each of the sections 710a, 710b, 710c, 710d includes a similar number of tiles 702 arranged in a grid pattern (e.g., each of the sections 710a, 710b, 710c, 710d can include 256 tiles arranged in 16 rows and 16 columns). The communication interface 708 is also divided into a plurality of sections: a first communication interface 7010A and a second communication interface 7010B arranged on either side of each of the sections 710a, 710b, 710c, 710d of the tile 702. The first communication interface 7010A can be coupled to the two tile sections 710a, 710c on the left side of the ASIC chip 700 via a controllable bus line. The second communication interface 7010B can be coupled to the two tile segments 710b, 710d on the right side of the ASIC chip 700 via a controllable bus line. As a result, the maximum distance that data travels to and / or from the communication interface 708 (and therefore the latency associated with data propagation) can be halved compared to an arrangement where only a single communication interface is available. Other coupling arrangements of the tile 702 and the communication interface 708 can also reduce data latency. The coupling arrangement of the tile 702 and the communication interface 708 can be programmed by providing control signals to the transmitter elements and multiplexers of the controllable bus lines.

[0088] In some embodiments, one or more tiles 702 are configured to initiate read and write operations with respect to controllable bus lines and / or other tiles (referred to herein as "control tiles") within the ASIC 700. The remaining tiles within the ASIC 700 can be configured to perform calculations (e.g., inferring with a computational layer) based on input data. In some embodiments, the control tiles include the same components and configuration as other tiles within the ASIC 700. The control tiles can be added as one or more additional tiles, one or more additional rows, or one or more additional columns of the ASIC 700. For example, for a symmetrical grid of tiles 702, where each tile 702 is configured to perform calculations on input data, one or more additional rows of control tiles can be included to handle read and write operations of tiles 702 that perform calculations on input data. For example, each of the sections 710a, 710b, 710c, 710d includes 18 rows of tiles, where the last two rows of tiles can include control tiles. In some embodiments, providing separate control tiles increases the amount of memory available in other tiles for performing calculations. However, a separate tile dedicated to providing control as described herein is not necessary, and, in some cases, a separate control tile is not provided. Instead, each tile may store instructions in its local memory for initiating read and write operations for that tile.

[0089] In addition, although Figure 7 Each of the sections 710a, 710b, 710c, 710d shown in FIG. 1 includes tiles arranged in 18 rows by 16 columns, but the number of tiles 702 and their arrangement in the sections can be different. For example, in some cases, the sections 710a, 710b, 710c, 710d can include an equal number of rows and columns.

[0090] In addition, despite the Figure 7 702 is shown as being divided into four sections, but the tiles 702 can be divided into other different groupings. For example, in some embodiments, the tiles 702 are grouped into two different sections, such as a first section above the vector processing unit 704 (e.g., closer to the Figure 7 ) and a second section below the vector processing unit 704 (e.g., closer to Figure 7 700 ). In such an arrangement, each zone may contain, for example, 596 tiles arranged in a grid of 18 tiles down (along direction 703) by 32 tiles across (along direction 701). The zones may contain other total numbers of tiles and may be arranged in arrays of different sizes. In some cases, the divisions between zones are delineated by hardware features of ASIC 700. For example, Figure 7 As shown, segments 710A, 710B may be separated from segments 710C, 710D by the vector processing unit 704 .

[0091] Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory program carrier, for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information to be sent to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0092] The term "data processing apparatus" includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus can include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC. In addition to hardware, the apparatus can also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0093] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs, ASICs, or GPGPUs (general purpose graphics processing units).

[0094] Although this specification contains many specific implementation details, these should not be interpreted as limitations on any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination can be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.

[0095] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0096] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, although the bus lines are described as "controllable", not all bus lines need to have the same level of control. For example, there can be different degrees of controllability, where some bus lines can be controlled only when some bus lines are limited in the number of tiles from which they can obtain data or to which they can send data. In another example, some bus lines may be dedicated to providing data in a single direction (e.g., north, east, west, or south as described herein). In some cases, the actions recited in the claims can be performed in different orders and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for characterizing inter-chip waiting time, comprising: determining, for each pair of chips among a plurality of chips of a semiconductor device, a corresponding loop latency for round-trip data transmission between the pair of chips around a transmission path through the plurality of chips; identifying a maximum loop waiting time from the loop waiting times; determining a full path latency for a data transmission originating from a chip among the plurality of chips to be sent around the path and back to the chip; Half of the maximum loop latency is compared to one-Nth of the full path latency, where N is the number of chips in the transmission path of the chip, and the larger value is stored as the inter-chip latency of the semiconductor device, wherein the inter-chip latency represents an operating characteristic of the semiconductor device.

2. The method according to claim 1, wherein: Determining the loop latency of the round-trip data transmission between the pair of chips includes: sending first time-stamped data from a first chip in the pair of chips to a second chip in the pair of chips; determining a first relative one-way waiting time between the pair of chips based on the first time-stamped data; sending second time-stamped data from the second chip to the first chip; determining a second relative one-way latency between the pair of chips based on the second time-stamped data, and A loop latency for round-trip data transmission between the pair of chips is determined based on the first relative one-way latency and the second relative one-way latency.

3. The method according to claim 2, wherein: The first time-stamped data indicates a local counter time of the first chip when the first time-stamped data is sent.

4. The method according to claim 2, wherein: Determining a first relative one-way latency between the pair of chips includes calculating a difference between a time indicated in the time-stamped data and a local counter time of the second chip when the second chip receives the first time-stamped data.

5. The method according to claim 2, wherein: Determining a round trip latency for data transmission between the pair of chips includes calculating a difference between a first relative one-way latency and a second relative one-way latency.

6. The method according to claim 2, wherein: One or more of the plurality of chips are application specific integrated circuit (ASIC) chips configured to perform neural network operations.

Images