Data transfer between memory and distributed computing arrays

Abstract

An integrated circuit (IC) comprises a plurality of dies. The IC comprises a plurality of memory channel interfaces configured to communicate with IC memory, and the plurality of memory channel interfaces are disposed within a first die among the plurality of dies. The IC may comprise a computation array distributed across the plurality of dies and a plurality of remote buffers distributed across the plurality of dies. The plurality of remote buffers are coupled to the plurality of memory channels and the computation array. The IC further comprises a controller configured to determine that data is stored in each of the plurality of remote buffers and, in response to this determination, to broadcast a read enable signal to each of the plurality of remote buffers to initiate data transfer from the plurality of remote buffers to the computation array across the plurality of dies.

Description

Technology Field

[0001] The present disclosure relates to an integrated circuit (IC), and more specifically, to data transfer between a computational array distributed across a plurality of dies of the IC and a memory. Background Technology

[0002] A neural network processor (NNP) refers to a type of integrated circuit (IC) having one or more computation arrays capable of implementing a neural network. The computation array(s) receive data, for example, weights for the neural network implementation, from memory. The computation array(s) receive weights from memory in parallel through multiple memory channels. Skew typically occurs when transferring data from memory to the computation array(s). Consequently, data reaches different parts of the computation array(s) at different times. Data skew is attributed to the independence between memory channels while operating at least partially in parallel; in the case of a multi-die IC with computation array(s) distributed across multiple dies, the data wavefront from each memory channel is orthogonal to the computation array(s) of the multi-die IC. Whether viewed individually or cumulatively, this problem makes data transfer from memory to the computation array unpredictable and leads to inefficient and / or degraded use of available bandwidth from memory.

[0003] An exemplary implementation includes an integrated circuit (IC). The IC includes a plurality of dies. The IC includes a plurality of memory channel interfaces configured to communicate with memory, and the plurality of memory channel interfaces are located within a first die among the plurality of dies. The IC may include a compute array distributed across the plurality of dies and a plurality of remote buffers distributed across the plurality of dies. The plurality of remote buffers are coupled to the plurality of memory channels and the compute array. The IC further includes a controller configured to determine that data is stored in each of the plurality of remote buffers and, in response to this determination, broadcast a read enable signal to each of the plurality of remote buffers to initiate data transfer from the plurality of remote buffers to the compute array across the plurality of dies.

[0004] Another exemplary implementation includes a controller. The controller is placed within an IC having multiple dies. The controller includes a request controller configured to convert a first request for access to memory into a second request compatible with an on-chip communication bus, and the request controller provides the second request to multiple request buffer bus master circuit blocks configured to receive data from multiple channels of memory. The controller further includes a remote buffer read address generation unit coupled to the request controller and configured to monitor the fill level of each of multiple remote buffers distributed across multiple dies. Each of the multiple remote buffers is configured to provide data obtained from each block of the multiple request buffer bus master circuit blocks to a computation array also distributed across multiple dies. In response to determining that each of the multiple remote buffers is storing data based on the fill level, the remote buffer read address generation unit is configured to initiate data transfer from each of the multiple remote buffers to the computation array.

[0005] Another exemplary implementation includes a method. The method comprises: monitoring the fill level of a plurality of remote buffers distributed across a plurality of dies—each of the plurality of remote buffers is configured to provide data to a compute array also distributed across a plurality of dies—determining, based on the fill level, that each of the plurality of remote buffers is storing data; and, in response to the determination, initiating a data transfer from each of the plurality of remote buffers to a compute array distributed across a plurality of dies.

[0006] The content section of this invention is provided merely to introduce specific concepts but not to identify any core or essential features of the claimed subject matter. Other features of the array of the invention will be apparent from the accompanying drawings and the following detailed description. Brief explanation of the drawing

[0007] The arrangement of the present invention is illustrated by way of example in the accompanying drawings. However, the drawings should not be interpreted as limiting the arrangement of the present invention only to the specific implementations shown. Various aspects and advantages will become apparent when reviewing the following detailed description and referring to the drawings. FIG. 1 illustrates an exemplary plan view of a circuit architecture implemented within an integrated circuit (IC). Figure 2 illustrates an exemplary implementation of the circuit architecture of Figure 1. Figure 3 illustrates another exemplary implementation of the circuit architecture of Figure 1. Figure 4 illustrates an example of a balanced tree structure used to implement the circuit architecture of Figure 1. FIG. 5 illustrates an exemplary implementation of a request buffer-bus master (RBBM) circuit block as described in the present disclosure. Figure 6 illustrates an exemplary implementation of a master controller. Figure 7 illustrates an exemplary implementation of a request controller. FIG. 8 illustrates an exemplary method of transferring data between a high-bandwidth memory and a distributed computing array. Figure 9 illustrates an exemplary architecture of an IC. Specific details for implementing the invention

[0008] Although the present disclosure concludes with claims defining novel features, it is believed that the various features described in the present disclosure will be better understood by considering the description together with the drawings. The process(s), machine(s), manufactured(s), and any variations thereof described herein are provided for illustrative purposes. Specific structural and functional details described in the present disclosure should not be construed as limiting, but merely as a basis for the claims and as a representative basis to instruct those skilled in the art to make various uses of the features described in any suitably detailed structure. Furthermore, the terms and phrases used in the present disclosure are intended not to be limiting, but to provide an understandable description of the described features.

[0009] The present disclosure relates to an integrated circuit (IC), and more specifically, to data transfer between a computation array distributed across multiple dies of an IC and memory. A neural network processor (NNP) refers to a type of integrated circuit (IC) having one or more computation arrays capable of implementing a neural network. If the IC is a multi-die IC, the IC may implement a single, larger computation array distributed across two or more dies of the multi-die IC. Implementing a single, larger computation array distributed across multiple dies, in contrast to multiple smaller, independent computation arrays located on different dies, provides certain advantages, including but not limited to improved latency, improved weight storage capacity, and improved computational efficiency.

[0010] The computation array receives data, such as weights for a neural network, from high-bandwidth memory or HBM. For the purposes of explanation, memory accessed by memory channels is referred to as "high-bandwidth memory" or "HBM" throughout this publication to better distinguish it from other types of memory in the circuit architecture, such as buffers and / or queues. However, it should be understood that HBM can be implemented using any of various different technologies that support multiple independent and parallel memory channels that are communically linked to the exemplary circuit architecture described through appropriate memory controllers. Examples of HBM may include any various RAM type memory, such as double data rate RAM or other suitable memory.

[0011] Although distributed across multiple dies of the IC, the compute array is considered as a single compute array with weights supplied in parallel through memory channels available in HBM. For example, the compute array can be implemented as an array where each die implements one or more rows of the compute array. Each memory channel can provide data to one or more rows of the compute array.

[0012] In the case of a single compute array distributed across multiple dies, data transfer from HBM to the compute array often suffers from timing issues. For example, each memory channel typically has its own independent control pin, asynchronous clock, and refresh sleep mode. These features cause data skew across the memory channels. Consequently, different rows of the compute array often receive data at different times. Data skew is exacerbated because different rows of the compute array are located on different dies of the IC and thus have different distances from the HBM. For instance, the data wavefront (e.g., data propagation) from the memory channels is orthogonal to the IC's compute array. These issues contribute to the overall unpredictability of data transfer from HBM to the compute array.

[0013] According to the arrangement of the invention described in this disclosure, an exemplary circuit architecture is provided that can schedule read requests for HBM across memory channels while improving and / or maximizing HBM bandwidth utilization. The exemplary circuit architecture can also de-skew data transfer between HBM and compute arrays. As a result, data can be provided from HBM to different rows of compute arrays across the dies of multi-die ICs in a synchronized manner with reduced skew. This allows the compute array to remain busy while making more full use of the HBM read bandwidth.

[0014] The exemplary circuit architecture can also reduce the overhead and complexity of distributing computational arrays across multiple dies of a multi-die IC. The exemplary circuit architecture described herein can be adapted to multi-die ICs having different numbers of dies internally. As the number of dies of a multi-die IC changes by model and / or the area of ​​each die changes, the exemplary circuit architecture described herein can be adapted to these changes to improve data transfer between the various dies of a multi-die IC and the HBM where the computational array is distributed.

[0015] Throughout this disclosure, the AMBA (Advanced Microcontroller Bus Architecture) AXI (eXtensible Interface) (hereinafter "AXI") protocol and communication bus are used for illustrative purposes only. AXI defines an embedded microcontroller bus interface to be used to establish on-chip connections between compliant circuit blocks and / or systems. AXI is provided as an exemplary example of a bus interface and is not intended to be a limitation of the examples described in this disclosure. It should be recognized that other similar and / or equivalent protocols, communication buses, bus interfaces, and / or interconnects may be used instead of AXI, and that the various exemplary circuit blocks and / or signals provided in this disclosure will differ based on the specific protocol, communication bus, bus interface, and / or interconnect used.

[0016] Further aspects of the arrangement of the present invention are described in more detail below with reference to the drawings. For the sake of simplicity and clarity of example, the elements depicted in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated compared to others for clarity. Additionally, where deemed appropriate, reference numbers are repeated in the drawings to indicate corresponding, similar, or alike features.

[0017] FIG. 1 illustrates an exemplary plan view of a circuit architecture implemented in an IC (100). The IC (100) is a multi-die IC and includes a computation array. The computation array is distributed across the dies (102, 104, and 106) of the IC (100). For illustrative purposes, the IC (100) is illustrated with three dies. In other examples, the IC (100) may have fewer or more dies than illustrated.

[0018] In this example, the computation array is subdivided into 256 computation array rows. Computation array rows 0 through 95 are implemented in die (102). Computation array rows 96 through 191 are implemented in die (104). Computation array rows 192 through 255 are implemented in die (106). The computation array may include a digital signal processing (DSP) cascade chain connected together across dies (102, 104, and 106).

[0019] Data, e.g., weights, are obtained from an HBM (not shown) that is communicably connected to the IC (100) via multiple memory channels. In one aspect, the HBM is implemented in a separate IC (e.g., outside the package for the IC (100)) and on the same circuit board as the IC (100). In another aspect, the HBM is implemented as another die within the IC (100) (e.g., within the same package as the IC (100)). The HBM may be located along the bottom of the IC (100), for example, from left to right, adjacent to the bottom of the die (106). For some HBMs, the memory channels are referred to as pseudo channels (PCs). For illustrative purposes, the term "memory channel" is used to refer to the memory channels of the HBM and / or the PCs of the HBM.

[0020] In the example of FIG. 1, the die (106) includes 16 memory controllers 0 through 15. Each memory controller can service two memory channels (e.g., read and / or write). The memory controllers 0 through 15 of FIG. 1 are labeled in parentheses to indicate the specific memory channels that each memory controller services. For example, memory controller 0 services memory channels 0 and 1, and memory controller 1 services memory channels 2 and 3.

[0021] Each memory controller is connected to one or more request buffers and one or more bus master circuits (e.g., AXI masters). In one exemplary implementation illustrated in FIG. 1, each memory channel is coupled to one request buffer bus master circuit block through the memory controller. In FIG. 1, each combination of request buffer bus master circuit blocks (e.g., where the bus master may be an AXI master) is abbreviated as "RBBM circuit blocks" and is illustrated as "RBBM" in this figure. Since each memory controller can serve two memory channels, there are two RBBM circuit blocks located directly above each memory controller and coupled to this controller. Each RBBM circuit block is labeled for a specific memory channel served by the RBBM circuit block. Thus, the example in FIG. 1 includes RBBM circuit blocks 0 through 15. In this example, the memory controller and the RBBM circuit blocks are both located on one, for example, the same die of the IC (100).

[0022] Each of the dies (102, 104, and 106) includes a plurality of remote buffers. The remote buffers are distributed across the dies (102, 104, and 106). In the example of FIG. 1, each RBBM circuit block is connected to a plurality of remote buffers. In one example, each RBBM circuit block is connected to four different remote buffers. For example, RBBM circuit block 0 is connected to remote buffers 0 through 3 to provide data. RBBM circuit block 1 is connected to remote buffers 4 through 7 to provide data, and other RBBM circuit blocks are likewise connected. Each of the remaining RBBM circuit blocks can be connected to a consecutively numbered group of four remote buffers continuing from die (102) through the remote buffers of dies (104 and 106).

[0023] Each of the dies (102, 104, and 106) also includes a plurality of caches. Generally, the number of caches (e.g., 32) corresponds to the number of memory channels. Each cache can provide data to a plurality of compute array rows. In the example of FIG. 1, each cache can provide data to eight compute array rows. Die (102) includes caches 0 through 11, wherein cache 0 provides data to compute array rows 0 through 7; cache 1 provides data to compute array rows 8 through 15; cache 2 provides data to compute array rows 16 through 23, and other caches are likewise. Die (104) includes caches 12 through 23, wherein cache 12 provides data to compute array rows 96 through 103; cache 13 provides data to compute array rows 104 through 111; Cache 14 provides data to rows 112 through 119 of the computation array, and other caches are like this. Die (106) includes caches 24 through 31, where cache 24 provides data to rows 192 through 199 of the computation array; cache 25 provides data to rows 200 through 207 of the computation array; cache 26 provides data to rows 208 through 215 of the computation array, and other caches are like this.

[0024] In this example, data such as weights, for example, can be loaded from the HBM through 32 memory channels implemented in the die (106). Ultimately, the weights are supplied to the rows of the compute array as multiplication operands. The weights are input to the IC (100) via parallel memory channels through memory controllers 0 to 15 of the die (106). Within the die (106), RBBM circuit blocks per memory channel are placed next to each memory channel to handle flow control between the rows of the compute array associated with each memory channel. The RBBM circuit blocks are controlled by a master controller (108) to perform read and write requests (e.g., "access") to the HBM.

[0025] In the example of FIG. 1, the memory channel is located far from the remote buffer. The master controller (108) is also located far from the memory channel, which is located closer to the right side of the die (106). Additionally, the data wavefront enters the rows of the compute array (e.g., horizontally) in a direction orthogonal (e.g., vertically) to the data wavefront entering the IC (100) through the memory controller.

[0026] Data read from the HBM is written from the RBBM circuit block to each of the remote buffers 0 through 127 of the dies (102, 104, and 106). The read side of each remote buffer (e.g., the side connected to the cache) is controlled by a master controller (108). The master controller (108) controls the read side of each remote buffer 0 through 127 to perform read de-skew operations across the dies (102, 104, and 106) and supplies data to each cache 0 through 31 for various compute array rows 0 through 255.

[0027] The master controller (108) can coordinate data transfers entering and leaving the remote buffer so that data is dequeued. By coordinating reads from the remote buffer, the master controller (108) ensures that data, e.g., weights, are provided to each row of the compute array in a synchronized manner. Furthermore, the master controller (108) can improve and / or maximize HBM bandwidth usage. This allows the compute array to remain in use while making more full use of the HBM read bandwidth.

[0028] As mentioned, the IC (100) may include fewer or more dies than exemplified in FIG. 1. In this regard, the circuit architecture in the example of FIG. 1 can reduce the overhead and complexity of data distribution for computation arrays distributed across multiple dies of a multi-die IC, regardless of whether such an IC includes fewer or more than three dies. The exemplary architecture described herein can be adapted to multi-die ICs having a different number of dies than illustrated. Additionally, as illustrated in FIG. 1, the sizes of the dies (102, 104, and 106) are merely examples to better illustrate the components of each individual die. The dies may have the same size or different sizes.

[0029] FIG. 2 illustrates an exemplary implementation of the circuit architecture of FIG. 1. In the example of FIG. 2, the die boundaries are removed. The IC (100) may be placed on a circuit board connected to a host computer via a communication bus. For example, the IC (100) may be connected to the host computer via a PCIe (Peripheral Component Interconnect Express) connection or other suitable connection. The IC (100), for example, die (106), may include a PCIe direct memory access (DMA) circuit (202) to facilitate PCIe connection. The PCIe DMA circuit (202) is connected to a Block Random Access Memory (BRAM) controller (204) via a connection (206). In an exemplary implementation, one or more of the request buffer and / or remote buffer of the RBBM circuit block are implemented using BRAM.

[0030] The master controller (108) is connected to the BRAM controller (204). The BRAM controller (204) can operate as an AXI endpoint slave for integration with the AXI interconnect and the system master device to communicate with local storage (e.g., BRAM). In the example of FIG. 2, the BRAM controller (204) can operate as a bridge between PCIe and the master controller (108). In one aspect, the master controller (108) is a centralized controller driven by a command queue via a host computer-PCIe connection (e.g., received through the PCIe DMA circuit (202) and the BRAM controller (204)).

[0031] The master controller (108) can implement a plurality of different operations. For example, the master controller (108) can implement a narrow write request for the HBM to initialize the HBM. In this case, the master controller (108) can access all 32 memory channels of the HBM through the AXI master (31) and the memory controller (15) (of the RBBM circuit block (31)) using a global address (e.g., using only a global address).

[0032] The master controller (108) can also implement narrow read requests from the HBM. In this case, the master controller (108) can access all 32 memory channels of the HBM through the AXI master (31) and the memory controller (15) using global addresses (e.g., using only global addresses).

[0033] The master controller (108) can also implement a wide read request from the HBM. The master controller (108) can implement read requests (e.g., both sequential and random) from all 32 memory channels in parallel through bus master circuits 0 to 31 (e.g., of RBBM circuit blocks 0 to 31) for memory controllers 1 to 15 in parallel using local memory channel addresses.

[0034] In the example of FIG. 2, the master controller (108) can monitor and / or track various signals. Additionally, the master controller (108) can generate various different signals in response to detecting specific conditions in the monitored signals. For example, the master controller (108) can generate a signal (208). The signal (208) is a remote buffer read enable signal. The master controller (108) can generate the signal (208) and a broadcasting signal (208) (e.g., the same signal) to each of the remote buffers 0 through 127. In this way, the master controller (108) can perform a read that causes each remote buffer to dequeue data read from the remote buffer and provided to the rows of the compute array in a synchronized manner.

[0035] The signal (210) is a remote buffer write enable signal. Each bus master circuit 0 to 31 of the RBBM circuit blocks 0 to 31 can generate the signal (210) to the corresponding remote buffer. The master controller (108) can receive each remote buffer write enable signal generated by the bus master circuits 0 to 31 for each remote buffer. In one aspect, the master controller (108) can monitor the fill level of each remote buffer by tracking the write enable signal (210) from each remote buffer and the read enable signal (208) provided to each remote buffer.

[0036] Signal (212) is identical to signal (208). However, the signal is generated according to a clock signal different from signal (208) (e.g., axi_clk rather than sys_clk). In this way, the master controller (108) can also provide the remote buffer read enable signal, which is broadcast to each of remote buffers 0 through 127, to the AXI master of each of RBBM circuit blocks 0 through 31, for example, each of RBBM circuit blocks 0 through 31. In this way, the remote buffer fill level can also be tracked by a remote buffer pointer manager implemented locally in each RBBM circuit block. The remote buffer pointer manager(s) are described in more detail in relation to FIG. 5.

[0037] Signal (214) represents an AXI-AW / W / B signal, which the master controller (108) may provide to the RBBM circuit block (31) to initiate a narrow write as previously described. In this disclosure, AXI-AW refers to the AXI write address signal; AXI-W refers to the AXI write data signal; AXI-B refers to the AXI write response signal; AXI-AR refers to the AXI read address signal; and AXI-R refers to the AXI read data signal. The AXI master controller (108) also receives a signal (216) from each of the RBBM circuit blocks 0 through 31. The signal (216) may be an AR REQ ready signal (e.g., where "AR" refers to "address read"). The master controller (108) can also initiate HBM reading by broadcasting a signal (218), for example, an AR REQ broadcast, to each of RBBM circuit blocks 0 to 31.

[0038] The exemplary circuit architecture of FIG. 2 includes multiple different clock domains. dsp_clk is used to clock the compute array rows and the output ports of caches 0 through 31. In one example, dsp_clk is set to 710 MHz. Using an 8x16b connection between each cache 0 through 31 and the 8 compute array rows supplied by each individual cache, a data transfer rate of 0.355 TB / s is achieved (8x16x32 bits * 710 MHz).

[0039] sys_clk is used to clock the input ports (e.g., right) of caches 0 through 31 connected to the remote buffers and the output ports (e.g., left) of remote buffers 0 through 127 connected to caches 0 through 31. sys_clk is also used to clock a portion of the master controller (108) to broadcast a signal (208) to each remote buffer, for example. In one example, sys_clk is set to 355 MHz. Using the illustrated 8x32b connection between remote buffers 0 through 127 and caches 0 through 31, a data rate of 0.355 TB / s is achieved (8x32x32 bits * 355 MHz). For example, sys_clk can be set to half or about half the frequency of dsp_clk.

[0040] In the example of FIG. 2, caches 0 through 31 can not only cache data but also traverse the clock domain. More specifically, each of caches 0 through 31 can receive data at a sys_clk rate and output data to a compute array row at a dsp_clk rate, for example, twice the clock rate of the input. In one or more exemplary implementations, circuits such as remote buffers and RBBM circuit blocks may be implemented as programmable logic units having a slower clock rate than other hardwired circuit blocks, such as those that can be used to implement the compute array row. Thus, caches 0 through 31 can bridge this difference in clock rates.

[0041] axi_clk is used to clock the input ports of the remote buffer (e.g., right) and the output ports of the RBBM circuit blocks (e.g., left). axi_clk is also used to clock a portion of the master controller (108) to monitor, for example, the received signals (210 and 216) and the output signals (212, 214, and 218). In one example, axi_clk is set to 450 MHz. A data rate of 0.45 TB / s is achieved with a 4x64b connection between RBBM circuit blocks 0 through 31 and remote buffers 0 through 127 (4x64x32 bits * 450 MHz).

[0042] Each RBBM circuit block is coupled to a corresponding memory controller via a 256b connection to achieve a data rate of 0.45 TB / s (32 x 256 bits * 450 MHz). Memory controllers 0 through 15 can also be clocked at 450 MHz. Each memory controller supports two 64b memory channel connections (e.g., one for each memory channel) providing a data rate of 0.45 TB / s (2048 bits / T * 1.8 GT / s).

[0043] For the purposes of discussion, the term “memory channel interface” is used within this disclosure to refer to a specific RBBM circuit block and a corresponding part of the memory controller to which the RBBM circuit block is connected (e.g., a single channel). For example, RBBM circuit block 0 and a part of memory controller 0 connected to RBBM circuit block 0 (e.g., a data buffer (302-0) and a request queue (304-0) with reference to FIG. 3) are memory channel interfaces, whereas a part of memory controller 0 connected to RBBM circuit block 0 and RBBM circuit block 1 (e.g., a data buffer (302-1) and a request queue (304-1)) is considered another memory channel interface.

[0044] The master controller (108) can generate read and write requests in accordance with HBM read and write commands received via PCIe DMA (202) and BRAM controller (204). The master controller (108) can operate in a "hurry up and wait" mode. For example, the master controller (108) can send read requests to the request buffers of RBBM circuit blocks 0 through 31 until the data path, including the request buffer and the remote buffer, is full. In response to each read command, the master controller (108) can also initiate a data read operation (e.g., data transfer) from each remote buffer to the corresponding cache. Additionally, the master controller (108) releases some request buffer space and triggers the master controller (108) to generate a new read request to obtain additional data from the HBM based on the available space in the request buffer.

[0045] In one exemplary implementation, the HBM includes 16 banks within each PC and 32 columns within each row. By interleaving the banks, up to 16x32x256 bits (128Kb) can be read by each PC. If the size of the BRAM is 4x36Kb, one 36Kb BRAM can be used to buffer two rows of compute arrays. As such, the exemplary circuit architecture of FIG. 2 can read up to 16 interleaved pages from one PC in burst lengths of 512 bits to serve eight rows of compute arrays at a time.

[0046] FIG. 3 illustrates another exemplary implementation of the circuit architecture of FIG. 1. In the example of FIG. 3, the die boundaries are removed. Additionally, each memory controller 0 to 31 (abbreviated as "MC" in FIG. 3) is coupled to two memory channels. FIG. 3 provides a more detailed drawing of memory controllers 0 to 15 and RBBM circuit blocks 0 to 31.

[0047] In the example of FIG. 3, each memory controller 0 through 15 serves two memory channels. As such, each memory controller 0 through 15 includes one data buffer (302) for each memory channel being served and one request queue (304) for each memory channel being served. For example, memory controller 0 includes a data buffer (302-0) and a request queue (304-0) for serving memory channel 0, and a data buffer (302-1) and a request queue (304-1) for serving memory channel 1. Similarly, memory controller (15) includes a data buffer (302-30) and a request queue (304-30) for serving memory channel (30), and a data buffer (302-31) and a request queue (304-31) for serving memory channel (31).

[0048] As an exemplary example using the AXI protocol, the data buffer (302) may be implemented as an AXI-R (read) data buffer. Each data buffer (302) may contain 64x16 (1024) entries, where each entry is 256 bits. The request queue (304) may be implemented as an AXI-AR (address read) request queue. Each request queue (304) may contain 64 entries. Each data buffer (302) receives data from a corresponding memory channel. Each request queue (304) may provide commands, addresses, and / or control signals received from a corresponding AXI master to the corresponding memory channel.

[0049] Each of RBBM circuit blocks 0 through 31 includes a bus master circuit and a request buffer. For example, RBBM circuit block 0 includes a bus master circuit 0 and a request buffer 0. RBBM circuit block 1 includes a bus master circuit 1 and a request buffer 1. RBBM circuit block 30 includes a bus master circuit 30 and a request buffer 30. RBBM circuit block 31 includes a bus master circuit 31 and a request buffer 31. Accordingly, each bus master circuit has a data connection to a corresponding data buffer (302) and a control connection to a corresponding request queue (304) (e.g., for an address, a control signal, and / or a command).

[0050] Skew is introduced between data read from HBM through different memory channels due to HBM refresh, clock domain crossing, and two memory channels interleaved in a single memory controller. In the case of memory channel skew, data read from HBM through different memory channels is not aligned. This is the case even when all read requests for all 32 memory channels are issued in parallel by each of the 32 AXI masters in the same cycle. Data skew is at least partially due to HBM refresh.

[0051] An example is considered where the HBM has a global refresh period of 260 ns every 3900 ns. In this case, the HBM throughput is limited by refresh commands to 0.42 TB / s ((3900-260) / 2900*0.45 = 0.42). This also implies a refresh window period of 117 axi_clk cycles (260 * 0.45 = 117) every 1755 axi_clk cycles (3900*0.45 = 1755), during which no HBM read or write requests can be issued through the memory channels to the HBM. Since these refresh windows are not aligned across all 32 memory channels, the maximum skew between any two memory channels is 117 axi_clk cycles when there are no overlapping refresh periods between them. If the memory controller can generate new requests every two axi_clk cycles, the skew is that there can be up to 59 (117 / 2 = 59) HBM read requests between any two memory channels during the period when one of the two memory channels has already issued 59 read requests, while the other memory channel is blocked due to the performance of refresh.

[0052] In the example of FIG. 3, each request queue (304) can be used to queue up to 64 HBM read requests initiated by the master controller (108) through the corresponding bus master circuit during the refresh command period. In this case, 59 HBM read requests accumulated during the refresh command period can be absorbed into the 64 entry request queue (304). The master controller (108) can monitor the FIFO ready or full status of each request queue (304) by monitoring the signal (216) from each RBBM circuit block (e.g., AR REQ ready signals 0 to 31 from all memory channels for HBM for wide read requests). For example, based on the state of each request queue (304), in response to determining that space is available in each data buffer (302), the master controller (108) may generate a new HBM read request (e.g., for each memory channel) and broadcast this request to each of memory controllers 0 through 15 (e.g., via a bus master circuit). That is, the master controller (108) transmits the HBM read request to the request buffer. Each bus master circuit serves the request from the local request buffer. If any of the request queues (304) are full, the master controller (108) does not generate a new HBM read request.

[0053] Regarding the fill level of each request queue (304) across each of the 32 memory channels, two cases may occur for an HBM wide read request. The first case corresponds to a state where the circuit architecture is ready for a new HBM read request. In the first case, buffer space is available in each (e.g., all) request queue (304) to receive a new HBM read request from the master controller (108). The second case corresponds to a state where the circuit architecture is not ready for a new HBM read request. In the second case, one or more of the request queues (304) are full, and at least one other request queue (304) is neither full nor empty. A situation does not occur where some request queues (304) are full and others are empty because the maximum skew (e.g., 59) between any two memory channels is smaller than the buffer size (e.g., 64) of the request queue (304). In the second case, since there are still pending HBM read requests in all request queues (304), HBM throughput is not affected by serving new requests from the master controller (108).

[0054] Referring to both FIG. 2 and FIG. 3, there are a total of 32 data streams. Each data stream has a width of 256 bits and extends from the memory channel to the corresponding remote buffer. As shown in FIG. 1, some remote buffers are located on the die (102 or 104), while other remote buffers are located on the die (106) closer to the master controller (108). In some exemplary arrangements, since the data path for each memory channel is 256 bits wide, hardware resources can be minimized by keeping these data paths relatively short.

[0055] FIG. 4 illustrates an example of a balanced tree structure used to implement the circuit architecture of FIG. 1. The balanced tree structure is used to broadcast HBM wide read requests from the master controller (108) to the memory controller.

[0056] As illustrated in the example of FIG. 4, the master controller (108) broadcasts the signal (218) (AR REQ broadcasting signal) from left to right to various RBBM circuit blocks. For illustrative purposes, only RBBM circuit blocks (31, 16, and 0) are shown. The arrival times of the signal (218) in each RBBM circuit block are aligned in the same axi_clk cycle. Additionally, the master controller (108) may broadcast the signal (208) (e.g., remote buffer read enable) to each remote buffer. For illustrative purposes, only remote buffers (0 to 3, 64 to 67, and 124 to 127) are shown.

[0057] In the example of FIG. 4, each RBBM circuit block includes a remote buffer pointer manager (406) (illustrated as 406-31, 406-16, and 406-0). The remote buffer pointer manager (406) may be included as part of the request buffer or implemented separately from the request buffer within each individual RBBM circuit block. Each remote buffer pointer manager (406) may receive a signal (208) for the purpose of tracking the fill level of the corresponding remote buffer. Additionally, each remote buffer pointer manager (406) may output a signal (210) to the corresponding remote buffer (e.g., remote buffer write enable signals (210-31, 210-16, and 210-0)).

[0058] For example, a flip-flop (FF) (402) of die (104) receives a 256-bit wide data signal from the RBBM circuit block (31) of die (106). The FF (402) transmits the data to the FF (404) of die (102). The FF (404) transmits the data to remote buffers 0 through 3. The remote buffer point manager (406-31) can output a control signal (210-31) to the FF (408) of die (104). The FF (408) outputs the control signal (210-31) to the FF (410) of die (102). The FF (410) outputs the control signal (210-31) to remote buffers 0 through 3. The master controller (108) generates a signal (208), for example, a remote buffer ready signal, and broadcasts it to remote buffers 124 through 127 of the die (106). The signal (208) continues from the die (104) to the FF (412). The FF (412) outputs the signal (208) to remote buffers 64 through 67. The FF (412) outputs the signal (208) to the FF (414) of the die (102). The FF (414) provides the signal (208) to remote buffers 0 through 3.

[0059] The FF (416) of the die (104) receives a 256-bit wide data signal from the RBBM circuit block (16) of the die (106). The FF (416) transfers the data to remote buffers 64 through 67. The remote buffer point manager (406-16) can output a control signal (210-16) to the FF (418) of the die (104). The FF (418) outputs the control signal (210-16) to remote buffers 64 through 67.

[0060] RBBM circuit block 0 directly outputs a 256-bit wide data signal to remote buffers 124 to 127 of the die (106). The remote buffer pointer manager (406-0) directly outputs a control signal (210-0) to remote buffers 124 to 127 within the die (106).

[0061] Generally, the skew resolved by the arrangement of the invention described in this disclosure has several different components. For example, the skew includes components caused by HBM refresh and components caused by data propagation delays, such as signals crossing the clock domain, data latency along different paths on the same die, and data latency along different paths across different dies. HBM refresh is the primary cause of the skew. This aspect of the skew is primarily handled, for example, deskewed, by the read data buffer within the memory controller. Other skew components that contribute less to the total skew than HBM skew, even when considered collectively, may be handled by remote buffers. For this reason, there is no need to balance the HBM read data latency from the memory controller to the remote buffers across 32 memory channels. Since the remote buffer has sufficient storage space to tolerate the remote buffer write-side skew introduced by the described latency imbalance, the minimum pipeline stage can be used to transmit data from the memory controller to the remote buffer of the die (102, 104, and 106).

[0062] The example in FIG. 4 also illustrates a method by which a remote buffer can generate back pressure on a memory controller for each memory channel. The back pressure is generated at least partially by locally generating each control signal (210) at each individual remote buffer pointer manager (406) inside each RBBM circuit block very close to the memory channel. Additionally, the master controller (108) broadcasts a common signal (208) using a balanced tree structure.

[0063] Read and write pointers for the remote buffer are generated in each individual remote buffer across the dies (102, 104, and 106). The signal (210) must match the delay of the remote buffer write data. Due to the skew between the write side of the remote buffer receiving data from various memory channels and the skew between the signal (210), the write data latency, and the remote data read enable latency, the remote buffer back pressure may include some buffer margin space to absorb these skews. The signal (210) and the signal (216) (not shown in FIG. 4) are propagated back to the master controller (108). The delay of these signals may be balanced to ensure that the master controller (108) captures the signal in the same cycle to generate both a new HBM read request and the signal (208).

[0064] FIG. 5 illustrates an exemplary implementation of an RBBM circuit block as described in the present disclosure. In the example of FIG. 5, the bus master circuit (502) includes two read channels. The AXI master (502) includes an AXI-AR master (504) capable of processing read address data and an AXI-R master (506) capable of processing read data. The other three AXI write channels: AXI-AW (write address), AXI-W (write data), and AXI-B (write response) can be processed within the master controller (108). For illustrative purposes, the signal label "AXI_Rxxxx" is intended to refer to an AXI signal from a related communication standard having the prefix "AXI_R", and the signal label "AXI_ARxxxx" is intended to refer to an AXI signal from a related communication standard having the prefix "AXI_AR".

[0065] The request buffer (508) is the request buffer previously described as being part of the RBBM circuit block and is used to handle the HBM read request queue backpressure round-trip delay between the AR REQ ready (e.g., signal (216)) and the AR REQ signal (e.g., signal (218)). For example, the depth of the request buffer (508) must be greater than the backpressure round-trip delay (e.g., less than 32).

[0066] The AR REQ (address read request) signal (218) may specify a read start address, which is a 23-bit local address for a wide read request from the master circuit (108) via memory controllers 0 through 15, or a 28-bit global address for a narrow read request from the master circuit (108) performed only via memory controller (15). The AR REQ signal (218) may also include a read transaction identifier, which is 6 bits for narrow and wide identification and specifies other stream-related information. The AR REQ signal (218) may also specify a read burst length, which is 4 bits supporting a burst length of up to 16 bytes depending on the specific AXI protocol used. In one exemplary implementation, three combinatorial logic block memories (CLBMs), each CLBM being a 32x14 dual-port RAM, may be used to implement a 32x42 FIFO for the request buffer (508). In that case, the request buffer (508) can serve the AXI-AR master (504) with a new read request every axi_clk cycle.

[0067] The remote buffer pointer manager (406) is used to generate a signal (210) (remote buffer write enable) for read data received from the HBM. The remote buffer pointer manager (406) can also locally generate remote buffer back pressure on the AXI-R (data read) channel corresponding to the AXI-R master (506). Due to skew between different memory channels, each remote buffer pointer manager (406) can maintain the fill level of the corresponding remote buffer by tracking the remote buffer write enable and the remote buffer read enable. Each remote buffer pointer manager (406) can increase the remote buffer fill level when the HBM read data is valid by the indication of the BRAM_WVALID signal. Each remote buffer pointer manager (406) can also decrease the remote buffer fill level when receiving a broadcasted signal (208) (remote buffer read enable) from the master controller (108). The remote buffer pointer manager (406) can also generate a remote buffer back pressure signal (e.g., BRAM_WREADY) for each memory channel based on a predefined remote buffer fill level. The remote buffer fill threshold value for generating the back pressure signal must take into account both the skew between the remote buffer write side latency across the memory channel and the skew between the read side and the write side of the remote buffer.

[0068] As described, the AXI-R master (506) can output data that can be provided to the corresponding remote buffer(s).

[0069] FIG. 6 illustrates an exemplary implementation of a master controller (108). In the example of FIG. 6, the master controller (108) includes a remote buffer read address generation unit (remote buffer read AGU) (602) coupled to a request controller (604). In the example of FIG. 6, the master controller (108) converts both narrow read requests and wide read requests from the BRAM controller (204) into AXI-AR requests for request buffers 0 through 31 (narrow reads only for request buffer 31). The master controller (108) also converts narrow write requests only for the master circuit (31) into AXI-W and AXI-AW requests.

[0070] In one aspect, simultaneously, for an AXI-AR request, the estimated read burst length from the AXI-R is transmitted from the request controller (604) to the remote buffer read AGU (602). The remote buffer read AGU (602) may queue the received estimated read burst length in, for example, a queue or memory therein. The remote buffer read AGU (602) may maintain the remote buffer fill level for each remote buffer by tracking all signals (210) (e.g., all remote buffer write enable 0 through 31) and the signal (208) (e.g., common remote buffer read enable). The remote buffer read AGU (602) may trigger, for example, continue to trigger a remote buffer read operation (e.g., indicating the signal (208)) as long as the remote buffer fill level of all remote buffers exceeds the estimated burst length stored in the queue. The signal (208) is used by each request buffer to generate remote buffer back pressure for each memory channel and by each remote buffer for dequeue. Because the expected burst length is predetermined and queued inside the remote buffer read AGU (602), the reorder function that allows the output order to be different from the input order is disabled by forcing the AXI ID to be a single identical value.

[0071] FIG. 7 illustrates an exemplary implementation of the request controller (604) of FIG. 6. In the example of FIG. 7, the request controller (604) includes a transaction buffer (702) and a dispatcher (704). The transaction buffer (702) separates two asynchronous clock domains by using sys_clk as a write clock and axi_clk as a read clock. The request controller (604) further includes a plurality of controllers illustrated as an AR (address read) controller (706), an AXI-AW (address write) controller (708), an AXI-W (write) controller (710), and an AXI B detector (712) capable of detecting a valid response in an AXI B or AXI response channel.

[0072] In the example of FIG. 7, the AR controller (706) can check the request buffer back pressure as indicated by the signal (216) (e.g., AR REQ Ready signals 0 to 31). The AR controller (706) can check the AR REQ Ready signals 0 to 31 at the same time, for example, simultaneously. In one aspect, the AR controller (706) generates an AXI-AR REQ (HBM read request) only when space is available in each request buffer.

[0073] For example, for AXI read-related channels such as AXI-AR and AXI-R, a request buffer is required between the AXI master and the request controller (604). For example, for AXI write-related channels such as AXI-AW, AXI-W, and AXI-B, the request controller (604) (e.g., controllers 708, 710, and 712) communicates directly with the AXI master (31) corresponding to the memory controller (15). When an AXI ready indication and an AXI valid indication (AXI-XX-AWREADY / AWVALID or AXI-XX-WREADY / WVALID) are indicated in the same cycle, the request controller (604) outputs an AXI request and simultaneously reads a new request from the buffer.

[0074] The dispatcher (704) can route (e.g., dispatch) different AXI transactions from the transaction buffer (702) to the appropriate controller (706, 708, 710, and / or 712) based on the transaction type. The dispatcher (704) also schedules AXI access sequences between AXI writes and AXI reads; and schedules AXI access sequences between consecutive AXI write operations. For example, the dispatcher (704) does not issue a new AXI-AR request or a new AXI-AW transaction until a response from a previous AXI write transaction is received.

[0075] Referring to FIGS. 6 and 7, the remote buffer read AGU (602) receives a remote buffer write enable and a common remote buffer read enable corresponding to each memory channel. The throughput of the system can be measured at the write side of the remote buffer or the read side of the remote buffer by counting data transfer elements with a performance counter.

[0076] FIG. 8 illustrates a method (800) for transferring data between an HBM and a distributed computing array. The method (800) may be performed using a circuit architecture (referred to as a “system” with reference to FIG. 8) as described in the disclosure in connection with FIG. 1 through 7.

[0077] In block (802), the system can monitor the fill level of multiple remote buffers distributed across multiple dies. Each of the multiple remote buffers can be configured to provide data to a compute array also distributed across multiple dies. In block (804), the system can determine that each of the multiple remote buffers is storing data based on the fill level. In block (806), the system can initiate data transfer from each of the multiple remote buffers across multiple dies to the compute array in response to the determination. Data transfer can be synchronized (e.g., dequeued). For example, data transfers occurring at each die are synchronized. Minimal pipelining is used to facilitate synchronization from one die to another. Data that is output from the remote buffer to the compute array row is further dequeued.

[0078] In one aspect, the system initiates data transmission from each remote buffer by broadcasting a read enable signal to each of the plurality of remote buffers. The read enable signal is a common read enable signal broadcast to each remote buffer.

[0079] In another aspect, the system can monitor the fill level by tracking multiple write enables corresponding to multiple remote buffers on a one-to-one basis and tracking a read enable common to each of the multiple remote buffers.

[0080] In a specific implementation, the system can receive data from HBM through a plurality of individual memory channels within a plurality of RBBM circuit blocks disposed on a first die among a plurality of dies. The plurality of RBBM circuit blocks provide data to each of a plurality of remote buffers.

[0081] The system can also convert a first request for access to memory into a second request compatible with an on-chip communication bus and provide the second request to a communication bus master circuit corresponding to each of a plurality of RBBM circuit blocks.

[0082] The system can provide data from each of a plurality of remote buffers to a plurality of cache circuit blocks distributed across a plurality of dies, and each cache circuit block is connected to at least one of the plurality of remote buffers and a compute array. Each cache circuit block may be configured to receive data from a selected remote buffer at a first clock rate and output data to a compute array at a second clock rate exceeding the first clock rate.

[0083] FIG. 9 illustrates an exemplary architecture for a programmable device (900). The programmable device (900) is an example of a programmable IC and an adaptive system. In one aspect, the programmable device (900) is also an example of a System-on-Chip (SoC). The programmable device (900) may be implemented using multiple interconnected dies in which various programmable circuit resources illustrated in FIG. 9 are implemented across different interconnected dies. In one example, the programmable device (900) may be used to implement the exemplary circuit architecture described herein in relation to FIG. 1 through 8.

[0084] In the example, the programmable device (900) includes a data processing engine (DPE) array (902), programmable logic (PL) (904), a processor system (PS) (906), a network-on-chip (NoC) (908), a platform management controller (PMC) (910), and one or more fixed wiring circuit blocks (912). A configuration frame interface (CFI) (914) is also included.

[0085] The DPE array (902) is implemented as a plurality of interconnected and programmable data processing engines (DPE) (916). The DPE (916) may be arranged in an array and fixedly wired. Each DPE (916) may include one or more cores (918) and memory modules (abbreviated as “MM” in FIG. 9) (920). In one aspect, each core (918) may execute program code stored in a core-specific program memory contained within each individual core (not shown). Each core (918) may directly access the memory module (920) within the same DPE (916) and the memory module (920) of any other DPE (916) adjacent to the core (918) of the DPE (916) in the up, down, left, and right directions. For example, the core (918-5) can directly read memory modules (920-5, 920-8, 920-6, and 920-2). The core (918-5) regards each of the memory modules (920-5, 920-8, 920-6, and 920-2) as an integrated area of ​​memory (e.g., as part of local memory accessible to the core (918-5)). This facilitates data sharing between different DPEs (916) in the DPE array (902). In another example, the core (918-5) can be directly connected to the memory module (920) of another DPE.

[0086] The DPE (916) is interconnected by a programmable interconnect circuit. The programmable interconnect circuit may include one or more different and independent networks. For example, the programmable interconnect circuit may include a streaming network formed by streaming connections (shaded arrow) and a memory-mapped network formed by memory-mapped connections (crosshair arrow).

[0087] By loading configuration data into the control registers of the DPE (916) via memory-mapped connections, each DPE (916) and its components can be controlled independently. The DPE (916) can be enabled / disabled on a per-DPE basis. For example, each core (918) can be configured to access only the memory module (920) or a subset thereof as described to achieve isolation of the core (918) operating as a cluster or of multiple cores (918). Each streaming connection can be configured to establish a logical connection only between selected DPEs (916) to achieve isolation of the DPE (916) or of multiple DPEs (916) operating as a cluster. Since each core (918) can be loaded with program code specific to that core (918), each DPE (916) can implement one or more different kernels within it.

[0088] In another aspect, the programmable interconnect circuit within the DPE array (902) may include additional independent networks, such as a debug network that is independent (e.g., distinct and separate) from the streaming connections and memory-mapped connections, and / or the event broadcast network. In some aspects, the debug network is formed by memory-mapped connections and / or is part of a memory-mapped network.

[0089] A core (918) can be directly connected to an adjacent core (918) via a core-to-core cascade connection. In one aspect, the core-to-core cascade connection is a unidirectional and direct connection between the cores (918) as illustrated. In another aspect, the core-to-core cascade connection is a bidirectional and direct connection between the cores (918). The activation of the core-to-core cascade interface may be controlled by loading configuration data into the control register of each DPE (916).

[0090] In an exemplary implementation, the DPE (916) does not include cache memory. By omitting the cache memory, the DPE array (902) can achieve predictable, for example, deterministic performance. Additionally, significant processing overhead is avoided because there is no need to maintain consistency between cache memories located in different DPEs (916). In an additional example, the core (918) does not have an input interrupt. Therefore, the core (918) can operate without interruption. Omitting the input interrupt for the core (302) also allows the DPE array (902) to achieve predictable, for example, deterministic performance.

[0091] The SoC interface block (922) operates as an interface connecting the DPE (916) to other resources of the programmable device (900). In the example of FIG. 9, the SoC interface block (922) includes a plurality of interconnected tiles (924) organized in rows. In a specific embodiment, different architectures may be used to implement the tiles (924) within the SoC interface block (922), each of which supports communication with different resources of the programmable device (900). The tiles (924) are connected so that data can be propagated bidirectionally from one tile to another. Each tile (924) may operate as an interface to the DPE (916) column immediately above it.

[0092] Tile (924) is connected to adjacent tiles, to the DPE (916) immediately above, and to the circuit below using streaming and memory-mapped connections as illustrated. Tile (924) may also include a debug network connected to a debug network implemented in the DPE array (902). Each tile (924) may receive data from another source, such as, for example, a PS (906), a PL (904), and / or another fixed wiring circuit block (912). For example, tile (924-1) may provide a portion of the data addressed to the DPE (916) in the column above to that DPE (916) regardless of application or configuration, while transmitting data addressed to the DPE (916) in another column to another tile (924, e.g., 924-2 or 924-3), so that such tile (924) may route the data addressed to the DPE (916) in each column accordingly.

[0093] In one aspect, the SoC interface block (922) includes two different types of tiles (924). The first type of tile (924) has an architecture configured to serve as an interface only between the DPE (916) and the PL (904). The second type of tile (924) has an architecture configured to serve as an interface between the DPE (916) and the NoC (908), and also between the DPE (916) and the PL (904). The SoC interface block (922) may include the first and second types of tiles, or a combination of only the second type of tiles.

[0094] In one aspect, the DPE array (902) may be used to implement the computation array described herein. In this regard, the DPE array (902) may be distributed across a plurality of different dies.

[0095] PL (904) is a circuit that can be programmed to perform a specified function. For example, PL (904) can be implemented as a circuit of the field-programmable gate array type. PL (904) may include an array of programmable circuit blocks. As defined herein, the term “programmable logic” refers to a circuit used to construct a reconfigurable digital circuit. A programmable logic is sometimes formed by many programmable circuit blocks, also called “tiles,” that provide basic functions. The topology of PL (904) is highly configurable, unlike a fixed-wiring circuit. Each programmable circuit block of PL (904) generally includes a programmable element (926) (e.g., a functional element) and a programmable interconnect (942). The programmable interconnect (942) provides the highly configurable topology of PL (904). The programmable interconnection unit (942) can be configured in wire units to provide connectivity between programmable elements (926) of the programmable circuit block of the PL (904), and, for example, can be configured in bit units, unlike connectivity between DPEs (916) (e.g., when each wire carries a single bit of information).

[0096] Examples of programmable circuit blocks of PL (904) include configurable logic blocks having lookup tables and registers. Unlike fixed-wiring circuits, which are described below and sometimes referred to as hard blocks, these programmable circuit blocks have functions that are not defined at the time of manufacture. PL (904) may include other types of programmable circuit blocks that provide basic and defined functions along with more limited programmability. Examples of such circuit blocks may include digital signal processing blocks (DSPs), phase lock loops (PLLs), and block random access memory (BRAMs). These types of programmable circuit blocks are as numerous as other blocks of PL (904) and are mixed with other programmable circuit blocks of PL (904). These circuit blocks may also generally have an architecture that includes programmable interconnects (942) and programmable elements (926), and thus are part of the highly configurable topology of PL (904).

[0097] Before use, PL (904), e.g., programmable interconnects and programmable elements, must be programmed or “configured” by loading data referred to as a configuration bitstream into an internal configuration memory cell within it. Once the configuration bitstream is loaded, the configuration memory cell defines how PL (904) is configured (e.g., topology) and how it operates (e.g., specific functions are performed). In this disclosure, the “configuration bitstream” is not the same as program code executable by a processor or computer.

[0098] In one aspect, the PL (904) may be used to implement one or more of the components illustrated in FIGS. 1 through 7. For example, various buffers, queues, and / or controllers may be implemented using the PL (904). In this regard, the PL (904) may be distributed across multiple dies.

[0099] PS (906) is implemented as a fixed-wiring circuit manufactured as part of a programmable device (900). PS (906) may be implemented as or include any various different processor types capable of executing program code. For example, PS (906) may be implemented as an individual processor, for example, as a single core capable of executing program code. In another example, PS (906) may be implemented as a multi-core processor. In another example, PS (906) may include one or more cores, modules, coprocessors, I / O interfaces, and / or other resources. PS (906) may be implemented using any of various different types of architectures. Exemplary architectures that may be used to implement PS (906) include, but are not limited to, ARM processor architecture, x86 processor architecture, graphics processing unit (GPU) architecture, mobile processor architecture, DSP architecture, combinations of the aforementioned architectures, or other suitable architectures capable of executing computer-readable instructions or program code.

[0100] NoC (908) is a programmable interconnect network for sharing data between endpoint circuits of a programmable device (900). Endpoint circuits may be placed in a DPE array (902), PL area (904), PS (906), and / or selected fixed wiring circuit block (912). NoC (908) may include high-speed data paths with dedicated switching. In an example, NoC (908) includes one or more horizontal paths, one or more vertical paths, or both horizontal and vertical paths. The arrangement and number of areas shown in FIG. 9 are merely examples. NoC (908) is an example of a common infrastructure available within a programmable device (900) for connecting selected components and / or subsystems.

[0101] The nets to be routed through the NoC (908) within the NoC (908) are unknown until a user circuit design is created to implement them within the programmable device (900). The NoC (908) can be programmed by loading configuration data into an internal configuration register that defines how elements within the NoC (908), such as switches and interfaces, are configured and how they operate to transfer data between switches and between NoC interfaces to connect endpoint circuits. The NoC (908) is manufactured as part of the programmable device (900) (e.g., fixed-wired) and cannot be physically modified, but can be programmed to establish connectivity between different master circuits and different slave circuits of the user circuit design. When powered on, the NoC (908) does not implement any data paths or routes within it. However, once configured by the PMC (910), the NoC (908) implements data paths or routes between endpoint circuits.

[0102] The PMC (910) serves to manage the programmable device (900). The PMC (910) is a subsystem within the programmable device (900) capable of managing other programmable circuit resources throughout the programmable device (900). The PMC (910) can maintain a safe and secure environment during normal operation, boot the programmable device (900), and manage the programmable device (900). For example, the PMC (910) can provide integrated and programmable control over power-up, boot / configuration, security, power management, safety monitoring, debugging, and / or error handling for different programmable circuit resources of the programmable device (900). The PMC (910) acts as a dedicated platform manager that separates the PS (906) from the PL (904). Thus, the PS (906) and the PL (904) can be managed, configured, and / or powered on and / or turned off independently of each other.

[0103] In one aspect, the PMC (910) may operate as a Root-of-Trust for the entire programmable device (900). For example, the PMC (910) serves to authenticate and / or verify a device image containing configuration data for any of the programmable resources of the programmable device (900) that can be loaded into the programmable device (900). The PMC (910) may also protect the programmable device (900) from tampering during operation. By operating as a Root-of-Trust for the programmable device (900), the PMC (910) may monitor the operation of the PL (904), PS (906), and / or any other programmable circuit resources that may be included in the programmable device (900). The Root-of-Trust capability performed by the PMC (910) is distinguished and separated from any operation performed by the PS (906) and PL (904) and / or the PS (906) and / or PL (904).

[0104] In one aspect, the PMC (910) operates on a dedicated power supply. Thus, the PMC (910) is powered by an independent power supply device, separate from the power supply of the PS (906) and the power supply of the PL (904). Through this power independence, the PMC (910), PS (906), and PL (904) can be protected from each other in terms of electrical noise and faults. Additionally, while the PMC (910) continues to operate, one or both of the PS (906) and PL (904) can be powered down (e.g., suspended or placed in hibernate mode). This capability allows any part of the programmable device (900), such as the powered-down PL (904), PS (906), NoC (908), etc., to be woken up and restored to an operational state more quickly without the need to perform a complete power-up and boot process for the entire programmable device (900).

[0105] The PMC (910) may be implemented as a processor with dedicated resources. The PMC (910) may include multiple redundant processors. The processors of the PMC (910) may execute firmware. The use of firmware supports the configurability and granularity of global features of the programmable device (900), such as reset, clocking, and protection, for example, providing flexibility when creating separate processing domains (distinguished from "power domains" that may be specific to the subsystem). A processing domain may include a mix or combination of one or more different programmable circuit resources of the programmable device (900) (e.g., where the processing domain may include a DPE array (902), PS (906), PL (904), NoC (908), and / or other fixed wiring circuit blocks (912)).

[0106] The fixed-wiring circuit block (912) includes a special-purpose circuit block manufactured as part of the programmable device (900). Although fixed-wiring, the fixed-wiring circuit block (912) may be configured by loading configuration data into a control register to implement one or more different modes of operation. Examples of the fixed-wiring circuit block (912) may include an input / output (I / O) block and a transceiver for transmitting and receiving signals to and from circuits and / or systems, memory controllers, etc., outside the programmable device (900). Other examples of the I / O block may include single-ended and pseudo-differential I / O. Examples of the transceiver may include a high-speed differential clocked transceiver. Other examples of the fixed-wiring circuit block (912) include, but are not limited to, an encryption engine, a digital-to-analog converter (DAC), an analog-to-digital converter (ADC), etc. Generally, the fixed wiring circuit block (912) is an application-specific circuit block.

[0107] In one aspect, a fixed wiring circuit block (912) may be used to implement one or more of the components shown in FIGS. 1 through 7. For example, each of the various memory controllers and / or other controllers may be implemented as a fixed wiring circuit block (912). In this regard, one or more fixed wiring circuit blocks (912) may be distributed across multiple dies.

[0108] CFI (914) is an interface through which configuration data, e.g., a configuration bitstream, can be provided to PL (904) to implement different user-specified circuits and / or circuits within them. CFI (914) is coupled to PMC (910) to provide configuration data to PL (904) and is accessible by PMC (910). In some cases, PMC (910) may first configure PS (906) so that once PS (906) is configured by PMC (910), configuration data can be provided to PL (904) via CFI (914). In one aspect, CFI (914) has an internally integrated built-in cyclic redundancy checking (CRC) circuit (e.g., a CRC 32-bit circuit). In this way, any data loaded into CFI (914) and / or read back through CFI (914) can be checked for integrity by examining the value of the code attached to the data.

[0109] The various programmable circuit resources illustrated in FIG. 9 may be initially programmed as part of the boot process for the programmable device (900). During runtime, the programmable circuit resources may be reconfigured. In one aspect, the PMC (910) may initially configure the DPE array (902), PL (904), PS (906), and NoC (908). At any point during runtime, the PMC (910) may reconfigure all or part of the programmable device (900). In some cases, once the PS (906) is initially configured by the PMC (910), the PL (904) and / or NoC (908) may be configured and / or reconfigured.

[0110] The exemplary programmable device described in connection with FIG. 9 is for illustrative purposes only. In other exemplary implementations, the exemplary circuit architecture described herein may be implemented in a custom multi-die IC, e.g., an application-specific IC having multiple dies, and / or in a programmable IC such as a Field Programmable Gate Array (FPGA) having multiple dies, e.g. In addition, specific techniques used to communically connect dies within an IC package, e.g., a common silicon interposer with wiring connecting the dies, a multi-chip module, three or more stacked dies, etc., are not intended to limit the arrangement of the invention described herein.

[0111] For the sake of illustrative purposes, specific nomenclature is described to provide a complete understanding of the various progressive concepts disclosed herein. However, the terms used herein are intended solely to describe specific aspects of the progressive arrangements and are not intended to be limiting.

[0112] As defined herein, the singular forms, "one" and "he", are intended to also include plural forms unless the context clearly indicates otherwise.

[0113] As defined herein, the term “approximately” means nearly accurate or accurate, or close to but not accurate in value or quantity. For example, the term “approximately” may mean that a described characteristic, parameter, or value falls within a predetermined amount of an accurate characteristic, parameter, or value.

[0114] As defined herein, the terms “at least one,” “one or more,” and “and / or” are open expressions that are both conjunctive and disjunctive in action unless otherwise explicitly stated. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0115] As defined herein, the term "automatically" means the absence of human intervention. As defined herein, the term "user" means a human.

[0116] As defined in this specification, the term “if” means “when” or “upon” or “in response to” or “in response to” depending on the context. Accordingly, the phrase “when determined” or “when [specified condition or event] is detected” may be interpreted, depending on the context, to mean “when determining” or “in response to determination” or “when [specified condition or event] is detected” or “in response to detecting [specified condition or event]” or “in response to detecting [specified condition or event]”.

[0117] As defined herein, the terms "responding to" and similar language as described, e.g., "if," "when," or "at," mean to readily respond to or react to an action or event. The response or reaction is performed automatically. Therefore, if a second action is performed "in response" to a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The term "responding to" indicates a causal relationship.

[0118] As defined herein, the term "processor" means at least one hardware circuit. The hardware circuit may be configured to execute instructions contained in program code. The hardware circuit may be an integrated circuit or embedded in an integrated circuit.

[0119] As defined herein, the term "substantially" means that while the cited characteristic, parameter, or value does not necessarily have to be achieved exactly, deviations or variations, including, for example, tolerances, measurement errors, limitations on measurement accuracy, and other factors known to those skilled in the art, may occur in amounts that do not exclude the effect intended to be provided by the characteristic.

[0120] Terms such as first, second, etc. may be used in this specification to describe various components. These elements should not be limited by these terms, because these terms are used only to distinguish one element from another unless otherwise specified or otherwise indicated by the context.

[0121] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of progressive arrangement. In this regard, each block of a flowchart or block diagram may represent a module, segment, or part of an instruction containing one or more executable instructions for implementing a specified operation.

[0122] It should also be noted that in some alternative embodiments, the actions described in the blocks may occur out of the order indicated in the drawings. For example, two consecutively depicted blocks may be executed substantially simultaneously, or blocks may be executed in reverse order from time to time, depending on the accompanying functionality. In other examples, blocks may generally be executed in increasing numerical order, whereas in other examples, one or more blocks may be executed in a variable order in which the result is stored and utilized in a subsequent block or another block that does not follow immediately. It will also be noted that each block depicted in the block diagram and / or flowchart, and combinations of blocks within the block diagram and / or flowchart, may be implemented by a special-purpose hardware-based system that performs the specified functions or actions or performs a combination of special-purpose hardware and computer instructions.

[0123] The corresponding structures, materials, operations, and equivalents of all means or steps and functional elements that may be found in the claims below are intended to include any structures, materials, or operations for performing a function together with other specifically claimed elements.

[0124] The IC may include a plurality of dies. The IC may include a plurality of memory channel interfaces configured to communicate with memory, and the plurality of memory channel interfaces are disposed within a first die among the plurality of dies. The IC may include a compute array distributed across a plurality of dies and a plurality of remote buffers distributed across a plurality of dies. The plurality of remote buffers may be coupled to the plurality of memory channels and the compute array. The IC may also include a controller configured to determine that data is stored in each of the plurality of remote buffers and, in response to this determination, broadcast a read enable signal to each of the plurality of remote buffers to initiate data transfer from the plurality of remote buffers to the compute array across the plurality of dies.

[0125] Data transmission is synchronized so that data delivered by each transmission can be discarded.

[0126] The aforementioned implementation and other implementations may each optionally include one or more of the following features, either alone or in combination. One or more implementations may include all of the following features.

[0127] In one aspect, the IC may include a plurality of request buffer bus master circuit blocks disposed on a first die, and each request buffer bus master circuit block is connected to one of a plurality of memory channel interfaces and to at least one of a plurality of remote buffers.

[0128] In another aspect, the IC includes a plurality of cache circuit blocks distributed across a plurality of dies, and each cache circuit block is connected to at least one of a plurality of remote buffers and a compute array.

[0129] In another aspect, each cache circuit block may be configured to receive data from a selected remote buffer at a first clock rate and output data to a compute array at a second clock rate exceeding the first clock rate.

[0130] In another aspect, the computation array includes multiple rows, and each of the multiple dies includes two or more rows among the multiple rows.

[0131] In another aspect, each memory channel interface can provide data from memory to two or more rows of a computation array.

[0132] In another aspect, memory is high-bandwidth memory. In another aspect, memory is 2x speed random access memory.

[0133] In another aspect, the computation array implements a neural network processor, and the data specifies the weights applied by the neural network processor.

[0134] In another aspect, each memory channel interface provides data from memory to two or more rows of a computation array.

[0135] In one aspect, the controller is placed within an IC having multiple dies. The controller includes a request controller configured to convert a first request for access to memory into a second request compatible with an on-chip communication bus, and the request controller provides the second request to multiple request buffer bus master circuit blocks configured to receive data from multiple channels of memory. The controller further includes a remote buffer read address generation unit coupled to the request controller and configured to monitor the fill level of each of the multiple remote buffers distributed across multiple dies. Each of the multiple remote buffers is configured to provide data obtained from each block of the multiple request buffer bus master circuit blocks to a computation array also distributed across multiple dies. In response to each of the multiple remote buffers determining that it is storing data based on the fill level, the remote buffer read address generation unit is configured to initiate data transfer from each of the multiple remote buffers to the computation array across multiple dies.

[0136] Data transmission is synchronized so that data delivered by each transmission can be discarded.

[0137] The aforementioned implementation and other implementations may each optionally include one or more of the following features, either alone or in combination. One or more implementations may include all of the following features in combination.

[0138] In one aspect, the request controller can receive a first request at a first clock frequency and provide a second request at a second clock frequency.

[0139] In another aspect, the remote buffer read address generation unit can monitor the fill level of each of the multiple remote buffers by tracking multiple write enables corresponding to multiple remote buffers and tracking a read enable common to each of the multiple remote buffers.

[0140] The method may include the step of monitoring the fill level of a plurality of remote buffers distributed across a plurality of dies, wherein each of the plurality of remote buffers is configured to provide data to a compute array also distributed across a plurality of dies. The method may also include the step of determining that each of the plurality of remote buffers is storing data based on the fill level, and the step of initiating data transfer from each of the plurality of remote buffers to a compute array distributed across a plurality of dies in response to this determination.

[0141] Data transmission is synchronized so that data delivered by each transmission can be discarded.

[0142] The aforementioned implementation and other implementations may each optionally include one or more of the following features, either alone or in combination. One or more implementations may include all of the following features in combination.

[0143] In one aspect, the step of initiating data transmission from each remote buffer includes the step of broadcasting a read enable signal to each of the multiple remote buffers.

[0144] In another aspect, the step of monitoring the fill level may include the step of tracking a plurality of write enables corresponding to a plurality of remote buffers and the step of tracking a read enable common to each of the plurality of remote buffers.

[0145] In another aspect, the method may include the step of receiving data from memory through a plurality of individual memory channels within a plurality of request buffer bus master circuit blocks disposed on a first die among a plurality of dies, wherein the plurality of request buffer bus master circuit blocks provide data to each of a plurality of remote buffers.

[0146] In another aspect, the method may include the step of converting a first request for access to memory into a second request compatible with an on-chip communication bus, and the step of providing the second request to a communication bus master circuit corresponding to each of a plurality of request buffers.

[0147] In another aspect, the method may include the step of providing data from each of a plurality of remote buffers to a plurality of cache circuit blocks distributed across a plurality of dies, and each cache circuit block is connected to at least one of the plurality of remote buffers and a compute array.

[0148] In another aspect, each cache circuit block may be configured to receive data from a selected remote buffer at a first clock rate and output data to a compute array at a second clock rate exceeding the first clock rate.

[0149] The description of progressive arrangements provided herein is for illustrative purposes only and is not intended to be exhaustive or limiting to the disclosed forms and examples. The terms used herein are chosen to describe the principles of progressive arrangements, substantial applications of technologies found in the market, or technical improvements, and / or to enable those skilled in the art to understand the progressive arrangements disclosed herein. Modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described progressive arrangements. Accordingly, reference should be made to the following claims rather than the foregoing disclosure to indicate the scope of such features and implementations.

Claims

Claim 1 An integrated circuit comprising a plurality of dies, the integrated circuit comprising: a plurality of memory channel interfaces configured to communicate with memory, wherein the plurality of memory channel interfaces are disposed within a first die among the plurality of dies; a computation array distributed across the plurality of dies; a plurality of remote buffers distributed across the plurality of dies, wherein the plurality of remote buffers are coupled to the plurality of memory channel interfaces and the computation array; and a controller, wherein the controller determines that data is stored in each of the plurality of remote buffers and, in response to this determination, is configured to broadcast a read enable signal to each of the plurality of remote buffers to initiate data transmission from the plurality of remote buffers to the computation array across the plurality of dies. Claim 2 An integrated circuit according to claim 1, further comprising a plurality of request buffer bus master circuit blocks disposed on the first die, wherein each request buffer bus master circuit block is connected to one of the plurality of memory channel interfaces and to at least one of the plurality of remote buffers. Claim 3 An integrated circuit according to claim 1, further comprising a plurality of cache circuit blocks distributed across the plurality of dies, wherein each cache circuit block is connected to at least one of the plurality of remote buffers and the computation array. Claim 4 An integrated circuit according to paragraph 3, wherein each cache circuit block is configured to receive the data from a remote buffer selected at a first clock speed and output the data to the computation array at a second clock speed exceeding the first clock speed. Claim 5 An integrated circuit according to claim 1, wherein the calculation array comprises a plurality of rows, and each of the plurality of dies comprises two or more of the plurality of rows. Claim 6 An integrated circuit according to claim 5, wherein each memory channel interface provides data from the memory to two or more rows of the computation array. Claim 7 An integrated circuit according to claim 1, wherein the memory is a high-bandwidth memory. Claim 8 An integrated circuit according to claim 1, wherein the memory is a double data rate random access memory. Claim 9 An integrated circuit according to claim 1, wherein the computation array implements a neural network processor and the data specifies weights applied by the neural network processor. Claim 10 An integrated circuit according to claim 1, wherein each memory channel interface provides data from the memory to two or more rows of the computation array. Claim 11 A controller disposed within an integrated circuit having multiple dies, comprising: a request controller configured to convert a first request for access to memory into a second request compatible with an on-chip communication bus, wherein the request controller provides the second request to multiple request buffer bus master circuit blocks configured to receive data from multiple channels of the memory; and a remote buffer read address generation unit coupled to the request controller and configured to monitor the fill level of each of a plurality of remote buffers distributed across the plurality of dies, wherein each of the remote buffers is configured to provide data obtained from each block of the plurality of request buffer bus master circuit blocks to a computation array distributed across the plurality of dies, and wherein, in response to the remote buffer read address generation unit determining that each of the remote buffers among the plurality of remote buffers is storing data based on the fill level, the remote buffer read address generation unit is configured to initiate data transfer from each of the remote buffers among the plurality of remote buffers across the plurality of dies to the computation array. Claim 12 In paragraph 11, the request controller is a controller that receives the first request at a first clock frequency and provides the second request at a second clock frequency. Claim 13 In claim 11, the remote buffer read address generation unit is a controller that monitors the fill level of each of the plurality of remote buffers by tracking a plurality of write enables corresponding to the plurality of remote buffers and tracking a read enable common to each of the plurality of remote buffers. Claim 14 A controller according to claim 11, wherein the plurality of request buffer bus master circuit blocks comprises a plurality of individual request buffers, and the request controller is configured to initiate a read request for the memory in response to the request controller determining that space is available in each of the plurality of request buffers. Claim 15 A controller according to claim 11, wherein the request controller comprises: a transaction buffer configured to separate a first clock domain from a second clock domain; a dispatcher coupled to the transaction buffer; and a plurality of controllers coupled to the dispatcher, wherein a first subset of the plurality of controllers is configured to monitor the plurality of remote buffers and a second subset of the plurality of controllers is configured to monitor the plurality of request buffer bus master circuit blocks, and the dispatcher is configured to route a transaction to another controller among the plurality of controllers based on a transaction type.

Images