Static schedulable feed and drain structure for systolic array architecture

Abstract

A systolic array implemented in an integrated circuit is disclosed, comprising an array of processing elements including processing elements. The systolic array includes one or more feeder circuits communicatively coupled to the processing element array. Each of the one or more feeder circuits includes: a first region configured to receive data stored in a memory external to the integrated circuit; and a second region configured to transmit the received data to the processing element array, wherein data transferred from the memory to the processing element array is double-buffered by the first region and the second region. The systolic array also includes one or more drain circuits communicatively coupled to the processing element array, the drain circuits including one or more memory buffers configured to store data output by the processing element array.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 488635, filed April 21, 2017, entitled “Statically schedulable Feed and Drain Structure for Systolic Array Architecture,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to matrix multiplication operations. More specifically, this disclosure relates to methods and apparatus for implementing a systolic array matrix multiplier for matrix multiplication operations. Background Technology

[0004] This section is intended to introduce the reader to various aspects of the art that may be related to the aspects of this disclosure described and / or claimed below. It is believed that this discussion will help provide the reader with background information to better understand the aspects of this disclosure. Therefore, it should be understood that these statements are to be read in this context and should not be accepted as prior art.

[0005] Many numerical computing applications, such as high-performance computing, deep learning (e.g., research on artificial neural networks and related machine learning algorithms), and digital signal processing (DSP), rely on matrix multiplication computations. Performing matrix multiplication computations using systolic arrays in both hardware and software has been very successful. However, implementing systolic array architectures on Field Programmable Gate Array (FPGA) platforms can present challenges. For example, there may be challenges related to bandwidth limitations of external memory (e.g., memory outside the integrated circuit, off-chip memory) and on-chip memory limitations of the FPGA. Specifically, off-chip memory bandwidth may be insufficient to maintain the peak operating performance of the systolic array, while on-chip memory bandwidth, although higher, is still limited. Attached Figure Description

[0006] The advantages of this disclosure will become apparent when reading the following detailed description and referring to the accompanying drawings, in which:

[0007] Figure 1A This is a block diagram illustrating an exemplary architecture of a pulsating array according to an embodiment;

[0008] Figure 1B Entry and exit according to the embodiments Figure 1A A schematic diagram of data feed for the processing elements of the pulsating array;

[0009] Figure 2 According to the embodiments Figure 1A A schematic diagram of the microarchitecture of the feeder array of the pulsating array;

[0010] Figure 3 According to the embodiments Figure 1A A schematic diagram of the exhaust microarchitecture of the pulsating array;

[0011] Figure 4 According to the embodiments, by Figure 1A A schematic diagram of the matrix structure in matrix multiplication performed by a pulsating array;

[0012] Figure 5 According to the embodiments Figure 4 A schematic diagram of the pre-arrangement of the matrix structure;

[0013] Figure 6 The example illustrates a method for extracting from... Figure 5 The pre-arranged matrix structure towards Figure 1A A schematic diagram of the feeding mode for feeding matrix data in a pulsating array.

[0014] Figure 7 This is illustrated according to an embodiment. Figure 5 The entry of the pre-arranged matrix structure Figure 1A A schematic diagram of the data feed in the processing element array of the pulsating array;

[0015] Figure 8 The embodiments shown are illustrated and can be used to perform Figure 1A A block diagram of an integrated circuit system implementing a pulsating array; and

[0016] Figure 9 These are examples of electronic systems for processing datasets using the systems and methods of this disclosure, according to embodiments. Detailed Implementation

[0017] One or more specific embodiments of this disclosure will now be described. To provide a concise description of these embodiments, not all features of the actual implementation are described in the specification. It should be recognized that, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints, which may vary between one implementation and another. Furthermore, it should be recognized that such development efforts can be complex and time-consuming, but remain a common practice for those skilled in the art to undertake design, manufacture, and production. The techniques proposed and claimed herein are referenced and applied to tangible and concrete examples that can arguably improve the practical nature of present-day art, and are thus not abstract, intangible, or purely theoretical.

[0018] These embodiments relate to matrix multiplication, where each row of matrix A is multiplied by each column of matrix B. The data stream can be predetermined. Matrix A and B can be arranged such that matrix A is used as is and matrix B is transposed. This reduces the size of complex data arrangements and memory buffers used for reordering. Matrix data can be double-buffered at the feeder circuitry. Buffers for individual reverse pressure signals and intermediate data within the systolic array can be eliminated. Double buffering enables simultaneous data fetching from off-chip memory and data streaming to the systolic array. Double buffering also hides off-chip memory latency. This disclosure encourages high data reuse because each value of the matrix can be shared within the systolic array processing element (PE).

[0019] The feeder circuitry has two regions as its memory buffers (double buffering). One region is used for loading from outside the chip, and the other for streaming into the PE. Data is sequentially filled into each region; thus, there is no sequence generator or reordering. All feeder circuits can share the same feeder controller, enabling centralized feed control and reducing the number of control logic units. Completion of data loading from each region can be checked / verified (e.g., validity check). Each region can be considered valid based on its completion. Back pressure can be provided from the eject circuitry when the eject FIFO memory buffer is full. The repeat block can be used as a data reuse counter.

[0020] Regarding the ejection circuitry, pulsating array results can be received via static scheduling, as the generation of PE (Problem Execution) introduces a fixed periodic delay. The feeder region can be pre-buffered and allows for high data reuse to ensure a constant flow of the generated results. Pauses may directly affect the drive of the write bandwidth at the FIFO memory buffer level (e.g., write wait requests). All ejection circuits can share the same controller, enabling centralized ejection control and reducing the number of control logic units. The FIFO complete signal can be generated by monitoring the FIFO memory buffer level. The efficiency of the ejection circuitry can be proportional to the write bandwidth.

[0021] The proposed embodiments can be designed to achieve highly efficient matrix multiplication and any suitable application using matrix structures such as neural networks. Double buffering at the feeder circuitry enables statically scheduled PEs. Data and control can be fully fed forward within the systolic array. Efficiency may be directly driven by write bandwidth. Using this pre-patterned matrix data structure can reduce or eliminate complex data sequence generators. As a result, high-performance and high-frequency circuitry can be implemented with reduced control logic units, while most of the time can be spent on data pipelined operation.

[0022] In light of the above, the proposed embodiments relating to matrix multiplication systolic arrays provide methods for efficiently implementing systolic arrays and associated processing microarchitectures. These can be implemented on integrated circuits (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) hardware). Efficient circuitry with static scheduling for feeding and evacuating data enables high-performance and / or fully feedforward data flows without individual reverse pressure control signals and complex reordering circuitry. Specifically, data fed from off-chip memory to the matrix multiplication systolic array (e.g., a matrix A with row and column indices m and n) m,n And a matrix B with row and column indices k and l. k,l The data flow can be pre-arranged to eliminate other complex data arrangements and large memory buffers used for reordering. Data can be double-buffered at the feeder, removing individual reverse pressure signals and buffers within the systolic array for intermediate data, enabling simultaneous data fetching from off-chip memory and data streaming into the systolic array (e.g., reduced off-chip memory latency). Furthermore, data can be shared within the processing elements (PEs) of the systolic array to facilitate high data reuse (e.g., data interleaving). As a result, the statically scheduled feed and dump circuitry for systolic array architectures discussed herein can improve off-chip memory bandwidth, and these circuits can also be scaled to implement larger-sized systolic arrays.

[0023] Figure 1AA block diagram illustrating an exemplary architecture of a systolic array 10 implemented on integrated circuit 12 is shown. Integrated circuit 12 may represent, for example, a programmable logic device, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC) including circuitry designed and / or programmed in accordance with this disclosure. The systolic array 10 includes a two-dimensional (2D) processing element (PE) array 14 comprising a plurality of PEs 16 spanning a width 18 and a height 20. In the illustrated embodiment, the width 18 and height 20 of the PE array 14 are four and four, respectively (e.g., a 4-row × 4-column array of PEs 16). The PE array 14 is coupled to feeder arrays 22 and 26 along its orthogonal edges. Feeder array 22 may include a plurality of feeders 24 (e.g., a first feeder "Feeder Y1", a second feeder "Feeder Y2", etc.), and feeder array 26 may include a plurality of feeders 28 (e.g., a first feeder "Feeder X1", a second feeder "Feeder X2", etc.). Feeder array 22 has a height 30 and feeder array 24 has a width 32. In the illustrated embodiment, the height 30 of feeder array 22 is equal to the height 20 of PE array 14 (e.g., four feeders 24), and the width 32 of feeder array 26 is equal to the width 18 of PE array 14 (e.g., four feeders 28). Feeder arrays 22 and 26 are coupled to external or off-chip memory 34, such as external random access memory (RAM). Although memory 34 is described as being "external" to integrated circuit 12, it may also be on-chip or "internal" to integrated circuit 12.

[0024] During operation, matrix data is loaded from external memory 34 into feeder arrays 22 and 26. Feeder arrays 22 and 26 buffer the input matrix data and pass it to the PE array 14 for matrix multiplication. (As will be discussed later...) Figure 2As described above, such data buffers in feed arrays 22 and 26 enable simultaneous data retrieval from external memory 34 and data streaming into pulsating array 10. Similarly, each of the PEs 16 in PE array 14 has a memory 36 to act as a "storage" for temporarily storing computation results. Within PE array 14, data transmission is propagated linearly to subsequent PEs 16, as indicated by arrows 38 between PEs 16, passing through PE array 14 row by row or column. Each of the PEs 16 is responsible for processing its input and passing the processed data or results to the corresponding ejector or collector. In the illustrated embodiment, pulsating array 10 includes an ejector or collector array 40 having a plurality of ejectors or collectors 42 communicatively coupled to PE array 14. Ejector or collector array 40 may be arranged along one of the axes of PE array 14 (e.g., along feeder array 22 or along the edge or axis of feeder array 26). The width of ejector array 40 is equal to the width 32 of feeder array 26 (e.g., four ejectors 42). Each of the several dischargers 42 processes a portion of the results from each column of the PE array 14 and / or passes them to the appropriate location (e.g., the system host). Figure 3 Further discussion on the discharge array 40 is provided in the document.

[0025] Figure 1B Entry and exit according to the embodiments Figure 1A A schematic diagram of data feeding to PE 16 of the pulsating array 10. In the illustrated embodiment, data (e.g., a portion of data from a first matrix) can be fed to PE 16 from feeder array 22 as indicated by data path 23, and data (e.g., data from a second matrix) can be fed to PE 16 from feeder array 26 as indicated by data path 27. Processed / computed data or results can be fed to and stored in memory 36 of PE 16 as indicated by data path 37. Processed / computed data or results can be fed from memory 36 to discharge array 40, such as a corresponding discharger or collector, as indicated by data path 39. In one embodiment, processed / computed data or results can also be fed from memory 36 back to PE 16 as indicated by data path 41.

[0026] Similarly, data (e.g., a portion of the data from the first matrix) can be fed from feeder array 22 to another PE 16 (e.g., a PE in an adjacent row of PE array 14), as indicated by data path 43. In one embodiment, forward register 45 can be configured along data path 43 for forward data transmission. Data (e.g., a portion of the data from the second matrix) can be fed from feeder array 26 to another PE 16 (e.g., a PE in an adjacent column of PE array 14), as indicated by data path 47. Forward register (e.g., forward register 45) can also be configured along data path 47 for forward data transmission.

[0027] Figure 2 An example of a feeder microarchitecture 50 for a feeder array (e.g., feeder array 22 or feeder array 26) is shown. The feeder microarchitecture 50 may include feeder circuitry 52 for each feeder. In the illustrated embodiment, each feeder array has four feeders (e.g., ...). Figure 1A The feeder microarchitecture 50 includes four feeders 24 of feeder array 22 and four feeders 28 of feeder array 26; correspondingly, the feeder microarchitecture 50 includes four feeder circuits 52. Each of the feeder circuits 52 includes a memory buffer 54, which includes two regions (e.g., double buffering), a first region 56 and a second region 58. The first region 56 loads matrix data from off-chip memory 34 to the corresponding feeder, while the second region 58 streams the loaded matrix data to the PE array 14. The matrix data is sequentially loaded into each of the first and second regions 56 and 58. Thus, in some embodiments, the feeder microarchitecture 50 may not include a sequence generator or reordering engine. The feeder microarchitecture 50 may also include a feeder controller 60 operatively coupled to the feeder circuits 52. In the illustrated embodiment, all four feeder circuits 52 are coupled to the same feeder controller 60, thereby enabling centralized feed control and reducing the number of control logic units. In other embodiments, fewer than all of the feeder circuits 52 may be coupled to a feeder controller, and the remaining feeder circuits 52 may be coupled to different feeder controllers.

[0028] The feeder controller 60 may include a read controller 62, a valid check engine 64, and a reverse pressure check engine 66. The read controller 62 enables the reading of matrix data (e.g., matrix A) via the feeder circuitry 52. m,n Sum matrix B k,l(As indicated by data communication arrow 68). The validity check engine 64 can perform validity checks on each of the first region 56 and the second region 58. For example, it can check / verify (e.g., validity check) the completion of data loading from each of the first region 56 and the second region 58. Specifically, the validity check engine 64 may include a block or logic unit 70, as indicated by data communication arrow 72, to check whether data loading in the first region 56 and / or the second region 58 is complete. The reverse pressure check engine 66 can provide information related to reverse pressure from the discharge array 40. For example, as indicated by data communication arrow 74, information representing reverse pressure (e.g., write wait request) can be provided from the discharge circuit when the first-in-first-out (FIFO) memory buffer of the discharger (e.g., discharger 42) is full. When the reverse pressure check (e.g., data communication arrow 74) is cleared (e.g., the FIFO memory buffer is not full), the validity check engine 64 can send a read request, as indicated by data communication arrow 76, to obtain read matrix data to be fed into the PE array 14. The valid check engine 64 may include a block or logic unit 78 that determines a valid signal 80, which initiates the loading of read matrix data (e.g., matrix A) into the PE array 14 via the feeder circuit 52. m,n Sum matrix B k,l As indicated by data communication arrow 82. For example, a valid signal 80 is determined when both a valid check and a reverse pressure check are received (e.g., at box 78). In some embodiments, the reverse pressure check engine 66 may include a repeating block or logic unit 84 that serves as a data reuse counter. Examples of data reuse (e.g., data interleaving) will be discussed later.

[0029] Figure 3 An example of a discharge microarchitecture 90 for a discharge array (e.g., discharge array 40) is shown. The discharge microarchitecture 90 may include discharge circuitry 92 for each discharger. In the illustrated embodiment, each discharge array has four dischargers (e.g., Figure 1AThe discharge array 40 includes a discharger 42; correspondingly, the discharge microarchitecture 90 includes four discharge circuits 92. Each of the discharge circuits 92 includes a memory buffer 94, such as a FIFO memory buffer, for each corresponding PE 16. For example, data from each of the PE 16 is discharged to the corresponding memory buffer 94, such as the FIFO memory buffer corresponding to the discharge circuit 92. The discharge microarchitecture 90 also includes a discharge controller 96 operably coupled to the discharge circuits 92 via an adapter circuit 98. In the illustrated embodiment, all four discharge circuits 92 are coupled to the same discharge controller 96, thereby enabling centralized discharge control and reducing the number of control logic units. In other embodiments, fewer than all of the discharge circuits 92 may be coupled to the discharge controller, and the remaining discharge circuits 92 may be coupled to different discharge controllers. The discharge controller 96 may include a write controller 100 and a memory buffer monitor 102. The write controller 100 enables the writing of computation results (e.g., partial computation results) from each of the discharge circuits 92 to an appropriate location (e.g., the system host). For example, adapter 98 may include a data bus width converter coupled between the respective PE 16 and the external memory 34, and write controller 100 may operate the data bus width converter to control the streaming of data output from PE 16 to external memory 34. Specifically, PE 16 can stream data over a relatively large width compared to the relatively narrow data bus interface of external memory 34. In this case, write controller 100 may control the operation of adapter 98 to narrow / concentrate the output data streaming from PE 16 to external memory 34. In one embodiment, output data from PE 16 may be streamed in multiple packets of the same size. Memory buffer monitor 102 may monitor the FIFO level of each of the memory buffers 94. Specifically, write controller 100 may receive information indicating reverse pressure from memory buffers 94 and transmit this information to memory buffer monitor 102, as indicated by data communication arrow 74. When the memory buffer monitor 102 determines that the memory buffer 94 is full, the discharge controller 96 may send a signal to the feeders (e.g., feeder arrays 22 and 26), as indicated by data communication arrow 104, to stop feeding matrix data to PE 16.

[0030] On the other hand, in the absence of data communication indicating critical reverse pressure (e.g., memory buffer 94 is full) (e.g., as indicated by arrows 74 and 104), computation results (e.g., partial computation results) from PE array 14 can be received and written to the appropriate location (e.g., the system host) following a data path indicated by arrow 106. Computation results from systolic array 10 can be received in a statically scheduled manner because the results are generated by PE 16 with a fixed-period delay. Specifically, since feeder architecture 50 includes dual buffer regions (e.g., first and second regions 56 and 58) and the data flow is pre-arranged and / or predetermined to achieve high data reuse (e.g., interleaved data), computation results can be generated as a constant flow by systolic array 10. Therefore, based on the feeder microarchitecture 50 and exhaust microarchitecture 90 described above, pauses can be directly driven by write bandwidth (e.g., write wait requests, as indicated by data communication arrow 74), which depends at least in part on the FIFO memory buffer level of memory buffer 94. In this way, in at least some cases, the efficiency of the discharge circuit 92 can be proportional to the write bandwidth.

[0031] Next, the discussion will focus on matrix A. m,n Sum matrix B k,l Data streams are transmitted to the feeders in PE array 12 (e.g., data streams from feeder arrays 22 and 26 to PE array 14), and corresponding pre-arrangements of the matrix structure stored in off-chip memory 34. Figure 4-7 In the illustration shown, matrix data is pre-arranged to be fed into an 8×8 PE array via eight feeders along each orthogonal edge; however, it should be understood that the process can be used for much larger PE arrays (e.g., 10×10 PE arrays, 12×12 PE arrays, 16×16 PE arrays, 24×24 PE arrays, 50×50 PE arrays, 100×100 PE arrays, 200×200 PE arrays, 1000×1000 PE arrays or larger). It should be recognized that feeder data stream transmission with a feed pattern that achieves high data reuse without complex data arrangement and large memory buffers for reordering is possible because the matrix data (e.g., matrix A)... m,n Sum matrix B k,l The feeder data is pre-arranged and stored in the off-chip memory 34, which supports this feeder data stream transmission mode. The feeder data streamed to the PE array 14 via feeder arrays 22 and 26 and the pre-arranged matrix structure stored in the off-chip memory 34 may be closely related; therefore, the exchange matrix A... m,n Sum matrix B k,l The pre-arranged matrix structure will not support feeder data stream transmission with the feed mode described herein.

[0032] Figure 4 This shows matrix A stored on the external memory 34. m,n The initial matrix structure 110 and matrix B k,l A schematic diagram of the initial matrix structure 112 is provided. Both initial matrix structures 110 and 112 are stored in row-major order (e.g., successive elements in a row are adjacent to each other; row elements of an array are contiguous in memory). Each of the initial matrix structures 110 and 112 can be divided or partitioned into an appropriate number of regions. In the illustrated embodiment, initial matrix structure 110 is a 16×16 matrix (e.g., m = n = 16), and initial matrix structure 112 is also a 16×16 matrix (e.g., k = l = 16). Each of the initial matrix structures 110 and 112 can be divided into four regions, regions 0, 1, 2, and 3, as shown. Region 0 includes matrix data from rows 0-7 and columns 0-7, region 1 includes rows 0-7 and columns 8-15, region 2 includes rows 8-15 and columns 0-7, and region 3 includes rows 8-15 and columns 8-15.

[0033] Figure 5 This shows matrix A stored on the external memory 34. m,n The initial matrix structure 110, the pre-arranged matrix structure 114, and matrix B k,l A schematic diagram of the pre-arranged matrix structure 116 of the initial matrix structure 112. Each of the initial matrix structures 110 and 112 can be further divided or subdivided into an appropriate number of sub-regions. In the illustrated embodiment, each of the initial matrix structures 110 and 112 is divided into eight sub-regions, sub-region 1, sub-region 2, ... and sub-region 8, as shown. Specifically, for matrix A... m,n Region 0 is further divided into sub-regions 1 and 3, region 1 is further divided into sub-regions 5 and 7, region 2 is further divided into sub-regions 2 and 4, and region 3 is further divided into sub-regions 6 and 8, as shown. Then, sub-regions 1-8 are rearranged into a pre-arranged matrix structure 114.

[0034] For matrix A m,n As the initial matrix structure 110 is rearranged into the pre-arranged matrix structure 114, the orientation of the matrix changes. Specifically, rows 0-7 and columns 0-3 in the initial matrix structure 110 become rows 0-3 and columns 0-7 in the pre-arranged matrix structure 114, rows 8-15 and columns 0-3 in the initial matrix structure 110 become rows 0-3 and columns 8-15 in the pre-arranged matrix structure 114, rows 0-7 and columns 4-7 in the initial matrix structure 110 become rows 4-7 and columns 0-7 in the pre-arranged matrix structure 114, and so on (e.g., the orientation has changed).

[0035] On the other hand, for matrix B k,l The difference between the pre-arranged matrix structure 116 and the initial matrix structure 112 is the number of divisions. Specifically, the four regions 0-3 are further divided into sub-regions 1-8, while the orientation remains unchanged.

[0036] Each sub-region of the pre-arranged matrix structure 114 has a height of 118 × width of 120, and each sub-region of the pre-arranged matrix structure 116 has a height of 122 × width of 124. In the illustrated embodiment, both the pre-arranged matrix structures 114 and 116 have a size of four rows × eight columns (e.g., height 118 = height 122 = 4 times width 120 = width 124 = 8).

[0037] Figure 6 To illustrate, matrices A are used to feed from off-chip memory 34 to feeder arrays 22 and 26 from a pre-arranged matrix structure 114. m,n Data feeding mode 130 and matrix B fed from pre-arranged matrix structure 116 k,l A schematic diagram of feed mode 132. Feed modes 130 and 132 are arranged in the order of feeding matrix data relative to time coordinate 134.

[0038] For feed mode 130, the matrix data A in the pre-arranged matrix structure 114 is... m,n Feeding from external memory 34 to the first feeder Y1 is illustrated as an example. Specifically, a portion of data from sub-region 1 (e.g., the first column of eight columns of data in sub-region 1) is fed, followed by a portion of data from sub-region 2 (e.g., the first column), followed by a portion of data from sub-region 3 (e.g., the first column), and so on along time axis 134. Thus, the first feeder Y1 can receive segment 141 from sub-region 1, then segment 142 from sub-region 2, ..., then segment 143 from sub-region 3, ..., then segment 148 from sub-region 8. As shown, segments 141-148 are indicated as column 1. In the same manner, the second feeder Y2 can receive segments from column 2, the third feeder Y3 can receive segments from column 3, and so on.

[0039] It should be noted that, according to feed mode 130, segment 141 of sub-region 1 corresponds to A in the initial matrix structure 110. 0,0 A 0,1 A 0,2 A 0,3 Segment 142 of subregion 2 corresponds to A in the initial matrix structure 110. 8,0 A 8,1 A 8,2 A 8,3And so on. Thus, matrix A in the initial matrix structure 110... m,n The first row of each of regions 0-3 is received by feeder array 22. Specifically, matrix A m,n The first row of each of regions 0-3 is received by the first feeder Y1, matrix A m,n The second row of each of regions 0-3 is received by the second feeder Y2, matrix A m,n The third row of each of regions 0-3 is received by the third feeder Y3, and so on.

[0040] For feed mode 132, the matrix data B in the pre-arranged matrix structure 116 will be fed... k,l Feeding from external memory 34 to the first feeder X1 is exemplified. Specifically, portions of data from sub-regions 1 and 2 (e.g., the first column of eight columns of data in sub-region 1 and the first column of eight columns of data in sub-region 2) are fed, followed by portions of data from sub-regions 3 and 4 (e.g., the first column), followed by portions of data from sub-regions 5 and 6 (e.g., the first column), and then portions of data from sub-regions 7 and 8 (e.g., the first column). Thus, the first feeder X1 can receive segment 161 from sub-regions 1 and 2, then segment 162 from sub-regions 3 and 4, then segment 163 from sub-regions 5 and 6, then segment 164 from sub-regions 7 and 8, as shown. In the same manner, the second feeder X2 can receive segments from column 2 of each of sub-regions 1-8, the third feeder X3 can receive segments from column 3 of each of sub-regions 1-8, and so on.

[0041] It should be noted that, according to feed mode 132, segment 161 of sub-region 1-2 corresponds to B in the initial matrix structure 112. 0,0 B 0,8 B 1,0 B 1,8 B 2,0 B 2,8 B 3,0 and B 3,8 Segment 162 of subregion 3-4 corresponds to B in the initial matrix structure 112. 4,0 B 4,8 B 5,0 B 5,8 B 6,0 B 6,8 B 7,0 B 7,8 And so on. In this way, the first column from sub-regions 1-8 of region 0-3 is received by the first feeder X2, the second column from sub-regions 1-8 of region 0-3 is received by the second feeder X2, and the column from B... k,lThe third column of sub-regions 1-8 of region 0-3 is received by the third feeder X3, and so on.

[0042] Figure 7 Matrix A is shown from feeder arrays 22 and 26 to PE array 14. m,n The feeder data stream transmission 170 and matrix B k,l A schematic diagram of the feeder data stream transmission 172. Specifically, feeder data stream transmissions 170 and 172 receive matrix data A based on feeder arrays 22 and 26 according to feed modes 130 and 132. m,n and B k,l The order or arrangement of the feeder data streams 170 and 172 shown in the figure. PE array 14 is an 8×8 array (e.g., height 20 = width 18 = 8) and is coupled to feeder arrays 22 and 26, each of which includes eight feeders.

[0043] Matrix A in the pre-arranged matrix structure 114 m,n Data is fed into PE array 14, making the matrix data A in the initial matrix structure 110... m,n "The matrix data in regions 0-3 is repeated twice and interleaved." For example, consider feeder data stream transmission 170 using the first feeder Y1. Matrix data from each of segments 141-148 in regions 0-3 is fed twice (e.g., repeated twice). For example, A from segment 141... 0,0 A 0,0 A 0,1 A 0,1 A 0,2 A 0,2 A 0,3 A 0,3 A from section 142 8,0 A 8,0 A 8,1 A 8,1 A 8,2 A 8,2 A 8,3 A 8,3 The remainder of the data from segments 143-148 (not shown) is fed via the first feeder Y1. The term "interleaving" can refer to data fed through the same feeder being shared over time. For example, matrix data from each of segments 141-148 in regions 0-3 are interleaved (e.g., data fed through the first feeder Y1 is shared over time).

[0044] Once the data stream transmission 170 of the first feeder Y1 is completed, this data stream transmission 170 continues using the second feeder Y2, followed by the third feeder Y3, and so on, up to the eighth feeder Y8. Each of the feeders Y2-Y8 streams data from directly adjacent columns in sub-regions 1-8 of the pre-arranged matrix structure 114. For example, the second feeder Y2 streams data along with the data streamed by the first feeder Y1. 0,0 A 0,1 A 0,2 A 0,3 and A 8,0 A 8,1 A 8,2 A 8,3 (In the pre-arranged matrix structure 114) directly adjacent A 1,0 A 1,1 A 1,2 A 1,3 and A 9,0 A 9,1 A 9,2 A 9,3 For example, the third feeder Y3 transmits the A stream transmitted by the second feeder Y2. 1,0 A 1,1 A 1,2 A 1,3 and A 9,0 A 9,1 A 9,2 A 9,3 (In the pre-arranged matrix structure 114) directly adjacent A 2,0 A 2,1 A 2,2 A 2,3 and A 10,0 A 10,1 A 10,2 A 10,3 .

[0045] Matrix B k,l Data is fed into PE array 14, making matrix data B k,l "Interleaved and repeated twice in regions 0-3." Matrix data from each of segments 161-164 in regions 0-3 are fed twice (e.g., repeated twice). For example, B from segment 161 is interleaved and repeated twice in this precise order. 0,0 B 0,8 B 1,0 B 1,8 B 2,0 B 2,8 B 3,0 B 3,8The data is fed twice, and the remaining portions of the data from segments 162-164 (not shown) are fed twice in their respective order via the first feeder X1. The term "interleaving" can refer to data fed through the same feeder being shared over time. For example, matrix data from each of segments 161-164 in regions 0-3 are interleaved (e.g., data fed through the first feeder X1 is shared over time).

[0046] Once the data stream transmission 172 of the first feeder X1 is completed, this data stream transmission 172 continues using the second feeder X2, followed by the third feeder X3, and so on, up to the eighth feeder X8. Each of the feeders X2-X8 streams data from directly adjacent columns in sub-regions 1-8 of the pre-arranged matrix structure 116. For example, the second feeder X2 streams data along with the data streamed by the first feeder X1. 0,0 B 0,8 B 1,0 B 1,8 B 2,0 B 2,8 B 3,0 B 3,8 (In the pre-arranged matrix structure 116) directly adjacent B 0,1 B 0,9 B 1,1 B 1,9 B 2,1 B 2,9 B 3,1 B 3,9 For example, the third feeder X3 streams and the B stream is streamed by the first feeder X1. 0,1 B 0,9 B 1,1 B 1,9 B 2,1 B 2,9 B 3,1 B 3,9 (In the pre-arranged matrix structure 116) directly adjacent B 0,2 B 0,10 B 1,2 B 1,10 B 2,0 B 2,10 B 3,2 B 3,10 .

[0047] Considering the above, matrix A m,n and B k,l The dimensions of matrix A and PE array 14 are related. m,n and B k,l Matrix A can have different dimensions (e.g., m×n ≠ l×k). m,n and Bk,l The size of can be a multiple of the size of the pulsating array 10 or can be padded with zeros accordingly. In other words, m can be any suitable positive integer multiple of the height 20 of the PE array 14 (or the height 30 of the feeder array 22) or padded with zeros, and l can be any suitable positive integer multiple of the width 18 of the PE array 14 (or the width 32 of the feeder array 26) or padded with zeros. Furthermore, n must be equal to k so that A mn ×B kl The outer product operation works correctly, and n and k divided by 2 can be any appropriate positive integer. For example, matrix A m,n The size can be 704×256, and it is fed into a PE array 14 of size 22×17 via a feeder array 22 of size 22. Matrix B k,l The size can be 256×544, fed into a PE array 14 of size 22×17 via a feeder array 26 of size 17. In this case, m is 32 times the height 20 of PE array 14 (or the height 30 of feeder array 22), and n divided by 2 equals 128; l is 32 times the width 18 of PE array 14 (or the width 32 of feeder array 26), and k divided by 2 equals 128. It should also be noted that... Figure 5 Each of the subregions 1-8 in the matrix is ​​a 4×8 array because, in the illustrated instance, each PE 16 is a 4-input multiply-accumulate (MAC) mode PE (e.g., a PE performs four successive multiplication and accumulation operations), and each of the feeder arrays 22 and 26 contains eight feeders. It can be generalized that matrix A... m,n The size of each sub-region = MAC input mode × number of feeders (e.g., 24 feeders), and matrix B k,l The size of each sub-region = MAC input mode × number of feeders (e.g., 26 feeders). MAC input modes for matrix A m,n and B k,l It is constant because matrix A m,n Line and B k,l They must be of the same length (m = l).

[0048] Figure 8 A block diagram of a system 200 that can be used to program a systolic array onto an integrated circuit 210 is shown. The integrated circuit 210 can be reconfigured (e.g., a field-programmable gate array (FPGA)) or can be an application-specific integrated circuit (ASIC). Users can use, for example... The Quartus version of the design software 214 is used to implement the circuit design to be written onto the integrated circuit 210.

[0049] Design software 214 can be executed by one or more processors 217 of computing system 215. Computing system 215 may include any suitable device capable of executing design software 214, such as a desktop computer, laptop computer, mobile electronic device, server, etc. Computing system 215 can access, configure, and / or communicate with integrated circuit 210. Processor 217 may include multiple microprocessors, one or more other integrated circuits (e.g., application-specific integrated circuits, field-programmable gate arrays, reduced instruction set processors, etc.), or some combination thereof.

[0050] One or more memory devices 219 may store design software 214. Furthermore, memory devices 219 may store information related to integrated circuit 210, such as control software, configuration software, lookup tables, configuration data, etc. In some embodiments, processor 217 and / or memory devices 219 may be external to computing system 215. Memory devices 219 may include tangible, non-transitory, machine-readable media, such as volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., read-only memory (ROM)). Memory devices 219 may store various information and be used for various purposes. For example, memory devices 219 may store machine-readable and / or processor-executable instructions (e.g., firmware or software) for execution by processor 217, such as instructions for determining the speed of integrated circuit 210 or regions of integrated circuit 210, determining the criticality of pathways in designs written in integrated circuit 210 or regions of integrated circuit 210, writing designs in integrated circuit 210 or regions of integrated circuit 210, etc. The memory device 219 may include one or more storage devices (e.g., non-volatile storage devices), which may include read-only memory (ROM), flash memory, hard disk drive, or any other suitable optical, magnetic, or solid-state storage medium or any combination thereof.

[0051] Design software 214 can use compiler 216 to generate low-level circuit design program 218 (bitstream), sometimes referred to as a program object file for programming integrated circuit 210. That is, compiler 216 can provide machine-readable instructions representing the circuit design of integrated circuit 210. For example, integrated circuit 210 can receive one or more programs 218 (bitstreams), the description of which should be stored in the hardware implementation within integrated circuit 210. Program 218 (bitstream) can be written into integrated circuit 210 as configuration program 211.

[0052] like Figure 9As shown, integrated circuit 210 can operate within data processing system 200 to assist in processing dataset 226 (e.g., matrix data) using the systolic array system and methods of this disclosure. Data processing system 220 can represent, for example, a computing device in a data center, capable of processing network traffic, image data, video data, financial data, or any other suitable form of data. In some examples, machine learning or neural network algorithms employing the systolic array of this disclosure can be used to process dataset 226. Processor complex 224 can execute instructions (e.g., software or firmware) stored in memory and / or storage device 222 to receive and route dataset 226 and control integrated circuit 210. For example, processor complex 224 can run software to analyze and process network traffic, image data, video data, financial data, or any other suitable form of data, which is offloaded to operation of integrated circuit 210 well-suited for processing by the systolic array on integrated circuit 210. The memory and / or storage device 222 may store one or more programs 218 (bit streams) that can be used to write the programmable structure of the integrated circuit 210 (e.g., when the integrated circuit 210 is a programmable logic device such as a field-programmable gate array (FPGA)).

[0053] While the embodiments set forth in this disclosure are susceptible to various modifications and alternatives, specific embodiments have been illustrated by way of example in the accompanying drawings and described in detail herein. However, it should be understood that this disclosure is not intended to be limited to the specific forms disclosed. This disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure as defined by the following appended claims.

[0054] The techniques described and claimed herein are referenced and applied to physical and concrete examples with practical attributes that demonstrably improve the technical field, and are thus not abstract, intangible, or purely theoretical. Furthermore, if any claim appended to this specification contains one or more elements designated as “module for [performing] [function]…” or “step for [performing] [function]…”, it is intended that such elements be interpreted under 35U.SC112(f). However, for any claim containing elements designated by any other means, it is intended that such elements not be interpreted under 35U.SC112(f).

Claims

1. A pulsating array implemented in an integrated circuit circuit, comprising: A processing element array, the processing element array comprising processing elements; Communication is coupled to one or more feeder circuits of the processing element array, wherein each of the one or more feeder circuits includes a dual buffer comprising: A first buffer region is configured to receive data stored in a memory external to the integrated circuit; and A second buffer region is configured to send received data to the processing element array, wherein data transferred from the memory to the processing element array is double-buffered by the first buffer region and the second buffer region. One or more discharge circuits are communicatively coupled to the processing element array, the one or more discharge circuits including one or more memory buffers configured to store data output by the processing element array; A first controller communication-coupled to the one or more feeder circuits, wherein the first controller includes: A read controller configured to enable reading of the data stored in the memory external to the integrated circuit; and A valid check logic unit is configured to perform a valid check on the first buffer region, the second buffer region, or both.

2. The pulsating array according to claim 1, wherein, The first controller further includes: A reverse pressure check logic unit is configured to provide an indication related to the reverse pressure from the one or more discharge circuits to the read controller.

3. The pulsating array according to claim 2, wherein, The reverse pressure check logic unit provides an indication in response to one or more memories of the one or more discharge circuits being full.

4. The pulsating array of claim 1, comprising a second controller communicatively coupled to the one or more discharge circuits, wherein, The second controller includes: A write controller configured to enable writing of the data output by the processing element array; and A buffer monitor is configured to monitor the first-in-first-out (FIFO) level of each of the one or more memory buffers.

5. The pulsating array according to claim 4, wherein, The second controller sends a signal to stop feeding the data stored in the external memory to the processing element array in response to the one or more memory buffers being full.

6. The pulsating array according to claim 1, wherein, The data stored in the external storage includes: Matrix A rearranged using the first pre-arranged matrix structure m,n , wherein matrix A adopts the first pre-arranged matrix structure m,n The orientation of the matrix A with its initial arrangement m,n Different orientations; and Matrix B rearranged using the second pre-arranged matrix structure k,l .

7. The pulsating array according to claim 6, wherein, The first pre-arranged matrix structure includes matrices arranged to facilitate the matrix A. m,n Entering one or more sub-regions of a first feed mode of the processing element array, wherein the second pre-arranged matrix structure includes elements arranged to facilitate the matrix B. k,l Enter one or more sub-regions of the second feed mode of the processing element array.

8. The pulsating array according to claim 7, wherein, The first pre-arranged matrix structure and the second pre-arranged matrix structure are predetermined based on the size of the processing element array.

9. The pulsating array according to claim 1, wherein, The integrated circuit is a field-programmable gate array.

10. An integrated circuit device, comprising: An integrated circuit including a pulsating array, the pulsating array comprising: An array of processing elements, comprising processing elements arranged in a vertical and horizontal direction; The discharge array is communication-coupled to the processing element array; A first feeder array includes a first feeder communicatively coupled to the processing element in a vertical direction. The first feeder loads first matrix data from an external memory into the processing element array according to a first feed mode. The first matrix data includes a matrix A employing a first pre-arranged matrix structure. m,n Such that matrix A adopts the first pre-arranged matrix structure m,n The orientation of the matrix A m,n The initial orientations are different, wherein the first pre-arranged matrix structure includes a first plurality of sub-regions; and A second feeder array includes a second feeder communicatively coupled to the processing element in a horizontal direction. The second feeder loads second matrix data from the external memory to the processing element array according to a second feed mode. The second matrix data includes a matrix B employing a second pre-arranged matrix structure. k,l The second pre-arranged matrix structure includes a second plurality of sub-regions, and the first feed mode and the second feed mode are based on the first pre-arranged matrix structure and the second pre-arranged matrix structure, respectively, to achieve data reuse and reduce the bandwidth usage of the external memory.

11. The integrated circuit device of claim 10, comprising: One or more memory devices coupled to the integrated circuit; as well as One or more processors, the one or more processors being coupled to the one or more memory devices and configured to execute instructions stored in the one or more memory devices to control the operation of the integrated circuit.

12. The integrated circuit device according to claim 10, wherein, Each of the first feeder array and the second feeder array includes: Communication is coupled to one or more feeder circuits of the processing element array, wherein each of the one or more feeder circuits includes: A first region, configured to receive data from the external storage; and The second region is configured to send the received data to the processing element array.

13. The integrated circuit device of claim 12, comprising a feeder controller communicatively coupled to the first feeder array, the second feeder array, or both, wherein, The feeder controller includes: A read controller is configured to enable reading of the first matrix data, the second matrix data, or both; A valid check logic unit, configured to perform a valid check on the first region, the second region, or both; and A reverse pressure check logic unit is configured to provide an indication related to the reverse pressure from the discharge array to the read controller.

14. The integrated circuit device according to claim 10, wherein, The discharge array includes: Communicationally coupled to one or more outgoing circuits of the processing element array, the one or more outgoing circuits including one or more memory buffers configured to store data output by the processing element array; and A discharge controller that is communication-coupled to one or more discharge circuits.

15. The integrated circuit device according to claim 14, wherein, The discharge controller includes: A write controller configured to enable writing of the data output by the processing element array; and A buffer monitor is configured to monitor the first-in-first-out (FIFO) level of each of the one or more memory buffers.

16. The integrated circuit device according to claim 14, wherein, The discharge controller sends a signal in response to the one or more memory buffers being full to stop loading the first matrix data, the second matrix data, or both from the external memory to the processing element array.

17. A method for loading data onto a systolic array implemented in an integrated circuit, comprising: A first matrix employing a first pre-arranged matrix structure is stored in a memory outside the integrated circuit, wherein the first pre-arranged matrix structure has a matrix orientation different from the initial matrix orientation of the first matrix, and wherein the first pre-arranged matrix structure includes a first plurality of sub-regions. A second matrix employing a second pre-arranged matrix structure is stored in the memory outside the integrated circuit, wherein the second pre-arranged matrix structure has the same matrix orientation as the initial matrix orientation of the second matrix, and wherein the second pre-arranged matrix structure includes a second plurality of sub-regions; Based on the first pre-arranged matrix structure and according to the first feeding mode, the data of the first matrix is ​​loaded into the processing element array of the pulsating array; and The data of the second matrix is ​​loaded into the processing element array of the pulsating array based on the second pre-arranged matrix structure and according to the second feeding mode.

18. The method according to claim 17, wherein, Loading data according to the first feeding mode includes streaming data that is repeated twice and interleaving the streaming data from different regions of the first pre-arranged matrix structure.

19. The method according to claim 17, wherein, Loading data according to the second feed mode includes streaming the data twice and interleaving the data streams from different regions of the first pre-arranged matrix structure.

20. The method of claim 17, wherein, Loading the data for the first matrix includes loading the data for the first matrix using a dual-buffered feeder array that is communicatively coupled to the processing element array.

Images