Unified hardware architecture for calculating sparse convolution and sparse attention based on outer product

Abstract

The invention relates to a uniform hardware architecture for calculating sparse convolution and sparse attention based on an outer product, the calculation in the sparse convolution and the sparse attention layer is decomposed into basic outer product calculation, a uniform hardware structure is designed based on the basic outer product calculation to realize the sparse convolution and the sparse attention layer, and the hardware reuse rate is very high. And a layered butterfly routing network is also designed for routing a multiplication result generated in outer product calculation to an accumulation cache region, so that the bandwidth close to the crossbar can be achieved, but the layered butterfly routing network consumes less area than the crossbar, and meanwhile, the layered butterfly routing network has better expandability. In addition, a compact alignment encoding algorithm is designed to encode sparse data, so that when the data is loaded to the multiplier array, each period of the multiplier can be close to full load, and the utilization rate of the multiplier under the sparse condition is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image computing, and relates to a unified hardware architecture for computing sparse convolution and sparse attention based on outer products. Background Art

[0002] With the development of big data technology, convolution is a common type of calculation in neural networks, which takes up a lot of computing time in the entire neural network calculation. In order to speed up the calculation of convolution, researchers have sparsed the convolution calculation and designed corresponding hardware circuits to accelerate the convolution calculation by using the sparsity in matrix calculation. Sparse convolution generally requires special hardware support. Note that in the convolution process, each element of the convolution kernel needs to be multiplied with each element of the input image (except for some edge elements, which can be ignored because there are few edge elements). Therefore, existing research uses an SCNN circuit to implement sparse convolution, which uses two caches (Weight FIFO and IARAM) to store the convolution kernel and input image respectively, and only non-zero elements and their offset position information in the convolution kernel / image are stored. However, the aforementioned traditional technology only considers sparse convolution calculations in neural networks, and does not consider other types of calculations that also take a lot of time. For example, in recent visual neural networks, convolution layers and attention layers are combined to build neural networks with higher accuracy. In this kind of neural network, in addition to convolution calculations, the calculation of the attention layer also consumes a lot of time, so dedicated hardware needs to be designed for acceleration. However, existing solutions cannot efficiently implement this hybrid network. The area overhead of the hardware circuit is large and the overall utilization of the multiplier is reduced, resulting in technical problems such as insufficient comprehensive hardware performance. Summary of the invention

[0003] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes a unified hardware architecture for calculating sparse convolution and sparse attention based on outer products, which can effectively improve the overall performance of hardware.

[0004] In order to achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0005] Provided is a unified hardware architecture for calculating sparse convolution and sparse attention based on outer product, the unified hardware architecture is designed by decomposing the calculations in the sparse convolution and sparse attention layers into the same basic outer product calculation, and is used to realize the calculations of the sparse convolution and sparse attention layers, the unified hardware architecture includes two input buffers, a coordinate calculation module, a multiplier array, a hierarchical butterfly routing network, an accumulation buffer, a PPU module and a switching output buffer;

[0006] The two input buffers are respectively connected to the coordinate calculation module and the multiplier array, the hierarchical butterfly routing network is respectively connected to the coordinate calculation module, the multiplier array and the accumulation buffer, the PPU module is respectively connected to the accumulation buffer and the exchange output buffer, and the PPU module is used to read out the calculation results;

[0007] The sparse input data is encoded using a set alignment coding algorithm before being loaded into the multiplier array. The coordinate calculation module includes a convolution coordinate calculation unit, an attention coordinate calculation unit and a selector. The input ends of the convolution coordinate calculation unit and the attention coordinate calculation unit are respectively connected to the index output ends of the two input buffers, and the output ends of the convolution coordinate calculation unit and the attention coordinate calculation unit are respectively connected to the input end of the selector. The output end of the selector is connected to a hierarchical butterfly routing network. The activation of the convolution coordinate calculation unit and the attention coordinate calculation unit is configured by whether the current calculation is a convolution calculation or an attention calculation.

[0008] One of the above technical solutions has the following advantages and beneficial effects:

[0009] The above unified hardware architecture for calculating sparse convolution and sparse attention based on outer product decomposes the calculations in sparse convolution and sparse attention layers into basic outer product calculations, and designs a unified hardware structure based on the basic outer product calculations to realize sparse convolution and sparse attention layers. The proposed hardware architecture has a high hardware reuse rate when processing both calculations. A hierarchical butterfly routing network is also designed to route the multiplication results generated in the outer product calculation to the accumulation buffer area. The hierarchical butterfly routing network divides the process of routing the multiplication results to the accumulation buffer area into multiple cycles, and uses an optimized routing algorithm to route the multiplication results in each cycle, which enables the hierarchical butterfly routing network to achieve a bandwidth similar to that of a crossbar (called a crossbar switch matrix or a vertical and horizontal switch matrix). However, the hierarchical butterfly routing network consumes less area than a crossbar and has better scalability. A compact alignment coding algorithm is also designed to encode sparse data, and the coded data is dynamically partitioned so that the number of nonzero elements in each partition is as much as possible a multiple of the number of multipliers. This allows the multipliers to be nearly fully loaded in each cycle when loading data into the multiplier array, thereby improving the utilization of the multipliers in the sparse case. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 Schematic diagram of sparse convolution;

[0012] Figure 2 This is a schematic diagram of the overall architecture of SCNN;

[0013] Figure 3 A schematic diagram of a unified hardware architecture for computing sparse convolution and sparse attention based on outer products in one embodiment;

[0014] Figure 4 A schematic diagram of sparse convolution under an outer product angle in one embodiment;

[0015] Figure 5 A schematic diagram of a storage method of an input image, a convolution kernel, and an output image in a sparse convolution process in one embodiment;

[0016] Figure 6 A schematic diagram of the operation of each cycle in a sparse convolution process in one embodiment;

[0017] Figure 7 A schematic diagram of sparse matrix multiplication under the outer product angle in one embodiment;

[0018] Figure 8 A schematic diagram of a storage method of a left matrix, a right matrix and an output matrix in a sparse matrix multiplication process in one embodiment;

[0019] Fig. 9 A schematic diagram of the operation of each cycle in a sparse matrix multiplication process in one embodiment;

[0020] Fig.10 A schematic diagram of the overall structure of a hierarchical butterfly routing network in one embodiment;

[0021] Fig.11 A schematic diagram of a routing algorithm implementation of a butterfly routing network in one embodiment;

[0022] Fig.12 This is a schematic diagram of a common encoding method;

[0023] Fig.13 It is a schematic diagram of the swapping scheme corresponding to the common coding;

[0024] Fig.14 A schematic diagram of left matrix alignment coding in one embodiment;

[0025] Fig.15 A schematic diagram showing a comparison between alignment coding and common coding in one embodiment;

[0026] Fig.16 A schematic diagram of a swapping solution corresponding to an alignment code in an embodiment;

[0027] Fig.17 The figure is a schematic diagram comparing the utilization of multipliers in the aligned coding and common coding modes in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0029] It should be noted that the reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of this phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be appreciated by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the specification of the present invention and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0030] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0031] Figure 1 The convolution process of a 7x7 image and a 3x3 convolution kernel in the sparse case is shown. After the sparseness, a large number of zero elements appear in both the input image and the convolution kernel. These zero elements do not need to participate in the multiplication calculation, which can reduce the amount of convolution calculation. Sparse convolution generally requires special hardware support. Note that during the convolution process, each element of the convolution kernel needs to be multiplied with each element of the input image (except for some edge elements, which can be ignored because there are few edge elements). Therefore, existing research such as SCNN proposes to use Figure 2 The circuit shown implements sparse convolution. Figure 2Two buffers (weight buffer and image buffer) are used to store the convolution kernel and input image respectively, and only non-zero elements and offset position information in the convolution kernel / image are stored. Weight FIFO (IARAM) is designed to read out F convolution kernel non-zero elements (I image non-zero elements) and their position offset information at a time. These non-zero elements are sent to the F×I multiplier array, and F×I multiplication results can be calculated in each clock cycle, and the position offsets of these non-zero elements are sent to the Coordinate computation module to calculate the corresponding positions of these F×I multiplication results in the output image. After obtaining these position information, the F×I multiplication results are sent to the A block accumulator buffers through the arbitrated xbar module. The specific block of the A block accumulator buffer to be sent depends on the position of the multiplication result in the output image. These accumulator banks store partial and results of the output image, which are calculated in the previous cycle. The new multiplication result is accumulated with the partial sum of the previous output image to obtain a new partial sum result, which is still stored in the accumulation buffer.

[0032] When all elements in the Weight FIFO and IARAM have been traversed, all non-zero elements in the convolution kernel and all non-zero elements in the input image have been multiplied, and the multiplication results are sent to the corresponding positions of the output image for accumulation. At this time, the final result of the convolution output image is stored in the accumulation buffer of block A, which can be read out by the PPU (physical processing unit) module for subsequent processing.

[0033] In the above process, since only non-zero elements are stored in the Weight FIFO and IARAM, the calculations involving zero elements are automatically omitted, thereby improving the calculation speed of the convolution. In the above calculation process, the existing scheme uses a crossbar to send F×I multiplication results to the A block accumulation buffer. During the calculation process, among these F×I multiplication results, multiple multiplication results may need to be sent to the same buffer, resulting in conflicts. In order to resolve the conflict, the existing scheme sets the number of blocks A in the accumulation buffer to twice that of F×I, and sets a cache in front of the crossbar to avoid conflicts that cause multiplier blocking. This scheme has two disadvantages. One is that the hardware complexity of the crossbar is O(F×I×A)=O((I×A) 2 ), which makes it difficult to adapt to large-scale multiplier arrays. Second, the number of accumulator cache blocks is twice the number of multipliers, resulting in a large area overhead.

[0034] The size of the multiplier array is F×I, and each cycle requires F non-zero elements from the convolution kernel and I non-zero elements from the input image to fully load the multiplier array. If the remaining convolution kernel elements are less than F or the remaining image elements are less than I, the multiplier array cannot be fully loaded. For extremely sparse situations, the proportion of cycles in which the multiplier cannot be fully loaded will increase, resulting in a decrease in the overall utilization of the multiplier.

[0035] In one embodiment, Figure 3 As shown, a unified hardware architecture for calculating sparse convolution and sparse attention based on outer product is provided. The unified hardware architecture is designed by decomposing the calculations in the sparse convolution and sparse attention layers into the same basic outer product calculations, and is used to realize the calculations of sparse convolution and sparse attention layers. The unified hardware architecture includes two input buffers (hereinafter respectively referred to as cache A and cache B), a coordinate calculation module, a multiplier array, a hierarchical butterfly routing network, an accumulation buffer, a PPU module, and an exchange output buffer. The two input buffers are respectively connected to the coordinate calculation module and the multiplier array, the hierarchical butterfly routing network is respectively connected to the coordinate calculation module, the multiplier array, and the accumulation buffer, the PPU module is respectively connected to the accumulation buffer and the exchange output buffer, and the PPU module is used to read out the calculation results.

[0036] In this embodiment, an alignment coding coding idea is proposed, and a specific coding algorithm (referred to as the alignment coding algorithm in this specification, and the specific implementation process is described below) is implemented based on the coding idea. Therefore, the set alignment coding algorithm is used to encode the sparse input data before it is loaded into the multiplier array. The coordinate calculation module includes a convolution coordinate calculation unit, an attention coordinate calculation unit and a selector. The input ends of the convolution coordinate calculation unit and the attention coordinate calculation unit are respectively connected to the index output ends of the two input buffers, and the output ends of the convolution coordinate calculation unit and the attention coordinate calculation unit are respectively connected to the input ends of the selector. The output end of the selector is connected to the hierarchical butterfly routing network. The activation of the convolution coordinate calculation unit and the attention coordinate calculation unit is configured by whether the current calculation is a convolution calculation or an attention calculation.

[0037] The above-mentioned unified hardware architecture for calculating sparse convolution and sparse attention based on outer product decomposes the calculations in sparse convolution and sparse attention layers into basic outer product calculations, and designs a unified hardware structure based on the basic outer product calculations to implement sparse convolution and sparse attention layers. The proposed hardware architecture has a high hardware reuse rate when processing both calculations. A hierarchical butterfly routing network is also designed to route the multiplication results generated in the outer product calculation to the accumulation buffer area. The hierarchical butterfly routing network divides the process of routing the multiplication results to the accumulation buffer area into multiple cycles, and uses an optimized routing algorithm to route the multiplication results in each cycle, which enables the hierarchical butterfly routing network to achieve a bandwidth similar to that of the crossbar. However, the hierarchical butterfly routing network consumes less area than the crossbar and has better scalability. A compact alignment coding algorithm is also designed to encode sparse data, and the coded data is dynamically partitioned so that the number of nonzero elements in each partition is as much as possible a multiple of the number of multipliers. This allows the multipliers to be nearly fully loaded in each cycle when loading data into the multiplier array, thereby improving the utilization of the multipliers in the sparse case.

[0038] It can be understood that this embodiment proposes a unified hardware architecture for processing sparse convolution layers and sparse attention layers in sparse convolution-attention hybrid neural networks. This hardware architecture can reuse most of the hardware structure when processing sparse convolution and sparse attention layers, thereby improving the utilization efficiency of the hardware and better supporting the calculation of hybrid neural networks. Its core innovations are three: First, by analyzing the convolution layer and the attention layer, the two calculations of the sparse convolution layer and the sparse attention layer are converted into the same basic calculation, and a unified hardware architecture is used for implementation; second, a hierarchical butterfly routing network is used to implement the routing network in the unified hardware architecture, which improves the scalability of the routing network and reduces the area overhead of the routing network and the accumulator cache; third, a novel alignment encoding method is proposed for encoding sparse input data, so that the data can be loaded into the multiplier array in a more compact form, thereby improving the utilization efficiency of the multiplier array. The specific implementation of these three parts is introduced in turn below.

[0039] 1. Unified hardware architecture for sparse convolution and sparse attention layers

[0040] The unified hardware architecture is generally similar to SCNN, such as Figure 3 As shown in , but with the addition of support for sparse attention layers. Figure 3 As shown, the unified hardware architecture includes two input buffers (such as cache A and cache B), a coordinate calculation module, a multiplier array of size WA×WB, a hierarchical butterfly routing network, an A-block accumulation buffer (which can use ping-pong buffering), and a switching output buffer.

[0041] In one embodiment, the widths of the two input buffers are respectively designed to be integer multiples of the length and width of the multiplier array.

[0042] Specifically, the non-zero elements of the two sparse input tensors and their coordinates are stored in input buffer A and input buffer B, whose widths are designed to be WA and WB times the data width, respectively. In each clock cycle, the input buffer supports reading out WA non-zero elements in one tensor and WB non-zero elements in the other tensor, and sends them to a two-dimensional multiplier array of size WA×WB, which multiplies them to obtain WA×WB multiplication results. At the same time, depending on whether the current calculation is a convolution calculation or an attention calculation, the coordinates of these input data are also sent to the corresponding coordinate calculation module, thereby generating the coordinates of WA×WB multiplication results. Guided by the coordinates output by the coordinate calculation module, the WA×WB multiplication results are sent to their corresponding accumulation buffers through the routing network and accumulated with the partial sum results previously stored there.

[0043] By studying and analyzing the way SCNN handles sparse convolution, we can find that SCNN mainly uses the feature that each convolution kernel element and each input image element are multiplied during the convolution process, which is the characteristic of outer product calculation. The main calculation type of the sparse attention layer is sparse matrix-matrix multiplication, and in this manual, matrix-matrix multiplication is also converted into outer product calculation. The following introduces the hardware implementation of sparse convolution and sparse matrix multiplication in turn.

[0044] (1) Sparse convolution: Figure 4 As shown in Figure 1, the convolution calculation process is shown from the perspective of the outer product. Figure 4 In the example, a 3×5 sparse image is convolved with a 2-channel 2×2 sparse kernel (i.e., kernel_0 and kernel_1). From the perspective of the outer product, each non-zero element in the kernel is multiplied by the entire input image to obtain an image of size 2×4. All the resulting 2×4 images are added together to obtain the final convolution output. For example, when generating the output image_0 corresponding to the first channel, the blue (red) kernel element is multiplied by all the pixels in the blue (red) box in the input image to obtain two 2×4 images. Adding these two 2×4 images together gives the convolution result of the first channel, and the same is true for the convolution of the second channel.

[0045] Figure 5 and Figure 6 Shows Figure 4 Detailed data flow of sparse convolution in . For ease of explanation, in this example, the size of the multiplier array is set to WA×WB=2×2, and the number of accumulation buffer blocks A=WA×WB=4. Figure 5It shows how the input image, convolution kernel and output image are organized in buffer A, buffer B and accumulation buffer respectively.

[0046] In one embodiment, the convolution kernels of the multiplier array are stored in an output channel-first format.

[0047] Specifically, the convolution kernel is stored in an output channel-first format: the convolution kernel is expanded into a one-dimensional vector in the order of output channel-row coordinate-column coordinate, and then the expanded vector is sparsely encoded and stored element by element. Since there are 3 non-zero elements in these two convolution kernels, 2 entries in cache A (2×WA≥3) are required to store them (the non-zero element position index represented by the offset form is also stored together with the non-zero value, but not in Figure 5 ). The pixels in the input image are stored in row-major format, i.e., they are ordered by row. The 5 non-zero pixels in the input image occupy 3 entries in buffer B (3×WB≥5).

[0048] For convolution with multiple input channels, the convolution kernels and input image data of different input channels are stored separately: the convolution kernels and image data from the same input channel are stored continuously in buffers A and B, respectively, and then the data from the next input channel is stored. Each accumulation buffer only stores a part of the output image, so the 2-channel output image is divided into equal parts and stored in these accumulation buffers. Figure 5 Different colors are used to show that the output image is divided into 4 parts.

[0049] Figure 6 The figure shows four calculation cycles in the sparse convolution process. In each cycle, WA convolution kernel elements and WB image pixels are read from cache A and cache B to the multiplier array and multiplied to generate WA×WB multiplication results. At the same time, the coordinate calculation module calculates the position of the multiplication result in the output image according to the following formula in this paragraph: (row, col, and ch represent the row, column, and channel index in the image or kernel, respectively, the subscript out represents the output, img represents the image, and ker represents the sparse convolution kernel. Assume there is no padding, and the index starts from 0):

[0050] row out =row img -row ker ,col out =col img -col ker ,ch out =ch ker .

[0051] These output coordinates guide the routing network to send the multiplication results to their corresponding accumulation buffers for accumulation.

[0052] (1) Sparse matrix multiplication: Figure 7 As shown in the figure, the process of matrix multiplication calculation is shown from the perspective of outer product. Figure 7 The multiplication between a 4×2 left matrix and a 2×4 right matrix is ​​shown. In the outer product calculation process, the first (second) column of the left matrix is ​​multiplied with the first (second) row of the right matrix to generate two 4×4 partial sum matrices; then the two 4×4 matrices are added to obtain the final result of the entire matrix multiplication.

[0053] Figure 8 and Fig. 9 Shows Figure 7 The detailed data flow of sparse matrix multiplication is shown in FIG. 1. In this example, it is still assumed that WA=WB=2 and A=WA×WB=4. Figure 8 The storage of the left matrix, right matrix and output matrix in buffer A, buffer B and A block accumulation buffer respectively is shown. For the left (right) matrix, the elements are stored column (row) by column, and elements of different columns (rows) cannot be stored in the same address of buffer A (buffer B) because different columns (rows) need to be multiplied with different elements in the inner product operation. Therefore, 3 addresses are required in buffer A to store the non-zero elements of the left matrix (2 addresses for the first column and 1 address for the second column). Similarly, 3 entries are required in buffer B to store the right matrix. Similar to sparse convolution, the output matrix is ​​divided into 4 equal parts and stored in 4 accumulation buffers.

[0054] Fig. 9 Four calculation cycles in this matrix multiplication are shown. Similar to the case of sparse convolution, in each cycle, WA non-zero elements of the left matrix and WB non-zero elements of the right matrix are read from cache A and cache B to the multiplier array to generate WA×WB multiplication results. At the same time, the coordinate calculation module calculates the position of the multiplication result in the output matrix according to the following formula (row and col represent the row and column indexes in the matrix, the subscript out indicates the output, left indicates the left matrix, and right indicates the right matrix):

[0055] row out =row left ,col out =col right .

[0056] These output coordinates guide the routing network modules to send the multiplication results to their corresponding accumulation buffers for accumulation.

[0057] 2. Hierarchical Butterfly Routing Network

[0058] Since the multiplication results generated by the multiplier array in one cycle may go to the same accumulation cache block, conflicts may occur. This specification proposes a hierarchical butterfly routing network for routing multiplier results. Compared with the traditional crossbar routing network, this hierarchical butterfly routing network has better scalability and requires fewer accumulation cache blocks. The following takes the case of WA=WB=4 as an example to introduce the implementation of the hierarchical butterfly routing network.

[0059] In one embodiment, the hierarchical butterfly routing network includes a cascaded wide 4-input 4-output butterfly network and four narrow 4-input 4-output butterfly networks, the data bit width of the wide 4-input 4-output butterfly network is 4 times that of the four narrow 4-input 4-output butterfly networks. The input end of the wide 4-input 4-output butterfly network is respectively connected to the output end of the multiplier array and the output end of the coordinate calculation module, and the output ends of the four narrow 4-input 4-output butterfly networks are connected to the accumulation buffer.

[0060] (1) Overall structure: Specifically, the overall structure of the hierarchical butterfly routing network is as follows: Fig.10 As shown in Figure 1, the routing network routes 16 multiplication results to 16 accumulation buffers in 4 cycles. The hierarchical butterfly routing network consists of two levels. The first level is a wide 4-input 4-output butterfly network, and the second level contains four narrow 4-input 4-output butterfly networks. The data bit width of the first level is 4 times that of the second level. In sparse convolution and sparse matrix multiplication, the 16 accumulation buffers are arranged as follows: Fig.10 The methods shown are divided into 4 groups.

[0061] For sparse convolution, the output channel can be represented as 4×k+i (k is an integer, i=0,1,2,3) output image pixels are stored in the i-th group of accumulation buffers. Since each row of multipliers in the two-dimensional multiplier array shares the same convolution kernel elements, each row of multiplication results has the same output channel and will be sent to the same group of accumulation buffers. Therefore, each row of multiplication results can be routed as a whole by the first-level butterfly routing network, and then routed to each block of the accumulation buffer in the group by the second-level narrower butterfly routing network.

[0062] For sparse matrix multiplication, the accumulation buffer is divided according to the output matrix row number: the i-th (i=0,1,2,3) group of buffers stores the output matrix elements whose row numbers can be expressed as 4×k+i (k is an integer). The multiplication results of each row of multipliers have the same row number in the output matrix, so they can be routed as a whole in the first-level butterfly routing network and dispersed in the second-level butterfly routing network.

[0063] In one embodiment, the hierarchical butterfly routing network divides the process of routing the multiplication results generated in the outer product calculation to the accumulation buffer into multiple cycles, and uses an optimized routing algorithm to route the multiplication results in each cycle; the optimized routing algorithm checks the highest bit of the destination field of all entries in the two input queues in each cycle, determines the destination output port to which the entry is sent, and maximizes the routing bandwidth.

[0064] Specifically, (2) optimized routing algorithm: Fig.11 The figure shows the internal structure of each crossbar switch in the butterfly routing network. Two input queues (input [0] and input [1]) are used to cache input data and their destinations. Since the radix of the butterfly routing network is 2, the destination port of the data in the queue depends only on the most significant bit (MSB) of its destination field. Therefore, in the routing implementation, the MSB of the destination field is used to control the multiplexer ( Fig.11 The solid line signal in the figure will output the data from the correct port. If the data in the two queues need to be sent to the same output port, only one output is valid ( Fig.11 The dashed signal in the middle shows the output validity).

[0065] In the design of this specification, a more advanced routing algorithm is used to achieve higher throughput. In each cycle, the routing algorithm of this specification checks the MSB of the destination of all entries in the two queues. If two entries from different queues need to be sent to different output ports, these entries are sent to the corresponding output ports to maximize bandwidth. If this condition is not met, one of the entries in the longer queue is sent to the destination output port. Since the queue depth is small (in the design of this specification, each queue contains 4 entries), it can be implemented using a register file, so only simple hardware is required to arbitrate the MSB of all entries in the queue.

[0066] The hierarchical butterfly routing network can adapt to larger WA and WB situations. It only needs to modify the first-level and second-level butterfly routing networks to WA to WA and WB to WB respectively (the number of levels of both is also increased to log 2 WA and log 2 WB)

[0067] 3. Novel Alignment Coding Algorithm

[0068] In the process of loading the input data into the multiplier array, if the remaining convolution kernel elements (left matrix elements) are less than WA or the remaining image elements (right matrix elements) are less than WB, the multiplier array will not be able to run at full load. For extremely sparse situations, the proportion of cycles in which the multiplier cannot be fully loaded will increase, resulting in a decrease in the overall utilization of the multiplier. In order to improve the utilization of the multiplier under extremely sparse conditions, the present embodiment proposes a novel alignment coding algorithm. Taking sparse matrix multiplication as an example, the following first introduces the ordinary coding method, and on this basis introduces the alignment coding algorithm proposed in this embodiment.

[0069] (1) Common encoding method: Fig.12 The multiplication between an M×K left matrix and a K×N right matrix is ​​shown. Due to the limited capacity of the accumulation buffer, when calculating large-scale matrix multiplication, the output matrix needs to be divided into several blocks for calculation, and one block is calculated in each round. In the following example, it is assumed that the size of a block in the output matrix is ​​m×n=32×32. In the case of ordinary coding, in the first round of calculation ( Fig.12 A1 round in the left matrix), the first m rows of the left matrix ( Fig.12 The first m×n block of the output matrix is ​​generated by multiplying the first n columns of the right matrix with block 1 in the output matrix. Since the number of non-zero elements in each column (row) of block 1 (block 2) may not be an integer multiple of WA (WB), the multiplier utilization is reduced.

[0070] The cumulative cache swapping method corresponding to this common encoding method is as follows Fig.13 As shown, during round A1, the ping-pong portion of the accumulation buffer (implemented in the form of ping-pong storage) is used to accumulate multiplication results, and the pong portion stores the calculation results of the previous round and is read out by the PPU module. Then, in the next round (round A2), block 2 is multiplied by block A to generate the second m×n block in the output matrix. At this time, the multiplication results are accumulated to the pong portion of the accumulation buffer, and the previous calculation results of the ping-pong portion (i.e., the results of round A1) are read out by the PPU module, thereby making room for the next round of calculation (round A3).

[0071] In one embodiment, in sparse matrix multiplication of a multiplier array, the length of a stripe of an input multiplier array is dynamically divided according to sparsity, so that the number of non-zero elements in the stripe is an integer multiple of the number of rows of the multiplier array.

[0072] Specifically, (2) Alignment coding algorithm: In order to improve the utilization of the multiplier array, the compact coding method adopted in this embodiment is not to encode non-zero data in strips with the same length (such as Fig.12The length of each column in block 1 is the same), but the length of the strip is allowed to change dynamically according to the sparsity, so that the number of non-zero elements in most strips is exactly an integer multiple of WA. For the sake of convenience, take the left matrix in sparse matrix multiplication as an example, such as Fig.14 As shown, Fig.14 Each column in the matrix is ​​divided unevenly so that the number of non-zero elements in each part is as much as possible an integer multiple of WA.

[0073] In the specific alignment coding algorithm, each column of the left matrix is ​​processed independently and divided into M / (m / 2) stripes. For each column, the starting range of the stripe marked with the number 1 is first determined (strip-1 for short, the starting position of stripe-1 is the first row, and its end position is limited by the size of the cumulative cache). Here, the end position is required to be between the (m / 2)th row and the (3m / 2)th row (the reason can be seen from the swap-out strategy below). Under this constraint, the end position is adjusted so that the number of non-zero elements contained in the stripe is as much as possible an integer multiple of WA. Then determine the range of stripe-2: its starting position is the end position of stripe-1, and its end position must be between the mth row and the 2mth row. The end position is still adjusted so that the number of non-zero elements contained in stripe-2 is also as much as possible an integer multiple of WA under such a constraint. The remaining stripes are also divided in a similar way. Since the number of non-zero elements in most stripes is exactly a multiple of WA, the utilization rate of the multiplier array is improved.

[0074] Fig.15The encoding of the first 48 elements of a column (horizontally displayed) using normal encoding and aligned encoding is compared, assuming WA = 4 and m = 32. For normal encoding, the entire column is divided into stripes of fixed length (m = 32), so stripe-1 contains elements from rows 1 to 32, and stripe-2 contains elements from rows 33 to 64. Under this division, stripe-1 contains 9 non-zero elements, so it takes 3 cycles to load all of them into the multiplier array, and only 1 element is loaded in the third cycle, which reduces the utilization of the multiplier array. For compact aligned encoding, the column is dynamically divided into several stripes. Since stripe-1 must contain 5 non-zero elements from the first 16 rows, 3 more non-zero elements are taken from rows 17 to 48 and added to stripe-1, so that the number of non-zero elements in stripe-1 is exactly twice WA (8 = 2 × 4). The remaining 5 non-zero elements from rows 17 to 48 are included in subsequent stripes. Therefore, in this example, the end position of stripe-1 is 26, and stripe-2 starts at 27. Since there is only one non-zero element between rows 27 and 32, and it must be included in stripe-2, 3 non-zero elements are taken from subsequent rows and added to stripe-2 so that it contains exactly 1×WA(4) non-zero elements. Therefore, the end position of stripe-2 is 41, and stripe-3 starts at 42. Subsequent stripes are determined in a similar manner.

[0075] In one embodiment, in the sparse convolution of the multiplier array, after each input channel of the convolution kernel is flattened into a column vector, all elements in each input channel after being flattened into a column vector are encoded using the encoding left matrix in the sparse matrix multiplication.

[0076] It can be understood that for the convolution kernel of the convolution layer, the alignment encoding method is similar to the encoding method in sparse matrix multiplication. In sparse convolution, all elements in one input channel of the convolution kernel need to be multiplied by all pixels in the same channel of the input image, which is similar to the situation in sparse matrix multiplication where all elements in a column of the left matrix need to be multiplied by all elements in the same row of the right matrix. Therefore, each input channel in the convolution kernel is flattened into a one-dimensional vector and then encoded in the same way as each column of the left matrix in sparse matrix multiplication. For kernels with multiple input channels, they are converted into matrices by flattening each input channel into a column vector, and then the matrix is ​​encoded in the same way as the left matrix.

[0077] In one embodiment, the cumulative cache swapping method of the exchange output buffer is that only one (m / 2×n)-shaped block is completed in each round of calculation, and the PPU module swaps out the (m / 2×n)-shaped block completed in the previous round, and a partially completed (m×n)-shaped block remains in the cumulative cache; m and n are the number of rows and columns of the output matrix block, respectively.

[0078] It can be understood that since each column of the matrix is ​​not evenly divided, the swapping scheme needs to be changed, such as Fig.16 As shown. In round A1, since stripe-1 may contain elements with row coordinates between 0 and 3m / 2, the actual output matrix block size is 3m / 2×n, so this block needs to occupy 3 / 4 of the ping-pong accumulation cache space. However, in round A1, only the first (m / 2×n) block (red part) is completely calculated, and the remaining (m×n) blocks (yellow part) are only partially calculated. This is because some elements with row coordinates between m / 2 and 3m / 2 may not be included in stripe s-1. In round A2, the next (m / 2×n) block of the output matrix is ​​calculated, and another (m×n) block is only partially completed. At the same time, the (m / 2×n) block calculated in round A1 is read out by the PPU module, so the accumulation and swapping of round A2 are as shown. Fig.16 In the subsequent calculations, the situation is similar to round A2: each round of calculation only completes one (m / 2×n)-shaped block, the PPU module swaps out the (m / 2×n)-shaped block completed in the previous round, and a partially completed (m×n)-shaped block remains in the accumulator cache.

[0079] The above-mentioned unified hardware architecture for calculating sparse convolution and sparse attention based on outer product decomposes the sparse convolution and sparse attention layers into basic, similar outer product calculations, so that the two calculations can be processed by the unified hardware architecture. In the processing of sparse convolution layers and sparse attention layers, most of the hardware components in the unified hardware architecture (such as cache A, cache B, two-dimensional multiplier array, hierarchical butterfly routing network and accumulation buffer area, etc.) can be reused. Only a few modules such as coordinate calculation modules need to be designed separately for sparse convolution and sparse attention calculations, and these modules require only a few resources in the entire system. In the implementation of the present invention, the modules that need to be designed separately occupy less than 3% of the area of ​​the entire system. Therefore, the unified hardware architecture proposed by the present invention can realize two types of calculations, sparse convolution and sparse attention, with a higher hardware resource reuse rate.

[0080] Compared with the traditional crossbar routing network, the hierarchical butterfly routing network proposed in the present invention reduces the area overhead of the routing network and the accumulation buffer while maintaining the same bandwidth, and has better scalability. The throughput of the hierarchical butterfly routing network with a queue depth of 4 per level (a total of 4 levels, an equivalent queue depth of 16) is comparable to that of a 16x32 crossbar equipped with an input queue with a depth of 16 (the former has an average number of valid outputs per cycle of 12.6, while the latter is 12.2). However, the area of ​​the former is only 60% of the latter, and only 16 accumulation buffers are required (the latter requires 32). The scalability of the hierarchical butterfly routing network is also better than that of the crossbar: for a multiplier array of size WA×WB, the number of levels of the hierarchical butterfly routing network is proportional to log(WA×WB), and the number of switches per level is proportional to WA×WB. Therefore, the complexity of the hierarchical butterfly routing network is O((WA×WB)×log(WA×WB)), while the complexity of the crossbar is O((WA×WB)2).

[0081] The proposed alignment encoding method improves the utilization of the multiplier array. Fig.17 The multiplier utilization caused by ordinary coding and aligned coding is compared. Assume that the number of rows of the left matrix is ​​M = 256, the number of rows of the block output matrix is ​​m = 32, and the sparsity of the left matrix is ​​random. When the sparsity reaches 90%, aligned coding can still achieve 94% multiplier utilization, while ordinary coding can only achieve 68%.

[0082] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection of the invention. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A unified hardware architecture for computing sparse convolution and sparse attention based on outer products, characterized in that: The unified hardware architecture is designed by decomposing the calculations in the sparse convolution and sparse attention layers into the same basic outer product calculation, and is used to implement the calculations of the sparse convolution and sparse attention layers. The unified hardware architecture includes two input buffers, a coordinate calculation module, a multiplier array, a hierarchical butterfly routing network, an accumulation buffer, a PPU module, and a switching output buffer; The two input buffers are respectively connected to the coordinate calculation module and the multiplier array, the hierarchical butterfly routing network is respectively connected to the coordinate calculation module, the multiplier array and the accumulation buffer, the PPU module is respectively connected to the accumulation buffer and the exchange output buffer, and the PPU module is used to read out the calculation result; The sparse input data is encoded using a set alignment coding algorithm before being loaded into the multiplier array. The coordinate calculation module includes a convolution coordinate calculation unit, an attention coordinate calculation unit and a selector. The input ends of the convolution coordinate calculation unit and the attention coordinate calculation unit are respectively connected to the index output ends of the two input buffers, and the output ends of the convolution coordinate calculation unit and the attention coordinate calculation unit are respectively connected to the input end of the selector. The output end of the selector is connected to the hierarchical butterfly routing network. The activation of the convolution coordinate calculation unit and the attention coordinate calculation unit is configured by whether the current calculation is a convolution calculation or an attention calculation.

2. The unified hardware architecture for computing sparse convolution and sparse attention based on outer product according to claim 1, characterized in that: The hierarchical butterfly routing network includes a cascaded wide 4-input 4-output butterfly network and four narrow 4-input 4-output butterfly networks, wherein the data bit width of the wide 4-input 4-output butterfly network is four times that of the four narrow 4-input 4-output butterfly networks; The input ends of the wide 4-input 4-output butterfly network are respectively connected to the output ends of the multiplier array and the output ends of the coordinate calculation module, and the output ends of four narrow 4-input 4-output butterfly networks are connected to the accumulation buffer.

3. The unified hardware architecture for computing sparse convolution and sparse attention based on outer product according to claim 1 or 2, characterized in that: The hierarchical butterfly routing network routes the multiplication result generated in the outer product calculation to the accumulation buffer area in a process of dividing it into multiple cycles, and uses an optimized routing algorithm to route the multiplication result in each cycle; The optimized routing algorithm checks the highest bit of the destination field of all entries in the two input queues in each cycle, determines the destination output port to which the entry is sent, and maximizes the bandwidth of the routing.

4. The unified hardware architecture for computing sparse convolution and sparse attention based on outer product according to claim 3, characterized in that: In the sparse matrix multiplication of the multiplier array, the length of the stripe input to the multiplier array is dynamically divided according to the sparsity situation, so that the number of non-zero elements in the stripe is an integer multiple of the number of rows of the multiplier array.

5. The unified hardware architecture for computing sparse convolution and sparse attention based on outer product according to claim 4, characterized in that: In the sparse convolution of the multiplier array, after each input channel of the convolution kernel is flattened into a column vector, all elements in each input channel after being flattened into a column vector are encoded using a method of encoding a left matrix in sparse matrix multiplication.

6. The unified hardware architecture for computing sparse convolution and sparse attention based on outer product according to claim 3, characterized in that: The cumulative cache swapping mode of the exchange output buffer is that only one (m / 2×n) shaped block is completed in each round of calculation, and the PPU module swaps out the (m / 2×n) shaped block completed in the previous round, and at the same time, a partially completed (m×n) shaped block remains in the cumulative cache; m and n are the number of rows and columns of the output matrix block respectively.

Images