Sparse convolutional neural network accelerator and implementation method thereof

Abstract

The application belongs to the technical field of neural network acceleration, and discloses a sparse convolutional neural network accelerator and an implementation method thereof. The accelerator comprises an operation unit array, a hash coding module and an output storage module. The output storage module comprises a plurality of accumulation memories. The operation unit array comprises a plurality of operation unit groups. Each operation unit group is connected with one hash coding module and one accumulation memory, and the operation unit group only accesses the corresponding accumulation memory. Each operation unit group comprises a plurality of operation units. The application can improve the operation speed of the accelerator and reduce the hardware resource cost.

Description

Technical Field

[0001] This application relates to the field of neural network acceleration technology, and in particular to a sparse convolutional neural network accelerator and its implementation method. Background Technology

[0002] Convolutional Neural Networks (CNNs) are widely used across various industries due to their excellent performance in computer vision. They are characterized by a large number of parameters and high data reuse rate, which allows custom circuits designed for CNNs to have a higher energy efficiency ratio compared to CPUs and GPUs. Furthermore, given the sparsity of CNNs, custom hardware accelerators can be designed to speed up network computation and effectively save power.

[0003] Currently, hardware accelerators designed using the Cartesian product property of convolution operations can maximize the advantages of sparsity for acceleration. However, accelerators designed using the Cartesian product property typically have randomly generated output addresses, leading to frequent hash collisions, which cause computational latency. Furthermore, address allocation results in high memory resource overhead and high data line interconnect density. Therefore, existing Cartesian product accelerators suffer from hash collisions, high memory resource overhead, and high interconnect density. Summary of the Invention

[0004] This application provides a sparse convolutional neural network accelerator and its implementation method, which can reduce the probability of hash collisions, thereby improving the accelerator's computing speed, while reducing hardware resource overhead caused by memory and line interconnect density, achieving higher energy efficiency and lower resource consumption.

[0005] In a first aspect, embodiments of this application provide a sparse convolutional neural network accelerator, which includes a computational unit array, a hash encoding module, and an output storage module, wherein the output storage module includes multiple accumulators.

[0006] The arithmetic unit array includes multiple arithmetic unit groups. Each arithmetic unit group is connected to a hash encoding module and an accumulation memory. Each arithmetic unit group only accesses its corresponding accumulation memory. Each arithmetic unit group contains multiple arithmetic units.

[0007] The hash encoding module is used to receive the input weight sparse encoding and generate a lookup table based on the input weight sparse encoding. The lookup table is used to mark the intermediate results of convolution operations that have hash collisions.

[0008] The operation unit is used to receive input activation data, input activation sparse coding, input weight data and lookup table, calculate multiple intermediate results of convolution operation based on input activation data and input weight data, search the lookup table based on input activation sparse coding to determine the intermediate results of convolution operation that have hash collisions among the multiple intermediate results of convolution operation, sum the intermediate results of convolution operation that have hash collisions, and send the summed output result to the accumulator memory corresponding to the operation unit for storage.

[0009] The input activation data is configured to contain at least two lines of input activation values ​​with a preset interval.

[0010] In one embodiment, the input weight data includes multiple input weights; each row of input activation values ​​includes one or more input activation values.

[0011] The arithmetic unit includes a multiplier array and a rolling summation module, wherein the multiplier array contains multiple multipliers;

[0012] A multiplier array is used to broadcast a single input activation value to multiple multipliers for multiplication with different input weights, resulting in multiple intermediate convolution operations;

[0013] The rolling summation module is used to activate the sparse coding lookup table based on the input to determine the intermediate convolution operation results that have hash collisions among multiple intermediate convolution operation results, sum the intermediate convolution operation results that have hash collisions, and send the summed output result to the accumulator memory corresponding to the operation unit.

[0014] In one embodiment, the rolling summation module includes multiple rolling units, each of which is connected one-to-one with each multiplier in the multiplier array to receive the intermediate results of the convolution operation output by the multiplier.

[0015] Each scrolling unit is also interconnected with multiple scrolling units adjacent to it on the left, so as to receive multiple intermediate results of neighbor convolution operations output by the multiple scrolling units adjacent to it on the left.

[0016] The rolling unit is used to select the intermediate results of the neighboring convolution operation to be summed from multiple intermediate results of neighboring convolution operations based on the input activation sparse coding and lookup table, store them, sum the stored intermediate results of the neighboring convolution operation to be summed with the intermediate results of the convolution operation that have hash collisions received in the next clock cycle, and send the summed output to the accumulator memory corresponding to the operation unit.

[0017] In one embodiment, the rolling unit includes a register, a first multiplexer, a second multiplexer, a third multiplexer, and a first adder;

[0018] The input data of the second multiplexer are the intermediate results of the first neighbor convolution operation output by the first rolling unit on the left, the intermediate results of the second neighbor convolution operation output by the second rolling unit on the left, and the zero value when there is no summation relationship;

[0019] The first multiplexer is used to retrieve the hash code corresponding to the input activation sparse code in the lookup table, and controls the second multiplexer to select the intermediate result of the neighbor convolution operation to be summed from the intermediate result of the first neighbor convolution operation and the intermediate result of the second neighbor convolution operation and send it to the register for storage.

[0020] The register is used to store the intermediate results of the nearest convolution operation to be summed;

[0021] The first adder is used to sum the intermediate results of the adjacent convolution operation to be summed and the intermediate results of the convolution operation received by the rolling unit when the next clock cycle arrives, and send the summed output to the first adder of the corresponding accumulation memory of the operation unit.

[0022] The third multiplexer is used to send the intermediate results of the convolution operation received by the scrolling unit to the two scrolling units adjacent to the right of the scrolling unit in sequence.

[0023] In one embodiment, the output storage module further includes a crossbar switch router;

[0024] Crossbar routers are used to send the output of each arithmetic unit to the corresponding accumulator memory of that arithmetic unit.

[0025] In one embodiment, the accumulator memory includes a second adder and a memory;

[0026] The second adder is used to retrieve the current value of the memory address corresponding to the output result, sum the current value and the output result, and send the sum back to the memory address corresponding to the output result.

[0027] In one embodiment, the memory is configured to be divided into blocks only by rows, not by columns.

[0028] In one embodiment, the accelerator further includes an input storage module connected to a computer memory device;

[0029] The input storage module includes an input activation memory, an input activation encoding memory, an input weight memory, and a weight sparse encoding memory;

[0030] The arithmetic unit is connected to the input activation memory, the input activation encoding memory and the input weight memory respectively, and the hash encoding module is connected to the weight sparse encoding memory.

[0031] The input activation memory, input activation code memory, input weight memory, and weight sparse code memory are used to store the input activation data, input activation code, input weight data, and input weight sparse code sent by the computer memory device, respectively.

[0032] In one embodiment, the accelerator further includes an output activation memory and a post-processing module, the post-processing module being connected to the output activation memory and the output activation memory being connected to a computer memory device.

[0033] The output activation memory is used to retrieve and store output activation data from the accumulator memory. The output activation data is the output result data stored in the accumulator memory. The output activation memory is also used to send the stored output activation data to the computer memory device.

[0034] The post-processing module is used to process the output activation data stored in the output activation memory when it receives subsequent processing instructions.

[0035] Secondly, embodiments of this application provide a method for implementing a sparse convolutional neural network accelerator based on any of the above embodiments, the method comprising:

[0036] The input activation data in the input data of the sparse convolutional neural network accelerator is configured as at least two rows of input activation values ​​with a preset number of rows between them. The input data includes input activation data, input weight data, input activation sparse coding corresponding to the input activation data, and input weight sparse coding corresponding to the input weight data.

[0037] The input weight sparse encoding is fed into the hash encoding module, which generates a lookup table based on the input weight sparse encoding. The lookup table is used to mark the intermediate results of convolution operations that have hash collisions.

[0038] The input activation data, input activation sparse code, input weight data, and lookup table are input into the operation unit. The operation unit calculates multiple intermediate convolution operation results based on the input activation data and input weight data, and retrieves the lookup table based on the input activation sparse code to determine the intermediate convolution operation result that has a hash collision among the multiple intermediate convolution operation results.

[0039] The arithmetic unit sums the intermediate results of convolution operations that have hash collisions, and sends the summed output to the corresponding accumulator memory for storage.

[0040] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0041] This application provides a sparse convolutional neural network accelerator, which may include a computational unit array, a hash encoding module, and an output storage module. The output storage module includes multiple accumulators. The computational unit array includes multiple computational unit groups, each of which is connected to a hash encoding module and an accumulator. Each computational unit group only accesses its corresponding accumulator, effectively reducing interconnection density. Each computational unit group contains multiple computational units. The hash encoding module receives input weight sparse encoding and generates a lookup table based on the input weight sparse encoding. The lookup table is used to mark intermediate convolutional operation results that have hash collisions. The computational unit receives input activation data, input activation sparse encoding, input weight data, and the lookup table. It calculates multiple intermediate convolutional operation results based on the input activation data and input weight data. It retrieves the lookup table based on the input activation sparse encoding to determine the intermediate convolutional operation results that have hash collisions. It sums the intermediate convolutional operation results that have hash collisions and sends the summed output to the accumulator corresponding to the computational unit for storage, thereby avoiding hash collisions between columns. The input activation data is configured to contain at least two lines of input activation values ​​with a preset interval, i.e., the input activation data is split line by line. This eliminates the possibility of hash collisions between lines, thereby reducing computational latency. The aforementioned accelerator can reduce the probability of hash collisions, improve the accelerator's computational speed, and at the same time reduce hardware resource overhead caused by memory and interconnect density, achieving higher energy efficiency and lower resource consumption. Attached Figure Description

[0042] Figure 1 This is an example diagram illustrating the application of convolution operations on a Cartesian product data stream according to one embodiment of this application.

[0043] Figure 2 An example diagram illustrating the application of convolution operations on improved input activation data provided in one embodiment of this application.

[0044] Figure 3 A comparison diagram of data flow interconnect density provided for one embodiment of this application.

[0045] Figure 4 This is a structural example diagram of a sparse convolutional neural network accelerator provided in one embodiment of this application.

[0046] Figure 5 This is an example diagram illustrating the working principle of the data flow inside an accelerator according to one embodiment of this application.

[0047] Figure 6 This is a structural diagram of a rolling summation module provided in one embodiment of this application.

[0048] Figure 7This is an example diagram of a memory structure divided into rows by granularity according to an embodiment of this application.

[0049] Figure 8 A flowchart illustrating an implementation method of a sparse convolutional neural network accelerator provided in one embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The sparsity property of CNNs refers to the presence of a large number of zero values ​​in the network model parameters. Utilizing this sparsity property to skip the corresponding multiply-accumulate operations can effectively improve hardware performance. Traditional accelerators typically use address alignment to pre-remove input parameters with zero values. As data irregularity increases, the logical resources required for address alignment increase exponentially, leading to excessive logical overhead from latency-free address alignment. Cartesian product data streams can address this issue to some extent. Specifically, in traditional address alignment data streams, the entire computation process is divided into several cycles, each corresponding to a different position of the weight window on the input activation map. Each activation needs to be compared with the weight at its corresponding address, and both are retained only if both the activation and weight are non-zero. However, due to the Cartesian product property of convolution operations, even if a non-zero activation is removed in one cycle due to its zero weight, it will still undergo address alignment with all non-zero weights in the entire weight window in several other cycles. The result of these alignments will always be that both the activation and weight are non-zero. Therefore, the Cartesian product data stream does not perform any address comparisons during the input phase, but retains all non-zero activations and non-zero weights, and all Cartesian products generated from them are stored as output results.

[0052] Therefore, it is evident that Cartesian product data streams offer significant optimization compared to address comparison data streams, but they also have drawbacks. This is because in traditional data streams, the several operation results obtained in each cycle correspond to the same output address. However, in Cartesian product data streams, the operation results may reside at different output addresses each time. This means that a large amount of logic is needed to individually assign addresses to each operation result, including logic to handle hash collisions when output addresses are identical. It can be approximated that Cartesian product data streams transform the large amount of input address comparison logic of address comparison data streams into relatively small output control logic. Therefore, the advantage of Cartesian product data streams is that they maximize the utilization of the sparsity characteristics of activations and weights with relatively lower hardware resources. However, the disadvantages are that their random output address mapping problem leads to extremely high density of interconnections (due to address allocation) and computational latency (due to hash collisions).

[0053] Please see Figures 1 to 4 ,like Figure 1 The example shown illustrates a convolution operation on a Cartesian product data stream. Performing a Cartesian product on the left side using four IA (input activations) and four W (weights) will produce sixteen multiplication results. Based on the convolution operation rules and the addresses of the input activations and weights, the output mapping address of each multiplication result can be derived. However, when the addresses of several multiplication results overlap, a hash collision occurs. Figure 1 On the right is the OA (Output Activation), where each grayscale square represents the number of multiplication results that have encountered hash collisions. In a traditional accelerator framework, hash collisions directly cause the crossbar switch module to experience a delay equal to the maximum number of address collisions. In the application example, if the output activation has two overlapping hash collisions and three overlapping hash collisions, then the crossbar switch module needs a total of three clock cycles to process all the multiplication results.

[0054] To overcome the high interconnectivity and high computational latency caused by hash collisions in Cartesian product data streams, this application provides a sparse convolutional neural network accelerator. The accelerator includes an array of computational units, a hash encoding module, and an output storage module, wherein the output storage module includes multiple accumulators.

[0055] The processing unit array comprises multiple processing unit groups. Each processing unit group is connected to a hash encoding module and an accumulation memory. Each processing unit group only accesses its corresponding accumulation memory. Distinguishing different processing units and memories by group reduces interconnections between different groups, thereby reducing data transmission power consumption and saving logic resources. Each processing unit group contains multiple processing elements (PEs).

[0056] The accelerator's input data consists of four categories: input activation data (IA data), the corresponding input activation sparse code (IA-id), input weight data (W data), and the corresponding input weight sparse code (W-id). The input activation sparse code is the sparse code corresponding to the input activation data, and the input weight sparse code is the sparse code corresponding to the input weight data. All of these sparse codes use a flow coding method to represent the row distance between every two variables in the data, i.e., the number of rows between them.

[0057] The hash encoding module is used to receive the input weight sparse encoding and generate a lookup table based on the input weight sparse encoding. The lookup table is used to mark the intermediate results of convolution operations that have hash collisions.

[0058] The operation unit is used to receive input activation data, input activation sparse coding, input weight data and lookup table, calculate multiple intermediate results of convolution operation based on input activation data and input weight data, search the lookup table based on input activation sparse coding to determine the intermediate results of convolution operation that have hash collisions among the multiple intermediate results of convolution operation, sum the intermediate results of convolution operation that have hash collisions, and send the summed output result to the accumulator memory corresponding to the operation unit for storage.

[0059] The input activation data is configured to include at least two lines of input activation values ​​with a preset interval.

[0060] In practice, the input activation values ​​of each row of calculations within the same period can be separated by ksl-1 rows, where ksl is the kernel side length of the weight window. This can avoid hash collisions between rows of calculation results. At the same time, by searching the lookup table, it can be determined whether there are hash collisions in other calculation results. Calculation results with hash collisions are summed and output, thereby eliminating hash collisions between input activation values ​​of different columns in the same row, that is, eliminating hash collisions between columns.

[0061] In this embodiment, the input activation data is improved by configuring it to contain at least two lines of input activation values ​​with a preset interval. For example... Figure 2 The example shown illustrates the application of convolution operations on the improved input activation data, which is also a Cartesian product generated by four IAs and four Ws, but... Figure 1 The difference is that its OA no longer has dark gray pixels, meaning no hash collisions occur. First, in this example, the input IA is split into rows, with a row spacing of ksl-1. Then, according to the convolution properties, the OAs obtained from the IA operations in different rows will be independent of each other, thus eliminating hash collisions between rows.

[0062] like Figure 3 The diagram compares the interconnect density of a traditional Cartesian product dataflow with that of this embodiment, where Psum represents the partial sum and Acc buffer represents the accumulation buffer. Interconnect density directly impacts line resources and the generation of gating logic. In Application Specific Integrated Circuit (ASIC) design, the impact of interconnect density is relatively small due to the larger memory area overhead. However, in Field Programmable Gate Array (FPGA) design, high interconnect density significantly limits FPGA performance. In a traditional dataflow, the multipliers within a PE (Programmable Element) need to interconnect with the memory of the entire two-dimensional mapping space. In this embodiment, the dataflow segments the input activation, which is equivalent to segmenting the output activation mapping memory space. This means that each multiplier within a PE only needs to interconnect with the corresponding memory in the mapping memory space, thereby reducing the dataflow interconnect density. Reducing interconnect density effectively optimizes timing and reduces logic resource overhead.

[0063] The sparse convolutional neural network accelerator provided in the above embodiments may include a computational unit array, a hash encoding module, and an output storage module. The output storage module includes multiple accumulators. The computational unit array includes multiple computational unit groups, each computational unit group being connected to a hash encoding module and an accumulator. Each computational unit group only accesses its corresponding accumulator, which can effectively reduce interconnection density. Each computational unit group contains multiple computational units. The hash encoding module is used to receive input weight sparse encoding and generate a lookup table based on the input weight sparse encoding. The lookup table is used to mark intermediate results of convolutional operations that have hash collisions. The computational unit is used to receive input activation data, input activation sparse encoding, input weight data, and the lookup table. It calculates multiple intermediate results of convolutional operations based on the input activation data and input weight data. It retrieves the lookup table based on the input activation sparse encoding to determine the intermediate results of convolutional operations that have hash collisions among the multiple intermediate results of convolutional operations. It sums the intermediate results of convolutional operations that have hash collisions and sends the summed output to the accumulator corresponding to the computational unit for storage, thereby avoiding hash collisions between columns. The input activation data is configured to contain at least two lines of input activation values ​​with a preset interval, i.e., the input activation data is split line by line. This eliminates the possibility of hash collisions between lines, thereby reducing computational latency. The aforementioned accelerator can reduce the probability of hash collisions, improve the accelerator's computational speed, and at the same time reduce hardware resource overhead caused by memory and interconnect density, achieving higher energy efficiency and lower resource consumption.

[0064] In some embodiments, the input weight data includes multiple input weights; each row of input activation values ​​includes one or more input activation values.

[0065] The arithmetic unit includes a multiplier array and a rolling summation module, wherein the multiplier array contains multiple multipliers;

[0066] A multiplier array is used to broadcast a single input activation value to multiple multipliers for multiplication with different input weights, resulting in multiple intermediate convolution operations;

[0067] The intermediate result of the convolution operation is the multiplication result calculated by a multiplier.

[0068] The rolling summation module is used to activate the sparse coding lookup table based on the input to determine the intermediate convolution operation results that have hash collisions among multiple intermediate convolution operation results, sum the intermediate convolution operation results that have hash collisions, and send the summed output result to the accumulator memory corresponding to the operation unit.

[0069] In practice, the rolling summation module can work synchronously with the multiplier array to ensure that the accelerator performance is not reduced.

[0070] Specifically, in the Cartesian product data stream, each PE (Activity Processor) receives *a* sparse activations and *w* input weights as multiplier inputs, and their Cartesian product produces *a*×*w* multiplication results. Simultaneously, the rolling summation module receives *a* sparse encoded values ​​to confirm the interval between the sparse activations at this moment and the sparse activations in the same row at the previous moment. Furthermore, it uses a hash lookup table to retrieve the hash collision relationships between the *a*×*w* outputs. That is, it first confirms the hash collisions between the outputs of the same row of activations. The rolling summation module is responsible for pre-summing the conflicting outputs, summing the multiplication results with hash collisions before sending them out, avoiding the computational delay caused by hash collisions in traditional cross-switch structures. At the same time, the rolling window characteristic based on convolution can effectively reduce the resource consumption of this module.

[0071] The working principle of the data flow inside the aforementioned accelerator can be found by referring to... Figure 5 For example, consider a convolution operation with a window size of 3×3 and a stride of 1. Figure 5 (a) is a schematic diagram of the input activations and input weights of the accelerator, showing a total of four input activations (IAs) and four weights (Ws). Specifically, within one clock cycle, a single PE is responsible for computing the Cartesian product of one IA and four Ws, while at the next clock cycle, the PE will accept the next IA in the same row as input, while the Ws remain unchanged.

[0072] Figure 5(b) is a schematic diagram of the output activation result obtained by performing a convolution operation on the input in (a). From the Cartesian product property of the convolution operation, it is easy to see that four IAs and four Ws will produce sixteen partial sums (psum). These sixteen psums will be mapped to different output addresses depending on the addresses of the input variables. At this point, hash collisions may occur; that is, when several partial sums have the same mapped output address, they will occupy the same data path and memory access path. Figure 5 In (b), hash collisions occurred in ①②③ where two output addresses overlapped.

[0073] Traditional sparse structures resolve hash collisions by requesting a delay across the entire accelerator, using multiple clock cycles to clear the blocked data. Taking ① as an example, the multiplication result of activating 'a' and weight A has the same mapped output address as the multiplication result of activating 'b' and weight B. Traditional sparse structures would consume two clock cycles to process the result obtained by the multiplier in one clock cycle; that is, a×A would be output in the first clock cycle, and b×B in the second. In particular, due to the pipelined operation of the data, the multiplier would stop operating in the second clock cycle, which directly and significantly reduces computational efficiency.

[0074] The rolling summation module proposed in this embodiment can directly output the result of a×A+b×B within one clock cycle with extremely low resource cost, without affecting pipelined operation.

[0075] Figure 5 (c) is a schematic diagram of data transmission between the hash encoding module and the PE, where K and stride represent the convolution window size and stride size of the corresponding convolution operation, respectively, and will be used as the input to the hash encoding module. Figure 5 The four types of data mentioned in (a) are used as inputs to corresponding modules to obtain output results. Specifically: IA data and W data are sent to the multiplier to obtain the multiplication result (multr), W-id is sent to the hash encoding module to obtain the look-up table (LUT), and IA-id is directly used as input to the rolling summation module. Finally, the rolling summation module receives the LUT, IA-id, and multr as inputs, sums the results of hash collisions in multr, and directly outputs the final result.

[0076] Figure 5 (d) is a schematic diagram of the internal data flow of the rolling summation module, showing how the relationship between multr is determined through lookup tables and IA-id. First, the LUT will provide ksl lookup tables. Retrieving the value of IA-id can determine which table should be used as the input to the rolling summation module.

[0077] by Figure 5Taking a convolution operation with a window size of 3×3 and a stride of 1 as an example, the LUT generates three tables corresponding to the three cases: IA-id = 0, 1, and IA-id >= 2. The specific encoding method of the lookup table is based on the size of IA-id, i.e., the distance between two adjacent rows of activations, to determine the overlap relationship of the weight windows and represent it using relative position encoding. For example, when the distance between any two rows of activations is 2 (IA-id = 1), the multr of the previous activation and weight A will have a hash collision with the multr of the next activation and weight B. The encoding will then mark A as summed with B in the cell to its right, i.e., the encoding value is 1 (when a hash collision occurs), and the corresponding encoding value of 0 indicates that no hash collision occurs. It can be seen that activations a and b satisfy the distance of 2, and activations b and c also satisfy the distance of 2. Therefore, they will both retrieve the lookup table corresponding to IA-id = 1, and use this to tell the rolling summation module the conflict multr that needs to be processed.

[0078] In particular, LUTs have the same reuse characteristics as weights. All Cartesian product operations (the entire input channel) involving the same set of weights can be retrieved by the same LUT to save power consumption.

[0079] This embodiment provides a sparse convolutional neural network accelerator, which also provides a rolling summation module to eliminate hash collisions between input activations in different columns of the same row. The rolling summation module can also retrieve the lookup table, sum the multiplication results that have hash collisions, and then send them out, directly eliminating the computation delay caused by hash collisions in traditional accelerators.

[0080] In some embodiments, the rolling summation module includes multiple rolling units, each of which is connected one-to-one with each multiplier in the multiplier array to receive intermediate results of convolution operations output by the multipliers.

[0081] Each scrolling unit is also interconnected with multiple scrolling units adjacent to it on the left, so as to receive multiple intermediate results of neighbor convolution operations output by the multiple scrolling units adjacent to it on the left.

[0082] The number of rolling units in the rolling summation module is the same as the number of multipliers in the multiplier array.

[0083] The rolling unit is used to select the intermediate results of the neighboring convolution operation to be summed from multiple intermediate results of neighboring convolution operations based on the input activation sparse coding and lookup table, store them, sum the stored intermediate results of the neighboring convolution operation to be summed with the intermediate results of the convolution operation that have hash collisions received in the next clock cycle, and send the summed output to the accumulator memory corresponding to the operation unit.

[0084] In practice, each rolling unit includes several registers (reg), a multiplexer (MUX), and an adder to achieve the above functions.

[0085] In this embodiment, the rolling summation module can sum the intermediate results of convolution operations that have hash collisions through multiple rolling units.

[0086] Based on the above embodiments, in one implementation, such as Figure 6 As shown, the rolling unit includes a register (reg), a first multiplexer (MUX0), a second multiplexer (MUX1), a third multiplexer (MUX2), and a first adder;

[0087] The input data of the second multiplexer are the intermediate results of the first neighbor convolution operation output by the first rolling unit on the left, the intermediate results of the second neighbor convolution operation output by the second rolling unit on the left, and the zero value when there is no summation relationship;

[0088] The first multiplexer is used to retrieve the hash code corresponding to the input activation sparse code in the lookup table, and controls the second multiplexer to select the intermediate result of the neighbor convolution operation to be summed from the intermediate result of the first neighbor convolution operation and the intermediate result of the second neighbor convolution operation and send it to the register for storage.

[0089] The register is used to store the intermediate results of the nearest convolution operation to be summed;

[0090] The first adder is used to sum the intermediate results of the adjacent convolution operation to be summed and the intermediate results of the convolution operation received by the rolling unit when the next clock cycle arrives, and send the summed output to the first adder of the corresponding accumulation memory of the operation unit.

[0091] The third multiplexer is used to send the intermediate results of the convolution operation received by the scrolling unit to the two scrolling units adjacent to the right of the scrolling unit in sequence.

[0092] In practice, Figure 6 MUX0 in the algorithm is responsible for retrieving the corresponding hash code from the LUT using IA-id. The inputs to MUX1 are the multr of the first and second rounds from the left, and the zero value when there is no summation relationship. When the hash code of the input to MUX1 in any round is 1 or 2, the multr of the first or second round from the left is sent to the register (reg) of that round via MUX1. The register retains the multr and sums it with the multr of that round at the next clock cycle. The entire rolling summation process is performed synchronously with the convolution multiplication, thus not degrading accelerator performance.

[0093] It is important to note that in the above implementation, each RUNIT is interconnected only with the two RUNITs on its left. This is determined by the selected window size and the order in which the weights are placed in memory. When the window size is 3×3, since the weights are stored row-wise first, it is easy to see that the address distance between weights located in the same row in the window will not exceed 3-1=2, meaning that only data interconnection within two RUNIT units is required.

[0094] According to the original Cartesian product structure, all multrs may generate hash collisions, but in the rolling summation module, a multr may only collide with the adjacent ksl-1 multrs. This means that the rolling summation module can achieve hash collision ablation without delay at a very low hardware resource cost.

[0095] Furthermore, such as Figure 6 As shown, the function of MUX2 is that when the interval between the three row activations is 1 (IA-id = 0), multr will be sent to the first RUNIT on the right in the first clock cycle and will continue to be sent to the second RUNIT on the right in the next clock cycle. That is, MUX2 can provide a data path for multr to flow continuously through two RUNITs, where IAcont (activation continuous flag) = IA-id & IA-id', and IA-id' represents the IA-id of the previous clock cycle.

[0096] In some embodiments, the output storage module further includes a crossbar switch router ( Figure 4 (Cross switch in the middle); the cross switch router is used to send the output of each arithmetic unit to the accumulator memory corresponding to that arithmetic unit.

[0097] In this embodiment, since each arithmetic unit is only connected to its corresponding accumulator memory, the connection density is reduced, effectively saving resource consumption. In some embodiments, the accumulator memory includes a second adder and a memory; the second adder is used to retrieve the current value of the storage address corresponding to the output result from the memory, sum the current value and the output result, and send the summation result back to the storage address corresponding to the output result for storage.

[0098] The accelerator provided in this embodiment can use the second adder to continuously update the stored data, thereby achieving rapid updates of the data stored in the accumulator memory.

[0099] In some embodiments, such as Figure 7 As shown, the memory is configured to be divided into blocks only by rows, not by columns.

[0100] The memory is configured to have only row block granularity and no column block granularity.

[0101] Specifically, while dividing the memory into rows, the number of rows should also be increased to meet storage requirements.

[0102] Currently, traditional Cartesian product architectures divide the accumulation memory into blocks according to three dimensions: rows, columns, and output channels. However, this embodiment eliminates the column-based block granularity of the memory, that is, it only divides the accumulation memory into blocks according to rows and output channels, while increasing the number of blocks by row.

[0103] This embodiment provides a memory granularity partitioning scheme. Compared with the traditional Cartesian product data stream, it increases the row partitioning granularity while eliminating the column partitioning granularity. This partitioning method can reduce memory resource overhead and significantly reduce the interconnection density of crossbar switch routers.

[0104] In specific implementation, such as Figure 7 The example shown is a specific application of memory partitioning by row granularity. A PEs contains m PEs, which share a memory with a granularity of (3*m+2)*3, where (3*m+2) represents the row partitioning granularity and *3 represents the output channel (OC) partitioning granularity. Figure 7 In this architecture, the output of each PE is connected to five row-level memory locations. For example, PE0 is interconnected with rows 0, 1, 2, 3, and 4, while PE1 is interconnected with rows 3, 4, 5, 6, and 7. When a PE is responsible for calculating the input activation of a block, it requires three loops to complete all the calculations.

[0105] Taking PE0 as an example: In the first loop, PE receives the input activation value of the first row and calculates the corresponding output activation values ​​for the three rows. These are then selected and stored in the corresponding memory locations for rows 0, 1, and 2. In the second loop, the same three rows of output activation values ​​are generated and allocated to the corresponding memory locations for rows 1, 2, and 3. The third loop similarly selects rows 2, 3, and 4. Different PEs within each loop will access different memory locations, thus preventing address conflicts between PEs. After all three loops have completed their calculations, the mapped addresses of the output activation values ​​will cover the entire memory space.

[0106] As described in the above embodiments, line-by-line input splitting activation can directly eliminate the possibility of hash collisions between lines, thus reducing computational latency.

[0107] Furthermore, to illustrate how the new block partitioning scheme can reduce memory resource overhead and significantly reduce interconnect density, consider the Cartesian product of *a* input activations and *w* weights. In a traditional dataflow, a*w*OC number of block-granularity memory resources are required, where OC represents the block granularity based on the output channel. Multiplying the number of input interfaces (*w) and the number of output interfaces (*w*OC) of the crossbar switch circuit yields the interconnect density of the crossbar switch circuit segment as (*w)^2*OC.

[0108] In the row-level granularity block partitioning scheme provided in this embodiment, the block granularity of the accumulated memory is (ksl*a+2)*OC. When w is greater than ksl, memory resources will be effectively saved. The interconnection density of the crossbar switch router is a*w*5*OC, which is equivalent to (w*a / 5) times the interconnection density gain.

[0109] In practical applications, although some access conflicts still exist in this embodiment, their probability is lower than that of the original hash conflict. Taking an a*w Cartesian product array as an example, in the original Cartesian product data stream, each array element may conflict with the same address as other a*w-1 elements. However, in the data stream of this embodiment, array elements may still conflict with other w-1 elements within the group, but not with other (a-1)*w elements. This is because the multrs with the same row and column address as the adjacent row activation have been resolved and masked by the rolling summation module. The aforementioned conflicts specifically refer to multrs in the same row but different columns. The latency caused by the new conflicts is far less than that of the original Cartesian product architecture.

[0110] In some embodiments, the accelerator further includes an input storage module connected to a computer memory device.

[0111] The computer memory device can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR).

[0112] The input storage module includes an input activation memory, an input activation code memory, an input weight memory, and a weight sparse code memory; the operation unit is connected to the input activation memory, the input activation code memory, and the input weight memory respectively, and the hash encoding module is connected to the weight sparse code memory; the input activation memory, the input activation code memory, the input weight memory, and the weight sparse code memory are used to store the input activation data, the input activation code, the input weight data, and the input weight sparse code sent by the computer memory device respectively.

[0113] This embodiment can provide an input activation memory, an input activation code memory, an input weight memory, and a weight sparse code memory to store the input activation data, input activation code, input weight data, and input weight sparse code sent by the computer memory device, so that they can be used by the computing unit array and the hash coding module.

[0114] In some embodiments, to facilitate the storage and processing of output data, the accelerator further includes an output activation memory and a post-processing module, the post-processing module being connected to the output activation memory and the output activation memory being connected to a computer memory device.

[0115] The output activation memory is used to retrieve and store output activation data from the accumulator memory. The output activation data is the output result data stored in the accumulator memory. The output activation memory is also used to send the stored output activation data to the computer memory device.

[0116] The post-processing module is used to process the output activation data stored in the output activation memory when it receives subsequent processing instructions.

[0117] This application also provides a method for implementing a sparse convolutional neural network accelerator based on the above embodiments, which may include the following steps:

[0118] Step S1: Configure the input activation data in the input data of the sparse convolutional neural network accelerator into at least two rows of input activation values ​​with a preset number of rows between them. The input data includes input activation data, input weight data, input activation sparse coding corresponding to the input activation data, and input weight sparse coding corresponding to the input weight data.

[0119] Step S2: Input the sparse encoding of the input weights into the hash encoding module. The hash encoding module generates a lookup table based on the sparse encoding of the input weights. The lookup table is used to mark the intermediate results of the convolution operation that have hash collisions.

[0120] Step S3: Input activation data, input activation sparse code, input weight data and lookup table are input into the operation unit. The operation unit calculates multiple intermediate convolution operation results based on the input activation data and input weight data, and retrieves the lookup table based on the input activation sparse code to determine the intermediate convolution operation result that has a hash collision among the multiple intermediate convolution operation results.

[0121] In step S4, the operation unit sums the intermediate results of the convolution operation that have hash collisions, and sends the summed output to the corresponding accumulator memory for storage.

[0122] For specific limitations on the implementation method provided in this embodiment, please refer to the above embodiment on sparse convolutional neural network accelerator, which will not be repeated here.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A sparse convolutional neural network accelerator, comprising: The accelerator includes a computing unit array, a hash encoding module, and an output storage module, wherein the output storage module includes multiple accumulators. The array of computing units includes multiple computing unit groups, each computing unit group is connected to a hash encoding module and an accumulation memory, and the computing unit group only accesses its corresponding accumulation memory; each computing unit group contains multiple computing units. The hash encoding module is used to receive the input weight sparse encoding and generate a lookup table based on the input weight sparse encoding. The lookup table is used to mark the intermediate results of convolution operations that have hash collisions. The operation unit is used to receive input activation data, input activation sparse coding, input weight data and the lookup table, calculate multiple intermediate convolution operation results based on the input activation data and the input weight data, retrieve the lookup table based on the input activation sparse coding to determine the intermediate convolution operation results that have hash collisions among the multiple intermediate convolution operation results, sum the intermediate convolution operation results that have hash collisions, and send the summed output result to the accumulation memory corresponding to the operation unit for storage. The input activation data is configured to include at least two lines of input activation values ​​with a preset interval.

2. The accelerator according to claim 1, characterized in that, The input weight data contains multiple input weights; each row of input activation values ​​contains one or more input activation values. The arithmetic unit includes a multiplier array and a rolling summation module, wherein the multiplier array contains multiple multipliers; The multiplier array is used to broadcast a single input activation value to the multiple multipliers for multiplication with different input weights, thereby obtaining multiple intermediate results of convolution operations; The rolling summation module is used to retrieve the lookup table based on the input sparse coding to determine the intermediate convolution operation results that have hash collisions among the multiple intermediate convolution operation results, sum the intermediate convolution operation results that have hash collisions, and send the summed output result to the accumulation memory corresponding to the operation unit.

3. The accelerator according to claim 2, characterized in that, The rolling summation module includes multiple rolling units, each of which is connected one-to-one with each multiplier in the multiplier array to receive the intermediate results of the convolution operation output by the multiplier. Each of the scrolling units is also interconnected with multiple scrolling units adjacent to it on the left, so as to receive multiple intermediate results of neighbor convolution operations output by the multiple scrolling units adjacent to it on the left. The rolling unit is used to select the intermediate results of the neighbor convolution operation to be summed from the intermediate results of the plurality of neighbor convolution operations according to the input activation sparse coding and the lookup table, store them, and sum the stored intermediate results of the neighbor convolution operation to be summed with the intermediate results of the convolution operation that have hash collisions received in the next clock cycle, and send the summed output result to the accumulation memory corresponding to the operation unit.

4. The accelerator according to claim 3, characterized in that, The rolling unit includes a register, a first multiplexer, a second multiplexer, a third multiplexer, and a first adder; The input data of the second multiplexer are the intermediate results of the first neighbor convolution operation output by the first left rolling unit of the rolling unit, the intermediate results of the second neighbor convolution operation output by the second left rolling unit, and the zero value when there is no summation relationship; The first multiplexer is used to retrieve the hash code corresponding to the input activation sparse code in the lookup table, and controls the second multiplexer to select the intermediate result of the neighbor convolution operation to be summed from the intermediate result of the first neighbor convolution operation and the intermediate result of the second neighbor convolution operation according to the hash code and send it to the register for storage; The register is used to store the intermediate results of the nearest convolution operation to be summed; The first adder is used to sum the intermediate results of the neighbor convolution operation to be summed and the intermediate results of the convolution operation received by the rolling unit when the next clock cycle arrives, and send the summed output result to the first adder of the accumulation memory corresponding to the operation unit. The third multiplexer is used to send the intermediate results of the convolution operation received by the scrolling unit to the two scrolling units adjacent to the right of the scrolling unit in turn.

5. The accelerator according to claim 1, characterized in that, The output storage module also includes a crossbar switch router; The crossbar router is used to send the output of each arithmetic unit to the corresponding accumulator memory of that arithmetic unit.

6. The accelerator according to any one of claims 1 to 5, characterized in that, The accumulator memory includes a second adder and a memory; The second adder is used to retrieve the current value of the storage address corresponding to the output result from the memory, sum the current value and the output result, and send the summation result back to the storage address corresponding to the output result for storage.

7. The accelerator according to claim 6, characterized in that, The memory is configured to be divided into blocks only by rows, not by columns.

8. The accelerator according to claim 1, characterized in that, The accelerator also includes an input storage module, which is connected to a computer memory device; The input storage module includes an input activation memory, an input activation encoding memory, an input weight memory, and a weight sparse encoding memory. The computation unit is connected to the input activation memory, the input activation encoding memory and the input weight memory respectively, and the hash encoding module is connected to the weight sparse encoding memory; The input activation memory, the input activation encoding memory, the input weight memory, and the weight sparse encoding memory are respectively used to store the input activation data, the input activation encoding, the input weight data, and the input weight sparse encoding sent by the computer memory device.

9. The accelerator according to claim 1, characterized in that, The accelerator also includes an output activation memory and a post-processing module, the post-processing module being connected to the output activation memory, and the output activation memory being connected to a computer memory device; The output activation memory is used to retrieve and store output activation data from the accumulator memory, wherein the output activation data is the output result data stored in the accumulator memory. The output activation memory is also used to send the stored output activation data to the computer memory device; The post-processing module is used to process the output activation data stored in the output activation memory when a subsequent processing instruction is received.

10. A method for implementing a sparse convolutional neural network accelerator based on any one of claims 1 to 9, characterized in that, The method includes: The input activation data in the input data of the sparse convolutional neural network accelerator is configured as at least two rows of input activation values ​​with a preset number of rows between them. The input data includes the input activation data, input weight data, input activation sparse coding corresponding to the input activation data, and input weight sparse coding corresponding to the input weight data. The input weight sparse encoding is input into the hash encoding module, and the hash encoding module generates a lookup table based on the input weight sparse encoding. The lookup table is used to mark intermediate results of convolution operations in which hash collisions occur. The input activation data, the input activation sparse code, the input weight data, and the lookup table are input into the operation unit. The operation unit calculates multiple intermediate convolution operation results based on the input activation data and the input weight data, and retrieves the lookup table based on the input activation sparse code to determine the intermediate convolution operation result that has a hash collision among the multiple intermediate convolution operation results. The operation unit sums the intermediate results of the convolution operation that has caused hash collisions, and sends the summed output to the corresponding accumulator memory for storage.

Images