A data processing unit of a sparse convolutional neural network
By introducing weight storage and activation cache modules into the sparse convolutional neural network accelerator, and utilizing first-in-first-out queues and parallel reading techniques, the problems of address congestion and port contention are solved, the loading speed and multiplier utilization are improved, and the addressing logic is simplified.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2022-12-15
- Publication Date
- 2026-04-28
AI Technical Summary
Existing sparse convolutional neural network accelerators fail to effectively control the loading order when loading non-zero activation data and non-zero weight data, leading to addressing congestion and memory port contention, increasing chip area and reducing performance.
It employs a weight storage module, an activation cache module, a weight decoding module, an activation decoding module, a weight loading queue, an activation loading queue, and a computation and output cache module. By using a first-in-first-out queue and parallel reading of multiple convolution weights, it simplifies the addressing logic and avoids addressing congestion and port contention.
It improves loading speed, ensures that the multiplier is effectively used for most clock cycles, fundamentally solves the problems of address congestion and port contention, and simplifies the cache addressing logic of convolution results.
Smart Images
Figure CN115983338B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically, to a data processing unit for a sparse convolutional neural network. Background Technology
[0002] Sparse convolutional neural networks generate a significant number of zero values during both training and inference. Since zero values contain no information, they can be removed during storage and computation to save storage space and improve the efficient computation ratio. In sparse convolution computation, theoretically only non-zero activation data and non-zero weight data need to be fed into a multiply-accumulate array composed of multipliers and adders for operation. To alleviate the pressure on hardware resources caused by the massive computational demands of sparse convolutional neural networks, several sparse convolutional neural network accelerators have been developed.
[0003] However, in existing sparse convolutional neural network accelerators, the loading order of non-zero activation data and non-zero weight data onto the multiplier array is not effectively controlled, leading to the following problems: The calculation results of multiple multipliers within the same cycle may be interconnected and need to be accumulated before being stored in the same output cache address, causing addressing congestion; the output cache addresses of multiple multiplier calculation results within the same cycle may also be located in the same memory block, resulting in memory read / write port contention. To alleviate addressing congestion and port contention, some accelerators have proposed adding FIFOs and cross switches between the multiplier array and the output cache address, and employing complex control logic to solve the above problems. However, the added components also lead to a corresponding increase in the accelerator's chip area, and performance degradation due to the need to pause the multiplier array's computation. None of the above accelerators have fundamentally solved the problem of partial multiplication-accumulation result cache congestion. Summary of the Invention
[0004] To address the addressing congestion and port contention issues in existing sparse convolutional neural network accelerators, this application provides a data processing unit for sparse convolutional neural networks that enables multipliers to read multiple convolution weights in parallel, greatly improving loading speed. A first-in-first-out queue ensures that the multipliers are effectively utilized for most clock cycles. The convolution result caching addressing logic is simple, fundamentally avoiding addressing congestion and port contention problems.
[0005] To achieve the above objectives, this application adopts the following technical solution:
[0006] Firstly, embodiments of this application provide a data processing unit for a sparse convolutional neural network.
[0007] In one possible implementation, the data processing unit includes a weight storage module, an activation cache module, a weight decoding module, an activation decoding module, a weight loading queue, an activation loading queue, and a computation and output cache module.
[0008] The weight storage module is electrically connected to the weight decoding module, the weight decoding module is electrically connected to the weight loading queue, and the weight loading queue is electrically connected to the operation and output buffer module.
[0009] The activation cache module is electrically connected to the activation decoding module, the activation decoding module is electrically connected to the activation loading queue, and the activation loading queue is electrically connected to the operation and output cache module.
[0010] The weight storage module is used to cache multiple encoded weight data;
[0011] The activation cache module is used to cache multiple encoded activation data;
[0012] The weight decoding module is used to decode multiple encoded weight data to obtain multiple decoded weight data and the coordinates of each decoded weight data.
[0013] The activation decoding module is used to decode multiple encoded activation data to obtain multiple decoded activation data and the coordinates of each decoded activation data.
[0014] The weight loading queue is used to temporarily store multiple decoded weight data and the coordinates of each decoded weight data.
[0015] The activation loading queue is used to temporarily store multiple decoded activation data and the coordinates of each decoded activation data;
[0016] The operation and output buffer module is used to perform multiplication and accumulation operations on each of the decoded weight data and each of the decoded activation data to obtain the multiplication and accumulation operation result.
[0017] The computation and output caching module is further configured to calculate the coordinates of the multiply-accumulate operation result using the coordinates of each of the decoded weight data and the coordinates of each of the decoded activation data, and to save the multiply-accumulate operation result based on the coordinates of the multiply-accumulate operation result.
[0018] In one possible implementation, the operation and output buffer module includes a multiplier array, an adder array, and an output buffer array; the multiplier array is electrically connected to the adder array, and the adder array is electrically connected to the output buffer array.
[0019] The multiplier array comprises N×M multipliers, where N is the number of rows in the multiplier array and M is the number of columns in the multiplier array; the multiplier array is used to multiply each of the decoded activation data and each of the decoded weight data pairwise to obtain the multiplication result and the coordinates of the multiplication result;
[0020] The adder array includes N×M adders, which are used to accumulate the multiplication results with the same coordinates to obtain the multiplication-accumulation operation result.
[0021] The output buffer array includes N×M storage blocks, each storage block corresponding to an adder; each adder is used to temporarily store the multiplication and accumulation operation result in the output buffer array according to the coordinates of the multiplication result;
[0022] The output buffer array is further used to store the operation result into the storage block in the i-th row and j-th column when the column number of the operation result modulo N is i and the output channel number of the operation result modulo M is j.
[0023] In one possible implementation, the operation and output buffer module is further configured to simultaneously transmit the multiplication result obtained by each multiplier with column number m1 and row number n1 and the coordinates of the multiplication result to three adders with column number m1 and row numbers n1, n1+1 and n1-1.
[0024] The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by the multiplier with row number 0 to the three adders with row numbers n1=0, n1=1 and n1=N-1.
[0025] The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by the multiplier with row number n1=N-1 to the three adders with row numbers n1=0, n1=N-2 and n1=N-1.
[0026] Each of the adders simultaneously receives the three multiplication results and the coordinates of the multiplication results;
[0027] Each adder selects the multiplication result and its coordinates based on the column coordinates of the decoded weight data loaded into the column of the adder in the weight matrix, and accumulates the multiplication result with the multiplication result with the same coordinates stored in the output buffer, and writes the accumulated result into the same address in the output buffer.
[0028] In one possible implementation, the decoded activation data and the decoded weight data loaded onto the multiplier array during any clock cycle belong to the same input channel.
[0029] The multipliers in the a-th row of the multiplier array are used to receive the same decoded activation data;
[0030] The b-th column of the multiplier array is used to receive the same decoded weight data.
[0031] In one possible implementation, during any given clock cycle, the decoded weight data loaded on different columns of the multiplier array is different and belongs to different output channels.
[0032] In one possible implementation, the weight loading queue comprises M first-in-first-out queues;
[0033] The weight loading queue is used to load each of the decoded weight data and the coordinates of each of the decoded weight data into the operation and output cache module;
[0034] The weight loading queue is also used to add the decoded weight data to the first-in-first-out queue numbered j when the output channel number in the coordinates of the decoded weight data is modulo M.
[0035] The weight loading queue is also used to synchronously output the decoded weight data and the coordinates of each decoded weight data at the front end of the queue within one clock cycle when loading each decoded weight data and the coordinates of each decoded weight data into the calculation and output cache module.
[0036] If the first-in-first-out queue is empty, the decoded weight data output from the empty queue is marked as invalid.
[0037] In one possible implementation, the activation loading queue comprises N first-in-first-out queues;
[0038] The activation loading queue is used to load each of the decoded activation data and the coordinates of each of the decoded activation data into the operation and output cache module;
[0039] The activation loading queue is also used to add the decoded activation data to the first-in-first-out queue numbered i when the column number of the coordinates of the decoded activation data is modulo i with N.
[0040] The activation loading queue is also used to synchronously output the decoded activation data and the coordinates of each decoded activation data at the front end of the queue within one clock cycle when loading each decoded activation data and the coordinates of each decoded activation data into the computing module.
[0041] If the first-in-first-out queue is empty, then the decoded activation data output from the empty queue is marked as invalid.
[0042] The beneficial effects of the embodiments of this application include, for example:
[0043] This application provides a data processing unit for a sparse convolutional neural network, including a weight storage module, an activation cache module, a weight decoding module, an activation decoding module, a weight loading queue, an activation loading queue, and a computation module. The weight storage module and activation cache module temporarily store data, while the weight decoding module and activation decoding module decode the sparsely encoded data. The weight loading queue and activation loading queue load data into the computation module. The computation and output cache module performs multiplication and accumulation operations on the weights and activation data and temporarily stores the results of these operations. This sparse convolutional neural network data processing unit allows the multiplier to read multiple convolution weights in parallel, greatly improving loading speed. The first-in-first-out queue ensures that the multiplier is effectively used within most clock cycles. The convolution result cache addressing logic is simple, fundamentally avoiding address congestion and address contention problems. Attached Figure Description
[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 A schematic diagram of the structure of a data processing unit for a sparse convolutional neural network provided in an embodiment of this application;
[0046] Figure 2 A schematic diagram illustrating the sparse coding method provided in the embodiments of this application;
[0047] Figure 3 This is a schematic diagram illustrating the operation of the activation loading queue provided in an embodiment of this application;
[0048] Figure 4 A schematic diagram of the structure of a multiplier array provided in an embodiment of this application;
[0049] Figure 5 This is a schematic diagram of the structure of an output buffer array provided in an embodiment of this application;
[0050] Figure 6 This is a schematic diagram of the structure of the operation and output cache module provided in an embodiment of this application.
[0051] Icons: 110 - Weight storage module; 120 - Activation cache module; 130 - Weight decoding module; 140 - Activation decoding module; 150 - Weight loading queue; 160 - Activation loading queue; 170 - Calculation and output cache module. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0053] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0054] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0055] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.
[0056] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0057] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.
[0058] Convolutional Neural Networks (CNNs) typically consist of numerous convolutional layers and multiple fully connected layers. Convolutional layers extract features from the input data; fully connected layers combine all local features into global features and classify targets in the input data. The input feature map is denoted as the IFM (Input Feature Map), and each element in the input feature map is called activation data. Convolutional filters, also known as convolutional kernels, are typically found in multiple convolutional layers. The output feature map is denoted as the OFM (Output Feature Map). During convolution, each convolutional kernel slides through the IFM in a sliding window manner. Within each window, only the weights of convolutional kernels belonging to the same input channel are multiplied bit-by-bit with the IFM image data. The results of all multiplications are then summed to obtain an element in one channel of the OFM. The image data for one channel of the OFM is obtained after one convolutional kernel is convolved with the IFM; the complete OFM data is obtained after all convolutional kernels are convolved with the IFM.
[0059] For ease of description, IFM_CH_NUM will be used to represent the number of input channels in the initial activation data, and the height of each input channel of the input feature image will be I. h Width is I w Each initial activation data uses This indicates that ifm_ch is the input channel number where the activation data is located, and a r Its line number, a c Give it a column number;
[0060] The number of output data channels is OFM_CH_NUM, and the height of each output channel of the output feature image is O. h Width is O w Each output data is used This indicates that ofm_ch is the output channel number where the data is located, and o r Its line number, o c Give it a column number;
[0061] The number of convolutional kernels in each convolutional layer is OFM_CH_NUM, the number of channels in each convolutional kernel is IFM_CH_NUM, and the height of each channel in the convolutional kernel is K. h Width is K w Each weight uses This indicates that the kernel number is also the output channel number, i.e., ofm_ch, and the channel number within the kernel is also the input channel number, i.e., ifm_ch. r Its line number, w c Give it a column number.
[0062] Please refer to Figure 1 This embodiment provides a data processing unit for a sparse convolutional neural network. The data processing unit includes a weight storage module 110, an activation cache module 120, a weight decoding module 130, an activation decoding module 140, a weight loading queue 150, an activation loading queue 160, and an operation and output cache module 170.
[0063] The weight storage module is electrically connected to the weight decoding module, the weight decoding module is electrically connected to the weight loading queue, and the weight loading queue is electrically connected to the operation and output buffer module.
[0064] The activation cache module is electrically connected to the activation decoding module, the activation decoding module is electrically connected to the activation loading queue, and the activation loading queue is electrically connected to the operation and output cache module.
[0065] The weight storage module is used to cache encoded weight data; the encoded weight data is weight data that has undergone sparse encoding.
[0066] The activation cache module is used to cache the encoded activation data; the encoded activation data is sparsely encoded activation data.
[0067] When weight data and activation data are cached in the weight storage module and activation cache module, they are stored in a sparse encoding format. For details on the encoding format, please refer to [link to relevant documentation]. Figure 2 .
[0068] When encoding a data vector, the number of zero values between two non-zero values, i.e., the offset, is recorded using 4 bits. If the number of zero values exceeds 15, the 16th zero value is treated as a non-zero value and recorded. For example... Figure 2 As shown, in the example data vector, there is one 0 before the starting position of the vector a, so its offset is recorded as 1. There are two 0s between b and c, so the offset of c is recorded as 2. There are 17 0s between c and d, which exceeds the maximum range of offsets. Therefore, the 16th 0 is treated as a non-zero element, and its offset is recorded as a hexadecimal number F. Then, the offset of d is calculated starting from 0.
[0069] In fact, except for the first convolutional layer, the IFM of each convolutional layer is the OFM of the previous convolutional layer. To reduce cache requirements, the calculation results of each convolutional layer should be sparsely encoded before proceeding to the next convolutional layer. Taking the second convolutional layer of the VGG-16 neural network as an example, the IFM size of this layer is 224×224×64, which means it contains 64 input channels, and the feature image size of each channel is 224×224. When loading the data to be processed into the data processing unit, the data is sparsely encoded channel by channel and row by row and temporarily stored in the activation cache module. That is, first, the data belonging to the 0th row of the 0th channel is input, then the data belonging to the 1st channel is input, and so on until the data belonging to the 64th channel is input. Then, the data belonging to the 1st row of the 0th to 64th channels is input sequentially. After traversing all channels and all rows of data, the input is complete.
[0070] When the size of each channel in the feature image is small, multiple rows of data can be concatenated for sparse coding. For example, the size of the 13th convolutional IFM layer in the VGG-16 neural network is 28×28×512. Within each channel, each row of IFM contains only 28 data points, and in sparse cases, the number of non-zero data points will be even smaller. In this case, two or four rows of data can be concatenated end-to-end into one row before sparse coding. During the concatenation process, it is essential to ensure that all data belong to the same input channel.
[0071] The weight decoding module is used to decode multiple encoded weight data to obtain multiple decoded weight data and the coordinates of each decoded weight data; wherein, the coordinates of the decoded weight data are the coordinates of the decoded weight data in the weight matrix.
[0072] The activation decoding module is used to decode multiple encoded activation data to obtain multiple decoded activation data and the coordinates of each decoded activation data; wherein, the coordinates of the decoded activation data are the coordinates of the decoded activation data in the input feature image.
[0073] The activation loading queue is used to temporarily store multiple decoded activation data and the coordinates of each decoded activation data;
[0074] In one embodiment, the activation loading queue comprises N First-In-First-Out (FIFO) queues. Generally, the number of FIFOs N should be a positive integer power of 2. Each position in the FIFO can store the decoded activation data and the original coordinate information including row number and column number.
[0075] like Figure 3 As shown, when the decoded activation data is input into the FIFO, the queue to be added is selected according to the column number of its original coordinate information. That is, if the column number of a decoded activation data is a... cThen the queue number it should join is a. c %N, in Figure 3 Assume N is 8. The left figure in Figure 3 shows the loading process of the decoded activation data, i.e., according to A0~A 55 The data is loaded sequentially and added to the corresponding queues, with 0 values represented by spaces in the diagram. Due to the FIFO nature of the algorithm, all non-zero decoded activation data will be tightly packed in the queues. The % operator performs a modulo operation, and the result is the remainder.
[0076] When the activation loading queue loads the decoded activation data and activation data coordinates into the computing module, each FIFO synchronously outputs the decoded activation data and activation data coordinates in each clock cycle. When any of the N FIFOs is empty, the decoded activation data output by the empty FIFO is marked as invalid.
[0077] The weight loading queue is used to temporarily store multiple decoded weight data and the coordinates of each decoded weight data.
[0078] In one embodiment, the weight loading queue includes M first-in-first-out (FIFO) queues; generally, M should be a positive integer power of 2. Each position in the FIFO can store the decoded weight data and the original coordinate information including the output channel number, row number, and column number.
[0079] Similar to the activation loading queue, when inputting the decoded weight data into the FIFO, the queue to be added is selected according to the output channel number of the original coordinate information of each weight. That is, if the output channel number of a decoded weight data is ofm_ch, then the queue number it should be added to is ofm_ch%M.
[0080] When the weight loading queue loads the decoded weight data into the computation module, each FIFO synchronously outputs the decoded weight data in each clock cycle. When any of the M FIFOs is empty, the decoded weight data output by the empty FIFO is marked as invalid. When loading data into the multiply-accumulate array, all N FIFOs in the activation loading queue and all M FIFOs in the weight loading queue synchronously output data until the end of the longest queue is reached. If a queue completes loading all non-zero data prematurely and becomes an empty queue, its output is marked as invalid. In this way, the decoded activation data and decoded weight data are loaded onto the multiplier array in a regular manner, and the utilization rate of the multiplier in each cycle is improved. It should be noted that due to the sparse randomness, the lengths of different queues may be different, which may actually lead to some rows of the multiplier array being idle in some cycles. This problem can be mitigated by balancing sparsity or increasing the number of non-zero data.
[0081] It is important to note that each loaded decoded activation data must remain in the queue for an extended period, and all non-zero decoded activation data in the activation loading queue must belong to the same input channel. During the loading process of non-zero decoded activation data and decoded weight data, each time a decoded weight data is updated, the data in the decoded activation data loading queue needs to be loaded cyclically until the decoded activation data and all decoded weight data belonging to the same input channel have completed the convolution operation.
[0082] The operation and output buffer module is used to perform multiplication and accumulation operations on each of the decoded weight data and each of the decoded activation data to obtain the multiplication and accumulation operation result.
[0083] The computation and output caching module is further configured to calculate the coordinates of the multiply-accumulate operation result using the coordinates of each of the decoded weight data and the coordinates of each of the decoded activation data, and to save the multiply-accumulate operation result based on the coordinates of the multiply-accumulate operation result.
[0084] In one embodiment, the arithmetic and output buffer module includes a multiplier array, an adder array, and an output buffer array; the multiplier array is electrically connected to the adder array, and the adder array is electrically connected to the output buffer array.
[0085] like Figure 4 As shown, the multiplicative array has a size of N×M, where N is the number of rows and M is the number of columns. It is generally recommended that N and M be positive integer powers of 2. Figure 4 In the embodiment shown, both N and M are 8. Figure 4 In ~ This is the decoded activation data. ~ This is the decoded weight data.
[0086] The multiplier array comprises N×M multipliers, where N is the number of rows in the multiplier array and M is the number of columns in the multiplier array; the multiplier array is used to multiply each of the decoded activation data and each of the decoded weight data pairwise to obtain the multiplication result and the coordinates of the multiplication result;
[0087] The adder array includes N×M adders, which are used to accumulate the multiplication results with the same coordinates to obtain the multiplication-accumulation operation result.
[0088] The output buffer array includes N×M storage blocks, each storage block corresponding to an adder; each adder is used to temporarily store the multiplication and accumulation operation result in the output buffer array according to the coordinates of the multiplication result;
[0089] The output buffer array is further configured to store the operation result in the storage block at row i and column j when the column number of the operation result modulo N is i and the output channel number of the operation result modulo M is j. This ensures that the product of the loaded decoded activation data and decoded weight data is meaningful in both cases.
[0090] Within any given clock cycle, the decoded activation data and decoded weight data loaded onto the multiplier array belong to the same input channel. The multipliers in row a of the multiplier array receive the same decoded activation data, and the multipliers in column b of the multiplier array receive the same decoded weight data. Within any given clock cycle, the decoded weight data loaded onto different columns of the multiplier array are different and belong to different output channels. This ensures that the calculation results of multipliers within the same column belong to different elements in the OFM, thus eliminating the need to accumulate the calculation results obtained by multipliers within the same column within the same clock cycle.
[0091] Furthermore, this method ensures that the partial results obtained by different column multipliers belong to different output channels, thus eliminating the need for accumulation between columns. This data loading method guarantees that all operations performed by the multiplier array are valid, while the calculation results of each multiplier are independent of each other.
[0092] In one embodiment, the weight loading queue includes M first-in-first-out queues;
[0093] The weight loading queue is used to load each of the decoded weight data and the coordinates of each of the decoded weight data into the operation and output cache module; the weight loading queue is also used to add the decoded weight data into the first-in-first-out queue numbered j when the output channel number in the coordinates of the decoded weight data is modulo j with M.
[0094] The weight loading queue is also used to synchronously output the decoded weight data and the coordinates of each decoded weight data at the front end of the queue within one clock cycle when loading each decoded weight data and the coordinates of each decoded weight data into the calculation and output cache module.
[0095] If the first-in-first-out queue is empty, the decoded weight data output from the empty queue is marked as invalid.
[0096] In one embodiment, the activation loading queue comprises N first-in-first-out queues;
[0097] The activation loading queue is used to load each of the decoded activation data and the coordinates of each of the decoded activation data into the operation and output cache module;
[0098] The activation loading queue is also used to add the decoded activation data to the first-in-first-out queue numbered i when the column number of the coordinates of the decoded activation data is modulo i with N.
[0099] The activation loading queue is also used to synchronously output the decoded activation data and the coordinates of each decoded activation data at the front end of the queue within one clock cycle when loading each decoded activation data and the coordinates of each decoded activation data into the computing module.
[0100] If the first-in-first-out queue is empty, then the decoded activation data output from the empty queue is marked as invalid.
[0101] In one embodiment, the operation and output buffer module is further configured to simultaneously transmit the multiplication result obtained by each multiplier with column number m1 and row number n1 and the coordinates of the multiplication result to three adders with column number m1 and row numbers n1, n1+1 and n1-1.
[0102] The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by the multiplier with row number 0 to the three adders with row numbers n1=0, n1=1 and n1=N-1.
[0103] The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by the multiplier with row number n1=N-1 to the three adders with row numbers n1=0, n1=N-2 and n1=N-1.
[0104] Each of the adders simultaneously receives the three multiplication results and the coordinates of the multiplication results;
[0105] Each adder selects the multiplication result and its coordinates based on the column coordinates of the decoded weight data loaded into the column of the adder in the weight matrix, and accumulates the multiplication result with the multiplication result with the same coordinates stored in the output buffer, and writes the accumulated result into the same address in the output buffer.
[0106] like Figure 5 As shown, simultaneously, the coordinates of the OFM data to which the product of the activation data and weight data belongs must also be calculated. The output channel to which the product belongs is determined by the ofm_ch number of the weights, and its position within the channel is determined by the following formula.
[0107]
[0108] The output buffer is used to store part of the accumulated results. Since the multiply-accumulate array can produce N×M results per clock cycle, we divide the output buffer array into N×M blocks to achieve fast caching. Figure 5 This demonstrates the memory block partitioning method when both N and M are 8. Each memory block corresponds to a multiplier. To implement pipelined processing in each layer of the neural network, the output buffer needs to be able to cache... Add 1 row of OFM data for sliding window calculation and pooling processing.
[0109] For a storage block Bank(i,j) with row number i and column number j, only the storage address satisfies o c The convolution result of %N=i, ofm_ch%M=j. Based on the constraints we set for activation data and weight loading, since the weights loaded into different columns of the multiplier array have different ofm_ch, the multiplier MUL( column number j) will have different values. The calculation results only need to be stored in Bank(j) Within j), where This represents any value of i. Also, since the activation data was already assigned to different rows of the multiplier array according to its column number when loading the activation data into the multiplier array, i.e., a c %N=i, therefore for column number o c The storage location of the multiplication result is determined by... Decision. When K w When =3, o c =ac ±1, meaning the result of the multiplier MUL(i,j) may need to be stored in Bank(i±1,j).
[0110] To support this storage requirement, embodiments of this application design each column in the computation and output cache module as follows: Figure 6 The structure shown indicates that the result of any multiplier in the array may need to be added to a portion of the result stored in the memory block above or below it. It is important to note the calculation results of the top and bottom two multipliers (including the result of multiplying the activation data and weights, and the output coordinates {o}). r o c The result of the multiplication should be transmitted to the other party using the same logic. The multiplexer (MUX) selects the signal based on the column coordinates of the weights (w). c The multiplier array provides the same selection signal because multipliers in the same column share the same weight. Therefore, multiplexers in the same column naturally use the same selection signal. The storage location of the multiply-accumulate result within each block of the output buffer (Obuf) is determined by (o... r o c The value is determined by / N, ofm_ch / M.
[0111] Furthermore, the arithmetic and output buffer module can be viewed as consisting of three arrays: a multiplier array, an accumulator array, and an output buffer array. Alternatively, a multiplier, an accumulator, and an output buffer block can be considered as a basic unit, and N×M basic units constitute the arithmetic module, such as... Figure 6 As shown.
[0112] The sparse convolutional neural network data processing unit provided in this embodiment has at least the following advantages:
[0113] The data processing unit of the sparse convolutional neural network provided in this application embodiment enables the multiplier to read multiple convolution weights in parallel, which greatly improves the loading speed; through the first-in-first-out queue, the multiplier can be effectively used in most clock cycles; the convolution result cache addressing logic is simple, which fundamentally avoids the problems of address congestion and address contention.
[0114] In summary, this invention provides a data processing unit for a sparse convolutional neural network, including a weight storage module 110, an activation cache module 120, a weight decoding module 130, an activation decoding module 140, a weight loading queue 150, an activation loading queue 160, and an operation and output cache module 170. The weight storage module 110 and activation cache module 120 are used to temporarily store data; the weight decoding module 130 and activation decoding module 140 are used to decode the sparsely encoded data; the weight loading queue 150 and activation loading queue 160 are used to load data into the operation module; and the operation and output cache module 170 is used to perform multiplication and accumulation operations on the weights and activation data and to temporarily store them in the output cache array. The data processing unit for a sparse convolutional neural network provided in this application allows the multiplier to read multiple convolution weights in parallel, greatly improving the loading speed; the first-in-first-out queue ensures that the multiplier can be effectively used within most clock cycles; and the convolution result cache addressing logic is simple, fundamentally avoiding address congestion and address contention problems.
[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A data processing unit for a sparse convolutional neural network, characterized in that, The data processing unit includes a weight storage module, an activation cache module, a weight decoding module, an activation decoding module, a weight loading queue, an activation loading queue, and a computation and output cache module; The weight storage module is electrically connected to the weight decoding module, the weight decoding module is electrically connected to the weight loading queue, and the weight loading queue is electrically connected to the operation and output buffer module. The activation cache module is electrically connected to the activation decoding module, the activation decoding module is electrically connected to the activation loading queue, and the activation loading queue is electrically connected to the operation and output cache module. The weight storage module is used to cache multiple encoded weight data; The activation cache module is used to cache multiple encoded activation data; The weight decoding module is used to decode multiple encoded weight data to obtain multiple decoded weight data and the coordinates of each decoded weight data. The activation decoding module is used to decode multiple encoded activation data to obtain multiple decoded activation data and the coordinates of each decoded activation data. The weight loading queue is used to temporarily store multiple decoded weight data and the coordinates of each decoded weight data. The activation loading queue is used to temporarily store multiple decoded activation data and the coordinates of each decoded activation data; The operation and output buffer module is used to perform multiplication and accumulation operations on each of the decoded weight data and each of the decoded activation data to obtain the multiplication and accumulation operation result. The operation and output buffer module is further configured to calculate the coordinates of the multiply-accumulate operation result using the coordinates of each of the decoded weight data and the coordinates of each of the decoded activation data, and save the multiply-accumulate operation result according to the coordinates of the multiply-accumulate operation result; The operation and output buffer module includes a multiplier array, an adder array, and an output buffer array; the multiplier array is electrically connected to the adder array, and the adder array is electrically connected to the output buffer array; The multiplier array includes N×M multipliers, where N is the number of rows in the multiplier array and M is the number of columns in the multiplier array; each row of the multiplier array stores the corresponding decoded activation data, and each column of the multiplier array stores the corresponding decoded weight data.
2. The data processing unit of the sparse convolutional neural network according to claim 1, characterized in that, The multiplier array is used to multiply each of the decoded activation data and each of the decoded weight data pairwise to obtain the multiplication result and the coordinates of the multiplication result; The adder array includes N×M adders, which are used to accumulate the multiplication results with the same coordinates to obtain the multiplication-accumulation operation result; The output buffer array includes N×M storage blocks, each storage block corresponding to an adder; each adder is used to temporarily store the multiplication and accumulation operation result in the output buffer array according to the coordinates of the multiplication result; The output buffer array is further used to store the operation result into the storage block in the i-th row and j-th column when the column number of the operation result modulo N is i and the output channel number of the operation result modulo M is j.
3. The data processing unit of the sparse convolutional neural network according to claim 2, characterized in that, The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by each multiplier with column number m1 and row number n1 and the coordinates of the multiplication result to three adders with column number m1 and row numbers n1, n1+1 and n1-1. The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by the multiplier with row number 0 to the three adders with row numbers n1=0, n1=1 and n1=N-1. The operation and output buffer module is also used to simultaneously transmit the multiplication result obtained by the multiplier with row number n1=N-1 to the three adders with row numbers n1=0, n1=N-2 and n1=N-1. Each of the adders simultaneously receives the three multiplication results and the coordinates of the multiplication results; Each adder selects the multiplication result and its coordinates based on the column coordinates of the decoded weight data loaded into the column of the adder in the weight matrix, and accumulates the multiplication result with the multiplication result with the same coordinates stored in the output buffer, and writes the accumulated result into the same address in the output buffer.
4. The data processing unit of the sparse convolutional neural network according to claim 2, characterized in that, During any clock cycle, the decoded activation data and the decoded weight data loaded onto the multiplier array both belong to the same input channel. The multipliers in the a-th row of the multiplier array are used to receive the same decoded activation data; The b-th column of the multiplier array is used to receive the same decoded weight data.
5. The data processing unit of the sparse convolutional neural network according to claim 2, characterized in that, During any given clock cycle, the decoded weight data loaded on different columns of the multiplier array is different and belongs to different output channels.
6. The data processing unit of the sparse convolutional neural network according to claim 2, characterized in that, The weight loading queue includes M first-in-first-out queues; The weight loading queue is used to load each of the decoded weight data and the coordinates of each of the decoded weight data into the operation and output cache module; The weight loading queue is also used to add the decoded weight data to the first-in-first-out queue numbered j when the output channel number in the coordinates of the decoded weight data is modulo M. The weight loading queue is also used to synchronously output the decoded weight data and the coordinates of each decoded weight data at the front end of the queue within one clock cycle when loading each decoded weight data and the coordinates of each decoded weight data into the calculation and output cache module. If the first-in-first-out queue is empty, the decoded weight data output from the empty queue is marked as invalid.
7. The data processing unit of the sparse convolutional neural network according to claim 2, characterized in that, The activation loading queue includes N first-in-first-out queues; The activation loading queue is used to load each of the decoded activation data and the coordinates of each of the decoded activation data into the operation and output cache module; The activation loading queue is also used to add the decoded activation data to the first-in-first-out queue numbered i when the column number of the coordinates of the decoded activation data is modulo i with N. The activation loading queue is also used to synchronously output the decoded activation data and the coordinates of the decoded activation data at the front end of the queue within one clock cycle when loading the decoded activation data and the coordinates of the decoded activation data into the operation and output buffer module. If the first-in-first-out queue is empty, then the decoded activation data output from the empty queue is marked as invalid.