Accelerator and method of operating an accelerator

Abstract

An accelerator and an operating method of the accelerator are provided. The operating method comprises: accessing, by a first processing circuit, a weight buffer to perform in-block preprocessing of a first weight tensor in the weight buffer to form a first preprocessed weight tensor in the weight buffer, and accessing, by the first processing circuit, the weight buffer to perform inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor in the weight buffer, wherein the in-block preprocessing comprises: moving a first element of a first weight block of the first weight tensor by one position in a forward direction or in a side direction by storing the first element at a second position of the weight buffer, and the inter-block preprocessing comprises: moving a first row of a first weight block of the first preprocessed weight tensor by one position in the forward direction or by one position in the side direction by storing the first row at a fourth position of the weight buffer.

Description

[0001] This application claims priority and benefit to U.S. Provisional Application No. 63 / 024,676, filed May 14, 2020, entitled “HIERARCHICAL WEIGHT PREPROCESSING FOR NEURAL NETWORK ACCELERATOR,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] One or more aspects of embodiments of this disclosure relate to accelerators for neural networks, and more specifically, to a system and method for hierarchical weight preprocessing for neural network accelerators. Background Technology

[0003] The computations performed in a neural network can involve tensor products of weights and activations. The tensors involved can be relatively sparse, so computing each element-wise product would be significantly inefficient, as a large portion of these products would be equal to zero.

[0004] Therefore, a technique is needed to improve the operating efficiency of accelerators used for neural networks. Summary of the Invention

[0005] According to an embodiment of the present invention, an operation method for an accelerator is provided, comprising: accessing a weight buffer via a first processing circuit to perform intra-block preprocessing of a first weight tensor in the weight buffer to form a first preprocessed weight tensor in the weight buffer; and accessing the weight buffer via the first processing circuit to perform inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor in the weight buffer, wherein the intra-block preprocessing comprises: storing a first element of a first weight block of the first weight tensor at a second position in the weight buffer, and moving the first element of the first weight block of the first weight tensor by one position in a forward direction or a side-view direction, the second position being a position adjacent to the first position where the first element is stored in the forward direction or a side-view direction; and the inter-block preprocessing comprises: storing a first row of a first weight block of the first preprocessed weight tensor at a fourth position in the weight buffer, and moving the first row of the weight block of the first preprocessed weight tensor by one position in a forward direction or a side-view direction, the fourth position being a position adjacent to the third position where the first row is stored in the forward direction or a side-view direction.

[0006] According to an embodiment of the present invention, an accelerator is provided, comprising: a first processing circuit configured to: access a weight buffer to perform intra-block preprocessing of a first weight tensor in the weight buffer to form a first preprocessed weight tensor in the weight buffer; and access the weight buffer to perform inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor in the weight buffer, wherein the intra-block preprocessing comprises: storing a first element of a first weight block of the first weight tensor at a second position in the weight buffer, and moving the first element of the first weight block of the first weight tensor by one position in a forward direction or a side-view direction, the second position being a position adjacent to the first position where the first element is stored in the forward direction or a side-view direction; and the inter-block preprocessing comprises: storing a first row of a first weight block of the first preprocessed weight tensor at a fourth position in the weight buffer, and moving the first row of the weight block of the first preprocessed weight tensor by one position in a forward direction or a side-view direction, the fourth position being a position adjacent to the third position where the first row is stored in the forward direction or a side-view direction.

[0007] According to an embodiment of the present invention, an accelerator is provided, comprising: means for processing, the means for processing being configured to: access a weight buffer to perform intra-block preprocessing of a first weight tensor in the weight buffer to form a first preprocessed weight tensor in the weight buffer; and access the weight buffer to perform inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor in the weight buffer, wherein the intra-block preprocessing comprises: storing a first element of a first weight block of the first weight tensor of the first weight tensor at a second position in the weight buffer, and moving the first element of the first weight block one position in a forward direction or a side-view direction, the second position being a position adjacent to the first position where the first element is stored in the forward direction or a side-view direction; and the inter-block preprocessing comprises: storing a first row of the first weight block of the first preprocessed weight tensor at a fourth position in the weight buffer, and moving the first row of the weight block of the first preprocessed weight tensor one position in a forward direction or a side-view direction, the fourth position being a position adjacent to the third position where the first row is stored in the forward direction or a side-view direction.

[0008] According to an embodiment of the present invention, a hierarchical weight preprocessing method is provided, comprising: performing intra-block preprocessing of a first weight tensor to form a first preprocessed weight tensor, and performing inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor. The intra-block preprocessing includes: within a first weight block of the first weight tensor, shifting a first element of the first weight block by one position along a forward direction or along a side-view direction, and the inter-block preprocessing includes: shifting a first row of the weight block of the first preprocessed weight tensor by one position along a forward direction or along a side-view direction.

[0009] In some embodiments, in-block preprocessing includes: within the first weight block of the first weight tensor, shifting the first element of the first weight block in the forward direction to a position.

[0010] In some embodiments, inter-block preprocessing includes shifting the first row one position in the forward direction.

[0011] In some embodiments, the preprocessing within the block further includes: within the second weight block of the first weight tensor, shifting the first element of the second weight block by one position along the side-view direction.

[0012] In some embodiments, the first row is a row of the second weight block of the first preprocessed weight tensor, and the inter-block preprocessing includes: moving the first row from the second weight block to the first weight block along the side-view direction.

[0013] In some embodiments, inter-block preprocessing further includes: creating a block sparsity graph corresponding to the first preprocessed weight tensor, the block sparsity graph having: columns for each weight block of the first preprocessed weight tensor, and rows for each row of the weight blocks, the block sparsity graph indicating the position of empty rows of the weight blocks of the first preprocessed weight tensor.

[0014] In some embodiments, the block sparsity graph has one dimension less than the dimension of the first preprocessed weight tensor.

[0015] In some embodiments, the method further includes identifying the first row based on a block sparsity graph.

[0016] In some embodiments, the method further includes multiplying the first row by the first activation vector to form a first dot product, wherein the multiplication step includes extracting the activation vector from a column of the activation buffer, the column of the activation buffer being a second column in the activation buffer.

[0017] In some embodiments, the method further includes: multiplying a first row with a first activation vector in a first processing element circuit to form a first dot product; multiplying a second row of a weight block of a first preprocessed weight tensor with a second activation vector in a second processing element circuit to form a second dot product; and adding the first dot product and the second dot product.

[0018] In some embodiments, inter-block preprocessing further includes shifting the second row of the first preprocessed weight tensor by one position in the forward direction.

[0019] In some embodiments, the method further includes identifying the second row based on a block sparsity graph.

[0020] According to an embodiment of the present invention, a hierarchical weight preprocessing system is provided, comprising: a first processing circuit configured to: perform intra-block preprocessing of a first weight tensor to form a first preprocessed weight tensor, and perform inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor, wherein the intra-block preprocessing includes: within a first weight block of the first weight tensor, shifting a first element of the first weight block by one position in a forward direction or in a side-view direction, and the inter-block preprocessing includes shifting a first row of the weight block of the first preprocessed weight tensor by one position in a forward direction or in a side-view direction.

[0021] In some embodiments, in-block preprocessing includes: within the first weight block of the first weight tensor, shifting the first element of the first weight block in the forward direction to a position.

[0022] In some embodiments, inter-block preprocessing includes shifting the first row one position in the forward direction.

[0023] In some embodiments, the preprocessing within the block further includes: within the second weight block of the first weight tensor, shifting the first element of the second weight block by one position along the side-view direction.

[0024] In some embodiments, the first row is a row of the second weight block of the first preprocessed weight tensor, and the inter-block preprocessing includes: moving the first row from the second weight block to the first weight block along the side-view direction.

[0025] In some embodiments, inter-block preprocessing further includes: creating a block sparsity graph corresponding to the first preprocessed weight tensor, the block sparsity graph having: columns for each weight block of the first preprocessed weight tensor, and rows for each row of the weight blocks, the block sparsity graph indicating the position of empty rows of the weight blocks of the first preprocessed weight tensor.

[0026] In some embodiments, the system further includes a second processing circuit, the second processing circuit including: a first processing element circuit and a second processing element circuit, wherein: the first processing element circuit is configured to multiply a first row by a first activation vector to form a first dot product; and the second processing element circuit is configured to: multiply a third row of a weight block of a first preprocessed weight tensor by a second activation vector to form a second dot product, and add the first dot product and the second dot product.

[0027] According to an embodiment of the present invention, a hierarchical weight preprocessing system is provided, comprising: a processing device configured to: perform intra-block preprocessing of a first weight tensor to form a first preprocessed weight tensor, and perform inter-block preprocessing of the first preprocessed weight tensor to form a second preprocessed weight tensor, wherein the intra-block preprocessing includes: within a first weight block of the first weight tensor, shifting a first element of the first weight block by one position in a forward direction or in a side-view direction, and the inter-block preprocessing includes: shifting a first row of the first preprocessed weight tensor by one position in a forward direction or in a side-view direction. Attached Figure Description

[0028] These and other features and advantages of this disclosure will be appreciated and understood by referring to the specification, claims and drawings, in which:

[0029] Figure 1 These are block diagrams based on embodiments of the present disclosure;

[0030] Figure 2A This is a data organization diagram according to an embodiment of the present disclosure;

[0031] Figure 2B This is a data organization diagram according to an embodiment of the present disclosure;

[0032] Figure 2C This is a data organization diagram according to an embodiment of the present disclosure;

[0033] Figure 2D This is a data organization diagram according to an embodiment of the present disclosure;

[0034] Figure 2E This is a data organization diagram according to an embodiment of the present disclosure;

[0035] Figure 2F This is a data organization diagram according to an embodiment of the present disclosure;

[0036] Figure 2G This is a data organization diagram according to an embodiment of the present disclosure;

[0037] Figure 2H This is a data organization diagram according to an embodiment of the present disclosure;

[0038] Figure 2I This is a data organization diagram according to an embodiment of the present disclosure;

[0039] Figure 2J This is a data organization diagram according to an embodiment of the present disclosure;

[0040] Figure 3 This is a simplified circuit diagram according to an embodiment of the present disclosure;

[0041] Figure 4A This is a diagram of tensors according to embodiments of the present disclosure;

[0042] Figure 4B This is a table of axes according to embodiments of the present disclosure;

[0043] Figure 4C This is a flowchart based on an embodiment of the present disclosure;

[0044] Figure 4D This is a diagram of tensors according to embodiments of the present disclosure;

[0045] Figure 4E This is a flowchart based on an embodiment of the present disclosure;

[0046] Figure 4F This is a table of costs according to embodiments of the present disclosure;

[0047] Figure 4G These are hybrid block diagrams and schematic diagrams based on embodiments of the present disclosure;

[0048] Figure 4H These are hybrid block diagrams and schematic diagrams according to embodiments of the present disclosure; and

[0049] Figure 4I This is a schematic diagram according to an embodiment of the present disclosure. Detailed Implementation

[0050] The specific embodiments described below with reference to the accompanying drawings are intended as exemplary descriptions of systems and methods for hierarchical weight preprocessing for neural network accelerators provided in this disclosure, and are not intended to represent the only forms in which this disclosure can be constructed or utilized. This description, in conjunction with the illustrated embodiments, illustrates the features of this disclosure. However, it will be understood that the same or equivalent functions and structures can be implemented by different embodiments that are also intended to be included within the scope of the disclosure. As indicated elsewhere herein, the same element numbers are intended to indicate the same elements or features.

[0051] In computations performed on neural networks (e.g., convolutional neural networks), various operations can be performed on the tensors of weights and the tensors of activations (the latter may be referred to as input feature maps (IFM)). One such operation is the tensor product, in which some dimensions (which may be referred to as reducible dimensions) are summed, and some dimensions (which may be referred to as irreducible dimensions) are not summed. The computation of this tensor product can be performed by processing circuitry, which may include, for example... Figure 1 The diagram shows an array of processing element circuitry 105 (or simply "processing element" or "tile"). Each tile can be fed weights from weight buffer 110 and activations from activation buffer 115. Each tile may include a multiplier array 120 and an adder tree 125. In operation, in each computation cycle (which may include a single cycle of the system clock or several cycles of the system clock), each tile may perform the following operations: (i) read activation vectors from columns of the (shared) activation buffer, (ii) read weight vectors (or "rows") from rows of its weight buffer, and multiply the vectors to form a dot product. The multiplication of the vectors may include (in the multipliers) calculating the element-wise product of the activation vectors and the weight rows, and (in the adder tree) summing the products. The summed product may be referred to as the dot product. In one example, the weight buffers may be included inside or outside an accelerator used for a neural network.

[0052] When some elements of the weight tensor are zero, this system can skip multiplication. For example, refer to... Figure 2AEach of the four weight arrays (e.g., block #1 to block #4) can include some elements as zero or "empty" (as shown by the unshaded square) and some elements as non-zero (as shown by the shaded square). The weight array corresponding to a block can be called a "weight block". The horizontal entries of the weight block refer to channels (e.g., ch1, ch2, ch3, and ch4), and the vertical entries refer to computation cycles (e.g., cycles 1 to 5). For example, the computation cycle indicated by "cycle 1" precedes the computation cycle indicated by "cycle 2". In each of the first, second, and third weight blocks, if processed before a computation cycle, the element marked "forward" (or "lookahead") will make its currently occupied row completely empty (i.e., all elements in that row will be zero), meaning that processing can skip that row and proceed directly to the next row. For example, the weight array of block 1 (i.e., the first weight block, labeled "block #1") has a single non-zero element (labeled "forward") in the second row, and an empty element in the first row and in the same column as the non-zero element. Therefore, moving the element labeled "forward" to the previous row (which can be referred to as the "forward" direction) will empty the second row. Such out-of-order processing can be achieved by modifying the weight tensor and (as discussed further in detail below) by using a multiplexer to allow blocks to access elements from columns other than the current column in the active buffer.

[0053] exist Figure 2AIn the example, the second row of the fourth weight block includes an element (labeled "forward") that can be moved to the position directly above it (i.e., to the first row in the same column), and another element in the second row that cannot be moved directly upward because the element directly above it is not empty (labeled "lookaside"). However, because the element to the upper left of the element labeled "lookaside" is empty, the element labeled "lookaside" can move up one row and to the adjacent column (i.e., up and from the fourth column to the third column). The direction in which the element labeled "lookaside" moves can be referred to as the "lookaside" direction. In this example, the element of the second row, second column of the fourth weight block is moved one position forward by storing it at the second position of the weight buffer. The second position is adjacent to the first position (e.g., the position where the element of the first row, second column is stored) along the forward direction. Furthermore, by storing the element of the second row and fourth column of the fourth weight block at the fourth position of the weight buffer, the element of the second row and fourth column of the fourth weight block is moved one position along the side-look direction. The fourth position is a position along the side-look direction adjacent to the third position where the element of the second row and fourth column is stored (e.g., the position where the element of the first row and third column is stored). In one example, the forward direction (or forward look mode) indicates the direction along which the first element moves to the zero element if there is a zero element in the column where the first element of the weight block is located; the side-look direction indicates the direction along which the first element moves to the zero element if there is no zero element in the column where the first element is located but there is a zero element in an adjacent column.

[0054] Once all modifications have been made to the weight array, the modified weight tensor can be used as follows: Figure 2B As shown in the diagram. Weights are elements of a three-dimensional tensor comprising four (two-dimensional) weight blocks (each weight block is itself a tensor). As mentioned above, each array of weights associated with a block may be referred to as a "weight block" (as used herein, the word "block" without the qualifier "weight" refers to a processing element). Figure 2B The modified tensor shown can be referred to as the "preprocessed weight tensor" or the "first preprocessed weight tensor"; because the decision about which elements will move within any weight block is based solely on the position of the empty element within the weight block, modifying the tensor to form the first preprocessed weight tensor can be referred to as "intra-block preprocessing". In one example, processing circuitry included in the accelerator accesses a weight buffer to perform intra-block preprocessing of the first weight tensor in the weight buffer to form the first preprocessed weight tensor in the weight buffer. This processing circuitry and processing element circuitry 105 can be the same processing circuitry or different processing circuitry.

[0055] Because the activation buffer can be configured to send the same activation vector to all blocks during any computation cycle, Figure 1 Some embodiments may fail to utilize blank lines generated by preprocessing within blocks. Therefore, if the system skips a second computation cycle in which the first, third, and fourth weight blocks have blank lines, the correct product will not be calculated based on the weights in the second line of the second weight block. In some embodiments, multiple activation buffers are employed, one for each block, to skip lines at different times.

[0056] In some embodiments, the use of multiple activation buffers can be avoided by performing an additional preprocessing step, which may be referred to as "inter-block preprocessing." As discussed in further detail below, in inter-block preprocessing, elements of weight blocks can be moved based on empty elements of other weight blocks, and elements can be moved from one weight block to another. Inter-block preprocessing can form another preprocessed weight tensor (which may be referred to as "second preprocessed weight tensor") based on the first preprocessed weight tensor; thus, intra-block preprocessing and inter-block preprocessing can form two levels of weight tensor preprocessing hierarchy. In one example, the processing circuitry accesses a weight buffer to perform inter-block preprocessing of the first preprocessed weight tensor to form the second preprocessed weight tensor in the weight buffer.

[0057] To perform inter-block preprocessing, a block sparsity graph (with one less dimension than the weight tensor) can be generated first. Figure 2C It shows Figure 2B The first preprocessed weight tensor block sparsity map. T1, T2, T3, and T4 indicate the first, second, third, and fourth weight blocks, respectively. Figure 2C In the diagram, c2 to c5 can indicate the second to fifth computation cycles. The block sparsity graph can have one element per row for each weight block of the first preprocessed weight tensor; if the row is empty, the element is 0 (or e), and if the row is not empty, the element is 1. In other embodiments, the same information can be represented additionally (e.g., a single sign indicating an empty row, and zero indicating a non-empty row), or additional information (such as...) can be captured by the block sparsity graph. Figure 2C As in the example, the symbol in the square indicates whether the row is empty, and if it is not empty, the symbol in the square indicates which computation cycle it corresponds to before performing inter-block preprocessing.

[0058] Block sparsity graphs can be used to identify further modifications to the weight tensor (e.g., elements of the block sparsity graph to be moved), allowing the system to utilize empty rows generated by preprocessing within blocks. For example, it can be seen that by moving the third element of the first column up one position, moving the second and third elements of the third column up one position each, and moving the third element of the fourth column up one position, the block sparsity graph can be modified to achieve the desired effect. Figure 2D As shown, the third row is empty. This result implies that: (i) further adjustments to the weight tensor result in all weight blocks having empty rows in the third computation cycle, and (ii) if the weights are adjusted in this way, the third computation cycle can be skipped in a system where the activation buffer is configured to send the same activation vector to all blocks during each computation cycle.

[0059] Figure 2E The sequence of computation cycles for two different cases is shown: Case 1 (“Case 0”) and Case 2 (“Case 1”). In Case 1 (“Case 0”), the weight tensor is not modified and no computation cycle is skipped (because there is no computation cycle in which all weight blocks have empty lines). In Case 2 (“Case 1”), the third computation cycle is skipped and the third line of each weight block is empty due to the first intra-block preprocessing and the second inter-block preprocessing. Figure 2E It is also shown that, for the second case, an inter-block forward (as described above) can be performed to obtain a block sparsity graph with empty rows. Each inter-block forward involves moving along the forward direction (i.e., in...). Figure 2C and Figure 2E In the representation, moving all weights (or equivalently, all non-zero weights) in the row of the weight block upwards by one position, such that, for example (e.g., ... Figures 2C to 2E As shown in the diagram, after inter-block preprocessing is complete, the second row of the first block of the second preprocessed weight tensor is the third row of the first block of the first preprocessed weight tensor. When the row of the weight block moves one position forward, the activation vector multiplied in the operation can be the second vector in activation buffer 115 (instead of the current (first) vector in activation buffer 115) (i.e., the vector in activation buffer 115 that would be processed in the next computation cycle if no forward operation was performed during preprocessing). In this example, the third row of the first block of the first preprocessed weight tensor is moved one position forward by storing it at a second position in the weight buffer. The second position is a position adjacent to the first position where the third row of the first block is stored (e.g., the position where the second row of the first block is stored) in the forward direction.

[0060] In some cases, it is possible to skip additional computation cycles by performing block look-through (i.e., moving weights from one weight block to another within the weight tensor). Figure 2F An example of a weighted block (after preprocessing within the block) is shown, and Figure 2G It shows Figure 2F The block sparsity graph of the weighted blocks. Figure 2G In this context, cycle'1 to cycle'5 can respectively indicate the first to the fifth calculation cycle. Figure 2H A method for performing inter-block preprocessing is shown, and Figure 2I The diagram shows the block sparsity after inter-block preprocessing. Figure 2H and Figure 2I This illustrates a combination of block forward and block side-look operations that can be rearranged using a block sparsity graph (and the weight tensor can be modified in a similar manner), allowing two computation cycles to be skipped. Each block side-look involves moving a row of weights from one weight block to a row in another weight block in an earlier processing cycle, such that, for example (e.g.) Figure 2H As shown, after inter-block preprocessing is complete, the fourth row of the first weight block of the second preprocessed weight tensor is the fifth row of the second weight block of the first preprocessed weight tensor. This movement can be (similar to the intra-block look-ahead movement described above) referred to as a movement along the "look-ahead" direction. In this example, by storing the fifth row of the second weight block of the first preprocessed weight tensor at the fourth position in the weight buffer, the fifth row of the second weight block of the first preprocessed weight tensor is moved one position along the look-ahead direction. The fourth position is adjacent along the look-ahead direction to the third position where the fifth row of the second weight block is stored (e.g., the position where the fourth row of the first weight block is stored).

[0061] Figure 2J The sequence of computation cycles for two different scenarios is shown: Scenario 1 ("Scenario 1") and Scenario 2 ("Scenario 2"). In Scenario 1, inter-block preprocessing involves only block forward, while in Scenario 2, inter-block preprocessing involves both block forward and block look-ahead. Figure 2J As can be seen in this example, if block look-ahead is used, additional computation cycles can be skipped. Therefore, the number of cycles used for neural network operations can be reduced, thereby improving the operating efficiency of accelerators used for neural networks.

[0062] Figure 3Circuitry for performing tensor computations is shown in some embodiments. When the weights of the weight tensor are shifted to make period skipping possible, the activations are also shifted such that each activation is multiplied by the appropriate weight. As described above, weights can be shifted during preprocessing, for example, by processing circuitry (discussed in further detail below), and activations can be shifted during operation using a multiplexer, as shown. A group of the first multiplexer 305 is used to perform the shifting of activations, the shifting of activations corresponding to: (1) shifting weights by intra-block forward followed by inter-block forward, and (2) shifting weights by intra-block side-by-side followed by inter-block forward (this second part is supplied as input to the multiplier as explained in the following sentences). A group of the second multiplexer 310 is used to: (i) perform the selection of activations corresponding to intra-block forward followed by inter-block forward, and (ii) send the result to one of the input ports of the multiplexer 305. The output of each adder tree 315 is connected to a demultiplexer 320, which performs the following operations: (i) transmits the results of block look-ins (which are look-ins in irreducible dimensions) back to the block where the weights were originally located (before the weights were preprocessed), and (ii) sends other results to an accumulator 325. Control signals for the multiplexer are generated while performing weight preprocessing, and these control signals are fed back to each block from the corresponding control bit buffer 330, which runs in parallel with the weight buffer 110.

[0063] The ideas discussed above for the case of a three-dimensional weight tensor (one dimension is reducible and the other is irreducible) can be generalized to tensors with a larger number of dimensions. Figure 4A A general N-dimensional weight tensor with one or more reducible dimensions and zero or more irreducible dimensions is illustrated. Dimensions can be ordered from innermost to outermost (see nested loops that can be used to process tensors). Each spatial parallel hyperplane in the weight tensor is processed simultaneously within a computation cycle. Some embodiments reduce the number of computation cycles to accelerate processing by providing compression of the sparse weight tensor within the time axis. Figure 4B This is a table showing which axes are reducible and which are irreducible for two different computations (convolution and GEMV (FC) (General Matrix-Vector Multiplication)), where C, K, R, and S are the indices of the input channel, output channel, height, and width of the convolution, respectively. Figure 4C This is a flowchart illustrating the iterative process involved in preprocessing an N-dimensional tensor in both reducible and irreducible dimensions, starting with Hierarchical Weighted Processing (HWP) of the N-dimensional weighted tensor. Figure 4CIn this context, LWP denotes the "local weight preprocessing" of the two-dimensional weight tensor (which can be represented as a matrix) as the lowest level of weight preprocessing, involving both forward and side-looking operations. For example, in the three-dimensional case (i.e., N=3), this processing can be performed as follows. Figure 4C In step 1 (indicated by the circled "1"), the N-dimensional weight tensor (where, for example, N=3) is prepared to be processed using Hierarchical Weight Processing (HWP) to accelerate computation. In step 2, the three dimensions can be divided into three categories (e.g., 3 dimensions = 1 (time axis) + 2 (spatial axis) = 1 (time axis D)). Time )+1(Irreducible axis D) IS (Multiple blocks) + 1 (Reducible axis D) RS (Multiple multipliers in the processing element)). In step 3, the three-dimensional weight tensor can be considered as a one-dimensional (1 = N - 2 = 3 - 2) ordered set of two-dimensional tensors, each of which can be or can be similar to Figure 2A One of the two-dimensional weight blocks shown in the diagram. Figure 4C In the middle, D Inner-most The innermost spatial axis is indicated. In step 4, a first-level weight preprocessing (which in this case is intra-block preprocessing) can be performed by using forward and side-looking operations. In steps 5 and 6, each two-dimensional tensor of the ordered set can be summarized in a one-dimensional sparsity vector to generate an (N-1)-dimensional sparsity reduction map (SRM) (in the case of three-dimensional hierarchical weight processing, the two-dimensional SRM is a block sparsity map). The process can then return to step 2. The first-level preprocessing has been completed, the tensor of interest for the next iteration becomes a two-dimensional ((N-1)-dimensional) sparsity reduction map, and the process repeats the preprocessing steps to complete the second-level preprocessing, which is inter-block preprocessing. In the processing circuit including the array of processing element circuits 105, the spatial parallel dimension can be three, and the weight tensor dimension can be four (1 time + 3 spatial). In this case, it is possible to Figure 4C Perform three iterations to produce, as follows Figure 2I The synchronous four-dimensional tensor is shown in the figure. Figure 4C The three-dimensional example can be extended to higher dimensions (such as four or five dimensions). Figure 4D It shows the way Figure 4C The method shown in the figure reduces the cycle time. Figure 4E yes Figure 4C A more detailed flowchart of the method is shown in the figure. Figure 4F This is a table showing the (processing and overhead) costs of the hardware used to perform preprocessing along the reducible and irreducible axes, where C0 and K0 correspond to the input and output dimensions, respectively. (h i d i h' i ,d'i The elements of ) are forward relative to the reducible axis, side view relative to the reducible axis, forward relative to the irreducible axis, and side view relative to the irreducible axis.

[0064] Figure 4G A block diagram illustrating the hardware for reducible and irreducible dimensions is shown. In the reducible dimension, activation rearrangement can be performed in two stages to match the weight rearrangement performed during preprocessing. The activation initiator (AP) can perform preprocessing with inter-block processing (e.g., in...). Figures 2G to 2J In the example, the repositioning of the activations (moving one row of weight at a time) is performed, and the multiplexer 405 can perform preprocessing within the block (e.g., in...). Figure 2A and Figure 2B In the example, the repositioning of activations corresponds to moving one weight at a time. In the irreducible dimension, to be consistent with the weight repositioning performed in preprocessing, the activation repositioning may only involve the first (activation initiator) phase, and after the multiplication is performed, the demultiplexer 410 can be used to transfer the result of the block look-in (which is the look-in in the irreducible dimension) back to the block where the weight originally resided. Figure 4G In the irreducible case, similar to the activation broadcasting of multiple blocks that can be used in 3D hierarchical weighting, it can be broadcast to multiple internal levels of HW. i-1 To share the activation register. Figure 4G In the middle, A i It is for HW i The number of activation cycles supplied (equal to the number of activation registers). Figure 4H This is a schematic block diagram illustrating the operation of activating the launcher, which can be started from A. i Select consecutive A cycles from the activation cycles. (i-1) Each activation cycle. This operation can be similar to the operation of a sliding window. Figure 4I An implementation of activating the initiator is shown in some embodiments, wherein a multiplexer 415 with two inputs is used to support one computation cycle forward between blocks (as shown in...). Figure 3 As in the example, where the second multiplexer 310 corresponds to the activation initiator. In other embodiments, the multiplexer may have more inputs, and a larger number of inter-block forward computation cycles can be supported. Figure 4I In the middle, WS is through HW i All active initiators within the unit share a multiplexed (muxed) signal. Figure 4I In this context, cyc can indicate the period, from act0 to act. Ai-1) It can indicate the activation of the output from the multiplexer 415.

[0065] As used herein, “part” of something means “at least some” of that thing, and therefore can mean less than or all of that thing. Thus, “part” of something includes the whole thing as a special case (i.e., an example where the whole thing is a part of something).

[0066] The term "processing circuitry" is used herein to refer to any combination of hardware, firmware, and software for processing data or digital signals. Processing circuitry hardware may include, for example, application-specific integrated circuits (ASICs), general-purpose or special-purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices (such as field-programmable gate arrays (FPGAs)). In processing circuitry, as used herein, each function is performed by hardware configured (i.e., hardwired) to perform said function, or by more general-purpose hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. Processing circuitry may be fabricated on a single printed circuit board (PCB) or distributed across several interconnected PCBs. Processing circuitry may include other processing circuitry; for example, processing circuitry may include two processing circuits, an FPGA, and a CPU interconnected on a PCB.

[0067] As used herein, the term "array" refers to an ordered set of numbers, regardless of how they are stored (e.g., whether they are stored in contiguous memory locations or in a linked list). As used herein, the term "rectangle" includes, as a special case, a square (i.e., a square is an example of a rectangle). As used herein, when a second number is "within" "Y%" of a first number, this means that the second number is at least (1-Y / 100) times the first number and at most (1+Y / 100) times the first number. As used herein, the term "or" should be interpreted as "and / or," such that, for example, "A or B" means any one of "A," "B," and "A and B."

[0068] As used herein, when a method (e.g., adjustment) or a first quantity (e.g., a first variable) is referred to as “based on” a second quantity (e.g., a second variable), it means that the second quantity is an input to the method or affects the first quantity (e.g., the second quantity may be an input to a function that calculates the first quantity (e.g., a unique input or one of some inputs), or the first quantity may be equal to the second quantity, or the first quantity may be the same as the second quantity (e.g., stored in the same one or more locations in memory)).

[0069] It will be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. Unless the context otherwise indicates, these terms are used only to distinguish one element, component, region, layer, or portion from another. Therefore, without departing from the spirit and scope of the inventive concept, the first element, component, region, layer, or portion discussed herein may be referred to as the second element, component, region, layer, or portion. Examples further indicated by the context include phrases such as “the column in the active buffer is second” or “the row in the matrix is ​​first.”

[0070] As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of…” following a list of elements modify the entire list of elements, not individual elements in the list. Furthermore, the use of “may” when describing embodiments of the inventive concept refers to “one or more embodiments of this disclosure.” Additionally, the term “exemplary” is intended to indicate an example or illustration. As used herein, the terms “use,” “being used,” and “being exploited” may be considered synonymous with the terms “utilizing,” “being exploited,” and “being exploited,” respectively.

[0071] It will be understood that when an element or layer is referred to as being "on" another element or layer, "connected to", "bonded to", or "adjacent to" another element or layer, the element or layer may be directly on, connected to, bonded to, or adjacent to the other element or layer, or one or more intermediate elements or intermediate layers may be present. Conversely, when an element or layer is referred to as being "directly on" another element or layer, "directly connected to", "directly bonded to", or "immediately adjacent to" another element or layer, no intermediate elements or intermediate layers are present.

[0072] Any numerical range described herein is intended to include all subranges of the same numerical precision contained within the described range. For example, the range “1.0 to 10.0” or “between 1.0 and 10.0” is intended to include all subranges between the described minimum value 1.0 and the described maximum value 10.0 (and includes both the described minimum value 1.0 and the described maximum value 10.0) (i.e., all subranges with a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as 2.4 to 7.6). Any maximum numerical limit described herein is intended to include all lower numerical limits contained therein, and any minimum numerical limit described in this specification is intended to include all higher numerical limits contained therein.

[0073] Although exemplary embodiments of systems and methods for hierarchical weight preprocessing for neural network accelerators have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it will be understood that systems and methods for hierarchical weight preprocessing for neural network accelerators constructed according to the principles of this disclosure may be implemented in ways other than those specifically described herein. The invention is also defined by the appended claims and their equivalents.

Claims

1. A method of operating an accelerator, comprising: The weight buffer is accessed through the first processing circuit to perform intra-block preprocessing of the first weight tensor in the weight buffer, thereby forming a first preprocessed weight tensor in the weight buffer. The weight buffer is accessed through the first processing circuit to perform inter-block preprocessing of the first preprocessed weight tensor, thereby forming a second preprocessed weight tensor in the weight buffer. The preprocessing within the block includes: storing the first element of the first weight block of the first weight tensor at the second position of the weight buffer; moving the first element of the first weight block of the first weight tensor one position along the forward direction or along the side-view direction; the second position being a position adjacent to the first position where the first element is stored along the forward direction or along the side-view direction. Inter-block preprocessing includes: storing the first row of the first weight block of the first preprocessed weight tensor at the fourth position of the weight buffer; and shifting the first row of the first weight block of the first preprocessed weight tensor one position forward or one position sideways. The fourth position is adjacent to the third position where the first row is stored, either forward or sideways. The forward direction indicates the direction along which the first element moves to the zero element when there is a zero element in the column where the first element is located in the first weighted block, and the side-looking direction indicates the direction along which the first element moves to the zero element when there is no zero element in the column where the first element is located but there is a zero element in the column adjacent to the first element.

2. The operating method as described in claim 1, wherein, The preprocessing within the block also includes: within the second weight block of the first weight tensor, shifting the first element of the second weight block one position along the side-view direction.

3. The operating method as described in claim 1 or 2, wherein: The first line is the second weight block of the first preprocessed weight tensor. Inter-block preprocessing includes: moving the first row from the second weighted block to the first weighted block along the side-view direction.

4. The operating method as described in claim 3, wherein, Inter-block preprocessing also includes: creating a block sparsity graph corresponding to the weight tensor after the first preprocessing, the block sparsity graph having: Columns, for each weight block of the first preprocessed weight tensor, and For each row in the weighted block, The block sparsity graph indicates the location of empty lines in the weight blocks of the first preprocessed weight tensor.

5. The operating method as described in claim 4, wherein, The block sparsity graph has one less dimension than the first preprocessed weight tensor.

6. The operating method as described in claim 4, further comprising: The first row is identified based on the block sparsity graph.

7. The operating method as described in claim 4, further comprising: Multiply the first row by the first activation vector to form the first dot product. The multiplication step includes extracting the first activation vector from the column corresponding to the first row of the activation buffer.

8. The operating method as described in claim 4, further comprising: In the first processing element circuit of the second processing circuit of the accelerator, the first row is multiplied by the first activation vector to form a first dot product. In the second processing element circuit of the second processing circuit of the accelerator, the second row of the weight block of the first preprocessed weight tensor is multiplied by the second activation vector to form a second dot product, and Add the first dot product and the second dot product together.

9. The operating method as described in claim 4, wherein, Inter-block preprocessing also includes shifting the second row of the first preprocessed weight tensor by one position in the forward direction.

10. The operating method as described in claim 9, further comprising: The second row is identified based on the block sparsity graph.

11. An accelerator, comprising: The first processing circuit is configured as follows: Access the weight buffer to perform intra-block preprocessing of the first weight tensor in the weight buffer, to form the first preprocessed weight tensor in the weight buffer, and Access the weight buffer to perform inter-block preprocessing of the first preprocessed weight tensor, thereby forming a second preprocessed weight tensor in the weight buffer. The preprocessing within the block includes: storing the first element of the first weight block of the first weight tensor at the second position of the weight buffer; moving the first element of the first weight block of the first weight tensor one position along the forward direction or along the side-view direction; the second position being a position adjacent to the first position where the first element is stored along the forward direction or along the side-view direction. Inter-block preprocessing includes: storing the first row of the first weight block of the first preprocessed weight tensor at the fourth position of the weight buffer; and shifting the first row of the first weight block of the first preprocessed weight tensor one position forward or one position sideways. The fourth position is adjacent to the third position where the first row is stored, either forward or sideways. The forward direction indicates the direction along which the first element moves to the zero element when there is a zero element in the column where the first element is located in the first weighted block, and the side-looking direction indicates the direction along which the first element moves to the zero element when there is no zero element in the column where the first element is located but there is a zero element in the column adjacent to the first element.

12. The accelerator of claim 11, wherein, The preprocessing within the block also includes: within the second weight block of the first weight tensor, shifting the first element of the second weight block one position along the side-view direction.

13. The accelerator as claimed in claim 11 or 12, wherein: The first line is the second weight block of the first preprocessed weight tensor. Inter-block preprocessing includes: moving the first row from the second weighted block to the first weighted block along the side-view direction.

14. The accelerator of claim 13, wherein, Inter-block preprocessing also includes: creating a block sparsity graph corresponding to the weight tensor after the first preprocessing, the block sparsity graph having: Columns, for each weight block of the first preprocessed weight tensor, and For each row in the weighted block, The block sparsity graph indicates the location of empty lines in the weight blocks of the first preprocessed weight tensor.

15. The accelerator of claim 14, wherein, The block sparsity graph has one less dimension than the first preprocessed weight tensor.

16. The accelerator of claim 14, wherein, The first processing circuit is configured to identify the first row based on the block sparsity map.

17. The accelerator of claim 14, further comprising: The second processing circuit includes: The first processing element circuit, and Second processing element circuit, in: The first processing element circuit is configured to multiply the first row by the first activation vector to form a first dot product; and The second processing element circuit is configured as follows: Multiply the second row of the weight block of the first preprocessed weight tensor with the second activation vector to form the second dot product, and Add the first dot product and the second dot product together.

18. The accelerator of claim 14, wherein, Inter-block preprocessing also includes shifting the second row of the first preprocessed weight tensor by one position in the forward direction.

19. The accelerator of claim 18, wherein, The first processing circuit is configured to identify the second row based on the block sparsity map.

20. An accelerator, comprising: The processing apparatus is configured as follows: Access the weight buffer to perform intra-block preprocessing of the first weight tensor in the weight buffer to form a first preprocessed weight tensor in the weight buffer, and Access the weight buffer to perform inter-block preprocessing of the first preprocessed weight tensor, thereby forming a second preprocessed weight tensor in the weight buffer. The preprocessing within the block includes: storing the first element of the first weight block of the first weight tensor at the second position of the weight buffer; moving the first element of the first weight block of the first weight tensor one position along the forward direction or along the side-view direction; the second position being a position adjacent to the first position where the first element is stored along the forward direction or along the side-view direction. Inter-block preprocessing includes: storing the first row of the first weight block of the first preprocessed weight tensor at the fourth position of the weight buffer; and shifting the first row of the first weight block of the first preprocessed weight tensor one position forward or one position sideways. The fourth position is adjacent to the third position where the first row is stored, either forward or sideways. The forward direction indicates the direction along which the first element moves to the zero element when there is a zero element in the column where the first element is located in the first weighted block, and the side-looking direction indicates the direction along which the first element moves to the zero element when there is no zero element in the column where the first element is located but there is a zero element in the column adjacent to the first element.

Images