An approximate multiply-accumulate operation device with shift compensation

Abstract

The application provides a multiply-accumulate approximation operation device with shift compensation, comprising an original weight decoding module, an input feature map rearrangement network module and a rearrangement index shift convolution RiSC algorithm operation module, the original weight decoding module is used for decoding input original weight data to generate base weight signals, weight deviation signals and exchange network control signals, the input feature map rearrangement network module is used for rearranging input original input feature map sequences according to the exchange network control signals generated by the original weight decoding module, and outputting rearranged feature map sequences, and the rearrangement index shift convolution RiSC algorithm operation module is used for splitting the input rearranged feature map sequences into multiple groups of subsequences and performing RiSC operations on the subsequences respectively, generating calculation results according to the RiSC algorithm and outputting the calculation results. The application can reduce the multiplication operation amount in the multiply-accumulate operation in the deep neural network and the model weight storage amount.

Description

Technical Field

[0001] This invention relates to the field of digital circuit design technology, and more specifically, to a multiply-accumulate approximation arithmetic device employing shift compensation. Background Technology

[0002] Deep learning is widely used in edge scenarios (such as face recognition, vehicle recognition, etc.). The complexity of deployment scenarios drives more complex model structures, which in turn leads to a larger amount of model weight storage. This has an adverse effect on edge devices with limited storage capacity (such as mobile phones and microcontrollers).

[0003] Although deep learning uses deep neural networks, which are black-box models, observing the distribution patterns of weights in classic deep neural network models (such as AlexNet and ResNet) reveals that the weight distribution resembles a Gaussian distribution. After a specific reordering of the model weights, it can be observed that the deviation of each element within a local area of ​​the model weight is almost zero; this deviation can be represented by a power of 2.

[0004] It is evident that the multiplication operations in the multiply-accumulate operation and the storage requirements for model weights in existing deep neural networks are very large. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention discloses a multiply-accumulate approximation operation device employing shift compensation, comprising an original weight decoding module, an input feature map rearrangement network module, and a rearranged index shift convolution RiSC algorithm operation module, characterized in that...

[0006] The raw weight decoding module is used to decode the input raw weight data to generate base weight signals, weight bias signals, and switching network control signals.

[0007] The input feature map rearrangement network module is used to rearrange the original input feature map sequence according to the exchange network control signal generated by the original weight decoding module, and output the rearranged feature map sequence.

[0008] The rearranged index shift convolution (RiSC) algorithm operation module is used to split the input rearranged feature map sequence into multiple sub-sequences and perform RiSC operations on each sub-sequence, generate calculation results according to the RiSC algorithm, and output them.

[0009] This invention can reduce the amount of multiplication operations and model weight storage in the multiply-accumulate operation of deep neural networks. At the same time, due to the redundancy of the neural network model weights, the neural network has a certain fault tolerance and error correction capability for fine-tuning the model weights. While reducing the number of model weight parameters, the recognition accuracy remains almost unchanged, making this device suitable for a wide range of applications. Compared with a MAC unit that uses multipliers and adders to achieve the same function, this multiply-accumulate approximation operation device with shift compensation has the following advantages: (1) It replaces some multiplication operations with shift operations, reducing the hardware implementation area. (2) It compresses the weight storage, reducing the actual bit width required for the weight memory. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a multiply-accumulate approximation arithmetic device with shift compensation provided in one embodiment of the present invention;

[0011] Figure 2 This is a structural diagram of the original weight decoding module risc_decode provided in one embodiment of this disclosure;

[0012] Figure 3 This is a structural diagram of the input feature map rearrangement network module risc_bar provided in one embodiment of this disclosure;

[0013] Figure 4 This is a structural diagram of the RiSC algorithm operation module risc_pe provided in one embodiment of this disclosure;

[0014] Figure 5 This is a diagram illustrating an example of the RiSC algorithm computation module risc_pe subsequence computation pipeline provided in one embodiment of this disclosure;

[0015] Figure 6 This is a second example of the RiSC algorithm operation module risc_pe subsequence calculation pipeline provided in one embodiment of this disclosure;

[0016] Figure 7 This is a RiSC 8x8 matrix-vector multiplication topology diagram provided in one embodiment of this disclosure;

[0017] Figure 8 This is a RiSC 8x8 matrix multiplication topology diagram provided in one embodiment of this disclosure;

[0018] Figure 9 This is a transformation graph of the input sequence and output sequence of the risc_bar input feature map rearrangement network module provided in one embodiment of this disclosure. Detailed Implementation

[0019] The following is in conjunction with the appendix Figures 1 to 9The embodiments of the present invention will be described in detail below. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Those skilled in the art should understand that the following specific embodiments or implementation methods are a series of optimized configurations listed to further explain the specific content of the invention. These configurations can be combined or used in conjunction with each other, unless the invention explicitly states that some or a specific embodiment or implementation method cannot be associated with or used in conjunction with other embodiments or implementation methods. Furthermore, the following specific embodiments or implementation methods are merely optimized configurations and are not intended to limit the scope of protection of the invention.

[0021] See Figure 1 In one embodiment, the present invention discloses a multiply-accumulate approximation operation device with shift compensation, comprising an original weight decoding module, an input feature map rearrangement network module, and a Reorderindex Shift Convolution (RiSC) algorithm operation module, characterized in that,

[0022] The raw weight decoding module is used to decode the input raw weight data to generate base weight signals, weight bias signals, and switching network control signals.

[0023] The input feature map rearrangement network module is used to rearrange the original input feature map sequence according to the exchange network control signal generated by the original weight decoding module, and output the rearranged feature map sequence.

[0024] The rearranged index shift convolution (RiSC) algorithm operation module is used to split the input rearranged feature map sequence into multiple sub-sequences and perform RiSC operations on each sub-sequence, generate calculation results according to the RiSC algorithm, and output them.

[0025] In this embodiment, in practical applications, a set of sequences will be split into multiple subsequences that satisfy the above-mentioned operational rules, for example in... The sequence is divided into 4 groups. The algorithm performs RiSC operations in parallel on each subsequence. Since the number of shifters in a CPU is insufficient, and the conditional checks introduced in the proc operation of RiSC are not well-suited for GPGPUs, a dedicated hardware computing device needs to be designed for this algorithm.

[0026] The present invention has extremely high scalability. This is mainly reflected in the following points: (1) The RiSC algorithm can be applied to any multiplication and addition operation with similar patterns between data. (2) The scale of the RiSC hardware computing unit can be arbitrarily expanded. (3) The basic components of RiSC can be interconnected to realize a computing array of arbitrary size.

[0027] The application scope of this invention is wide, mainly in the following aspects: (1) RiSC can be applied to vector inner product operations, using the RiSC algorithm and hardware acceleration for vector inner product operations. (2) RiSC can be applied to matrix-vector multiplication operations, dividing matrix-vector multiplication operations into block matrix-vector multiplication operations, using the RiSC algorithm and hardware acceleration for block matrix-vector multiplication operations. (3) RiSC can be applied to matrix multiplication operations, dividing matrix multiplication operations into block matrix multiplication operations, using the RiSC algorithm and hardware acceleration for block matrix multiplication operations. (4) RiSC can be applied to deep neural network algorithms, using the RiSC algorithm and hardware acceleration for multiplication-accumulation operations in convolution operations, which account for a large proportion of computation in deep neural network algorithms. (5) The RiSC algorithm can be applied to GPGPU algorithms to replace the original multiplication-accumulation operations.

[0028] RiSC's basic components include Figure 1 As shown, the approximate multiplication-accumulation calculation function is achieved through combination. Here, orig_wt represents the original data retrieved from the weight memory, and orig_fm represents the original input feature map retrieved from the input feature map memory.

[0029] The original weight decoding module risc_decode structure is as follows: Figure 2 As shown, it is responsible for processing the original weight binary input and decoding to generate the following three signals: base weight wt_base, weight bias wt_diff, and the network control signal bar_ctrl, which is the rearranged index of each element.

[0030] In another embodiment, the RiSC algorithm takes the following form:

[0031]

[0032]

[0033]

[0034]

[0035] Where S represents the result of multiplication and accumulation, and n represents the number of accumulation groups. Represents weight, Represents the input feature map, This represents the width of the encoded storage bits. This represents the deviation between weights. << represents a shift operation function, and << represents a left shift operation of binary bits. This represents a weighted recoding operation.

[0036] For this embodiment, consider the following fixed-point arithmetic expression:

[0037]

[0038] Where n represents the number of accumulated groups. Represents weight, S represents the input feature map, and S represents the multiplication and accumulation result.

[0039] Perform the following transformation to extract the sub-expressions of each term. Combined into one item:

[0040]

[0041]

[0042] Simplified to:

[0043]

[0044] make This indicates the deviation between the weights:

[0045]

[0046] It can be approximated as the size of the power of 2 closest to it, and then the multiplication operations of the terms other than the first term in the above formula are transformed into arithmetic left shift operations (except for...). situation):

[0047]

[0048] in:

[0049]

[0050] in Function representation The sign of a number is 1 for positive and -1 for negative.

[0051] To further simplify the calculation, let ,so All are non-negative numbers, for The function can be simplified as follows:

[0052]

[0053] right Perform the following encoding, assuming the storage width of this encoding is... ,but

[0054] encode ( di f i )={ 0, di f i =0 min( ( log 2 di f i +1),( 2 W dif -1)), di f i >0 encode ( di f i )∈[0, 2 W dif -1]

[0055] The function is transformed into:

[0056]

[0057] In the final form of the RiSC algorithm It can be generated offline. Compared to the original multiply-accumulate expression, the RiSC algorithm is equivalent to replacing a multiplication operation with an addition and an arithmetic left shift. The RiSC algorithm can also be understood from another perspective: the leftmost term of the expression is equivalent to an approximate operation of the whole multiply-accumulate, and the rightmost term is equivalent to using a shift operation to compensate for the approximate calculation result.

[0058] In practical computation, since it's difficult for most elements of the original weight sequence to satisfy a power of 2 relationship, a manual rearrangement operation is needed to construct multiple subsequences that satisfy the RiSC operation prerequisites from the original sequence. Because the rearranged weight sequences that satisfy the RiSC operation prerequisites do not correspond one-to-one with the original input feature map sequences, the same rearrangement operation needs to be performed online on the input activation sequence during computation. Since the rearranged weight sequences can be generated offline, the rearrangement index can also be generated offline. Software or hardware can then perform the rearrangement operation on the corresponding elements of the input feature map sequence based on the weight rearrangement index. This algorithm requires offline storage of each group of RiSC subsequences. and The value also needs to store the reorder index of each element.

[0059] When splitting an original sequence into multiple subsequences that satisfy the prerequisites for RiSC operations, let the length of the original sequence be... The subsequence length is The base weights (i.e., the first weight of each RiSC subsequence) have a bit width of 1. The weighted bias encoding bit width is The input feature map element bit width is Then the following relationship exists:

[0060] (1) Number of subsequences

[0061] (2) Number of weighted biases

[0062] (3) Rearrange the index encoding bit width

[0063] (4) RiSC weight storage width

[0064] The RiSC weighted rearrangement algorithm is implemented using a general sorting algorithm.

[0065] Scoring function: a weighted sum of the deviations of all elements except the basic element from the basic element and the deviations in their second power form.

[0066] Optimization objective: Find the highest-scoring permutation from all possible combinations of the original sequence.

[0067] Reorder the index encoding of each element in the generated original input feature map sequence: whether it is the first number in the subsequence + which subsequence it belongs to.

[0068] In another embodiment, the input feature map rearrangement network module includes multiple to Arbitrator with priority, where The length of the original input feature map sequence. is the length of the subsequence.

[0069] In this embodiment, since the positional relationships of elements other than the first element within a subsequence are not fixed, the rearrangement network can automatically sort the elements other than the first element based on this rule. The sorting relationship is determined by the relative size of the port numbers when these elements are input into the rearrangement network. Therefore, the actual rearrangement index encoding bit width for each element only needs to be... For example, for bits Only 2 bits are needed to generate the rearranged index for each element, which reduces the number of bits by 1 compared to the rearranged index using the swap switch scheme, effectively reducing the index storage amount.

[0070] The input feature map rearrangement network module risc_bar is responsible for rearranging the input feature map sequence according to the control signals generated by the decoding unit, and outputting the rearranged sequence. Multiple... to A structure for combining arbitrators with priority. The risc_bar structure is as follows: Figure 3 As shown, arbiter is an arbiter with priority.

[0071] In another embodiment, all arbitrators receive the same is_first signal, and the valid signal corresponding to each element in the original input feature map sequence when it is input into the arbitrator is generated based on whether the subsequence number indicated in its rearrangement index is the same as the arbitrator number.

[0072] In this embodiment, for example, an input feature map sequence [1,2,3,4] has a rearrangement index of [0,1,0,1], and this input feature map sequence broadcasts two 4-input 2-output arbitrators. For arbitrator 0, the valid signal for each element of the sequence is [1,0,1,0], and for arbitrator 1, the valid signal for each element of the sequence is [0,1,0,1]. 5. The apparatus according to claim 1, wherein the rearrangement network employs a Banyan network, a Benes network, or a cross switch.

[0073] In another embodiment, the rearrangement network uses a Banyan network, a Benes network, or a cross switch.

[0074] In another embodiment, the results of the operations on all subsequences in the RiSC algorithm operation module are summed by a multi-input adder to obtain the final result.

[0075] In this embodiment, the RiSC algorithm operation module splits the input sequence into multiple groups, performs RiSC operations on each group separately, and generates the calculation results according to the RiSC algorithm. The structure of the RiSC algorithm operation module risc_pe is as follows: Figure 4 As shown, the dashed box represents the operation within a RiSC subsequence, and the results of the operations of all subsequences are summed by a multi-input adder to obtain the final result.

[0076] When the subsequence length At that time, the risc_pe subsequence computation pipeline is as follows: Figure 5 As shown, it takes 3 clock cycles from the initiation of input data to the capture of the output result. The reduce operation corresponds to... Figure 4 The adder in the middle.

[0077] When the subsequence length At that time, the risc_pe subsequence computation pipeline is as follows: Figure 6 As shown, it takes 3 clock cycles from the initiation of input data to the capture of the output result. The reduce operation corresponds to... Figure 4 The adder in the middle.

[0078] In another embodiment, the original weight decoding module further includes an input clock signal and a reset signal.

[0079] In this embodiment, the control signals of the original weight decoding module risc_decode are shown in Table 1.

[0080]

[0081] Table 1

[0082] In another embodiment, the input feature map rearrangement network module further includes an input clock signal, a reset signal, and a switching network control signal.

[0083] In this embodiment, the control signals of the input feature map rearrangement network module are shown in Table 2.

[0084]

[0085] Table 2

[0086] In another embodiment, the RiSC algorithm operation module further includes an input clock signal, a reset signal, a base weight signal, and a weight bias signal.

[0087] In this embodiment, the control signals of the RiSC algorithm operation module are shown in Table 3.

[0088]

[0089] Table 3

[0090] In another embodiment, the control signals of the RiSC top layer are shown in Table 4.

[0091]

[0092] Table 4

[0093] In another embodiment, the RiSC 8-element vector inner product is implemented by performing an inner product of an 8-element vector (input feature map) and an 8-element vector (weights), generating an approximate multiplicative summation result. The topological connection structure is as follows: Figure 1 As shown, orig_fm contains 8 elements.

[0094] In another embodiment, the RiSC 8x8 matrix-vector multiplication operation is implemented. A matrix-vector multiplication is performed between an 8x8 matrix (input feature map) and an 8-element vector (weights) to generate an 8-element vector. The topological connection structure is as follows: Figure 7 As shown, the wt_base and wt_diff are the same for each risc_pe input and are transmitted in a broadcast manner.

[0095] In another embodiment, the RiSC 8x8 matrix multiplication operation is specifically implemented. Matrix multiplication of an 8x8 matrix (input feature map) and an 8x8 matrix (weights) is performed to generate an 8x8 matrix. The topological connection structure is as follows: Figure 8 As shown, the wt_base and wt_diff are the same for each column of risc_pe input, and are transmitted in a broadcast manner.

[0096] Some optimizations were made here, reducing the number of risc_bars to 8, and also changing the bit width of the input elements of the risc_bars to... ,in It is the number of rows in the array. Figure 8 Transformation of input and output sequences in risc_bar as follows Figure 9 As shown, assuming the original feature map matrix is ​​stored in row-major order, before inputting it into risc_bar, the feature map matrix is ​​split into 8 parts according to columns and input into the 8 input ports of risc_bar respectively. When recovering the feature map data to be input into risc_pe from the 8 output ports of risc_bar, it is reorganized into 8 groups according to the row direction and then input into the 8 risc_pe respectively.

[0097] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A multiply-accumulate approximation operation device with shift compensation, comprising an original weight decoding module, an input feature map rearrangement network module, and a rearranged index shift convolution RiSC algorithm operation module, characterized in that, The raw weight decoding module is used to decode the input raw weight data to generate base weight signals, weight bias signals, and switching network control signals. The input feature map rearrangement network module is used to rearrange the original input feature map sequence according to the exchange network control signal generated by the original weight decoding module, and output the rearranged feature map sequence. The rearranged index shift convolution (RiSC) algorithm operation module is used to split the input rearranged feature map sequence into multiple sub-sequences and perform RiSC operation on each sub-sequence, generate calculation results according to the RiSC algorithm and output them. in, The RiSC algorithm takes the following form: ， ， ， ， Where S represents the result of multiplication and accumulation, and n represents the number of accumulation groups. Represents weight, Represents the input feature map, Represents the width of the encoded storage bits. This represents the deviation between weights. << represents a shift operation function, and << represents a left shift operation of binary bits. This represents a weighted recoding operation; The input feature map rearrangement network module includes multiple to Arbitrator with priority, where The length of the original input feature map sequence. The length of the subsequence; All arbitrators receive the same is_first signal. The valid signal corresponding to each element in the input feature map sequence is generated based on whether the subsequence number indicated in its rearrangement index is the same as the arbitrator number.

2. The apparatus according to claim 1, wherein the rearrangement network employs a Banyan network, a Benes network, or a cross switch.

3. In the apparatus according to claim 1, the operation results of all subsequences in the rearranged index shift convolution RiSC algorithm operation module are summed by a multi-input adder to obtain the final result.

4. The apparatus according to claim 1, wherein the original weight decoding module further includes an input clock signal and a reset signal.

5. The apparatus according to claim 1, wherein the input feature map rearrangement network module further includes an input clock signal, a reset signal, and a switching network control signal.

6. The apparatus according to claim 1, wherein the rearranged index shift convolution RiSC algorithm operation module further includes an input clock signal, a reset signal, a base weight signal, and a weight bias signal.

Images