Multiply-accumulate approximate operation device adopting shift compensation

Abstract

The invention provides a multiply-accumulate approximation operation device adopting shift compensation, which comprises an original weight decoding module, an input feature map rearrangement network module and a rearrangement index shift convolution RiSC algorithm operation module, the original weight decoding module is used for decoding and generating a base weight signal, a weight deviation signal and a switching network control signal; the input feature map rearrangement network module is used for rearranging an input original input feature map sequence according to the switching network control signal generated by the original weight decoding module and outputting a rearranged feature map sequence; and the rearrangement index shift convolution RiSC algorithm operation module is used for splitting the input rearranged feature map sequence into a plurality of groups of subsequences, respectively carrying out RiSC operation, generating a calculation result according to the RiSC algorithm and outputting the calculation result. According to the method, the multiplication operation amount and the model weight storage amount in the multiply-accumulate operation in the deep neural network can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of digital circuit design, and in particular to a multiplication-accumulation approximate operation device using shift compensation. Background Art

[0002] Deep learning is widely used in edge scenarios (such as face recognition and vehicle recognition). The complexity of deployment scenarios drives more complex model structures and leads to larger storage of model weights, which is an adverse effect on edge devices with limited storage capacity (such as mobile phones and single-chip computers).

[0003] Although the deep neural network used in deep learning is a black box model, by observing and counting the distribution of weights of some classic deep neural network models (such as Alexnet, Resnet, etc.), it can be found that the weight distribution of deep neural networks is similar to Gaussian distribution. After a specific reordering of the model weights, it can be observed that the deviation of each element in the local model weight is almost zero, and this deviation can be expressed as a power of 2.

[0004] It can be seen that the amount of multiplication operations in the existing deep neural networks and the amount of model weight storage are very large. Summary of the invention

[0005] In order to solve the above-mentioned technical problems, the present invention discloses a multiplication-accumulation approximate operation device using shift compensation, including an original weight decoding module, an input feature map rearrangement network module and a rearrangement index shift convolution RiSC algorithm operation module, characterized in that:

[0006] The original weight decoding module is used to decode the input original weight data to generate a base weight signal, a weight deviation signal and a switching network control signal:

[0007] The input feature map rearrangement network module is used to rearrange the input original input feature map sequence according to the switching network control signal generated by the original weight decoding module, and output the rearranged feature map sequence;

[0008] The re-indexed shifted convolution RiSC algorithm operation module is used to split the input re-ordered feature map sequence into multiple groups of sub-sequences and perform RiSC operations on each of them, generate and output calculation results according to the RiSC algorithm.

[0009] The present invention can reduce the amount of multiplication in the multiplication-accumulation operation in the deep neural network and the amount of model weight storage. At the same time, due to the redundancy of the neural network model weight, the neural network has a certain fault tolerance and error correction capability for fine-tuning the model weight. While the model weight parameter amount is reduced, the recognition accuracy remains almost unchanged, which makes the device have a full range of application scenarios. Compared with the MAC unit with the same function implemented by multipliers and adders, the multiplication-accumulation approximate operation device using shift compensation has the following advantages: (1) Part of the multiplication operation is replaced by shift operation, reducing the hardware implementation area. (2) The weight storage amount is compressed, reducing the actual required bit width of the weight memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the structure of a multiplication-accumulation approximate operation device using shift compensation provided in one embodiment of the present invention;

[0011] Figure 2 is a structural diagram of an original weight decoding module risc_decode provided in one embodiment of the present disclosure;

[0012] Figure 3 It is a structural diagram of the input feature map rearrangement network module risc_bar provided in one embodiment of the present disclosure;

[0013] Figure 4 It is a structural diagram of the RiSC algorithm operation module risc_pe provided in one embodiment of the present disclosure;

[0014] Figure 5 This is a diagram showing an example of a calculation pipeline of a risc_pe subsequence of a RiSC algorithm operation module provided in one embodiment of the present disclosure;

[0015] Figure 6 It is a second diagram of an example of a calculation pipeline of a risc_pe subsequence of a RiSC algorithm operation module provided in one embodiment of the present disclosure;

[0016] Figure 7 is a RiSC 8x8 matrix-vector multiplication topology diagram provided in one embodiment of the present disclosure;

[0017] Figure 8 is a multiplication topology diagram of RiSC 8x8 matrix multiplication provided in one embodiment of the present disclosure;

[0018] Fig. 9 It is a conversion diagram of the input sequence and output sequence of the input feature map rearrangement network module risc_bar provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is combined with Figures 1 to 9The embodiments of the present invention are described in detail. It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Those skilled in the art should know that the following specific embodiments or specific implementations are a series of optimized settings listed by the present invention to further explain the specific content of the invention, and these settings can be combined or used in association with each other, unless the present invention clearly states that some or a specific embodiment or implementation cannot be associated or used together with other embodiments or implementations. At the same time, the following specific embodiments or implementations are only used as the most optimized settings, and are not to be understood as limiting the scope of protection of the present invention.

[0021] See also Figure 1 In one embodiment, the present invention discloses a multiplication-accumulation approximate operation device using shift compensation, comprising an original weight decoding module, an input feature map rearrangement network module and a reorder index shift convolution (Reorderindex Shift Convolution) RiSC algorithm operation module, characterized in that:

[0022] The original weight decoding module is used to decode the input original weight data to generate a base weight signal, a weight deviation signal and a switching network control signal:

[0023] The input feature map rearrangement network module is used to rearrange the input original input feature map sequence according to the switching network control signal generated by the original weight decoding module, and output the rearranged feature map sequence;

[0024] The re-indexed shifted convolution RiSC algorithm operation module is used to split the input re-ordered feature map sequence into multiple groups of sub-sequences and perform RiSC operations on each of them, generate and output calculation results according to the RiSC algorithm.

[0025] In this embodiment, in actual application, a set of sequences will be split into multiple subsequences that meet the above operation rules. For example, The sequence is divided into 4 groups Subsequences are processed and RiSC algorithm operations are performed in parallel for each subsequence. Since the number of shifters in the CPU is not sufficient and the conditional judgment introduced in the RiSC proc operation is not friendly to GPGPU, a dedicated hardware computing device needs to be designed for this algorithm.

[0026] The present invention has strong scalability. This is mainly reflected in the following points: (1) The RiSC algorithm can be applied to any multiplication and addition operation formula with similar rules between data. (2) The scale of RiSC hardware computing unit can be expanded arbitrarily. (3) RiSC basic components can be interconnected to realize computing arrays of any scale.

[0027] The present invention has a wide range of applications, mainly in the following aspects: (1) RiSC can be applied to vector inner product operations, using the RiSC algorithm and hardware to accelerate vector inner product operations. (2) RiSC can be applied to matrix-vector multiplication operations, dividing the matrix-vector multiplication operations into block matrix-vector multiplication operations, using the RiSC algorithm and hardware to accelerate the block matrix-vector multiplication operations. (3) RiSC can be applied to matrix multiplication operations, dividing the matrix multiplication operations into block matrix multiplication operations, using the RiSC algorithm and hardware to accelerate the block matrix multiplication operations. (4) RiSC can be applied to deep neural network algorithms, using the RiSC algorithm and hardware to accelerate the multiplication-accumulation operations in the convolution operations that account for a large proportion of the operations in the deep neural network algorithms. (5) The RiSC algorithm can be applied to GPGPU algorithms to replace the original multiplication-accumulation operations therein.

[0028] The basic components of RiSC are Figure 1 As shown, the approximate multiplication and accumulation calculation function is realized by combination. Where orig_wt represents the original data taken out from the weight memory, and orig_fm represents the original input feature map taken out from the input feature map memory.

[0029] The original weight decoding module risc_decode structure is as follows Figure 2 As shown, it is responsible for processing the original weight binary input and decoding to generate the following three signals: base weight wt_base, weight deviation wt_diff, and exchange network control signal bar_ctrl, that is, the rearrangement index of each element.

[0030] In another embodiment, the RiSC algorithm is of the form:

[0031]

[0032]

[0033]

[0034]

[0035] Among them, S represents the multiplication and accumulation result, n represents the number of accumulation groups, represents the weight, represents the input feature map, Represents the encoding storage bit width, represents the deviation between weights, represents the shift processing function, << represents the binary bit left shift operation, Represents a weight re-encoding operation.

[0036] For this embodiment, consider the following fixed-point operation:

[0037]

[0038] Where n represents the number of accumulated groups, represents the weight, represents the input feature map, and S represents the multiplication and accumulation result

[0039] Make the following transformation to extract the sub-formula of each term , combined into one item:

[0040]

[0041]

[0042] Simplified to:

[0043]

[0044] make , which represents the deviation between weights:

[0045]

[0046] It can be approximated as the power of 2 that is closest to it, and then the multiplication operations of the other terms except the first term in the above formula are converted into arithmetic left shift operations (except situation):

[0047]

[0048] in:

[0049]

[0050] in Function Representation The positive and negative values ​​are 1 and -1 respectively.

[0051] To further simplify the operation, ,so are all non-negative numbers. The function can be simplified as follows:

[0052]

[0053] right Do the following encoding, assuming that the encoding storage width is ,but

[0054] encode ( d f i )={ 0, d f i =0 min( ( log 2 d f i +1),( 2 W dif -1)), d f i >0 encode ( d f i )∈[0, 2 W dif -1]

[0055] The function transforms to:

[0056]

[0057] The final form of the RiSC algorithm It can be generated offline. Compared with the original multiplication-accumulation operation, the RiSC algorithm is equivalent to replacing a multiplication operation with an addition and an arithmetic left shift. The RiSC algorithm can also be understood from another perspective: the leftmost term of the formula is equivalent to the overall approximate operation of multiplication-accumulation, and the right-hand term is equivalent to using a shift operation to compensate for the approximate calculation result.

[0058] In actual operations, since it is difficult for most of the elements in the original weight sequence to satisfy the relationship that the difference is a power of 2, it is necessary to manually perform a rearrangement operation to construct multiple subsequences that meet the RiSC operation prerequisite from the original sequence. Since the weight sequence that meets the RiSC operation prerequisite after rearrangement does not correspond one-to-one to the original input feature map sequence, it is necessary to perform the same rearrangement operation on the input excitation sequence online during the operation. Since the rearranged weight sequence can be generated offline, the rearrangement index can also be generated offline. The software or hardware can perform a rearrangement operation on the elements of the corresponding input feature map sequence based on the weight rearrangement index. This algorithm requires offline storage of each group of RiSC subsequences and The value of , also needs to store the reordering index of each element.

[0059] When an original sequence is split into multiple subsequences that meet the RiSC operation prerequisite, the length of the original sequence is , the subsequence length is , the base weight (i.e. the first weight of each RiSC subsequence) has a bit width of , the weight bias encoding bit width is , the input feature map element width is , then the following relationship exists:

[0060] (1) Number of subsequences

[0061] (2) Number of weight deviations

[0062] (3) Rearrange index encoding bit width

[0063] (4) RiSC weight storage bit width

[0064] The RiSC weight re-ranking algorithm is implemented using a general sorting algorithm.

[0065] Scoring function: the weighted sum of the deviations of the elements other than the base element and the deviations in the form of quadratic powers.

[0066] Optimization goal: Find the highest scoring permutation from all possible permutations of the original sequence.

[0067] Generate the reordered index encoding of each element of the original input feature map sequence: whether it is the first number in the subsequence + which subsequence it belongs to.

[0068] In another embodiment, the input feature map rearrangement network module includes multiple to With priority arbiter, is the original input feature map sequence length, is the subsequence length.

[0069] In this embodiment, since the positional relationship of the elements except the first element in the subsequence is not fixed, the rearrangement network can automatically sort the elements except the first element according to this rule, and the sorting relationship is based on the relative size of the port numbers when these elements are input into the rearrangement network. Therefore, in fact, the rearrangement index encoding bit width of each element only needs to be For example, for , only 2 bits are needed to generate the reordering index of each element, which is 1 bit less than the reordering index using the swap switch scheme, effectively reducing the index storage amount.

[0070] The input feature map rearrangement network module risc_bar is responsible for rearranging the input feature map sequence according to the control signal generated by the decoding unit and outputting the rearranged sequence. to The structure with priority arbiter combination. The risc_bar structure is as follows Figure 3 As shown, arbiter is an arbiter with priority.

[0071] In another embodiment, all arbitrators receive the same is_first signal, and a corresponding valid signal is generated when each element in the original input feature map sequence is input into the arbitrator based on whether the subsequence number indicated in its reordering index is the same as the arbitrator number.

[0072] In this embodiment, for example, there is an input feature map sequence [1, 2, 3, 4] whose rearrangement index is [0, 1, 0, 1]. The input feature map sequence is broadcast input to two 4-input 2-output arbiters. For arbiter 0, the valid signal of each element of the sequence is [1, 0, 1, 0], and for arbiter 1, the valid signal of each element of the sequence is [0, 1, 0, 1]. 5. According to the apparatus of claim 1, the rearrangement network adopts a banyan network, a benes network or a crossbar switch.

[0073] In another embodiment, the rearrangement network is implemented as a banyan network, a benes network or a crossbar switch.

[0074] In another embodiment, the operation results of all subsequences in the RiSC algorithm operation module are summed up through a multi-input adder to obtain a final result.

[0075] In this embodiment, the RiSC algorithm operation module divides the input sequence into multiple groups and performs RiSC operations on each group, and generates calculation results according to the RiSC algorithm. The structure of the RiSC algorithm operation module risc_pe is as follows: Figure 4 As shown, the dotted box represents the operation within a RiSC subsequence, and then the operation results of all subsequences are summed up through a multi-input adder to obtain the final result.

[0076] When the subsequence length When risc_pe subsequence calculation pipeline is as follows Figure 5 As shown in Figure 1, it takes three clock cycles from the time the input data is initiated to the time the output result is captured. The reduce operation corresponds to Figure 4 The adder in .

[0077] When the subsequence length When risc_pe subsequence calculation pipeline is as follows Figure 6 As shown in Figure 1, it takes three clock cycles from the time the input data is initiated to the time the output result is captured. The reduce operation corresponds to Figure 4 The adder in .

[0078] In another embodiment, the original weight decoding module further includes an input clock signal and a reset signal.

[0079] For this embodiment, the control signal of the original weight decoding module risc_decode is shown in Table 1.

[0080]

[0081] Table 1

[0082] In another embodiment, the input feature map rearrangement network module also includes an input clock signal, a reset signal and a switching network control signal.

[0083] For this embodiment, the control signal of the input feature map rearrangement network module is shown in Table 2.

[0084]

[0085] Table 2

[0086] In another embodiment, the RiSC algorithm operation module further includes an input clock signal, a reset signal, a base weight signal and a weight deviation signal.

[0087] For this embodiment, the control signal of the RiSC algorithm operation module is shown in Table 3.

[0088]

[0089] Table 3

[0090] In another embodiment, the control signals of the top layer of RiSC are shown in Table 4.

[0091]

[0092] Table 4

[0093] In another embodiment, RiSC implements the inner product of an 8-element vector. The inner product of an 8-element vector (input feature map) and an 8-element vector (weight) is performed to generate an approximate multiplication-accumulation result. The topological connection structure is as follows: Figure 1 As shown, orig_fm contains 8 elements.

[0094] In another embodiment, RiSC implements 8 x 8 matrix-vector multiplication. Perform matrix-vector multiplication of 8 x 8 matrix (input feature map) and 8-element vector (weight) to generate 8-element vector. The topological connection structure is as follows Figure 7 As shown, the wt_base and wt_diff for each risc_pe input are the same and are transmitted in broadcast form.

[0095] In another embodiment, RiSC implements 8 x 8 matrix multiplication operation. Perform matrix multiplication of 8 x 8 matrix (input feature map) and 8 x 8 matrix (weight) to generate 8 x 8 matrix. The topological connection structure is as follows Figure 8 As shown, the wt_base and wt_diff of each column of risc_pe input are the same and are transmitted in broadcast form.

[0096] Some optimizations are done here to reduce the number of risc_bars to 8 and make the input element width of risc_bar ,in is the number of array rows. Figure 8 The risc_bar input sequence and output sequence conversion are as follows Fig. 9 As shown, assuming that the original feature map matrix is ​​stored in row priority, the feature map matrix is ​​split into 8 parts by column before inputting into risc_bar and respectively input into the 8 input ports of risc_bar. When the feature map data to be input into risc_pe is recovered from the 8 output ports of risc_bar, it is reorganized into 8 groups in the row direction and then respectively input into 8 risc_pes.

[0097] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A multiplication-accumulation approximate operation device using shift compensation, comprising an original weight decoding module, an input feature map rearrangement network module and a rearrangement index shift convolution RiSC algorithm operation module, characterized in that: The original weight decoding module is used to decode the input original weight data to generate a base weight signal, a weight deviation signal and a switching network control signal: The input feature map rearrangement network module is used to rearrange the input original input feature map sequence according to the switching network control signal generated by the original weight decoding module, and output the rearranged feature map sequence; The re-indexed shifted convolution RiSC algorithm operation module is used to split the input re-ordered feature map sequence into multiple groups of sub-sequences and perform RiSC operations on each of them, generate and output calculation results according to the RiSC algorithm.

2. The device according to claim 1, preferably, the RiSC algorithm is in the following form: ， ， ， ， in, S represents the result of multiplication and accumulation, n represents the number of accumulation groups, represents the weight, represents the input feature map, Represents the encoding storage bit width, represents the deviation between weights, represents the shift processing function, << represents the binary bit left shift operation, Represents a weight re-encoding operation.

3. The device according to claim 1, wherein the input feature map rearrangement network module comprises a plurality of to With priority arbiter, is the original input feature map sequence length, is the subsequence length.

4. According to the device of claim 3, the is_first signal received by all arbitrators is the same, and the corresponding valid signal of each element in the input feature graph sequence when it is input into the arbitrator is generated according to whether the subsequence number indicated in its reordering index is the same as the arbitrator number. For example, the reordering index of an input feature graph sequence [1,2,3,4] is [0,1,0,1]. The input feature graph sequence is broadcast input into two 4-input 2-output arbiters. For arbitrator 0, the valid signal of each element of the sequence is [1,0,1,0], and for arbitrator 1, the valid signal of each element of the sequence is [0,1,0,1].

5. The device according to claim 1, wherein the rearrangement network adopts a banyan network, a benes network or a cross switch.

6. The device according to claim 1, wherein the operation results of all subsequences in the re-indexed shifted convolution (RiSC) algorithm operation module are summed up through a multi-input adder to obtain a final result.

7. The device according to claim 1, wherein the original weight decoding module further comprises an input clock signal and a reset signal.

8. According to the device according to claim 1, the input feature map rearrangement network module also includes an input clock signal, a reset signal and a switching network control signal.

9. The device according to claim 1, wherein the re-indexed shifted convolution (RiSC) algorithm operation module further comprises an input clock signal, a reset signal, a base weight signal and a weight deviation signal.

Images