Quantization method and system based on in-memory computing circuit architecture

Abstract

The present invention provides a quantization method and system based on an in-memory computing circuit architecture, wherein a parameter grouping step groups quantization weights into group quantization weights according to grouping values, and groups input excitation functions into group excitation functions according to grouping values. A multiplication-accumulation step performs a multiplication-accumulation operation on the grouped quantization weights and the grouped excitation functions to generate a convolution output. A convolution quantization step quantizes the convolution output into a quantized convolution output according to a convolution target bit. A convolution merging step performs a partial sum operation on the quantized convolution output according to the grouping values ​​to generate an output excitation function. Therefore, by grouping pairing and considering hardware limitations, and by combining the classification distribution of analog-to-digital converters and the robust properties of specific quantization methods with deep neural networks, better weight parameters can be learned.

Description

Technical Field

[0001] The present invention relates to a quantization method and a system thereof, and in particular to a quantization method and a system thereof based on a computing circuit architecture in a memory. Background Art

[0002] In the prior art, quantization methods are all aimed at quantizing the input activation function and weights of the neural network, and realizing low-bit neural networks through digital circuits. In recent years, in-memory computing has been proposed, and it is better than digital circuits in terms of energy efficiency. However, when the existing quantization methods are applied to in-memory computing, the weight parameters obtained by neural network training cannot effectively improve the accuracy. From the above, it can be seen that there is currently a lack of a quantization method and system based on the in-memory computing circuit architecture that can effectively improve the accuracy on the market, so relevant industry players are all seeking solutions. Summary of the invention

[0003] Therefore, the purpose of the present invention is to provide a quantization method and system based on the in-memory computing circuit architecture, which decomposes the input activation function and weight of the neural network to simulate the characteristics of the word line and bit line of the in-memory computing circuit, and quantizes the output result after the group convolution to simulate the limitation of the analog-to-digital converter, and finally completes the typical convolution operation with partial sum. The above quantization combined with the weight update of the neural network can improve the accuracy.

[0004] According to an embodiment of the method of the present invention, a quantization method based on an in-memory computing circuit architecture is provided, which includes a quantization parameter providing step, a parameter grouping step, a multiplication and accumulation step, a convolution quantization step, and a convolution merging step. The quantization parameter providing step is to provide a quantization parameter, and the quantization parameter includes an input excitation function, a quantization weight, and a grouping value. The parameter grouping step is to divide the quantization weight into a plurality of grouping quantization weights according to the grouping value, and to divide the input excitation function into a plurality of grouping excitation functions according to the grouping value. The multiplication and accumulation step is to perform a multiplication and accumulation operation on any grouping quantization weight and any grouping excitation function to generate a convolution output. The convolution quantization step is to quantize the convolution output into a quantized convolution output according to a convolution target bit. The convolution merging step is to perform a part of the sum operation on the quantized convolution output according to the grouping value to generate an output excitation function. The input excitation function may have an input channel number, and the grouping value is obtained by calculating the input channel number and a control parameter, and the grouping value is equal to the input channel number divided by the control parameter. The grouping value is a positive integer, and is greater than 1 and less than or equal to the input channel number. The convolution quantization step includes performing a direct estimation quantization method or a specific quantization method.

[0005] Therefore, the quantization method of the present invention can store the grouped quantization weights in advance in the memory unit, and achieve the convolution operation function in the memory through a specific in-memory operation circuit architecture, which can avoid the problem of large amounts of data transfer between hierarchical memories in the traditional architecture, thereby reducing system power consumption, reducing system latency, and improving energy efficiency. In addition, the present invention groups the quantization weights and the input excitation function, and considers the discretization of the memory unit, word line input, and convolution output of the finite bit line, and learns the weight parameters suitable for the hardware limitations of the operation in the non-volatile memory through the robust properties of the deep neural network.

[0006] Other embodiments of the aforementioned implementation are as follows: The aforementioned quantization method based on the computing circuit architecture in the memory may also include an iterative execution step, which repeats the multiplication and accumulation step, the convolution quantization step and the convolution merging step according to an iteration number, and the iteration number is equal to the grouping value.

[0007] Other embodiments of the aforementioned implementation are as follows: The aforementioned quantization parameter providing step may include a weight quantization step, which quantizes a weight into a quantized weight according to a weight target bit. The weight quantization step includes a weight target bit, a weight, a mapped quantized weight, a quantization equation, and a quantized weight, and the weight target bit is represented by b W , the weight is denoted as W l , the quantization equation is denoted as quan, and the quantization weight is denoted as W l q And it meets the following formula:

[0008] and

[0009]

[0010] Other embodiments of the aforementioned implementation are as follows: The aforementioned quantization method based on the in-memory computing circuit architecture may further include an excitation quantization step, which quantizes the output excitation function into a quantized output excitation function according to an excitation target bit. The excitation quantization step includes an output excitation function, a clipping function, a mapped excitation function, an excitation target bit, a quantization equation, and a quantized output excitation function, and the output excitation function is represented by A l , the clip function is represented by Clip, and the mapped activation function is represented by The excitation target bit is represented by b A , the quantization equation is expressed as quan, and the quantization output excitation function is expressed as And it meets the following formula:

[0011] and

[0012]

[0013] Other embodiments of the above-mentioned embodiment are as follows: The above-mentioned straight-through estimation quantization method includes a convolution output, a convolution target bit, a mapped convolution output, a quantization equation, and a quantized convolution output. The convolution output is represented by A l,k , the convolution target bit is represented by b AD , the convolution output after mapping is expressed as The quantized equation is denoted as quan, and the quantized convolution output is denoted as The direct estimation quantization method conforms to the following formula:

[0014]

[0015] and

[0016]

[0017] Other embodiments of the aforementioned implementation are as follows: The aforementioned specific quantization method includes convolution output, representative level, amplification factor, deviation factor, current digital value, excitation function signal, sigmoid function, standard deviation, specific function, temperature constant, classification probability, random sample, cluster value, excitation function quantization signal and quantized convolution output. The convolution output is represented by A l,k , representing the level as R AD , the amplification factor is represented by α, the deviation factor is represented by β, and the current digital value is represented by r i , the activation function signal is represented by a, the sigmoid function is represented by sigmoid, and the standard deviation is represented by σ i , the specific function is represented by Concrete, the temperature constant is represented by λ, and the classification probability is represented by π i , the random sample is represented by s i , the cluster value is represented by K, and the activation function quantization signal is represented by a q , the quantized convolution output is expressed as The specific quantification method conforms to the following formula:

[0018]

[0019]

[0020] s i =Concrete(π i , λ); and

[0021]

[0022] Other examples of the aforementioned implementation are as follows: The aforementioned specific functions include noise, temperature constant, classification probability, random sample, Gambel function and normalized exponential function, and the noise is represented by ni , Gumbel function is represented by Gumbel, normalized exponential function is represented by softmax, and the specific function conforms to the following formula:

[0023] n i =Gumbel(0,1); and

[0024]

[0025] According to an embodiment of the structural aspect of the present invention, a quantization system based on an in-memory computing circuit architecture is provided, which includes a parameter grouping module, a multiplication-accumulator, a convolution quantizer and an adder. The parameter grouping module divides a quantization weight into multiple grouping quantization weights according to a grouping value, and divides an input excitation function into multiple grouping excitation functions according to the grouping value. The multiplication-accumulator signal is connected to the parameter grouping module, and the multiplication-accumulator performs a multiplication-accumulation operation on any grouping quantization weight and any grouping excitation function to generate a convolution output. The convolution quantizer signal is connected to the multiplication-accumulator, and the convolution quantizer quantizes the convolution output into a quantized convolution output according to a convolution target bit. The adder signal is connected to the convolution quantizer, and the adder performs a part of the sum operation on the quantized convolution output according to the grouping value to generate an output excitation function. The input excitation function may have an input channel number, and the grouping value is obtained by calculating the input channel number and the control parameter, and the grouping value is equal to the input channel number divided by the control parameter. The grouping value is a positive integer, and the grouping value is greater than 1 and less than or equal to the input channel number. The convolutional quantizer performs a direct estimation quantization method or a specific quantization method.

[0026] Therefore, the quantization system based on the in-memory operation circuit architecture of the present invention can take into account the bit number limitation of the hardware specifications, and through the classification distribution of analog-to-digital converters and specific quantization methods, train a quantization deep neural network that is not only adaptable to quantization noise but also friendly to in-memory operations.

[0027] Other embodiments of the aforementioned implementation are as follows: the aforementioned product-accumulator, convolutional quantizer and adder may be repeatedly executed according to an iteration number, where the iteration number is equal to the grouping value.

[0028] Other embodiments of the aforementioned implementation are as follows: The aforementioned quantization system based on the in-memory computing circuit architecture may further include a weight quantizer, the weight quantizer signal is connected to the parameter grouping module, the weight quantizer quantizes a weight into a quantized weight according to a weight target bit, and the weight quantizer includes a weight target bit, a weight, a mapped quantized weight, a quantization equation, and a quantized weight. The weight target bit is represented by b W , the weight is denoted as W l , the quantization equation is denoted as quan, and the quantization weight is denoted as W l q, the weight quantizer is denoted as Q W And it meets the following formula:

[0029] and

[0030]

[0031] Other embodiments of the aforementioned implementation are as follows: The aforementioned quantization system based on the in-memory computing circuit architecture may further include an excitation function quantizer, the excitation function quantizer signal is connected to the adder, the excitation function quantizer quantizes the output excitation function into a quantized output excitation function according to an excitation target bit, and the excitation function quantizer includes an output excitation function, a clipping function, a mapped excitation function, an excitation target bit, a quantization equation, and a quantized output excitation function. The output excitation function is represented by A l , the clip function is represented by Clip, and the mapped activation function is represented by The excitation target bit is represented by b A , the quantization equation is expressed as quan, and the quantization output excitation function is expressed as The activation function quantizer is denoted as Q A And it meets the following formula:

[0032] and

[0033]

[0034] Other embodiments of the above-mentioned embodiment are as follows: The above-mentioned straight-through estimation quantization method includes a convolution output, a convolution target bit, a mapped convolution output, a quantization equation, and a quantized convolution output. The convolution output is represented by A l,k , the convolution target bit is represented by b AD , the convolution output after mapping is expressed as The quantized equation is denoted as quan, and the quantized convolution output is denoted as The direct estimation quantization method conforms to the following formula:

[0035]

[0036] and

[0037]

[0038] Other embodiments of the aforementioned implementation are as follows: The aforementioned specific quantization method includes convolution output, representative level, amplification factor, deviation factor, current digital value, excitation function signal, sigmoid function, standard deviation, specific function, temperature constant, classification probability, random sample, cluster value, excitation function quantization signal and quantized convolution output. The convolution output is represented by A l,k , representing the level as RAD , the amplification factor is represented by α, the deviation factor is represented by β, and the current digital value is represented by r i , the activation function signal is represented by a, the sigmoid function is represented by sigmoid, and the standard deviation is represented by σ i , the specific function is represented by Concrete, the temperature constant is represented by λ, and the classification probability is represented by π i , the random sample is represented by s i , the cluster value is represented by K, and the activation function quantization signal is represented by a q , the quantized convolution output is expressed as The specific quantification method conforms to the following formula:

[0039]

[0040]

[0041] s i =Concrete(π i , λ); and

[0042]

[0043] Other examples of the aforementioned implementation are as follows: The aforementioned specific function may include noise, temperature constant, classification probability, random sample, Gumbel function and normalized exponential function, Gumbel function is represented by Gumbel, noise is represented by n i , the normalized exponential function is expressed as softmax, and the specific function conforms to the following formula:

[0044] n i =Gumbel(0,1); and

[0045] BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic diagram showing a quantization system based on an in-memory computing circuit architecture according to a first embodiment of the present invention;

[0047] Figure 2 is a schematic diagram showing the architecture of a computing circuit in a memory according to a second embodiment of the present invention;

[0048] Figure 3 It is shown Figure 1 A schematic diagram showing the correspondence between the weight tensor of the parameter clustering module and the computing circuit architecture in the memory;

[0049] Figure 4 It is shown Figure 1 The convolution output distribution of the multiplier-accumulator corresponds to the cumulative current distribution of a single bit line (Bit Line; BL);

[0050] Figure 5 It is shown Figure 2 A schematic diagram of the classification distribution of analog-to-digital converters of the in-memory computing circuit architecture;

[0051] Figure 6 It is shown Figure 5 A schematic diagram of the classification probability of each current digital value;

[0052] Figure 7 is a flow chart showing a quantization method based on an in-memory computing circuit architecture according to a third embodiment of the present invention;

[0053] Figure 8 is a comparison diagram showing the accuracy of the specific quantization method of the present invention and the through estimation quantization method based on the number of bits of the analog-to-digital converter; and

[0054] Fig. 9 It is a comparison diagram showing the accuracy of the specific quantization method of the present invention and the direct estimation quantization method based on the control parameters.

[0055] The reference numerals are described as follows:

[0056] 100: Quantitative system

[0057] 110: Parameter grouping module

[0058] 120: Multiply-accumulate

[0059] 130: Adder

[0060] 200: In-memory computing circuit architecture

[0061] 210: Memory unit

[0062] 220: Digital Converter

[0063] 300: Quantitative Methods

[0064] S02: Quantization parameter provision step

[0065] S022: Weight quantization step

[0066] S04: Parameter grouping step

[0067] S06: Multiplication and accumulation step

[0068] S08: Convolution quantization step

[0069] S10: Convolutional merging step

[0070] S12: Iterative execution steps

[0071] S14: Excitation quantization step

[0072] a0, a1, a m : Activation function signal

[0073] A l : Output activation function

[0074] A l,k : Convolution output Input activation function Grouping Activation Function Quantizing convolution output Quantized output activation function b A : Stimulus target bit

[0075] b AD : Convolution target bits

[0076] b W : Weight target bit

[0077] Bit: bit number BL[0], BL[n]: bit line C in : Number of input channels

[0078] FP32: 32-bit symbol number Current k, l: variable

[0079] NG: Negative weight group

[0080] PG: Positive Weight Group

[0081] p: distribution

[0082] P: Cumulative probability density function

[0083] Q A : Activation Function Quantizer

[0084] Q W : Weight quantizer

[0085] Q AD : Convolutional Quantizer

[0086] r i , r0, r1, r2, r3: current digital value

[0087] SL[0], SL[n]: Source lines

[0088] w 0,0 , w 1,0 , w m,0 , w 0,n , w 1,n , w m,n : Weight value

[0089] W l : Weight

[0090] W l q : Quantization weight

[0091] Grouping quantization weight

[0092] WK: weight tensor

[0093] WL[0], WL[1], WL[8], WL[m]: word line

[0094] x: input signal

[0095] τ: control parameter

[0096] σ i , σ0, σ1, σ2, σ3: standard deviation DETAILED DESCRIPTION

[0097] The following will describe multiple embodiments of the present invention with reference to the accompanying drawings. For the purpose of clarity, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, in order to simplify the drawings, some conventional structures and elements will be shown in the drawings in a simple schematic manner; and repeated elements may be represented by the same number.

[0098] In addition, in this article, when a certain element (or unit or module, etc.) is "connected" to another element, it may refer to that the element is directly connected to another element, or it may refer to that a certain element is indirectly connected to another element, that is, there are other elements between the element and the other element. When it is explicitly stated that a certain element is "directly connected" to another element, it means that there are no other elements between the element and the other element. The terms first, second, third, etc. are only used to describe different elements, and there is no restriction on the elements themselves. Therefore, the first element can also be renamed as the second element. Moreover, the combination of elements / units / circuits in this article is not a generally known, conventional or existing combination in this field. Whether the elements / units / circuits themselves are existing cannot be used to determine whether their combination relationship is easy to be easily completed by those with ordinary knowledge in the technical field.

[0099] The present invention covers quantizing the weights and activation functions of deep neural networks. The motivation for quantizing the weights of neural networks is to reduce the size of the model and accelerate the calculations during training and inference. In terms of hardware, less memory and computing resources can be used to perform high-precision artificial intelligence (AI) applications. The motivation for quantizing the activation function is to replace the inner product operation with a binary operation and reduce the data between layers. In terms of hardware, in addition to reducing the temporary storage memory, it can also reduce the bandwidth and latency required for internal transmission in the system.

[0100] Please also read Figures 1 to 6 ,in Figure 1 is a schematic diagram showing a quantization system 100 based on an in-memory computing circuit architecture 200 according to a first embodiment of the present invention; Figure 2 is a schematic diagram showing an in-memory computing circuit architecture 200 according to a second embodiment of the present invention; Figure 3 It is shown Figure 1 A schematic diagram of a weight tensor WK of the parameter clustering module 110 corresponding to a computing circuit architecture 200 in a memory; Figure 4 It is shown Figure 1 The convolution output A of the product accumulator 120 l,k distribution, which corresponds to a single bit line (i.e. Figure 2 The cumulative current distribution of one of BL[0] to BL[n]; Figure 5 It is shown Figure 2 Classification and distribution of the analog-to-digital converter 220 of the in-memory computing circuit architecture 200 Schematic diagram of Figure 6 It is shown Figure 5 The digital value of each current r i As shown in the figure, the quantization system 100 based on the in-memory computing circuit architecture 200 includes a weight quantizer Q W , parameter grouping module 110, product accumulator 120, convolution quantizer Q AD , adder 130 and excitation function quantizer Q A .

[0101] Weight Quantizer Q W According to a weight target bit b W Set a weight W l Quantized into a quantized weight W l q , weight quantizer Q W Contains weight target bit b W , weight W l , quantized weights after mapping Quantization equation quan and quantization weight W l q , and the weight quantizer Q W It satisfies the following equations (1) and (2):

[0102]

[0103]

[0104] The above weight W l is a 32-bit symbol number (FP32), and the quantization weight W l q is the weight target bit b W The numeric value of .

[0105] The parameter clustering module 110 is connected to the weight quantizer Q W And receive the quantized weight W l q The parameter clustering module 110 quantizes the weight W according to the clustering value K. l q Divide into multiple groups and quantize weights And according to the cluster value K, an input activation function Divide into multiple group activation functions In detail, the input activation function With input channel number C in , the cluster value K is obtained by inputting the number of channels C in The cluster value K is equal to the number of input channels C. in Divide by the control parameter τ. The grouping value K is a positive integer, greater than 1 and less than or equal to the number of input channels C in Specifically, the cluster value K can be equal to the number of input channels C in , that is, the control parameter τ is equal to 1, but the present invention is not limited to this.

[0106] The multiplication and accumulation device 120 is connected to the parameter grouping module 110. The multiplication and accumulation device 120 quantizes the weight of any grouping. And any cluster activation function Perform multiplication and accumulation operations to produce the convolution output A l,k .

[0107] Convolutional Quantizer Q AD The signal is connected to the product-accumulator 120, the convolution quantizer Q AD According to the convolution target bit b AD The convolution output A l,k Quantize to quantize convolution output In detail, the convolutional quantizer Q ADA straight-through-estimator-based quantization (STE-based quantization) or a concrete-based quantization (Concrete-based quantization) can be performed. The straight-through-estimator quantization method includes the convolution output A l,k , convolution target bit b AD , convolution output after mapping Quantization equation quan and quantized convolution output The direct estimation quantization method complies with the following equations (3) to (5):

[0108]

[0109]

[0110]

[0111] The main concept of the straight-through estimation quantization method is to consider the limited number of bits of the analog-to-digital converter 220 (ADC) of the in-memory computing circuit architecture 200 during the quantization process, wherein the convolution output after mapping The convolution output A is converted based on the bit number of the analog-to-digital converter 220. l,k The reason for this is that the nonvolatile in-memory computing (nvIMC) is limited by the current that the analog-to-digital converter 220 can withstand (e.g. Figure 2 of ) is limited, the present invention uses the extreme value of the analog-to-digital converter 220 bits as the tolerable current limit, thereby converting the convolution output A l,k Restricted to a specific range to simulate quantization behavior. Then, the quantized convolution output is obtained by linear quantization

[0112] The specific quantization rules include the convolution output A l,k , represents the level R AD , amplification factor α, deviation factor β, current digital value r i , activation function signal a, sigmoid function sigmoid, standard deviation σ i , concrete function Concrete, temperature constant λ, classification probability π i , random samples i , grouping value K, activation function quantization signal a q And quantized convolution output The specific quantification method conforms to the following equations (6) to (9):

[0113]

[0114]

[0115] s i =Concrete(π i ,λ) (8);

[0116]

[0117] The above representative level R AD equal Initial magnification factor in Deviation factor β = 0, standard deviation σ i =α / 3. The concrete function Concrete contains the noise n i , temperature constant λ, classification probability π i , random samples i , Gumbel function and normalized exponential function softmax, the concrete function Concrete conforms to the following equations (10) to (11):

[0118] n i =Gumbel(0,1) (10);

[0119]

[0120] The above specific quantization method can construct a learnable representative level R through the amplification factor α and the deviation factor β AD , and for the noise distribution p(η) and the classification distribution The probability model is established, where and For the noise distribution p(η), its noise model is assumed to be a logistic distribution (L) with a mean of zero and a standard deviation of σ i means; that is, p(η)=L(0,σ i ), while the categorical distribution As shown in Figure 5. Taking any input signal x as an example, when the input signal x falls into the noise model, it can be distributed in each category The corresponding tail end probability is calculated, such as the shaded area in Figure 5. Then, these tail end probabilities based on the input signal x are used as the input signal x for each current digital value r i The classification probability of Figure 6 As shown. Taking a deep neural network as an example, each input convolution quantizer QAD The activation function signal a(a∈A l，k ) conforms to the following equations (12) to (13):

[0121]

[0122] Where P represents the Cumulative Distribution Function (CDF). Since the CDF of the logistic distribution is a sigmoid function, the assumption of the logistic distribution can benefit from two advantages: the first advantage is that the logistic distribution is very similar to the normal distribution and also satisfies the bit line current distribution; the second advantage is that the sigmoid function can perform partial differential operations in the backward propagation stage during training, and no virtual gradients need to be generated, further solving the gradient mismatch problem. From the above, it can be seen that the main concept of the specific quantization method is to establish a probability model for the noise distribution of the analog-to-digital converter 220, and to obtain the classification probability π i Standardization. By classification probability π i The establishment of can not only simulate the quantization noise of the analog-to-digital converter 220, but also prevent the deep neural network from overfitting.

[0123] The adder 130 signal is connected to the convolution quantizer Q AD The adder 130 converts the quantized convolution output into Perform a partial sum operation to generate the output activation function A l .

[0124] Excitation function quantizer Q A The signal is connected to the adder 130, the excitation function quantizer Q A According to the excitation target bit b A The output activation function A l Quantized into a quantized output activation function Excitation function quantizer Q A Contains the output activation function A l , Clip function Clip, mapped excitation function Stimulus target bit b A , quantization equation quan and quantization output excitation function Excitation function quantizer Q A It meets the following equations (14) and (15):

[0125]

[0126]

[0127] The above-mentioned product accumulator 120 and convolution quantizer Q AD The adder 130 may be repeatedly executed according to an iteration number, which is equal to the grouping value K. Therefore, the grouping quantization weight And the group activation function The convolution operation can be completed in pairs.

[0128] Therefore, the quantization system 100 based on the in-memory computing circuit architecture 200 of the present invention can take into account the bit number limitation of the hardware specification and can also achieve the desired result by using the classification distribution of the analog-to-digital converter 220. and specific quantization methods to train a quantized deep neural network that is not only adaptable to quantization noise but also friendly to in-memory operations.

[0129] The in-memory computing circuit architecture 200 includes a memory unit 210 and an analog-to-digital converter 220, wherein the memory unit 210 is signal-connected to the analog-to-digital converter 220 and includes a plurality of memory cells, a plurality of word lines (such as Figure 2 WL[0]~WL[m] and Figure 3 WL[0]~WL[8]), multiple bit lines (such as Figure 2 BL[0]~BL[n] and Figure 3 Positive-Weight Group (PG) and Negative-Weight Group (NG)) and multiple source lines (such as Figure 2 Each memory cell is connected to a word line, a bit line and a source line. Each memory cell stores a weight value (such as Figure 2 w 0,0 ~w m,n ), each word line transmits an excitation function signal a (such as Figure 2 a0~a m ), each bit line transmits the convolution output A l,k (like Figure 2 Current ), each source line is grounded. Specifically, the memory unit 210 may be a resistive random access memory (ReRAM), but the present invention is not limited thereto.

[0130] Raise Figure 3 For example, assuming that the size of the input feature map is H×W×C in , the size of the weight tensor WK is 3×3×C in ×C out ( Figure 3 Only part of the weight tensor WK is shown. After convolution, the output size is H×W×Cout The output feature map of . Where H and W represent the length and width of the input feature map, respectively, and C in Represents the number of input channels, C out Represents the number of output channels, and the number of word lines is equal to 9 (i.e. WL[0] to WL[8]). The convolution operation process of non-volatile memory operation (nvIMC) is as follows: a partial input feature map of size 3×3×1 is mapped to the word line input, and combined with the grouped quantization weights on multiple bit lines (3×3 weight values) are used for calculation, and the number of bit lines is C out After one cycle time, nvIMC will output C out The output value of the matrix-vector multiplication (MVM) operation is then passed through the waiting C in After the cycle time, C in ×C out MVM output values, and finally these MVM output values ​​are combined to obtain 1×1×C out The output feature map is obtained by replacing W l Finally, we complete a convolution operation and get W×H×C out Therefore, the nvIMC of the present invention can quantize the weights of the clusters The convolution operation function in the memory is achieved by pre-storing in the memory unit 210 and using a specific in-memory computing circuit architecture 200, which can avoid the problem of large amounts of data transfer between hierarchical memories in the traditional architecture, thereby reducing system power consumption, reducing system latency, and improving energy efficiency. In addition, the present invention uses the convolution of ReRAM to quantize the weight W l q With input activation function The memory cells 210, word line inputs and convolution outputs A are grouped and considered to have a limited number of bit lines. l,k The discretization of , through the robust nature of deep neural networks, learns weight parameters suitable for the hardware limitations of nvIMC.

[0131] Please also read Figures 1 to 7 ,in Figure 7 2 is a flow chart showing a quantization method 300 based on the in-memory computing circuit architecture 200 according to the third embodiment of the present invention. The quantization method 300 based on the in-memory computing circuit architecture 200 comprises a quantization parameter providing step S02, a parameter grouping step S04, a product accumulation step S06, a convolution quantization step S08, a convolution merging step S10, an iterative execution step S12 and an excitation quantization step S14.

[0132] The quantization parameter providing step S02 is to provide a quantization parameter, which includes an input excitation function Quantization weight W l q And the cluster value K. In detail, the input activation function With input channel number C in , the cluster value K is obtained by inputting the number of channels C in The cluster value K is equal to the number of input channels C. in Divide by the control parameter τ. The grouping value K is a positive integer, greater than 1 and less than or equal to the number of input channels C in Furthermore, the quantization parameter providing step S02 includes a weight quantization step S022, which is based on the weight target bit b W The weight W l Quantized into a quantized weight W l q The weight quantization step S022 includes a weight target bit b W , weight W l , quantized weights after mapping Quantization equation quan and quantization weight W l q , quantized weight W l q It complies with the above equations (1) and (2).

[0133] The parameter grouping step S04 is to quantize the weight W according to the grouping value K. l q Divide into multiple groups and quantize weights And according to the cluster value K, the input activation function Divide into multiple group activation functions

[0134] The multiplication and accumulation step S06 is to quantize the weight of any group And any cluster activation function Perform a multiply-accumulate operation to generate a convolution output A l,k .

[0135] The convolution quantization step S08 is based on a convolution target bit b AD The convolution output A l,k Quantized into a quantized convolution output Specifically, the convolution quantization step S08 includes executing a direct estimation quantization method or a specific quantization method, wherein the direct estimation quantization method includes convolution output A l,k , convolution target bit b AD , convolution output after mapping Quantization equation quan and quantized convolution output The direct estimation quantization method complies with the above equations (3) to (5). As for the specific quantization rule, it includes the convolution output A l,k , represents the level R AD , amplification factor α, deviation factor β, current digital value r i , activation function signal a, sigmoid function sigmoid, standard deviation σ i , concrete function Concrete, temperature constant λ, classification probability π i , random samples i , grouping value K, activation function quantization signal a q And quantized convolution output The specific quantization method conforms to the above equations (6) to (9). In addition, the specific function Concrete contains the noise n i , temperature constant λ, classification probability π i , random samples i , Gumbel function Gumbel and normalized exponential function softmax, the specific function Concrete conforms to the above equations (10)~(11).

[0136] The convolution merging step S10 is to combine the quantized convolution output according to the cluster value K. Perform a partial sum operation to generate an output activation function A l .

[0137] The iterative execution step S12 repeatedly executes the product-addition step S06, the convolution quantization step S08 and the convolution merging step S10 according to the number of iterations, and the number of iterations is equal to the clustering value K.

[0138] The excitation quantization step S14 is based on an excitation target bit b A The output activation function A l Quantized into a quantized output activation function The excitation quantization step S14 includes outputting the excitation function A l , Clip function Clip, mapped excitation function Stimulus target bit b A , quantization equation quan and quantization output excitation function Quantized output activation function This is consistent with the above equations (14) and (15).

[0139] The above quantization method 300 can be illustrated by Table 1. Table 1 shows that the quantization method of the present invention is used to train a neural network, which converts the input activation function of the neural network into Weight W l And the MVM output value (i.e., the convolution output A l,k ) are quantized to b A, b W With b AD bits, and considers the limited number of words in the memory, and introduces the value τ as the control parameter of the partial sum. Where k and l are variables.

[0140] Table 1

[0141]

[0142] The first to 12th rows of Table 1 represent the quantization process of the inference part of an L-depth neural network, and the 13th and 14th rows represent the back propagation and weight update rules of the neural network, respectively, where SGD stands for Stochastic Gradient Decent, and ADAM stands for Adaptive Moment Estimation. The above back propagation and weight update rules are all existing technologies, and their details are not repeated here.

[0143] The 1st and 12th rows of Table 1 represent the operations from the 1st layer (l=1) to the Lth layer, and each layer will perform the operations from the 2nd to the 11th rows.

[0144] The second row of Table 1 is the weight W of the floating point value. l Linear quantization is performed to the weight target bit b W , to obtain the quantized weight W l q .

[0145] The third row of Table 1 is based on the control parameter τ and the quantization weight W l q And the input activation function The number of input channels C in Perform calculations to obtain the clustering value K of the clustered convolution of the current layer. In deep learning, the quantized weight W l q With input activation function In the training phase, it is a four-dimensional tensor, where the quantized weight W l q The four dimensions are "convolution kernel length, convolution kernel width, number of input channels C in , Output channel number C out "; Input activation function The four dimensions are "batch size, input feature map length H, input feature map width W, input channel number C in ”.

[0146] Rows 4 to 9 of Table 1 illustrate how to perform grouped convolution through iterative operations. This part simulates the limited number of word lines in the computing circuit architecture 200 in the memory and the conversion of the analog-to-digital converter 220.

[0147] The fourth row of Table 1 is based on the cluster value K in the third row, and the quantization weight W is l q And the input activation function After the split operation, the split tensor will become a five-dimensional tensor; in other words, the quantized weight W l q The dimensions will be changed from the original "convolution kernel length, convolution kernel width, number of input channels C in , Output channel number C out "Through the input channel number C in Cut and change to "convolution kernel length, convolution kernel width, control parameter τ, number of output channels C out , grouping value K"; and the input activation function The dimensions will be changed from the original "batch size, input feature map length H, input feature map width W, input channel number C in ” is divided into “batch size, input feature map length H, input feature map width W, control parameter τ, and cluster value K”. Therefore, from the fifth dimension, we can get K groups of weights and tensors after cutting; that is, we can use (convolution kernel length × convolution kernel width × τ) to determine the number of open character lines (W l ), which is determined by the control parameter τ and the convolution kernel size.

[0148] The 5th and 9th rows of Table 1 are iterative operations, which represent the execution from k=1 to k=K, and the operations from the 6th to the 8th rows are executed each time.

[0149] The sixth row of Table 1 is the quantization weight of the cluster And the clustering activation function Perform convolution operation. Take the first group (k = 1) for the quantized weights of the clusters And the clustering activation function For example, the clustering quantization weight And the clustering activation function The tensor dimensions are "convolution kernel length, convolution kernel width, control parameter τ, number of output channels C out " and "batch size, input feature map length H, input feature map width W, control parameter τ", after convolution operation between the two, a floating-point value convolution output A can be generated and output l,k (i.e., MVM output value).

[0150] The seventh row of Table 1 is the convolution output A for floating point values. l,kAfter quantization, a b AD Quantized convolution output This portion simulates the quantization behavior of the analog-to-digital converter 220 .

[0151] The eighth row of Table 1 is the quantized value (i.e., the quantized convolution output ) performs a partial sum operation and adds the values ​​of each iteration to achieve the purpose that the proposed grouped convolution is equivalent to the traditional convolution operation.

[0152] The 10th row of Table 1 is the partial sum completed by accumulation (i.e., the output activation function A l ) performs linear quantization of the activation function and obtains b A Quantized Output Excitation Function Line 11 performs other operations of the deep neural network (such as pooling), which is a prior art and its details are not repeated here. After executing line 12, it means that the L-layer neural network has been passed and the forward propagation of the network has ended.

[0153] Table 2 shows the test error rate of different control parameters τ on various data sets, which simulates the number of word lines turned on by the control parameter τ. The number of word lines turned on determines the size and extreme value of the accumulated current on the bit line. Different numbers of word lines turned on will have different degrees of influence on the test error rate. The control parameter τ of this embodiment is {1,2,4}. For the convolution layer with a kernel size of 3×3, the number of word lines turned on is equivalent to {9,18,36}. It can be demonstrated from the experimental results that due to the existence of the analog-to-digital converter 220, too many word lines turned on will lead to an increase in the error rate. In b W -b A -b AD Under the test conditions of 2-2-4 and CIFAR-10 datasets, the error rate based on ReRAM convolution (τ=1) can be reduced by 0.48% compared with that without ReRAM convolution (τ=V). i The ST dataset can also achieve a 0.5% error rate reduction.

[0154] Table 2

[0155] <![CDATA[b W ]]> <![CDATA[b A ]]> <![CDATA[b AD ]]> τ MNIST(%) CIFAR-10 (%) 2 2 4 V 0.75 18.43 2 2 4 4 0.76 19.03 2 2 4 2 0.75 18.02 2 2 4 1 0.70 17.95

[0156] Figure 8 is a comparison chart showing the accuracy of the specific quantization method of the present invention and the straight-through estimation quantization method based on the bit number of the analog-to-digital converter, and the test conditions include using the image classification data set CIFAR-10, the weight target bit b W is 2 bits and the excitation target bit b Ais 2 bits. In different convolution target bits b AD When the convolutional quantizer Q is based on a specific distribution AD At each bit, the convolutional quantizer Q based on the straight-through estimation quantization method is compared AD Better performance, with an average improvement of 0.88% from 3-bit to 8-bit, especially in the convolution target bit b AD When the convolutional quantizer Q is 2 bits, the specific distribution AD The accuracy rate can be improved by 9.52%.

[0157] Fig. 9 is a comparison chart showing the accuracy of the specific quantization method of the present invention and the direct estimation quantization method based on the control parameter τ, and the test conditions include using the image classification data set CIFAR-10, the weight target bit b W is 2 bits, the excitation target bit b A is 2 bits and the convolution target bit b AD is 2 bits. Under different control parameters τ, the convolutional quantizer Q based on the specific distribution AD Each value is compared to the convolutional quantizer Q based on the straight-through estimation quantization method. AD There was an average improvement of 2.92%.

[0158] It can be seen from the above embodiments that the present invention has the following advantages: First, the quantization system based on the in-memory operation circuit architecture can take into account the bit number limitation of the hardware specifications, and through the classification distribution of the analog-to-digital converter and the specific quantization method, a quantization deep neural network that is not only adaptable to quantization noise but also friendly to in-memory operations can be trained. Second, the quantization method based on the in-memory operation circuit architecture can pre-store the grouped quantization weights in the memory unit, and achieve the convolution operation function in the storage memory through a specific in-memory operation circuit architecture, which can avoid the problem of large amounts of data transfer between hierarchical memories in the traditional architecture, thereby achieving the goal of reducing system power consumption, reducing system delays and improving energy efficiency. Third, the present invention groups and pairs the quantization weights with the input excitation function, and considers the discretization of the memory unit, word line input and convolution output of the finite bit line, and learns the weight parameters suitable for the hardware limitations of the non-volatile memory operation through the robust nature of the deep neural network.

[0159] Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A quantization method based on a memory computing circuit architecture, characterized in that: The following steps are involved: A quantization parameter providing step is to provide a quantization parameter, wherein the quantization parameter includes an input excitation function, a quantization weight and a grouping value; a parameter grouping step, which is to divide the quantized weight into a plurality of grouped quantized weights according to the grouped value, and to divide the input excitation function into a plurality of grouped excitation functions according to the grouped value; A multiplication-accumulation step is to perform a multiplication-accumulation operation on any of the group quantization weights and any of the group excitation functions to generate a convolution output; a convolution quantization step, quantizing the convolution output into a quantized convolution output according to a convolution target bit; and A convolution merging step is to perform a partial sum operation on the quantized convolution output according to the group value to generate an output excitation function; The input excitation function has an input channel number, the group value is obtained by calculating the input channel number and a control parameter, the group value is equal to the input channel number divided by the control parameter, and the group value is a positive integer greater than 1 and less than or equal to the input channel number; The convolution quantization step includes executing a direct estimation quantization method or a specific quantization method.

2. The quantization method based on the in-memory computing circuit architecture according to claim 1, characterized in that: Also includes: An iterative execution step is to repeatedly execute the product-addition step, the convolution quantization step and the convolution merging step according to an iteration number, and the iteration number is equal to the grouping value.

3. The quantization method based on the in-memory computing circuit architecture according to claim 1, characterized in that: The quantization parameter providing step comprises: A weight quantization step is to quantize a weight into the quantized weight according to a weight target bit, the weight quantization step includes the weight target bit, the weight, a mapped quantized weight, a quantization equation and the quantized weight, the weight target bit is represented by b W , the weight is denoted as W l , the quantization equation is denoted as quan, and the quantization weight is denoted as And it meets the following formula: and 4. The quantization method based on the in-memory computing circuit architecture according to claim 1, characterized in that: Also includes: An excitation quantization step is to quantize the output excitation function into a quantized output excitation function according to an excitation target bit. The excitation quantization step includes the output excitation function, a clipping function, a mapped excitation function, the excitation target bit, a quantization equation and the quantized output excitation function. The output excitation function is represented by A l , the clip function is represented by Clip, the mapped excitation function is represented by, and the excitation target bit is represented by b A , the quantization equation is expressed as quan, and the quantization output excitation function is expressed as And it meets the following formula: and 5. The quantization method based on the in-memory computing circuit architecture according to claim 1, characterized in that: The direct estimation quantization method includes the convolution output, the convolution target bit, a mapped convolution output, a quantization equation and the quantized convolution output, and the convolution output is represented by A l,k , the convolution target bit is represented by b AD , the convolution output after the mapping is expressed as The quantization equation is expressed as quan, and the quantized convolution output is expressed as The direct estimation quantization method conforms to the following formula: and 6. The quantization method based on the in-memory computing circuit architecture according to claim 1, characterized in that: The specific quantization method includes the convolution output, a representative level, an amplification factor, a deviation factor, a current digital value, an excitation function signal, a sigmoid function, a standard deviation, a specific function, a temperature constant, a classification probability, a random sample, the group value, an excitation function quantization signal and the quantized convolution output, and the convolution output is represented by A l,k , the representative level is expressed as R AD , the amplification factor is represented by α, the deviation factor is represented by β, and the current digital value is represented by r i , the activation function signal is represented by a, the sigmoid function is represented by sigmoid, and the standard deviation is represented by σ i , the specific function is represented by Concrete, the temperature constant is represented by λ, and the classification probability is represented by π i , the random sample is represented by s i , the cluster value is represented by K, and the excitation function quantization signal is represented by a q , the quantized convolution output is expressed as The specific quantification method conforms to the following formula: s i =Concrete(π i , λ); and 7. The quantization method based on the in-memory computing circuit architecture according to claim 6, characterized in that: The specific function includes a noise, the temperature constant, the classification probability, the random sample, a Gambel function and a normalized exponential function. The noise is represented by n i , the Gumbel function is represented by Gumbel, the normalized exponential function is represented by softmax, and the specific function conforms to the following formula: n i =Gumbel(0,1); and 8. A quantization system based on an in-memory computing circuit architecture, characterized in that: Include: a parameter grouping module, which divides a quantized weight into a plurality of grouped quantized weights according to a grouping value, and divides an input excitation function into a plurality of grouped excitation functions according to the grouping value; a multiply-accumulate-adder, signal-connected to the parameter grouping module, the multiply-accumulate-adder performs a multiply-accumulate operation on any of the grouping quantization weights and any of the grouping excitation functions to generate a convolution output; a convolution quantizer, signal-connected to the product-accumulator, the convolution quantizer quantizing the convolution output into a quantized convolution output according to a convolution target bit; and an adder, the signal of which is connected to the convolution quantizer, and the adder performs a partial sum operation on the quantized convolution output according to the group value to generate an output excitation function; The input excitation function has an input channel number, the group value is obtained by calculating the input channel number and a control parameter, the group value is equal to the input channel number divided by the control parameter, and the group value is a positive integer greater than 1 and less than or equal to the input channel number; The convolutional quantizer performs a direct estimation quantization method or a specific quantization method.

9. The quantization system based on the in-memory computing circuit architecture as claimed in claim 8, characterized in that: The product-accumulator, the convolutional quantizer and the adder are repeatedly executed according to an iteration number, which is equal to the grouping value.

10. The quantization system based on the in-memory computing circuit architecture as claimed in claim 8, characterized in that: Also includes: A weight quantizer, signal-connected to the parameter grouping module, the weight quantizer quantizes a weight into the quantized weight according to a weight target bit, the weight quantizer includes the weight target bit, the weight, a mapped quantized weight, a quantization equation and the quantized weight, the weight target bit is represented by b W , the weight is denoted as W l , the quantization equation is denoted as quan, and the quantization weight is denoted as The weight quantizer is denoted as Q W And it meets the following formula: and 11. The quantization system based on the in-memory computing circuit architecture as claimed in claim 8, characterized in that: Also includes: An excitation function quantizer, the signal is connected to the adder, the excitation function quantizer quantizes the output excitation function into a quantized output excitation function according to an excitation target bit, the excitation function quantizer includes the output excitation function, a clipping function, a mapped excitation function, the excitation target bit, a quantization equation and the quantized output excitation function, the output excitation function is represented by A l , the clip function is represented as Clip, and the mapped activation function is represented as The excitation target bit is represented by b A , the quantization equation is expressed as quan, and the quantization output excitation function is expressed as The activation function quantizer is represented by Q A And it meets the following formula: and 12. The quantization system based on the in-memory computing circuit architecture as claimed in claim 8, characterized in that: The direct estimation quantization method includes the convolution output, the convolution target bit, a mapped convolution output, a quantization equation and the quantized convolution output, and the convolution output is represented by A l,k , the convolution target bit is represented by b AD , the convolution output after the mapping is expressed as The quantization equation is expressed as quan, and the quantized convolution output is expressed as The direct estimation quantization method conforms to the following formula: and 13. The quantization system based on the in-memory computing circuit architecture as claimed in claim 8, characterized in that: The specific quantization method includes the convolution output, a representative level, an amplification factor, a deviation factor, a current digital value, an excitation function signal, a sigmoid function, a standard deviation, a specific function, a temperature constant, a classification probability, a random sample, the group value, an excitation function quantization signal and the quantized convolution output, and the convolution output is represented by A l,k , the representative level is expressed as R AD , the amplification factor is represented by α, the deviation factor is represented by β, and the current digital value is represented by r i , the activation function signal is represented by a, the sigmoid function is represented by sigmoid, and the standard deviation is represented by σ i , the specific function is represented by Concrete, the temperature constant is represented by λ, and the classification probability is represented by π i , the random sample is represented by s i , the cluster value is represented by K, and the excitation function quantization signal is represented by a q , the quantized convolution output is expressed as The specific quantification method conforms to the following formula: s i =Concrete(π i , λ); and 14. The quantization system based on the in-memory computing circuit architecture according to claim 13, characterized in that: The specific function includes a noise, the temperature constant, the classification probability, the random sample, a Gumbel function and a normalized exponential function. The Gumbel function is represented by Gumbel, and the noise is represented by n i , the normalized exponential function is expressed as softmax, and the specific function conforms to the following formula: n i =Gumbel(0,1); and

Images