A neural network computing processing method, device and computer readable medium

Abstract

The present invention discloses a neural network operation processing method, device and computer readable medium, comprising: receiving input values ​​of two functions of operation; checking the value range of the input value of at least one of the functions; and selecting one of the input values ​​of the two functions to perform the operation according to the above-mentioned inspection result. The present invention provides a neural network operation processing method and device that reduces power consumption, shortens processing time and thus improves performance, and can maximize the reuse of hardware for activation-normalization, normalization-activation and activation-weight architecture design.

Description

Technical Field

[0001] The present invention discloses a neural network operation processing method, device and computer-readable medium, and relates to engineering technical fields such as low-power design and neural network operation. Background Art

[0002] With the rapid development of artificial intelligence technology, neural network operations have been widely and successfully applied in fields with massive data such as images, voice and text. In specific application scenarios such as deep neural networks (DNNs) and convolutional neural networks (CNNs), multiplication is one of the most basic operations in many applications, so the power consumption and processing time of multiplication operations usually account for a large part of the total power consumption and processing time.

[0003] In the prior art, a common traditional method is to check whether one of the two inputs of the multiplication operation is zero. For example, the Eyeriss accelerator checks whether the input from the feature map is zero to prevent the MAC data path from switching when the input is zero. If the input is n-bit data, an N-bit comparator is required. For example, the N-bit data in the Eyeriss accelerator is 16-bit data. When the input data size is a runtime variable, such as 16-bit, 8-bit, or 4-bit data, the comparator is required to have the ability to compare this variable bit length data. For details, see reference 1: Y.-H. Chen, T. Krishna, J. Emer, V. Sze, "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks," IEEE Journal of Solid State Circuits (JSSC), ISSCC Special Issue, Vol. 52, No. 1, pp. 127-138, January 2017.

[0004] In order to realize a deeper neural network, such as a 1001-layer neural network, contrary to the traditional "post-activation" scheme, He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian proposed a "pre-activation" scheme for the weight layer in Reference 2: "Identity Mapping in Deep Residual Networks". Figure 1 As shown, Figure 1 (a) is a schematic diagram of “post-activation”. Figure 1 (b) is a schematic diagram of "pre-activation". In this way, the activation function ReLu (linear rectification unit) is performed after BN (batch normalization). In Reference 2, the weight layer is called a convolutional layer. Figure 1Each block shown, such as BN, ReLu, Weight, and Addition, is a layer of the neural network model. "Layer" and "function" are used interchangeably. Among them:

[0005] ReLu function: f(x)=max(0,x); if the input x is a negative value, the output of ReLu f(x) is f(x)=0, otherwise the output of ReLu is x.

[0006] BN (Batch Normalization): For details, see reference 3: Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML. (2015).

[0007] Since there are other types of normalization, the calculation can be summarized as: g(x) = (x-μ)*λ. The simplified diagram is as follows Figure 2 When A is the input (x-μ) and B is the normalized scaling factor λ, the output of the normalized activation is f(C) = f(B x A), where f is the activation function (ReLu in this case).

[0008] In addition, in traditional methods, such as the method used in Eyeriss, it is necessary to store the PE results in memory, such as the Global Buffer in Eyeriss, read the data from the memory, perform ReLu, and then store the ReLu results in memory (such as DRAM in Eyeriss). Eyeriss performs run-length coding (RLC) on the ReLu results before storing them in DRAM. Eyeriss then reads back the compressed ReLu results, performs RLC decoding, and stores the decoded results together with the filter weights in the Global Buffer, and finally uses a PE array with multiple MACs to perform multiplication or multiplication-addition / multiplication-accumulation. This method requires storing the ReLu results in memory first and then reading them from memory to perform "layer weight" processing, which relatively prolongs the processing time.

[0009] In summary, existing technologies often lead to huge computing requirements, memory usage requirements, and high bandwidth requirements when performing a large number of multiplication calculations, which puts high demands on the hardware to implement large-scale neural networks. Summary of the invention

[0010] The technical problem to be solved by the present invention is: in view of the defects of the prior art, a neural network operation processing method, device and computer-readable medium are provided, which can reduce the power consumption and processing time of multiplication operations by checking the sign or value range information of at least one of its inputs.

[0011] The present invention adopts the following technical solutions to solve the above technical problems:

[0012] In a first aspect, the present invention discloses a neural network operation processing method, the method comprising: receiving input values ​​of two functions to be operated; checking the value range of the input value of at least one of the functions; and determining the execution strategy of the function based on the above-mentioned inspection results.

[0013] Furthermore, the checking of the value range of the input value of at least one function includes checking the sign of the input value of at least one function, specifically checking the sign information or sign bit of the input value of the function; and the operation includes a multiplication operation.

[0014] Furthermore, the execution strategy of the determined function includes: if the value range of the input value of the checked function is positive, then executing one of the two functions in the first mode; otherwise, executing one of the two functions in the second mode.

[0015] Executing one of the two functions in the first mode includes performing a multiplication operation.

[0016] Executing one of the two functions in the second mode includes: assigning an output of one of the two functions to a set constant value.

[0017] Furthermore, the method also includes: predicting the range of operation results by checking the value range of the input value of at least one function; if the range of the predicted operation result is a positive value, executing the function, otherwise, not executing the function.

[0018] The execution strategy of the determined function specifically includes: if the value range of the input value of the checked function indicates that the input value of the checked function is a negative value, then the function is not executed; if the value range of the input value of the checked function indicates that the input value of the checked function is a positive value, then the function is executed.

[0019] If the function is not executed, the generated constant result is 0.

[0020] The execution strategy of the determined function may also include: if the value range of the input value of the checked first function is positive, then selecting the first coefficient as the input value of the second function; if the value range of the input value of the checked first function is negative, then selecting the second coefficient as the input value of the second function.

[0021] The execution of the operation consists in multiplying the input value of the checked function by the selected coefficient.

[0022] Furthermore, the execution of the function also includes: if the value range of the input value of the checked first function is a positive value, selecting the first coefficient as input and multiplying the input value of the first function with the selected first coefficient; if the value range of the input value of the checked first function is a negative value, selecting the first coefficient and the second coefficient as input and multiplying the input value of the first function with the selected first coefficient and the second coefficient.

[0023] As a preferred embodiment of the present invention, the multiplication operation is implemented as a multiplier, a shifter, a series of adders and shifters, or a series of AND gates and adders.

[0024] As a preferred embodiment of the present invention, the two functions are respectively a normalization function and an activation function. The normalization function includes: a layer normalization function, an instance normalization function, a group normalization function or a switchable normalization function; the activation function includes a ReLu function, a PReLu function, a ReLu6 function or a ReLuN function.

[0025] In the second aspect, the present invention also discloses a neural network operation processing device, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor includes a plurality of multipliers, and is characterized in that the processor implements the following steps when executing the computer program: receiving input values ​​of two functions of operation; checking the value range of the input value of at least one of the functions; and determining the execution strategy of the function based on the above-mentioned inspection results.

[0026] In a third aspect, the present invention further discloses a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the neural network operation processing method.

[0027] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: providing a neural network operation processing method and device that reduces power consumption, shortens processing time and thus improves performance.

[0028] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, preferred embodiments are given below, together with

[0030] The attached drawings are described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the "post-activation" scheme and the "pre-activation" scheme in the prior art.

[0032] Figure 2 This is a schematic diagram of executing the activation function ReLu after BN in the prior art.

[0033] Figure 3 It is a schematic diagram of the processing method of the present invention.

[0034] Figure 4 This is a schematic diagram of the neg indicator.

[0035] Figure 5 is a schematic diagram of an exemplary PE array.

[0036] Figure 6 is an exemplary circuit schematic for skip detection.

[0037] Figure 7 This is a schematic diagram of the application of “ReLu–Weight” fusion.

[0038] Figure 8 Schematic diagram of the “ReLu–Weight” fusion method in the present invention.

[0039] Fig. 9 is another exemplary circuit schematic for skip detection.

[0040] Fig.10 It is a schematic diagram of the “PReLu–Weight” fusion method in the present invention. DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0042] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0043] The processing method of the present invention is shown in the schematic diagram Figure 3 As shown, the normalization function takes the BN function as an example, and the activation function takes the ReLu function as an example. The ReLu function can be replaced by other types of activation functions such as PReLu, ReLu6, and ReLuN. The BN function can be replaced by other types of normalization functions, such as layer normalization, instance normalization, group normalization, switchable normalization, etc.

[0044] The first design innovation of the present invention is that when the activation function ReLu is executed after the BN function, the proposed method is applied to reduce power consumption and processing time. The details are as follows:

[0045] When the ReLu function is applied to the multiplication result C = AxB, that is, Figure 2 As shown, f(C) = f(AxB), the proposed method is to check the sign of A and the sign of B. If their signs are different, C will be negative and f(c) will be zero. Therefore, unnecessary multiplication calculations can actually be avoided by checking the signs of the two inputs.

[0046] In a specific embodiment of the present invention, as shown in the following table, A=-1 and B=-7 are displayed in 4-bit 2's complement form. And f(C)=7. In 2's complement form, the sign bit of A is 1'b1, and the sign bit of B is also 1'b1.

[0047]

[0048] like Figure 4 As shown, the exemplary detection circuit for checking whether C is negative (neg) can be implemented as a simple XOR gate. If the neg indicator is true (ie, neg=1'b1), indicating that C is negative, the multiplication for calculating C=AxB can be avoided or skipped to reduce power consumption.

[0049] In some implementations, multiplication operations require multiple cycles. For example, Eyeriss described in the background implements a two-stage pipeline multiplier. Therefore, avoiding multiplication can also shorten processing time.

[0050] In another embodiment, a bit-serial multiplier is used, so skipping multiplication can reduce the number of clock cycles required by the bit-serial multiplier to perform the bit-serial multiplication operation. The bit-serial multiplier can be implemented with only a shifter for representing a power of two multiplication. The bit-serial multiplier can also use an add-shift or an AND-shift architecture to represent arbitrary coefficients instead of powers of two coefficients.

[0051] In another embodiment, if Figure 5As shown, using a PE array with multiplication or multiplication-addition / multiplication-accumulation (MAC) units that can be executed in parallel, if all neg indicators (neg 1, neg 2, neg 3, neg 4) indicate that C1 to C4 will be 0, the processing time can be shortened because no multiplication or MAC operation needs to be performed. If any neg indicator N indicates that CN will be positive, its associated MAC unit still needs to calculate CN, so the total processing time cannot be reduced. However, the capacity of other MAC units whose associated neg indicators indicate that the result is a negative value can be reduced.

[0052] The following table shows an example of A=1 and B=-7 in the form of 4-bit signed values. In this example, since C=-7, f(C)=0.

[0053]

[0054] The method can still use the simple XOR gate described above to generate a neg indicator with the sign bit of A and the sign bit of B. Other number systems besides 2's complement and sign magnitude can also be used.

[0055] The proposed method can also be combined with a check to see if the input is zero. Figure 6 , an example of such a detection circuit is shown.

[0056] In another embodiment, input B is designed as an unsigned data type. In this case, if the circuit is specifically designed for this use case, the XOR gate can be omitted, and the sign bit of A can be directly used as a neg indicator.

[0057] The second design innovation of the present invention is: an optimization method proposed for the "pre-activation" of the weight layer, that is, executing the activation function ReLu (linear correction unit) after executing the BN function. Figure 7 There are two pre-activation operations shown in 1 →ReLu 1 and BN 2 →ReLu 2 . Among them: BN 1 and BN 2 Refers to the first and second BN layers respectively. Similarly, ReLu 1 and ReLu 2 denote the first and second ReLu layers respectively.

[0058] We will introduce another optimization method for the second aspect of the present invention. In conventional design, we can use the weight layer 1 and layer BN 2 The layer fusion of , thereby optimizing the multiplication amount of these two layers. Applying our proposed method to the layer ReLu 2and layer Weight 2 This not only reduces data access but also reduces the power consumption of performing multiplication operations.

[0059] As mentioned in the background of this application, traditional methods, such as the method used in Eyeriss, require storing the PE result in memory (such as the Global Buffer in Eyeriss), reading data from the memory, performing ReLu, and then storing the ReLu result in memory (such as DRAM in Eyeriss). Eyeriss performs run-length coding (RLC) on the ReLu result before storing it in DRAM. Eyeriss then reads back the compressed ReLu result, performs RLC decoding, and stores the decoded result together with the filter weights in the Global Buffer, and finally uses a PE array with multiple MACs to perform multiplication or multiplication-addition / multiplication-accumulation.

[0060] The conventional method is to perform the ReLu function after the multiplier-adder unit, while the present invention performs layer fusion of the "ReLu layer" and the "Weight layer". In addition, the present method uses a very simple circuit to perform the ReLu function before the operation unit, such as a multiplier or a multiplier-adder unit. Through this fusion, the present invention no longer needs to store the results of executing the ReLu function in the memory and then read them from the memory to perform the "Weight layer" processing. This method is also effective for other activation functions used in the design.

[0061] A 1×1 convolution is used as a weight layer with a 4-channel MxN input feature map. The output feature map at position (0,0) is:

[0062] y0(0,0)=w10*x0(0,0)+w11*x1(0,0)+w12*x2(0,0)+w13*x3(0,0);

[0063] Among them, w10~w13 are the weights of the first kernel of this weight layer. Usually, a weight layer has multiple kernels.

[0064] x0(0,0), x1(0,0), x2(0,0), and x3(0,0) are the input feature map data at position (0,0).

[0065] Since the previous layer of this weight layer is a ReLu layer, Figure 8This layer fusion of one embodiment is shown. A is the input of the ReLu layer, B is the weight of the Weight layer, and f(*) is the ReLu function. With such layer fusion, only a single memory access for reading the result of A and f(A) of the ReLu function is directly provided as one of the inputs of the multiplication or multiply-addition / multiply-accumulate operation. Figure 8 To simplify the diagram, the accumulation part of the weight layer is not shown. When either f(A) or B is zero, the multiplication step can be skipped.

[0066] As mentioned above, the ReLu function is: f(x)=max(0,x); that is, if the input x is a negative value, the output of the ReLu function is f(x)=0, otherwise the output of the ReLu function is x.

[0067] Checking whether f(A) is zero can be done by checking whether A is less than or equal to 0, such as Fig. 9 (a) is shown. Another exemplary implementation for such detection is to Fig. 9 (b) Figure 6 Furthermore, if the circuit is designed specifically for this use case, the XOR gate can be omitted and the sign bit of A can be used directly as the neg indicator.

[0068] If there are multiple PEs, for example 8 PEs, then there will be If all these skip signals indicate that the operation is to be skipped, then we can not only reduce power consumption but also skip processing cycles. In this case, a look-ahead circuit can be implemented to check the sign of several A's. Since the sign check circuit is very simple, the corresponding look-ahead circuit is also easy to implement.

[0069] For example, when A(s,t) is used to represent the input of the sth PE in cycle t, then:

[0070] A(0,0), A(1,0), A(2,0), ..., A(7,0) represent the input A of PE0 to PE7 in cycle 0;

[0071] A(0,1), A(1,1), A(2,1), ..., A(7,1) represent the input A of PE0 to PE7 in cycle 1;

[0072] A(0,2), A(1,2), A(2,2), ..., A(7,2) represent the input A of PE0 to PE7 in cycle 2;

[0073] A(0,3), A(1,3), A(2,3), ..., A(7,3) represent the input A of PE0 to PE7 in cycle 3;

[0074] If all of these A's skip signals indicate that the operation is to be skipped, four cycles can be saved, thus improving performance.

[0075] In addition to ReLu, other types of activation functions can also benefit from this layer fusion. For example, parameterized ReLu (PReLu). In another embodiment of the present invention, a PReLu function is used, where:

[0076] f(x)=max(αx,x); α∈[0,1), or, f(x)=max(αx,x); α∈(0,1).

[0077]

[0078] In the above embodiment, the calculation C=B xf(A) of the “PReLu–Weight” fusion can be changed to:

[0079] C = B x A, for A ≥ 0;

[0080] C=αx B x A,for A<0.

[0081] Fig.10 (a) shows one of the embodiments. αx B can be calculated in the pipeline, so there is no need to store the intermediate result BxA to memory first, nor to read it back to perform the PReLu function, i.e., αx(B x A).

[0082] In another embodiment, since B and α are constants, αx B does not have to be calculated at runtime, and αx B can be pre-calculated as B' offline. In addition, the sign of A can be checked first so that B or B' can be read from memory. Fig.10 As shown in (b), B represents data B or B' read from the memory according to the sign of A. Fig. 9 The neg and skip circuits shown in can be used in this embodiment.

[0083] If the fusion method described in the present invention is not used, two multiplication operations are required to obtain the final C, namely:

[0084] C=B×f(A)while,

[0085] f(A)=A,for A≥0;

[0086] f(A)=αx A,for A<0.

[0087] Similarly, the PReLu function can be used for normalization-activation fusion. When A is the input and B is the normalization scaling factor, the output of the normalization-activation is f(C) = f(B x A):

[0088] F(C)=B x A, for B x A≥0;

[0089] F(C)=λ*B x A, for B x A<0.

[0090] On the other hand, the above fusion method for “ReLu–Weight” fusion and “PReLu–Weight” fusion can also be applied to ReLu-Normalization and PReLu-Normalization fusion.

[0091] Another beneficial effect of the present invention is that hardware reuse can be maximized in the activation-normalization, normalization-activation and activation-weight architecture designs.

[0092] The above is a detailed description of the implementation mode of the present invention in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation mode. Various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present invention. The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical content disclosed above to an equivalent embodiment of equivalent changes without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principle of the present invention, still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A neural network computing method, characterized in that: The method comprises: Receiving input values ​​of two functions of an operation; the operation includes a multiplication operation; the two functions are a normalization function and an activation function respectively; The normalization function includes: a layer normalization function, an instance normalization function, a group normalization function or a switchable normalization function; The multiplication operation is implemented as a multiplier, a shifter, a series of adders and shifters, or a series of AND gates and adders; Checking the range of input values ​​of at least one of the functions; checking the sign of the input value of at least one of the functions; According to the above inspection results, determine the execution strategy of the function; Determining the execution strategy of the function includes: If the value range of the input value of the checked function is positive, one of the two functions is executed in the first mode, otherwise, one of the two functions is executed in the second mode; The performing one of the two functions in the first mode includes: performing a multiplication operation; Executing one of the two functions in the second mode includes: assigning an output of one of the two functions to a set constant value; The method further comprises: predicting the range of the operation result by checking the value range of at least one input value of the function; If the range of the predicted operation result is a positive value, the function is executed, otherwise, the function is not executed; Determining the execution strategy of the function specifically includes: If the range of values ​​of the input value of the checked function indicates that the input value of the checked function is a negative value, the function is not executed; If the range of the input value of the checked function indicates that the input value of the checked function is a positive value, executing the function; Determining the execution strategy of the function includes: If the input value range of the checked first function is positive, the first coefficient is selected as the input parameter of the second function; If the input value range of the checked first function is a negative value, the second coefficient is selected as an input parameter of the second function.

2. A neural network computing method as claimed in claim 1, characterized in that: Execution of the function consists in multiplying the input value of the checked function by the selected coefficient.

3. A neural network computing method as claimed in claim 1, characterized in that: The execution of the operation also includes: If the value range of the input value of the checked first function is positive, selecting a first coefficient as input and multiplying the input value of the first function by the selected first coefficient; If the value range of the input value of the checked first function is a negative value, a first coefficient and a second coefficient are selected as inputs and the input value of the first function is multiplied by the selected first coefficient and second coefficient.

4. A neural network computing method as claimed in claim 1, characterized in that: The checking of the value range of the input value of at least one function includes checking the sign information or the sign bit of the input value of the function.

5. A neural network computing method as claimed in claim 1, characterized in that: If the operation is not performed, the generated constant result is 0.

6. A neural network computing method as claimed in claim 1, characterized in that: The activation function includes a ReLu function, a PReLu function, a ReLu6 function or a ReLuN function.

7. A neural network operation processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor comprises a plurality of multipliers, characterized in that: When the processor executes the computer program, the following steps are implemented: Receive the input values ​​of the two functions to be operated; The operation includes a multiplication operation; the two functions are respectively a normalization function and an activation function; The normalization function includes: a layer normalization function, an instance normalization function, a group normalization function or a switchable normalization function; The multiplication operation is implemented as a multiplier, a shifter, a series of adders and shifters, or a series of AND gates and adders; checking a range of values ​​of at least one input value of the function; checking the sign of an input value of at least one function; According to the above inspection results, the execution strategy of the function is determined, including: If the value range of the input value of the checked function is positive, one of the two functions is executed in the first mode, otherwise, one of the two functions is executed in the second mode; The performing one of the two functions in the first mode includes: performing a multiplication operation; Executing one of the two functions in the second mode includes: assigning an output of one of the two functions to a set constant value; The method further comprises: predicting a range of a calculation result by checking a range of a value of an input value of at least one of the functions; If the range of the predicted operation result is a positive value, the function is executed, otherwise, the function is not executed; Determining the execution strategy of the function also includes: If the range of values ​​of the input value of the checked function indicates that the input value of the checked function is a negative value, the function is not executed; If the range of the input value of the checked function indicates that the input value of the checked function is a positive value, executing the function; Determining the execution strategy of the function also includes: If the input value range of the checked first function is positive, the first coefficient is selected as the input parameter of the second function; If the input value range of the checked first function is a negative value, the second coefficient is selected as an input parameter of the second function.

8. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 6.

Images