Method and apparatus for performing code generation for creating program code for artificial neural network computing in hardware environment

Abstract

A computer-implemented method for performing code generation for computing a Softmax function of a neural network, having: providing a displacement s and a multiplier m for a quantized representation of an input tensor of the Softmax function of the neural network; from this, a second lookup table (LUT2) is created for replacing a nested function for calculating an EXPONNEG (MULSAT ()) function from the CMSIS-NN library, in which element values of the input tensor normalized to a negative value field between 0 and a minimum value are used as arguments, wherein 0 is assigned to the maximum possible element value of the input tensor and the minimum value is assigned to the minimum element value of the input tensor; access to the second lookup table (LUT2) is implemented into the code generated for computing the Softmax function such that the code replaces the function call of the EXPONEG (MULSAT ()) function.

Description

Technical Field

[0001] This invention relates to the implementation of program code in a hardware environment, such as a control device controlled by a microcontroller. Furthermore, this invention relates to a method for efficiently calculating the Softmax function. Background Technology

[0002] Specific hardware environments, such as microcontrollers in control devices, require the creation of matching executable code to account for the characteristics and limitations of that specific hardware environment. Therefore, in particular, the available size of the working memory that the microcontroller can directly access may be limited, or the processor speed may be finite.

[0003] The computational steps used to compute the corresponding layers of a neural network require enormous computational overhead. Existing code generators specify during code generation which algorithms will be used for the computation of each layer. Here, the code generator often makes choices, particularly regarding the trade-offs between the required storage in working memory and computational speed.

[0004] Softmax is a widely distributed layer type in neural networks. Softmax generates a probability distribution from an arbitrary vector of input values, and is therefore commonly used in CNN classification networks, but also in modern transformer networks.

[0005] The computation of the Softmax layer is very computationally expensive, so attempts are made to avoid computation of the Softmax layer as much as possible, such as the scheme in "SOFT: Softmax-free Transformer with Linear Complexity." Neural Information Processing Systems (2021) by Lu, Jiachen et al.

[0006] The objective of this invention is to provide an improved method for accelerating Softmax computation in quantization models for use in code generators. Summary of the Invention

[0007] This task is accomplished by the method for executing code generation for creating code for neural network computation as described in claim 1, and by the device as described in the parallel claims.

[0008] Other design options are described in the dependent claims.

[0009] According to the first aspect, a computer implementation method for generating code for executing a Softmax function for computing a neural network is provided, comprising the following steps: - Provides a shift s and a multiplier m for the quantization representation of the input tensor of the Softmax function of the neural network; - A second lookup table is created based on the displacement s and the multiplier m to replace the nested function used to compute the EXP_ON_NEG(MUL_SAT()) function from the CMSIS-NN library, where the element values ​​of the input tensor normalized to the negative domain between 0 and the minimum value are used as the independent variables, where 0 corresponds to the largest possible element value of the input tensor and the minimum value corresponds to the smallest possible element value of the input tensor; - Implement access to the second lookup table into the code generated for calculating the Softmax function, such that this code replaces the function call to EXP_ON_NEG(MUL_SAT()).

[0010] The computation of the Softmax layer in a neural network is very expensive. For floating-point networks, the Softmax function is calculated as follows: Here, in the first step, before the Softmax calculation, the maximum value of the input tensor x is used. Normalize the element values ​​xi of the input tensor, for example Make the input tensor d All elements in the set are 0 or less, and then the Softmax function is calculated as follows: .

[0011] For quantized networks, the floating-point value is obtained using the displacement s and the multiplier m. With quantized value The relationship between them: , And similarly, it can be concluded that... .

[0012] The above method provides a code generation approach for implementing a quantized Softmax function using one or more lookup tables, which improves the computation duration by an order of magnitude.

[0013] exist Figures 1a-1dThe softmax implementation in CMSIS-NN is shown as prior art; see Liangzhen Lai and Naveen Suda and Vikas Chandra (2018), “CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs,” ArXiv. In the standardized version of Softmax, all exponents are 0 or negative. The CMSIS-NN implementation function EXP_ON_NEG (see...) Figure 1b The function is used to compute the exponential function of the negative independent variable and implements saturated multiplication MUL_SAT (see part 1c) to avoid numerical overflow. That is, the function MUL_SAT ensures that the result of multiplying two int32 integers (which may otherwise be outside the range of int32 numbers) is always restricted to the allowed range of int32 values.

[0014] Using these two functions, CMSIS-NN utilizes relationships .

[0015] The summation in the denominator of the above calculation formula used for Softmax Numerical overflow may still occur. For this reason, CMSIS-NN uses a factor of 2 during summation using the function DIV_POW2. ACCUM_BITS Scaling is performed so that the corresponding ,in , As in Figure 1a As shown in line 23 of the code.

[0016] The next step in CMSIS-NN is to reduce the denominator. The reciprocal evaluation is in (See) Figure 1a Lines 26-28 in the code.

[0017] use (See) Figure 1a In line 30 of the code, the result is , (See) Figure 1a Lines 34-37 in the code). Finally, the result must be placed in an 8-bit range (see...). Figure 1aLines 38-41 in the code.

[0018] The sub-function is shown for reference: Figure 1b EXP_ON_NEG in Figure 1c MUL_SAT and Figure 1d DIV_POW2 in the example.

[0019] The above method utilizes a lookup table to replace at least one calculation of the traditional Softmax function in order to implement the Softmax function in the code generator.

[0020] Parameters in the above equation It depends on the shift s (shift, mask) and the multiplier m (mult). The shift s and the multiplier m are constant and known at the time of code generation, such that for each Softmax function in the network, only It is a variable. A lookup table can be created in this way, which is related to... Relatedly includes All possible values, and the lookup table is the second lookup table in the above scheme: .

[0021] The lookup table LUT2 contains 32-bit values.

[0022] The method may include additional steps: - Create a first lookup table based on the displacement s and the multiplier m, to replace the nested function DIV_POW2(EXP_ON_NEG(MUL_SAT())) from the CMSIS-NN library, where the element values ​​of the input tensor normalized to the negative domain between 0 and the minimum value are used as arguments, where 0 corresponds to the largest possible element value of the input tensor and the minimum value corresponds to the smallest possible element value of the input tensor; - Implement access to the first lookup table into the code generated for calculating the Softmax function, such that the code corresponds to a function call to the DIV_POW2(EXP_ON_NEG(MUL_SAT())) function.

[0023] During calculation, Not every term in the summation is constant. Therefore, it is possible to consider the relationship described above. The total calculation for each added item in the table creates a first lookup table, which is represented as follows: And it again contains 32-bit values.

[0024] These two lookup tables replace the existing ones. Figures 1a-1d The computation is much slower in the standard implementation.

[0025] Because of the above relationship It depends on the combination of all input data. and Therefore, the necessity of two lookup tables is evident. For this reason, the expression cannot be fully expressed using a single lookup table. This method replaces computation with lookup tables as much as possible.

[0026] These two lookup tables are different for each combination of displacement *s* and multiplier *m*, which are derived from the quantization parameters of the corresponding Softmax layer in a given model. The quantization parameters are the result of quantization using other software tools unrelated to this method, such as the TensorFlow Lite model converter. For this reason, this technique cannot be used in library implementations of Softmax functions, but only in code generators that can produce lookup tables for each specific Softmax function during code generation.

[0027] In both Softmax versions, the value diff calculated in lines 21 and 32 is zero for the largest element in the input tensor and negative for all other values, where diff ∈ [-255, 0].

[0028] The minimum element value in the input tensor, input[col], yields the maximum negative value of the diff. It's possible that the minimum negative value of the diff is so small that it cannot be computed in the quantized representation and is therefore ignored. Both the CMSIS-NN version of Softmax and the LUT-based version use the variable diff_min to specify the corresponding threshold. This variable corresponds to the negative value computed from the quantized properties of the corresponding Softmax layer in the network.

[0029] Since the above equation is evaluated only if diff >= diff_min, the two lookup tables can only contain values ​​for diff ∈ [diff_min, 0], meaning each lookup table has a maximum size of 256 entries, which corresponds to a required memory of 1kB each. However, if diff_min > -255, the size of the lookup tables will be less than 256 entries. Furthermore, it is possible that the first lookup table... The minimum value, i.e., the first entry in the lookup table, is calculated to be 0. These values ​​are then removed from the lookup table, and the number of zeros removed is stored in the table value LUT1_offset, which is an additional function argument used for the new function.

[0030] Using these optimizations, the resulting lookup table is always as small as possible, and the required storage space in the data storage is minimized.

[0031] According to another approach, a method is provided for calculating the Softmax function using code generated using one of the methods described above.

[0032] According to another aspect, a device is provided for performing one of the above methods. Attached Figure Description

[0033] The embodiments are described in more detail below with reference to the accompanying drawings. Wherein: Figures 1a-1d The code for the traditional CMSIS-NN algorithm used to compute the Softmax function is shown; Figure 2 A schematic diagram of the platform used for code generation and implementation in the hardware environment is shown; Figure 3 This diagram illustrates the methodological steps used to generate code for the Softmax function. Figure 4 The code for the CMSIS-NN algorithm used to compute the Softmax function is shown, adapted by the code generator. Detailed Implementation

[0034] Figure 2 A block diagram is shown of a platform 1 for performing code generation and implementing the generated program code in a hardware environment 2. This hardware environment corresponds, for example, to a control device having a microcontroller, microprocessor, etc. Code generation is performed on a conventional computer 3 or workstation with a pre-given neural network configuration. The computer 3 is configured to perform memory planning and code generation in a pre-given order of computational steps, wherein memory planning first performs the configuration of memory regions for accommodating at least one input data block and at least one output data block for each computational step of the neural network. Code generation specifies the generation of code for implementing the computations of each computational step.

[0035] Once the code is generated, it is transferred to hardware environment 2 and implemented or executed there.

[0036] Combination Figure 3The flowchart describes a method for generating code to compute the Softmax function, where a first lookup table LUT1 and a second lookup table LUT2 are used to replace the code derived from... Figures 1a-1d One or both of the Softmax functions shown.

[0037] In step S1, the displacement s and multiplier m are determined by the 8-bit representation of the input tensor for the computation steps of the neural network.

[0038] The values ​​of s and m are computed in a manner known per se by the quantization parameters of the corresponding Softmax layer, specifically the scaling of the input tensor, and by the optional parameter beta of the Softmax layer. This is similar to the computations performed in other deployment solutions, such as TensorFlow Lite Micro. This is done separately for each layer of the neural network, not only the softmax layer but all other layers. The computation rules can be implemented similarly to, for example, TensorFlow Lite Micro.

[0039] In step S2, the calculation rules for rows 34-37 are as follows. ,in particular Figure 1a The traditional EXP_ON_NEG algorithm creates the lookup table LUT2.

[0040] The above equation Parameters in depending on Displacement s and multiplier m.

[0041] The displacement s and multiplier m are initially determined and constant based on the quantization parameters of the Softmax layer in the model, such that for each Softmax function in the network, only It is a variable. This method allows the creation of lookup tables, which are related to... Relatedly includes All possible values ​​of .

[0042] Due to the value of the quantized input vector It has a range The int8 format, and the value By from The result is obtained by subtracting the maximum value from the middle value. Maximum possible value .

[0043] The value diff calculated in lines 21 and 32 in both Softmax versions corresponds to this value. Correspondingly, diff equals zero for the largest element in the input tensor and is negative for all other elements, where diff∈[-255,0].

[0044] In order to create a lookup table, therefore for all these values, given s and m, ... Or, in other words, using diff to evaluate the expression. .

[0045] The lookup table LUT2 contains 32-bit values.

[0046] In step S3, the code generator implements access to the second lookup table LUT2 in the program code for the EXP_ON_NEG algorithm used to compute the Softmax function, which is shown in the figure. Figure 1b In China. This is in Figure 4 The code is shown in lines 34-37.

[0047] In step S4, it is determined that a first lookup table LUT1 is created, which should replace the calculation rule of row 23. .

[0048] During calculation, Not every term in the summation is constant. Therefore, it is possible to consider the relationship described above. The total calculation for each added item in the table creates a first lookup table, which is represented as follows: And it again contains 32-bit values.

[0049] Similar to LUT2 computation, in order to create the LUT1 table, for To evaluate the relationship using all possible values .

[0050] The minimum element value in the input tensor, input[col], yields the maximum negative value of the diff. It's possible that the minimum negative values ​​of the diff are so small that they cannot be computed in the quantized representation and are therefore ignored. Both the CMSIS-NN version and the LUT-based version of Softmax use the variable diff_min to specify the corresponding threshold. This variable corresponds to a negative value, similar to the multiplier m and shift s, that is computed from the quantization properties of the corresponding Softmax layer in the network or pre-defined.

[0051] Since the above equation is evaluated only if diff >= diff_min, the two lookup tables can only contain values ​​for diff ∈ [diff_min, 0], meaning each lookup table has a maximum size of 256 entries, which corresponds to a required memory of 1kB each. However, if diff_min > -255, the size of the lookup tables will be less than 256 entries. Furthermore, it is possible that the first lookup table... The minimum value, i.e., the first entry in the lookup table, is calculated to be 0. These values ​​are then removed from the lookup table, and the number of zeros removed is stored in the integer value LUT1_offset, which is an additional function argument used for the new function.

[0052] In step S5, the code generator uses nested functions for calculating the Softmax function. The algorithm's program code implements access to the first lookup table LUT1.

Claims

1. A computer implementation method for generating code to execute a Softmax function for computing a neural network, comprising the following steps: - Provides (S1) shifts s and multipliers m for the quantization representation of the input tensor of the Softmax function of the neural network; - Create a second lookup table (LUT2) based on the displacement s and multiplier m, to replace the nested function used to compute the EXP_ON_NEG(MUL_SAT()) function from the CMSIS-NN library, where the element values ​​of the input tensor normalized to the negative domain between 0 and the minimum value are used as the independent variables, where 0 corresponds to the largest possible element value of the input tensor and the minimum value corresponds to the smallest possible element value of the input tensor; - Implement access to the second lookup table (LUT2) (S5) into the code generated for calculating the Softmax function, such that the code replaces the function call of the EXP_ON_NEG(MUL_SAT()) function.

2. The method according to claim 1, further comprising the following steps: - Create a first lookup table (S4) based on the displacement s and multiplier m, in place of the nested function DIV_POW2(EXP_ON_NEG(MUL_SAT())) from the CMSIS-NN library, where the element values ​​of the input tensor normalized to the negative domain between 0 and the minimum value are used as the arguments, where 0 corresponds to the largest possible element value of the input tensor and the minimum value corresponds to the smallest possible element value of the input tensor; - Implement access to the first lookup table (LUT1) into the code generated for calculating the Softmax function, such that the code replaces the function call of DIV_POW2(EXP_ON_NEG(MUL_SAT())).

3. A method for calculating a Softmax function using code, said code being generated using any one of the methods according to claims 1 to 2.

4. An apparatus for performing the method according to any one of claims 1 to 2.

5. A computer program product comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 2.

6. A machine-readable storage medium comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 2.

Images