Method and apparatus for code generation for creating program code for calculating an artificial neural network in a hardware environment

Abstract

The invention relates to a computer-implemented method for performing code generation for calculating a softmax function of a neural network, comprising the following steps: - Providing (S1) a shift s and a multiplier m for the quantized representation of the input tensors of the softmax function of the neural network; - Create (S2) a second lookup table (LUT2) to replace a nested function for calculating the EXP_ON_NEG (MUL_SAT ()) function from a CMSIS-NN library depending on the shift s and the multiplier m, using as arguments the element values ​​of an input tensor normalized to a negative range between 0 and a minimum value, where 0 is assigned to a maximum possible element value of the input tensor and the minimum value is assigned to the smallest possible element value of the input tensor; - Implement (S5) an access to the second lookup table (LUT2) in code generated to calculate the softmax function, so that it replaces the function call of the EXP_ON_NEG (MUL_SAT 0) function.

Description

Technical field

[0001] The invention relates to the implementation of program code on a hardware environment, such as microcontroller-controlled control units and the like. The invention further relates to methods for the efficient calculation of a softmax function. Technical background

[0002] Certain hardware environments, such as microcontrollers in electronic control units (ECUs), require the creation of customized executable program code to accommodate the specific characteristics and limitations of that hardware environment. For example, the available amount of RAM that the microcontroller can directly access may be limited, or the processor speed may be restricted.

[0003] The computational steps for calculating the corresponding layers of neural networks require considerable computing power. Existing code generators determine which algorithms are used to calculate the individual layers during code generation. In particular, a trade-off between the required memory footprint and the code generator's processing speed is often necessary.

[0004] Softmax is a widely used type of layer in neural networks. Softmax generates a probability distribution from an arbitrary vector of input values ​​and is therefore frequently used in CNN classification networks, but also in modern transformer networks.

[0005] Computation of a softmax layer is very computationally intensive, and therefore attempts are made to avoid the computation of the softmax layer whenever possible, as shown by the approach from Lu, Jiachen et al. “SOFT: Softmax-free Transformer with Linear Complexity.” Neural Information Processing Systems (2021).

[0006] The object of the present invention is to provide an improved method for accelerating softmax calculation in quantized models for use in a code generator. Disclosure of the invention

[0007] This problem is solved by the method for performing code generation to create a code for the computation of a neural network according to claim 1 and by the device according to the dependent claim.

[0008] Further details are specified in the dependent claims.

[0009] According to a first aspect, a computer-implemented procedure for performing code generation for calculating a softmax function of a neural network is provided, with the following steps: - Providing a shift s and a multiplier m for the quantized representation of the input tensors of the computation steps of the neural network. - Create a second lookup table to replace a nested function for calculating the EXP_ON_NEG (MUL_SAT ()) function from a CMSIS-NN library depending on the shift s and the multiplier m, using as arguments the element values ​​of an input tensor normalized to a negative range between 0 and a minimum value, where 0 is assigned to a maximum possible element value of the input tensor and the minimum value is assigned to the smallest possible element value of the input tensor; - Implementing access to the lookup table in code generated to calculate the Softmax function, so that it replaces the function call of the EXP_ON_NEG (MUL_SAT ()) function.

[0010] Computing a softmax layer in neural networks is very computationally expensive. For float networks, the softmax function is described as... softmax(x)i=exi∑jexj The first step involves subtracting the element values ​​xi of an input tensor from the maximum value max ({x) before the softmax calculation. j}) of the input tensor x normalized, as di=xi−max({xj}) so that all element values ​​in the input tensor d are 0 or less, and the softmax function is then considered softmax(x)i=edi∑jedj calculated.

[0011] For quantized networks, the following relationship emerges: di≅xiq×2s×m231, between the float value x i and the quantized value xiq and similarly di≅diq×2s×m231 with the shift s and the multiplier m.

[0012] The above procedure provides a code generation method for implementing a quantized softmax function using one or more lookup tables, which improves the computation time by an order of magnitude.

[0013] In the Fig. 1a-1d is a Softmax implementation in CMSIS-NN presented as state of the art, see Liangzhen Lai and Naveen Suda and Vikas Chandra (2018), “CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs”, ArXiv

[0014] In the normalized version of Softmax, all exponents are 0 or negative. CMSIS-NN implements the function EXP_ON_NEG (see Fig. 1b) for calculating the exponential of negative arguments and a saturated multiplication MUL_SAT (see section 1c) to avoid numerical overflow, i.e. the function MUL_SAT ensures that the result of the multiplication of two int32 integers, which may otherwise lie outside the range of int32 numbers, always remains restricted to the permissible int32 range.

[0015] CMSIS-NN uses these two functions to establish the relationship E(djq,s,m)=edjq×2s×m=EXP_ON_NEG(MUL_SAT(djq×2s,m)).

[0016] The sum in the denominator of the above formula for calculating softmax Σ j e dj This could still lead to a numerical overflow. For this reason, CMSIS-NN scales by a factor of 2 during summation using the DIV_POW2 function. ACCUM_BITS , so that ∑jescalesdj=∑jDIV_POW2(E(djq,s,m),2ACCUM_BITS)with ACCUMBITS=12 corresponds to line 23 of the code. Fig. 1a shown.

[0017] The next step in CMSIS-NN is the evaluation of the inverse of the denominator Σ. j e dj as shifted_scaled=ONE_OVER1((sum>0?sum< <headroom:0)−(1<<31) where headroom=CLZ(∑jedj) (see lines 26-28 in the code of Fig. 1a).

[0018] With bits_over_mit=ACCUM_BITS−headroom+23 (see line 30 in the code of the Fig. 1a), is the result then softmax(djq)=DIV_POW2(MUL_SAT(shifted_scaled,E(djq,s,m)),bits_over_unit), (see lines 34-37 of the code of Fig. 1a). Finally, the result must be converted into an 8-bit range (see lines 38-41 in the code of the Fig. 1a).

[0019] For reference, the subfunctions EXP_ON_NEG are in Fig. 1b, MUL_SAT in Fig. 1c and DIV_POW2 in Fig. 1d shown.

[0020] The above method uses the replacement of at least one of the calculations of the conventional softmax function with a lookup table for the implementation of the softmax function in the code generator.

[0021] The size E(djq,s,m) in the equation above depends on djq, the shift s (shift, mask) and the multiplier m (mult). The shift s and the multiplier m are constant and known at the time of code generation, so that for each softmax function in the network only djq a variable. In this way, a lookup table can be created that depends on djq all possible values ​​of E(djq,s,m) contains and is the second lookup table in the above approach: LUT2[djq]=E(djq,s,m)=EXP_ON_NEG(MUL_SAT(edjq×22m)).

[0022] This lookup table LUT2 contains 32-bit values.

[0023] The procedure may include the following further steps: - Creating an initial lookup table to replace a nested function DIV_POW2(EXP_ON_NEG(MUL_SAT () )) from the CMSIS-NN library depending on the shift s and the multiplier m, where the argument is the element values ​​of an input tensor normalized to a negative range between 0 and a minimum value, where 0 is assigned to a maximum possible element value of the input tensor and the minimum value is assigned to the smallest possible element value of the input tensor; - Implementing access to the first lookup table in code generated to calculate the softmax function, so that it corresponds to the function call of the DIV_POW2(EXP_ON_NEG(MUL_SAT () )) function.

[0024] The calculation E(djq,s,m) The sum is not constant for every summand. Therefore, a first lookup table can be used to calculate each summand in the above relation. ∑Jescaleddjq=∑JDIV_POW2(E(djq,s,m),2ACCUM_BITS) are created that are considered LUT1[djq]=escaleddq=DIV_POW2(EXP_ON_NEG(MUL_SAT(djq×2s,m)),2ACCUM_BITS) is designated and in turn contains 32-bit values.

[0025] The two lookup tables replace the much slower calculations in the standard implementation of the Fig. 1a-1d.

[0026] The need for two lookup tables arises because the above relation softmax(djq)=DIV_POW2(MUL_SAT(shifted_scaled,E(djq,s,m)),bits_over_unit), by shifted_scaled escaleddjq and depends on `bits_over_unit`, which is calculated from all input data. Therefore, this expression cannot be fully expressed using a lookup table. The procedure replaces the calculation with lookup tables wherever possible.

[0027] The two lookup tables are different for each combination of shift s and multiplier m, which result from the quantization parameters of the respective softmax layer in the given model. The quantization parameters are a result of quantization performed with a separate software tool independent of the method, such as the TensorFlow Lite model converter. Therefore, this technique cannot be used in a library implementation of a softmax function, but only in a code generator that can create the lookup tables for each specific softmax function during code generation.

[0028] The value diff calculated in lines 21 and 32 in both Softmax versions is zero for the largest element value in the input tensor and negative for all other values ​​with diff ∈ [-255, 0].

[0029] The smallest element values ​​`input[col]` in the input tensor result in the largest negative values ​​of `diff`. It is possible that the smallest negative values ​​of `diff` are so small that they cannot be computed in the quantized representation and are therefore ignored. Both the CMSIS-NN version and the LUT-based version of Softmax use the variable `diff_min` to define a corresponding threshold. This corresponds to a negative value calculated from the quantization properties of the respective Softmax layer in the network.

[0030] Since the equations above are only evaluated if diff >= diff_min, both lookup tables may only contain values ​​for diff [diff_min, 0], meaning both lookup tables have a maximum size of 256 entries, corresponding to 1 kB of required memory each. However, if diff_min > -255, the size of the lookup tables will be less than 256 entries. Furthermore, it is possible that the calculation formula for the first lookup table for the smallest values ​​of djq, This means the first entries in the lookup table return 0. These values ​​are then removed from the lookup table, and the number of removed zeros is stored in a table value called LUT1_offset, which is an additional function argument for the new function.

[0031] With these optimizations, the generated lookup tables are always as small as possible, and the required storage space in the data store is minimized.

[0032] According to another aspect, a method for calculating a softmax function is provided using code generated by one of the above methods.

[0033] According to another aspect, a device for carrying out one of the above procedures is provided. Brief description of the drawings

[0034] The embodiments are explained in more detail below with reference to the accompanying drawings. These show: Fig. 1a-1d a code for a conventional CMSIS-NN algorithm to compute a softmax function; Fig. 2 a schematic representation of a platform for code generation and implementation in a hardware environment; Fig. 3. A flowchart illustrating the process steps for code generation for a softmax function; Fig. 4. A code for a CMSIS-NN algorithm adapted by the code generator to calculate a softmax function. Description of embodiments

[0035] Fig. Figure 2 shows a block diagram of a platform 1 for performing code generation and implementing the generated program code in a hardware environment 2. The hardware environment corresponds, for example, to a control unit with a microcontroller, microprocessor, or the like. Code generation takes place on a conventional computer 3 or workstation, based on the configuration of a neural network. The computer 3 is configured to perform memory allocation and code generation according to a predefined sequence of computation steps. Memory allocation first assigns memory areas for each computation step of the neural network to hold at least one input data block and at least one output data block. Code generation then generates the code for executing the calculations of each computation step.

[0036] Once the code is generated, it is transferred to the hardware environment 2 and implemented or executed there.

[0037] In conjunction with the flowchart of the Fig. Section 3 describes a method for generating a code to calculate a softmax function, where one or two functions from the set in the Fig. The Softmax function shown in 1a-1d is replaced by a first lookup table LUT1 and a second lookup table LUT2.

[0038] In step S1, a shift s and a multiplier m are determined for the 8-bit representation of the input tensors of the computation steps of the neural network.

[0039] The values ​​for s and m are calculated in a known manner from the quantization parameters of the respective softmax layer, specifically the scaling of the input tensor, and the selectable parameter beta of the softmax layers. This calculation is analogous to that used in other deployment solutions, such as TensorFlow Lite Micro. This process is performed separately for each layer of the neural network, including both softmax and all other layers. The calculation procedure can be implemented analogously to, for example, TensorFlow Lite Micro.

[0040] In step S2, the calculation rule for lines 34-37 is used. const in32_t res=DIV_POW2(MUL_SAT(shifted_scale,EXP_ON_NEG( MUL_SAT(diff*mask,mult))),bits_over_unit)+NN_Q7_MIN; in particular the EXP_ON_NEG algorithm of the conventional algorithm of Fig. 1a. A lookup table LUT2 was created.

[0041] The size E(djq,s,m) in the equation above LUT2[djq]=E(djq,s,m)=EXP_ON_NEG(MUL_SAT(edjq×22m)). depends on djq, the displacement s and the multiplier m.

[0042] The shift s and the multiplier m can be initially determined and are constant based on the quantization parameters of the softmax layer in the model, so that for each softmax function in the network only diq a variable. In this way, a lookup table can be created that depends on diq all possible values ​​of E(djq,s,m) contains.

[0043] Since the values xiq of the quantized input vector int8 format, with the value range xiq∈[−128,127] have and the values diq by subtracting the maximum value from the xiq result, can the diq maximum values diq∈[−255,0] assume.

[0044] The value diff calculated in lines 21 and 32 in both Softmax versions corresponds to this value. diq. Accordingly, diff is zero for the largest element value in the input tensor and negative for all other values. diff∈[−255,0].

[0045] To create the table, the expression is therefore used for given s and m. E(djq,s,m) for all these values diq or evaluated using diff.

[0046] This lookup table LUT2 contains 32-bit values.

[0047] In step S3, the code generator implements access to the second lookup table LUT2 in the program code for calculating the EXP_ON_NEG algorithm of the Softmax function, which is in Fig. 1b is shown. This is in the code of the Fig. 4 shown in lines 34-37.

[0048] In step S4, the first lookup table LUT1 is created, which contains the calculation rule. sum+=DIV_POW2(EXP_ON_NEG(MUL_SAT(diff*mask,mult)),ACCUM_BITS); The location to replace line 23 has been determined.

[0049] The calculation E(djq,s,m) The sum is not constant for every summand. Therefore, the first lookup table can be used for the entire calculation of each summand in the above relation. ∑jescaleddjq=∑jDIV_POW2(E(djq,s,m),2ACCUM_BITS) are created that are considered LUT1[djq]=escaleddq=DIV_POW2(EXP_ON_NEG(MUL_SAT(djq×2s,m)),2ACCUM_BITS) is designated and in turn contains 32-bit values.

[0050] Analogous to the calculation of LUT2, the relation is used to create the LUT1 table. LUT1[djq]=escaleddq=DIV_POW2(EXP_ON_NEG(MUL_SAT(djq×2s,m)),2ACCUM_BITS) for all possible values ​​of diq evaluated.

[0051] The smallest element values ​​`input[col]` in the input tensor result in the largest negative values ​​of `diff`. It is possible that the smallest negative values ​​of `diff` are so small that they cannot be computed in the quantized representation and are therefore ignored. Both the CMSIS-NN version and the LUT-based version of Softmax use the variable `diff_min` to define a corresponding threshold. This corresponds to a negative value that, analogous to the multiplier `m` and the shift `s`, is calculated from or predefined by the quantization properties of the respective Softmax layer in the network.

[0052] Since the equations above are only evaluated if diff >= diff_min, both lookup tables may only contain values ​​for diff ∈ [diff_min, 0], meaning both lookup tables have a maximum size of 256 entries, corresponding to 1 kB of required memory each. However, if diff_min > -255, the size of the lookup tables will be less than 256 entries. Furthermore, it is possible that the calculation formula for the first lookup table for the smallest values ​​of djq, This means the first entries in the lookup table result in 0. These values ​​are then removed from the lookup table, and the number of removed zeros is stored in an integer value LUT1_offset, which is an additional function argument for the new function.

[0053] In step S5, the code generator implements access to the first lookup table LUT1 in the program code for calculating the nested functions algorithm. DIV_POW2(EXP_ON_NEG(MUL_SAT(djq×2s,m)),2ACCUM_BITS. the Softmax function.

Claims

[1] Computer-implemented method for performing code generation for computation of a softmax function of a neural network, comprising the following steps: - Providing (S1) a shift s and a multiplier m for the quantized representation of the input tensors of the softmax function of the neural network; - Create (S2) a second lookup table (LUT2) to replace a nested function for calculating the EXP_ON_NEG (MUL_SAT ()) function from a CMSIS-NN library depending on the shift s and the multiplier m, using as arguments the element values ​​of an input tensor normalized to a negative range between 0 and a minimum value, where 0 is assigned to a maximum possible element value of the input tensor and the minimum value is assigned to the smallest possible element value of the input tensor; - Implement (S5) an access to the second lookup table (LUT2) in code generated to calculate the softmax function, so that it replaces the function call of the EXP_ON_NEG (MUL_SAT 0) function. [2] Method according to claim 1, comprising the further steps: - Create (S4) a first lookup table to replace a nested function DIV_POW2(EXP_ON_NEG(MUL_SAT () )) from the CMSIS-NN library depending on the shift s and the multiplier m, where the argument is the element values ​​of an input tensor normalized to a negative range between 0 and a minimum value, where 0 is assigned to a maximum possible element value of the input tensor and the minimum value is assigned to the smallest possible element value of the input tensor; - Implementing access to the first lookup table (LUT1) in code generated to calculate the softmax function, so that it replaces the function call of the DIV_POW2(EXP_ON_NEG(MUL_SAT () )) function. [3] Method for calculating a softmax function using code generated by one of the methods according to claims 1 to 2. [4] Device for carrying out one of the methods according to one of claims 1 to 2. [5] Computer program product comprising instructions which, when the program is executed by at least one data processing device, cause it to perform the steps of the method according to one of claims 1 to 2. [6] Machine-readable storage medium comprising instructions which, when executed by at least one data processing device, cause it to perform the steps of the method according to one of claims 1 to 2.

Images