Method and apparatus for code generation for the computation of a neural network with recalibratable parameters

Abstract

The invention relates to a computer-implemented method for performing code generation to determine a recalibratable code for the computation of a neural network, comprising the following steps: - Providing (S1) source code for implementing the neural network, which does not contain the network parameters defining the neural network, i.e., its configuration and parameterization, as constants, wherein the source code is designed to access a parameter data set with predefined network parameters, containing all the parameters required to recalibrate the neural network; - Compiling (S2) the source code so that a program code for the desired hardware environment is generated; - Providing (S3) the network parameters so that they are accessible through the program code; - Implementing (S4) the generated program code in the hardware environment.

Description

Technical field

[0001] The invention relates to the code generation of code for the execution of a neural network, in particular with the possibility of simple recalibration of the network parameters of the neural network. Technical background

[0002] Neural networks or similar data-driven functions are used in a wide range of applications. In the field of embedded systems, their application to sensor signals such as acceleration and velocity data, radar signals, power consumption, etc., is widespread in order to predict a derived quantity. The neural network represents a function f derived from the input values. x→ the output values y→ accordingly y→=f(x→) calculated. How well the function f can approximate the actual output values ​​depends, on the one hand, on the architecture of the neural network, i.e., the hyperparameters that specify the type, number, and dimensions of the individual layers, and on the network parameters that specify weight values, bias values, quantization parameters, kernel values, and the like for individual nodes (neurons).

[0003] While the architecture of a neural network represents a conscious design decision, the network parameters - weighting values, bias values ​​and quantization parameters - are determined during training through an optimization process.

[0004] In many areas where embedded systems are used, the developed software must be adapted to different product variants. In the automotive sector, for example, firmware for a control unit can be adapted to different vehicle variants.

[0005] This is typically achieved by retraining the neural network rather than recompiling the entire firmware, simply by adjusting the network parameters with new values. While the specific architecture of a neural network remains unchanged after a development phase in such use cases—for example, to obtain a one-time approval for the overall system—it is desirable to be able to adjust the trained network parameters subsequently.

[0006] Data-driven models in the automotive sector are frequently represented by one-dimensional or multi-dimensional characteristic maps. These maps can be subsequently adapted to specific vehicle characteristics, such as a particular engine. If this is to be achieved using neural networks, one option is to use a library that implements the individual layers of neural networks. The specific neural network can then be stored in the system, for example, as a flat buffer, and replaced as needed.

[0007] The advantage of this approach is its high degree of flexibility: neither the network parameters nor the specific architecture of the neural network (for network description) need to be defined. Both can be changed even after the firmware has been compiled. The disadvantage of such an approach is its low efficiency, as flash memory is reserved for the maximum size of the network description, even if much smaller neural networks are used in practice. The same applies to the required amount of RAM and also to the required processing time if the neural network is to run in a time-critical task.

[0008] Furthermore, using a library largely prevents the optimized implementation of the neural network on the desired hardware environment. If the specific architecture of the neural network is known, specific layer configurations can often be implemented much more efficiently using a code generator.

[0009] The object of the present invention is to provide a method for code generation for the implementation of a neural network with a defined architecture, in order to implement it in a hardware environment as optimally as possible with regard to runtime and memory consumption. Furthermore, if a parameter of the installed neural network is changed, it should be possible to modify it without changing the code or recompiling. Disclosure of the invention

[0010] This problem is solved by the code generation method according to claim 1 and a corresponding device according to the dependent claim.

[0011] Further details are specified in the dependent claims.

[0012] According to a first aspect, a computer-implemented procedure for performing code generation to determine a recalibratable code for the computation of a neural network is provided, with the following steps: - Providing source code for implementing the neural network, which does not contain the network parameters defining the neural network, i.e., its configuration and parameterization, as constants, wherein the source code is designed to access a parameter dataset with predefined network parameters, containing all the parameters required to recalibrate the neural network; - Compiling the source code so that program code for the desired hardware environment is generated; - Providing the network parameters so that they are accessible through the program code; - Implementing the generated program code in the hardware environment.

[0013] The above method involves providing source code for implementing the neural network that does not contain the network parameters defining the neural network—that is, its configuration and parameterization—as constants. For example, the network parameters can be stored in a file and read in at runtime so that they are used for the predetermined architecture (defined via hyperparameters). Alternatively, the network parameters can be stored in flash memory or other data storage, as is already done for characteristic curves.

[0014] In this way, the network parameters are not compiled as part of the source code, but the code is generated in such a way that a variable parameterization of a neural network can be read in using a parameter data set.

[0015] In this way, the trained parameters of the neural network are not part of the program code that is immutable after compilation, but can be provided separately.

[0016] This allows for easy recalibration of the network parameter set by simply providing it as a file or in a data storage device. The network parameter set can be referenced using a simple address pointer.

[0017] To make the generated code flexibly calibratable, several changes are necessary compared to conventional code generation. The parameter set can include a C struct with pointers to the network parameters or some other specification for initializing the pointers. How these pointers are initialized depends on the specific implementation on the target system's hardware environment.

[0018] For quantized networks, additional requirements arise for the method. Each layer in a quantized network has one or more scale parameters for the input vector, one or more scale parameters for the output vector, and one or more scale parameters for the weights, since for tensors in quantized layers in neural networks, it must also be specified which float32 value range corresponds to the quantized values ​​of the respective tensor.

[0019] Each scale parameter s float32is approximated by two integer values, a shift value s and a multiplier m. The relationship is given by sfloat32≈m×2s231.

[0020] In conventional program code, the shift value s and the multiplier m can be implemented as constant values ​​in the generated code. However, in calibratable code, they must be flexible because they depend on the trained network parameters. Therefore, the parameter set must include the scale parameter in addition to the network parameters. These parameters must then be passed as function arguments to the layer functions (functions for calculating the layers of the neural network).

[0021] The same applies to the offset of the input and output data, which indicates where the zero point lies within the quantized value range, and the minimum and maximum values ​​according to the activation function used.

[0022] If shift s, multiplier m and offsets of the input and output data are no longer defined, the implementations of certain helper functions in the generated code also change.

[0023] For example, if the sign of the shift s is defined at generation time in the Embedded AI Coder code generator, the code generator can use an optimized requantization function depending on whether the shift s is positive or negative. Another point concerns layers with activation functions such as RELU. RELU sets negative output values ​​to zero. In quantized networks, the zero point during quantization is therefore set to the respective minimum value of the quantization data type whenever possible. In the case of int8-quantized networks, this is -128. In these cases, the requantized values ​​can be restricted to the legal range q ∈ [-128..127] using an optimized function SSAT, provided the target architecture offers such a function.

[0024] In contrast, calibratable code cannot make assumptions about the signs of the shifts or the zero point of layers with a RELU activation function. The generated code must then use more general implementations for the requantization function instead of SSAT.

[0025] When calibratable code is recalibrated with new network parameters, the code generator must ensure that the new parameter data matches the generated code. To guarantee this, all topological properties of the network and all layers, as well as any code generator options, must be stored for all code generations. The necessary information includes the topology or architecture of the neural network—that is, complete information about which layers consume the output of which layers as input; the topological properties of each individual layer, such as the dimensions of matrices in FullyConnected layers; kernel sizes, dilation, padding, and stride for convolutions; hyperparameters that are not trained, such as the a-value of a LeakyRelu; and the code generator options that affect the generated code.

[0026] When a neural network is calibrated, this information must be compared to the original and must be identical. Additionally, code can be generated again in each calibration step and compared to the original, of course without recompiling it. Brief description of the drawings

[0027] The embodiments are explained in more detail below with reference to the attached drawings. Fig. 1 a schematic representation of a platform for code generation and implementation in a hardware environment; Fig. 2 a schematic representation of a process flow for generating a recalibratable program code for executing the computation of a neural network; Fig. 3a and Fig. 3b Comparisons of two code implementations for a conventional code generator and a code generator for creating recalibratable program code; Fig. 4. A code example of the form of a C struct for the parameters, which is taken into account when compiling the source code. Description of embodiments

[0028] Fig. Figure 1 shows a block diagram of a platform 1 for performing code generation and implementing the generated program code in a hardware environment 2. The hardware environment corresponds, for example, to a control unit with a microcontroller, microprocessor, or the like. Code generation takes place on a conventional computer 3 or workstation, using a neural network configuration. The computer 3 is configured accordingly to perform code generation.

[0029] Fig. Figure 2 schematically shows the process flow for generating recalibratable program code to implement a neural network NN1, NN2. In step S1, the code generator creates a source code code.c, e.g., in C or Rust or another target programming language, which accesses a parameter dataset with predefined network parameters containing all parameters required for recalibrating the neural network.

[0030] The source code is compiled in step S2 so that the generated program code can be implemented for the desired hardware environment.

[0031] In step S3, the network parameters are provided. This can be done as a binary file, such as "parameters.x", or in more specialized file formats, such as DCM files, which are widely used in the automotive industry.

[0032] They can be provided to the compiled network, for example, encapsulated as a C-struct or in some other encapsulated form.

[0033] The generated program code “bin” is then implemented in the hardware environment in step S4 and can access the provided network parameters.

[0034] If the neural network needs to be recalibrated (NN2), another parameter set 'parameters' can be provided, which is then implemented in hardware environment 2. In certain use cases, several different parameter sets can also be stored in the file system and flexibly exchanged.

[0035] The hardware environment then has access to the parameter data set, which can be stored, for example, in a parameter file or in a memory area of ​​an available data storage device. If the network needs to be recalibrated, only the parameter data set needs to be changed and made available in the hardware environment. When the program code is executed, it then accesses the updated parameter data set and can thus execute the recalibrated function.

[0036] Additionally, the parameter data set may contain information that influences the generated code and depends on the configuration of the recalibrated neural network.

[0037] In addition to the network parameters, each layer in a quantized network has one or more scale parameters for the input vector, one or more scale parameters for the output vector, and one or more scale parameters for the weights, since for tensors in quantized layers in neural networks, it is additionally necessary to specify which float32 value range corresponds to the quantized values ​​of the respective tensor.

[0038] Each scale parameter S float32 is approximated by two integer values, a shift value s and a multiplier m. The relationship is given by sfloat32≈m×2s231.

[0039] In calibratable code, the shift value s and multiplier m must be flexible, as they depend on the trained network parameters. Therefore, the parameter set must provide the scale parameter in addition to the network parameters. These must then be passed as function arguments to the layer functions (functions for calculating the layers of the neural network).

[0040] The same applies to the offset of the input and output data, which indicates where the zero point lies within the quantized value range, and the minimum and maximum values ​​according to the activation function used.

[0041] If Shift S, Multiplier m and Offset of the input and output data are no longer defined, the implementations of certain helper functions in the generated code also change.

[0042] For example, if the sign of the shift s is defined at generation time in the Embedded AI Coder code generator, the code generator can use an optimized requantization function depending on whether shift s is positive or negative. Another point concerns layers with activation functions like RELU. RELU sets negative output values ​​to zero. In quantized networks, the zero point during quantization is therefore set to the respective minimum value of the quantization data type whenever possible. In the case of int8-quantized networks, this is -128. In these cases, the requantized values ​​can be restricted to the legal range q ∈ [-128..127] using an optimized function SSAT, provided the target architecture offers such a function.

[0043] In calibratable code, the offsets are variable and a more general function must be used.

[0044] The requantization function requant(q, multiplier, shift) can be optimized if the sign of shift is known.

[0045] For example, the optimization can be illustrated using the corresponding CMSIS NN function `arm_nn_requantize`. This function uses two helper functions, `function1` and `function2`, whose operation is irrelevant here: 1 int32_t arm_nn_requantize(const int32_t val, 2 const int32_t multiplier, 3 const int32_t shift) 4 { 5 const int32_t temp = val * (1 << LEFT_SHIFT(shift)); 6 const int32_t dividend = function1 (temp, multiplier); 7 const int32_t exponent = RIGHT_SHIFT(shift); 8 9 return function2(dividend, exponent); 10}

[0046] The included macros LEFT_SHIFT and RIGHT_SHIFT are defined as: #define LEFT_SHIFT(shift) (shift > 0 ? shift : 0) #define RIGHT_SHIFT(shift) (shift > 0 ? 0 : -shift) Obviously, for shift>0 LEFT_SHIFT(shift) = shift RIGHT_SHIFT(shift) = 0 and for shift<=0 LEFT_SHIFT(shift) = 0 RIGHT_SHIFT(shift) = -shift

[0047] The optimization potential of two separate implementations for positive and negative shift is obvious.

[0048] Additionally, if `shift` <= 0, the sign of `shift` can be inverted at code generation time to avoid having to perform the inversion at runtime. With the resulting positive shift, this yields: 1 int32_t requantize_negative_shift(const int32_t val, 2 const int32_t multiplier, 3 const int32_t shift) 4 { 5 const int32_t dividend = function1 (val, multiplier); 6 7 return function2(dividend, shift); 8}

[0049] On certain hardware architectures, further optimization potential arises when the negative sign of shift is known. These additional optimizations are described in the Fig. 3a implements the function `requant_add_pns`. If the shift value is known to be >= 0, the following results instead: 1 int32_t requantize_positive_shift(const int32_t val, 2 const int32_t multiplier, 3 const int32_t shift) 4 { 5 const int32_t temp = val * (1 << shift); 6 const int32_t dividend = function1 (temp, multiplier); 7 8 return function2(dividend, 0); 9}

[0050] On certain hardware architectures, further optimization potential arises when the positive sign of shift is known. These additional optimizations are implemented in the function `requant_add_ps` (not shown). Due to these additional optimizations, the functions `requant_add_ms` in Fig. 3b, as well as requant_add_pns in Fig. 3a and the unshown function requant_add_ps output_offset as an additional function argument.

[0051] In most cases, activations in quantized layers are restricted to the range q ∈ [-128, 127]. This must be defined by the quantization algorithm itself. In rare cases, however, other ranges of values, such as q ∈ [-128, 124], can result depending on the parameters. The standard case q ∈ [-128, 127] can be implemented on certain hardware architectures using special instructions: q=SSAT(q);

[0052] In non-parameterizable code, this function SSAT can be used. In parameterizable code, the use of q=MAX(q_min,MIN(q,q_max)); necessary. The values ​​q_min and q_max must be coded as function arguments, as they can change during reparameterization.

[0053] In contrast, optimizations that adapt the generated code to the unchanging topology of the neural network are also possible in parameterizable code.

[0054] Examples of such optimizations include adapting to the exact dimensions of the matrix of a FullyConnected layer and adjusting to the kernel size, stride, dilation, and padding in convolutions. These optimizations can also be tailored to the specific target hardware. For this reason, the generated, parameterizable code offers significantly better performance than a library implementation.

[0055] Both points, where parameterizable code must use a more general, less performant implementation, are addressed by comparing the generated non-parameterizable code in Fig. 3a and the parameterizable code for the same fully connected layer in Fig. 3b clear.

[0056] Other examples of this, which can also occur in non-quantized models (i.e., those that work with float values ​​and therefore do not use shift and multiplier operations), are if-else conditions that depend on the parameters, such as in LSTM networks. In non-parameterizable code, such conditions can be resolved at code generation time, and only the relevant branch is generated as code. This is not possible in parameterizable code.

[0057] When calibratable code is recalibrated with new network parameters, the code generator must ensure that the new parameter data matches the generated code. To guarantee this, all topological properties of the network and all layers, as well as any code generator options, must be stored for all code generations. The necessary information includes the topology or architecture of the neural network—that is, complete information about which layers consume the output of which layers as input; the topological properties of each individual layer, such as the dimensions of matrices in fully connected layers; kernel sizes, dilation, padding, and stride for convolutions; hyperparameters that are not trained, such as the α value of a leaky relu; and the code generator options that affect the generated code.

[0058] When a neural network is calibrated, this information must be compared to the original and must be identical. Additionally, code can be generated again in each calibration step and compared to the original, of course without recompiling it.

[0059] Fig. 3a and Fig. 3b shows a comparison in the form of code examples of a fully connected layer implementation using a conventional code generator versus a code generator with recalibratable configuration and parameterization of the neural network.

[0060] Lines 1-11 of the code Fig. 3a and lines 1-10 of the code of Fig. 3b: The calibratable function additionally takes all integer parameters that can change during calibration as arguments. The conventional function defines these as constants.

[0061] Lines 12-13 of the code Fig. 3a and lines 14-15 of the code Fig. 3b: Both implementations process four rows of the matrix in parallel to reduce the number of times the input data needs to be loaded. This is an optimization during code generation, tailored to the specific layer of the neural network and the hardware environment. Since the layer topology remains unchanged, the calibratable code can also utilize this optimization.

[0062] Lines 52-59 of the code Fig. 3a and lines 54-61 of the code Fig. 3b: For requantization, the conventional code uses the Fig. 3a. The function `requant_add_pns`. This function can only be used if the shift value is negative. During code generation, the sign is then reversed. The calibratable code must use the less optimized function `requant_add_ms`, which does not make any assumptions about the shift.

[0063] Similarly, the optimized code can use the optimized function SSAT to restrict the data to the int8 value range. Since activation_min and activation_max cannot be assumed to be known in the calibratable code, it must restrict the requantized values ​​to the permissible range using MIN and MAX instead of SSAT.

[0064] From line 68 of the code of Fig. 3a and from line 70 of the code Fig. 3b: When calculating four rows in parallel, two rows remain, which are calculated separately.

[0065] Fig. Figure 4 shows a code example for the encapsulated form of a C struct, which is taken into account when compiling the source code.

Claims

[1] Computer-implemented method for performing code generation to determine a recalibrable code for the computation of a neural network, comprising the following steps: - Providing (S1) source code for implementing the neural network, which does not contain the network parameters defining the neural network, i.e., its configuration and parameterization, as constants, wherein the source code is designed to access a parameter data set with predefined network parameters, containing all the parameters required to recalibrate the neural network; - Compiling (S2) the source code so that a program code for the desired hardware environment is generated; - Providing (S3) the network parameters so that they are accessible through the program code; - Implementing (S4) the generated program code in the hardware environment. [2] Method according to claim 1, wherein the network parameters are stored in a file or data storage and can be read at runtime of the program code and are provided in encapsulated form, in particular as a C-struct. [3] Method according to claim 1 or 2, wherein for quantized networks, in addition to the network parameters, one or more scale parameters (S) are further specified for the input vectors, the output vectors and the network parameters of each layer. float32 ) are provided, which indicate the relationship between the quantized range of values ​​and the original float32 range of values. [4] Device for carrying out one of the methods according to one of claims 1 to 3. [5] Computer program product comprising instructions which, when the program is executed by at least one data processing device, cause it to perform the steps of the method according to any one of claims 1 to 3. [6] Machine-readable storage medium comprising instructions which, when executed by at least one data processing device, cause it to perform the steps of the method according to any one of claims 1 to 3.

Images