Generating program code to accelerate neural networks
By generating multiple lookup tables specific to neuron layers or groups and using integer operations and offsets to calculate the actual parameters of the activation function, the problem of low data processing efficiency in neural networks in microcontrollers is solved, and faster activation is achieved Function evaluation and higher flexibility for hardware platforms with limited computing power.
Patent Information
- Application Number
- CN202410877066.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-03
- Filing Date
- 2024-07-02
- Publication Date
- 2025-01-03
AI Technical Summary
On the hardware platform of neural networks running in microcontrollers or embedded systems, data processing efficiency is low, and when extending the hardware platform to increase processing efficiency, the production cost, installation space and power consumption faces limitations such as production cost, installation space and power consumption, and the evaluation of activation functions is relatively expensive. High, and prior art is difficult to flexibly manage lookup tables to improve processing speed.
Generate multiple lookup tables specific to neuron layers or groups, pre-calculate possible values of activation functions and store them in the lookup table, calculate the actual parameters of the activation function using integer operations and offsets, and provide program code to use hardware platform Quickly retrieve activation function values on it, suitable for different activation function architectures and hardware platforms.
Improves the processing speed and flexibility of neural networks, reduces memory requirements, simplifies the lookup table retrieval process, is suitable for hardware platforms with limited computing power but sufficient memory, and improves the processing efficiency of measured data and the response ability of manipulating signals. .
Smart Images
Figure CN119250166A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an implementation of a neural network for running on a hardware platform such as a microcontroller or a control device. Background Art
[0002] More and more devices controlled by microcontrollers and other embedded systems gain additional functionality through the use of neural networks. Processing data, especially measurement data detected by sensors, using neural networks is relatively time-consuming. However, it is not always easy to expand or enhance the hardware platform used. In addition to the increased production costs, the installation space, power consumption and / or heat release are often limited. In this case, of course, the existing hardware platform must be taken into account. In order to speed up data processing, the only lever is the processing efficiency.
[0003] An important part of the cost in the execution of a neural network flows into the evaluation of the so-called activation function. In many neural networks, the inputs fed to each neuron are summed in a weighted manner, and the output of the neuron is determined from the result by applying an activation function. The activation function is always nonlinear. The evaluation of certain activation functions can be accelerated by retrieving the predicted result from a lookup table. Such examples are disclosed in US 11361213 B1 and US 2021 / 132954 A1. Summary of the invention
[0004] The present invention provides a method for generating a program code for implementing a neural network with a predetermined architecture. When the generated program code is executed on a hardware platform, a neural network is implemented. The predetermined architecture has neurons organized in layers and / or other groups. Thus, for example, neurons of one layer in a multilayer stack can accept the output of the previous layer as input and provide their own output as input to neurons of a subsequent layer. The input layer can accept the input of the neural network as a whole, and the output layer can output the output of the neural network as a whole.
[0005] Within the scope of the method, a nonlinear activation function of the neurons in at least one layer and / or group of neurons is determined according to the predefined architecture. The possible values that the activation function can assume are precalculated and stored in a lookup table. This means that, in contrast to previously known solutions, there is no longer only a single lookup table, but rather the lookup table is specific to the respective layer and / or group of neurons. Thus, the program code for processing the entire neural network as a whole comprises a plurality of lookup tables.
[0006] Now a program code is generated which implements: processing the inputs of all neurons in a layer and / or group into corresponding outputs.
[0007] Specifically, the program code aggregates the inputs of the corresponding neurons into the actual parameters of the activation function according to a predetermined architecture. Based on the actual parameters, the program code determines the index at which the associated value of the activation function is stored in the lookup table for the corresponding layer or group.
[0008] Now, the program code determines the output of the neuron by retrieving the value from the lookup table using this index. Thus, the processing of the input from the corresponding neuron is completed.
[0009] Providing a lookup table for each layer and / or group, and therefore providing a plurality of lookup tables in total, requires additional memory. However, at the expense of the additional memory, a double advantage is obtained. On the one hand, flexibility is expanded so that the lookup table can also be used to implement an architecture in which not all neurons use exactly the same activation function. On the other hand, this also makes it easier to retrieve values from the lookup table. Since the lookup table is specific to the corresponding layer or group, there is a relatively simple association between the actual parameter of the activation function and the index storing the appropriate value of the activation function. Therefore, the time spent on retrieving values from the lookup table will not increase due to the complex calculation of the index or even due to the search in the lookup table. After all, the whole meaning of the lookup table is that it saves net time relative to directly calculating the value of the activation function.
[0010] In general, the advantage of generating a lookup table at the time of generating the code relative to providing the lookup table in the library is that the program code finally generated only contains those lookup tables that are actually used to process the specific pre-given neural network. On the other hand, for example, the library function that implements the activation function in the case of using the internal lookup table must be completely loaded into the memory together with all these lookup tables. Since the library is usually created for many use cases, only a small number of several of the lookup tables may be needed in a specific application. For example, as will be explained in more detail later, if an activation function related to free parameters with an internal lookup table is provided as a library function, the lookup table must be calculated and provided for all possible combinations of the free parameters of the activation function for the library. This will make the source code of the library huge. In addition, using the library will become extremely laborious because the user must manually select the correct lookup table for each layer. If the linker (Linker) used to finally complete the implementation of the neural network does not remove the unused lookup table, the final program code will also become very large.
[0011] In a particularly advantageous embodiment, a program code is generated which calculates the actual parameter of the activation function using integer arithmetic. In addition to the fact that integers can be calculated faster than floating point numbers, integers can also be easily converted into integer indices and thus into storage addresses for activation function values.
[0012] In another particularly advantageous embodiment, a program code is generated which calculates the index by adding an offset to the actual parameter. For example, if you always operate with integers that can take values between -128 and +127, you can add an offset of 128. The actual parameter -128 is then mapped to index 0 and thus to the beginning of the lookup table. The actual parameter +127 is mapped to index 255 and thus to the end of the lookup table. The acquisition of the value of the activation function is thus simplified to a combination of adding the offset and a memory access to the lookup table.
[0013] In another particularly advantageous embodiment, the values of the activation function are stored as integers in a lookup table. They can then be immediately used as inputs for the next neuron, which in turn can be used to perform integer arithmetic. In addition, hardware platforms that do not have a floating-point computing unit can also be used to execute the generated program code.
[0014] In a particularly advantageous embodiment, integer operations with 256 possible values are selected. As mentioned above, these can be, for example, signed int8 values from -128 to +127. In practice, 256 possible values have proven to be a very good compromise between the best possible accuracy on the one hand and the lowest possible memory requirement for the lookup table on the other hand. If even better accuracy is desired, it is more cost-effective to expand the neural network architecture and provide more neurons than to convert the operations to the next larger integer class (e.g. int16).
[0015] In a further particularly advantageous embodiment, a program code is generated which approximates the multiplication by an integer factor in combination with a subsequent bitwise shift. At least one multiplication with a non-integer factor. In this way, multiplication with floating-point factors can be approximated even on hardware platforms without floating-point units.
[0016] This can be particularly useful, for example, in the context of activation functions to take into account parameters that are not necessarily integers.
[0017] For example, an activation function with a free parameter can be selected, the value of which is the same in each group and / or layer of neurons. Thus, a new lookup table is required for each value of the free parameter. For example, groups of neurons can be formed in a targeted manner based on which neurons have the same activation function with the same free parameter value.
[0018] In another advantageous embodiment, a program code is generated which adds an offset when forming the actual parameters of the activation function and / or when determining the output of the neuron. The offset is then the same in each group and / or layer of neurons. For example, the groups of neurons can be specifically found based on the following aspects:
[0019] Input offset and / or output offset,
[0020] The type of activation function and
[0021] If necessary, the value of the free parameter of the activation function
[0022] The combinations formed are respectively identical. The lookup tables are respectively specific to this combination.
[0023] If the quantization for a neural network executed in integer arithmetic also depends on further parameters, such as the number of multipliers and / or the number of bits for bitwise shifting when converting between floating point numbers and integers (shift), the lookup table can also be specific to said parameters. That is, if the quantization parameters are changed at the transition from one layer and / or group of neurons to another, If changes occur when going to the next layer and / or group of neurons, a new lookup table is required.
[0024] In a particularly advantageous embodiment, a leaky rectified linear unit (Leaky ReLU) is selected as the activation function, which outputs positive parameters unchanged and multiplies negative parameters by a predetermined factor α. This activation function is particularly advantageous in cases where, on the other hand, a conventional rectified linear unit (ReLU) which maps negative parameters to 0 would make training difficult due to the constant partial function of the negative parameters, owing to the subsequent vanishing gradient. Furthermore, leaky ReLUs are increasingly being used precisely in very small neural networks, which are used in quantized form on microcontrollers.
[0025] In another particularly advantageous embodiment, the program code is generated which contains pointers to the lookup tables. In this way, the storage of multiple lookup tables in the memory can be designed with maximum flexibility. In particular, only at the time of generating the program code, under the boundary conditions of the free parameters of the activation function, the input offsets and / or the output offsets that may be set, is it known how many different lookup tables are required and which values must be contained therein, i.e., the calculation of the lookup tables is performed when the program code is generated.
[0026] In large neural networks, there can be a very large number of different combinations of free parameters for the activation function, such as the amplification factor α for the leaky ReLU as the activation function, the input offset and the output offset. Each such combination requires its own lookup table for the activation function value. If the memory requirement for this becomes too large overall, a portion of the memory can be saved while giving up a portion of the acceleration by recalculating those activation functions that are only relatively rarely called in a traditional, slower manner and method at each call. If the fact that multiple data can be processed in parallel using one instruction (single instruction, multiple data, i.e. SIMD) on the hardware platform is ignored, the computational cost of evaluating the activation function in, for example, a layer of neurons is measured in a first approximation according to the size of the input vector fed to the layer. For example, a layer that processes relatively short input vectors can evaluate the activation function according to a standard implementation that recalculates the value at each call. In contrast, a layer that processes longer input vectors can use a lookup table. The memory requirement for the corresponding lookup table does not depend on the length of the input vector.
[0027] Therefore, in another particularly advantageous embodiment, the following layers and / or groups are selected from the total existing layers and / or groups of the neural network based on the computational cost incurred for evaluating the activation function in the corresponding groups and / or layers, for which a lookup table and a program code for retrieving the value of the activation function from the lookup table are generated.
[0028] In another particularly advantageous embodiment, the program code is loaded onto a hardware platform and executed, thereby implementing the neural network. As described above, with the same computing power of the hardware platform, the neural network is executed faster, but at the expense of an increased memory requirement due to the lookup table. The method can therefore be used, for example, in particular to better utilize hardware platforms that have limited computing power but still have sufficient memory available.
[0029] In a further particularly advantageous embodiment, measurement data recorded by at least one sensor are fed to the neural network, with which the neural network determines an output with respect to a predefined task based on the measurement data.
[0030] For this purpose, for example, the neural network can be trained, in particular, using training data, which are labeled with target outputs with respect to a predetermined task. In such training, the training data can be processed by the neural network into outputs, and these outputs can be compared with the corresponding target outputs. The deviations resulting from this comparison can be evaluated using a cost function. For example, the weights or also other parameters characterizing the behavior of the neural network can then be optimized with respect to the following goal: to improve the evaluation performed by the cost function, wherein the inputs of the corresponding neurons are aggregated into actual parameters of the activation function using the weights. This training enables the neural network to generalize also to measurement data that have not been seen during training.
[0031] A control signal is determined from an output generated by the neural network from the measurement data. The control signal is used to control a vehicle, a driver assistance system, a robot, a system for monitoring an area, a system for quality control and / or a system for medical imaging. In this way, the probability is increased that the reaction of the corresponding technical system to the control using the control signal is suitable for the situation reflected in the measurement data.
[0032] The neural network can be designed in particular, for example, as an image classifier which assigns classification scores for one or more classes of a predefined classification to images or parts of these images as measurement data.
[0033] The method can in particular be implemented completely or partly by a computer. Therefore, the invention also relates to a computer program with machine-readable instructions which, when executed on one or more computers, cause the one or more computers to carry out the described method. In this sense, embedded systems of vehicle control devices and technical devices which are also capable of executing machine-readable instructions should also be regarded as computers.
[0034] The invention also relates to a machine-readable data carrier and / or a download product having the computer program. A download product is a digital product that can be transmitted via a data network, that is to say a digital product that can be downloaded by a user of a data network, which digital product can be sold for immediate downloading, for example in an online store.
[0035] Furthermore, the computer can be equipped with the computer program, the machine-readable data carrier or the download product. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The following is a description of the preferred embodiments of the present invention in conjunction with the accompanying drawings to illustrate further measures to improve the present invention in more detail.
[0037] Figure 1 An exemplary embodiment of a method 100 for generating a program code 10 for implementing a neural network 2 on a hardware platform 1 is shown;
[0038] Figure 2 An embodiment is shown in which a selection layer 3* is used to create the lookup tables 6a-6f on the neural network 2. DETAILED DESCRIPTION
[0039] Figure 1 is a schematic flow chart of an exemplary embodiment of a method 100 for generating a program code 10 which, when executed on a hardware platform 1, implements a neural network 2 having a predetermined architecture. The architecture of the neural network 2 is given, which specifies the number of neurons 4 and their organization in layers and / or other groups 3a-3f.
[0040] In step 110 , for at least one layer and / or group 3 a - 3 f of neurons 4 , nonlinear activation functions 5 a - 5 f of the neurons 4 in this layer and / or group 3 a - 3 f are determined according to the predefined architecture.
[0041] According to block 111 , an activation function 5a - 5f may be selected having at least one free parameter, the value of which is the same within each group of neurons 4 and / or layer 3a - 3f .
[0042] According to block 112 , a leaky corrected linear unit may be selected as the activation function 5 a - 5 f , which outputs positive real parameters unchanged and multiplies negative real parameters by a predetermined factor α.
[0043] In step 120, possible values that the activation functions 5a-5f can take are pre-calculated and stored in look-up tables 6a-6f.
[0044] According to block 121 , the values of the activation functions 5a - 5f may be stored as integers in the lookup tables 6a - 6f .
[0045] In step 130, program code is generated for implementing the neurons 4 in the layers and / or groups 3a-3f.
[0046] According to box 131, the program code aggregates the input of the corresponding neuron 4 into the actual parameter 7 of the activation function 5a-5f according to the pre-given architecture. According to box 132, it further determines the index 8 according to the actual parameter 7, under which the corresponding value of the activation function 5a-5f is stored in the lookup table 6a-6f for the corresponding layer or group 3a-3f. Finally, according to box 133, the program code 10 determines the output 9 of the neuron 4 by retrieving the value from the lookup table 6a-6f with the index 8.
[0047] In particular, according to block 131a, a program code 10 can be generated which calculates the actual parameter 7 of the activation function 5a-5f using integer arithmetic. In this case, integer arithmetic with 256 possible values can be selected, for example.
[0048] According to block 131 b , a program code 10 can be generated which, when forming the actual parameters 7 of the activation functions 5 a - 5 f , adds the same offset for all neurons 4 in a group and / or layer 3 a - 3 f .
[0049] According to block 132a, program code 10 may be generated that calculates index 8 by adding an offset to actual parameter 7.
[0050] According to block 133a, a program code 10 can be generated which, when forming the output 9 of the neuron 4, adds the same offset for all neurons 4 within a group and / or layer 3a-3f.
[0051] According to block 134 , program code 10 may be generated which contains pointers to the look-up tables 6 a - 6 f .
[0052] According to box 135, the following layers and / or groups 3* can be selected from the total existing layers and / or groups 3a-3f of the neural network (2) based on the computing cost generated for the evaluation of the activation function 5a-5f in the corresponding group and / or layer 3a-3f, for which lookup tables 6a-6f and program code 10 for retrieving the value of the activation function 5a-5f from the lookup table 6a-6f are generated. Figure 2 This is explained in more detail in .
[0053] exist Figure 1 In the example shown, in step 140 , the program code 10 is loaded onto the hardware platform 1 and executed, thereby implementing the neural network 2 .
[0054] In step 150, the measurement data 11 recorded by at least one sensor 12 are then fed to the neural network 2. Based on these measurement data 11, in step 160, the output 13 is determined with the neural network 2 in view of the predefined task.
[0055] In step 170, a control signal 14 is determined from the output 13 thus obtained. In step 180, the control signal 14 is used to control the vehicle 50, the driver assistance system 51, the robot 60, the system for monitoring the area 70, the system for quality control 80 and / or the system for medical imaging 90.
[0056] Figure 2 1 shows an exemplary architecture of a neural network 2 on which the method 100 can be performed. Figure 2 In the example shown, the layer 3* for which the lookup tables 6 a - 6 f and the associated program code 10 are created overall is selected, in particular on the basis of the computational effort.
[0057] The neural network 11 processes the input 11 present as a 1x9 vector into an output 13 comprising only a single number (1x1 vector). The neural network 11 comprises six layers 3a-3f. Figure 2 In the example shown, all of these layers are fully connected layers, in which all outputs of the respective preceding layer are available as input to each neuron 4 .
[0058] The first layer 3a has an activation function 5a which expects a 1x215 vector as argument 7. It outputs a 1x215 vector to the second layer 3b.
[0059] The second layer 3b has an activation function 5b which expects a 1x50 vector as argument 7. It outputs a 1x50 vector to the third layer 3c.
[0060] The third layer 3c has an activation function 5c which expects a 1x85 vector as argument 7. It outputs a 1x85 vector to the fourth layer 3d.
[0061] The fourth layer 3d has an activation function 5d which expects a 1x36 vector as argument 7. It outputs a 1x36 vector to the fifth layer 3e.
[0062] The fifth layer 3e has an activation function 5e which expects a 1x77 vector as argument 7. It outputs a 1x77 vector to the sixth layer 3f.
[0063] The sixth layer 3f directly compresses this 1x77 vector into a 1x1 output 13.
[0064] The expenditure required for calculating the activation functions 5a-5e depends substantially on the length of the input to be processed, since the activation functions 5a-5e should be re-evaluated for each element of the input. The longer the respective input is, the more time can be saved by evaluating via the lookup tables 6a-6e. In contrast, the memory requirement of each lookup table 6a-6e is independent of the length of the input.
[0065] exist Figure 2 In the example shown, the two layers 3a and 3c, which together account for 64.8% of the total computing effort for calculating the activation functions 5a-5e, are selected as layers 3* for which the activation functions 5a or 5c and the lookup tables 6a or 6c and the associated program code 10 are implemented. For this purpose, only 40% of the additional memory requirements that would be required to implement all activation functions 5a-5e and lookup tables 6a-6e are required. Therefore, Figure 2 Option 3* shown in is a good compromise between processing speedup on the one hand and additional memory consumption on the other hand.
Claims
1. A method (100) for generating a program code (10) which, when executed on a hardware platform (1), implements a neural network (2) having a predetermined architecture, wherein: The architecture has neurons (4) organized in layers and / or other groups (3a-3f), and wherein the method (100) comprises the following steps: Determining (110) for at least one layer and / or group (3a-3f) of neurons (4) a nonlinear activation function (5a-5f) of the neurons (4) in the layer and / or group (3a-3f) according to the predetermined architecture; The possible values that the activation function (5a-5f) can take are pre-calculated and stored (120) in a look-up table (6a-6f); Generate (130) a program code (10) which, for all neurons (4) in the layer and / or group (3a-3f), respectively: o According to the predetermined architecture, the inputs of the corresponding neurons (4) are aggregated (131) The actual parameter (7) of the activation function (5a-5f) is formed. o determining (132) an index (8) according to the actual parameter (7), under which, The respective values of the activation functions (5a-5f) are stored in the lookup tables (6a-6f) for the corresponding layers or groups (3a-3f), and o determining (133) the output (9) of the neuron (4) by retrieving a value from the lookup table (6a-6f) using the index (8).
2. The method (100) according to claim 1, wherein a program code (10) is generated for calculating (131a) the actual parameter (7) of the activation function (5a-5f) using integer arithmetic.
3. The method (100) according to claim 2, wherein a program code (10) is generated for calculating (132a) the index (8) by adding an offset to the actual parameter (7).
4. The method (100) according to any one of claims 2 to 3, wherein the value of the activation function (5a-5f) is stored (121) as an integer in the look-up table (6a-6f).
5. The method (100) according to any one of claims 2 to 4, wherein integer operations with 256 possible values are selected.
6. A method (100) according to any one of claims 1 to 5, wherein an activation function (5a-5f) is selected (111) having at least one free parameter, the value of which is the same within each group and / or layer (3a-3f) of neurons (4).
7. The method (100) according to any one of claims 1 to 6, wherein generating a program code (10) which adds (131b, 133a) an offset when forming the actual parameter (7) of the activation function (5a-5f) and / or when determining the output (9) of the neuron, respectively, and The offset is the same within each group and / or layer (3a-3f) of neurons (4).
8. The method (100) according to any one of claims 1 to 7, wherein: A leaky rectified linear unit is selected (112) as activation function (5a-5f), which outputs positive real parameters unchanged and multiplies negative real parameters by a predetermined factor α.
9. The method (100) according to any one of claims 1 to 8, wherein a program code (10) is generated (134) comprising a pointer to the look-up table (6a-6f).
10. The method (100) according to any one of claims 1 to 9, wherein: The method comprises selecting (135) from the total existing layers and / or groups (3a-3f) of the neural network (2) those layers and / or groups (3*) for which a lookup table (6a-6f) and a program code (10) for retrieving the value of the activation function (5a-5f) from the lookup table (6a-6f) are generated based on the computational effort resulting from the evaluation of the activation function (5a-5f) in the corresponding group and / or layer (3a-3f).
11. The method (100) according to any one of claims 1 to 10, wherein the program code (10) is loaded onto a hardware platform (1) and executed (140) to implement the neural network (2).
12. The method (100) according to claim 11, wherein feeding (150) the neural network (2) with measurement data (11) recorded by at least one sensor (12), using the neural network (2) to determine (160) an output (13) for a predetermined task based on the measured data (11), determining (170) a control signal (14) based on said output (13), and Using the control signal (14) to control (180) a vehicle (50), a driver assistance system (51), a robot (60), a system (70) for monitoring an area, a system (80) for quality control and / or a system (90) for medical imaging.
13. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or computing instances, cause the one or more computers or computing instances to perform the method (100) according to any one of claims 1 to 12.
14. Machine-readable data carrier and / or download product having a computer program according to claim 13.
15. One or more computers and / or computing instances having a computer program according to claim 13 and / or having a machine-readable data carrier and / or a download product according to claim 14.