Method and device for memory planning for code generation for program code for calculating an artificial neural network in a hardware environment

Abstract

The invention relates to a computer-implemented method for performing memory planning for code generation to determine a code for the computation of a neural network in a hardware environment (2), comprising the following steps: - Providing (S1) successive computation steps of the neural network, wherein several of the computation steps provide for the computation of a weight prefetching layer for which weight prefetching is applicable; - Assigning (S3) a possible weight prefetching to at least some of the multiple weight prefetching layers to obtain several different combinations of weight prefetching layers, wherein the weight prefetching involves preloading network parameters into a working memory of the hardware environment for the respective weight prefetching layer; - Performing (S4) memory planning for the code to be created for the several different combinations, wherein the respective memory planning for the successive computation steps of the neural network is carried out taking into account the respective combination of the weight prefetching layers for which weight prefetching is provided, wherein for the respective memory planning for each computation step the memory area of ​​the respective input data block, output data block and at least one network parameter block in main memory is determined and a maximum required memory space of main memory is determined; - Select (S5) one of the memory schedules depending on the maximum required memory space.

Description

Technical field

[0001] The invention relates to the implementation of program code on a hardware environment, such as microcontroller-controlled control units and the like. The invention further relates to methods for memory planning for handling input data, output data, and network parameters. Technical background

[0002] Certain hardware environments, such as microcontrollers in electronic control units (ECUs), require the creation of customized executable program code to accommodate the specific characteristics and limitations of that hardware environment. For example, the available amount of RAM that the microcontroller can directly access may be limited, or memory move or copy operations from flash or external storage to RAM may be particularly resource-intensive due to hardware limitations.

[0003] The computational steps for calculating the corresponding layers of neural networks can require a considerable amount of storage space, since at least one input data block and at least one output data block must be stored in the working memory for each computational layer and be usable by the microcontroller.

[0004] Existing code generators determine during memory allocation which area of ​​main memory the data blocks required for each computation layer will be stored in. During memory allocation, in addition to assigning input and output data blocks to memory areas, a corresponding memory area is also allocated to the data generated during computation in a computation layer.

[0005] Weight prefetching is a technique for accelerating the computation of certain layers in neural networks. A defining characteristic of these layers is that each element of the trained parameters is applied not just to a single element of the input data, but to multiple layers. In many cases, fully connected layers correspond to simple matrix multiplication. An N-dimensional input vector is multiplied by an MxN weighting matrix to produce an M-dimensional output vector. In these cases, weight prefetching is not useful because each element of the weighting matrix is ​​used only once. In other cases, the input data corresponds to a PxN-dimensional matrix, which is multiplied by an NxM matrix to create a PxM matrix. In this case, each element of the NxM matrix is ​​used P times, and weight prefetching is beneficial.Layers that meet this requirement include, in particular, convolutions and depthwise convolutions, but also LSTM and fully connected layers if they are applied to a set of data (weight prefetching layers).

[0006] Weight prefetching is particularly helpful on systems without a cache. It exploits the fact that, for example, each element in the kernel is used many times during computation for calculating a convolution of the convolutional layer.

[0007] The Embedded Al Coder code generator from ETAS GmbH uses weight prefetching to accelerate neural networks containing the layers just listed.

[0008] Typically, the trained network parameters of a layer are stored in the hardware's data storage, such as flash memory, and then loaded into registers when needed for a computation. When weight prefetching is used, all network parameters—that is, the kernel, bias, and quantization parameters—are copied from data storage to main memory for quantized neural networks before the computation of that layer begins. All subsequent retrieval operations then obtain data from main memory instead of data storage, which is significantly faster.

[0009] After the calculation of the respective layer has been completed, the memory used for the network parameters of the layer in question can be used again for other data.

[0010] However, the additional memory consumption due to preloading network parameters during weight prefetching can significantly increase the maximum required memory capacity.

[0011] Very often, however, the increase in the maximum required memory is caused by only one or a few layers in the neural network, while memory remains unused when other layers perform weight prefetching. Weight prefetching on these layers has no negative impact on the amount of maximum required memory.

[0012] The object of the present invention is to provide improved memory management for faster computation of artificial neural networks, while keeping the maximum required working memory limited. Disclosure of the invention

[0013] This problem is solved by the method for performing a memory planning for a code generation of a code for the computation of a neural network according to claim 1 and by the device according to the dependent claim.

[0014] Further details are specified in the dependent claims.

[0015] According to a first aspect, a computer-implemented procedure for performing memory planning for code generation to determine a code for the computation of a neural network in a hardware environment is provided, with the following steps: - Providing successive computation steps of the neural network, where several of the computation steps include the computation of a weight prefetching layer for which weight prefetching is applicable; - Assigning a possible weight prefetching to at least some of the multiple weight prefetching layers in order to obtain several different combinations of weight prefetching layers, where the weight prefetching involves preloading network parameters into a working memory of the hardware environment for the respective weight prefetching layer; - Performing memory planning for the code to be created for the several different combinations, wherein the respective memory planning for the successive computation steps of the neural network is carried out taking into account the respective combination of the weight prefetching layers for which weight prefetching is provided, wherein for the respective memory planning for each computation step the memory area of ​​the respective input data block, output data block and at least one network parameter block in main memory is determined and a maximum required memory space of main memory is determined; - Select one of the memory schedules depending on the maximum required RAM space.

[0016] The individual computational steps for calculating the layers of a neural network are typically executed sequentially on a hardware environment. This means that the generated code defines a sequence for the hardware environment in which the input data is processed and the output data of each computational step is generated. Each computational step retrieves the input data from one or more input data blocks in main memory and stores the resulting output data in one or more output data blocks in main memory. Input data blocks and output data blocks represent memory areas that correspond to a contiguous address space. Additionally, network parameters for the computation of the individual computational steps (layers) can be cached in main memory to allow for faster access to these parameters.

[0017] Weight prefetching is an optimization technique used in the implementation of neural networks to improve the efficiency and speed of computational steps. It involves loading required network parameters from a data store (which is inaccessible during the computation step) into working memory before they are actually needed.

[0018] Prefetching can reduce the latency that would otherwise result from loading weights and utilize computing power more efficiently. Weight prefetching is particularly applicable for computational layers / steps of neural networks that repeatedly access the network parameters. Layers that meet this requirement include convolutions and depthwise convolutions, but also LSTM and fully connected layers when applied to a set of data (weight-prefetching layers).

[0019] Memory planning, prior to code generation, determines which memory areas in main memory the respective input and output data blocks will be placed in for the calculation steps. If weight prefetching is implemented for a calculation step, the memory plan also considers the placement of one or more network parameter blocks for storing the network parameters.

[0020] By performing numerous memory planning tests for neural network computations with weight-prefetching layers, for which weight prefetching is implemented in various combinations, the maximum required memory footprint can be determined for each layer. This enables a well-informed selection of the weight-prefetching layer combination that optimally balances the (for the user) increase in maximum memory footprint with the (for the user) faster execution time for the specific application.

[0021] Memory planning can begin by determining the maximum memory required for a neural network implementation (sequence of computation steps) without weight prefetching. Since weight prefetching speeds up the computation of computation steps, it can then be implemented for that combination of weight prefetching layers where the memory plan shows no increase in the maximum memory required compared to the implementation without weight prefetching. This implementation is then faster than the original implementation without weight prefetching, while the maximum memory requirement remains the same, and is therefore objectively better.

[0022] Alternatively, weight prefetching can be provided for the combination of weight prefetching layers where, during memory planning, the increase in the maximum required memory space compared to the maximum required memory space in an implementation of a neural network (sequence of computation steps) without weight prefetching is less than a predetermined relative proportion, and at the same time the number of weight prefetching layers where weight prefetching is provided is maximized.

[0023] Furthermore, a Pareto curve can be created for combinations of weight prefetching layers for which weight prefetching is implemented. This curve indicates the smallest maximum required memory space with respect to the shortest computation time for all calculation steps. This allows for improved selection of the specific combination of weight prefetching layers for which weight prefetching should be implemented.

[0024] The Pareto curve offers the advantage of enabling the selection of an implementation with the best possible compromise between the maximum required storage space and the computation time for calculating all computation steps.

[0025] The above methods can be very effectively combined with tiling to reduce the memory required for computing a convolutional neural network. According to a tiling factor, the computational steps for a sequence of layers eligible for tiling can be divided into multiple tiling steps, for which weight prefetching can be applied or omitted depending on the combinations provided. This increases the number of combinations / possibilities for selectively applying weight prefetching to the computational steps of a neural network.

[0026] It can therefore still be provided that the combinations of weight prefetching layers continue to take into account tiling computation steps when tiling is provided for weight prefetching layers suitable for tiling, in particular for convolution layers, depthwise convolution layers, pooling layers, as well as all element-wise working layers, with a multitude of possible tiling factors, so that for each combination of weight prefetching layers and sequences of tiling computation steps determined by the tiling factor, for which weight prefetching is provided, memory planning is carried out and a maximum required amount of main memory is determined.

[0027] In other words, combinations are created of weight prefetching layers not subject to tiling, layers to which weight prefetching is assigned or not, and tiling calculation steps for weight prefetching layers subjected to tiling with a specific tiling factor, where each tiling calculation step is assigned to weight prefetching or not. The combinations also include consideration of multiple tiling factors for the weight prefetching layers subjected to tiling with a specific tiling factor.

[0028] In particular, performing storage planning may involve applying an optimization procedure where the objective function takes into account minimizing the maximum required storage space.

[0029] Memory planning can be performed in a familiar manner using a so-called SMT solver. This begins with a list of sequential computation steps that define the neural network, and associated memory areas for input data blocks, network parameter blocks, and output data blocks that are not yet assigned to an address range in main memory. Brief description of the drawings

[0030] The embodiments are explained in more detail below with reference to the accompanying drawings. These show: Fig. 1 a schematic representation of a platform for code generation and implementation in a hardware environment; Fig. 2. A schematic flowchart illustrating a storage planning procedure for determining weight prefetching layers for which weight prefetching is planned; Fig. 3 an exemplary Pareto curve of combinations of weight-prefetching layers; Fig. 4 a structure of a convolutional neural network with 9 convolutional layers. Description of embodiments

[0031] Fig. Figure 1 shows a block diagram of a platform 1 for performing code generation and implementing the generated program code in a hardware environment 2. The hardware environment corresponds, for example, to a control unit with a microcontroller, microprocessor, or the like. Code generation takes place on a conventional computer 3 or workstation, using a predefined neural network configuration. The computer 3 is configured to perform memory allocation and code generation. The memory allocation process first assigns memory areas for each computation step of the neural network to hold at least one input data block and at least one output data block. The network parameter block(s) comprise all network parameters required for the calculation of the respective computation step, such as kernel values, weights, and bias values ​​of a convolutional layer.

[0032] Once the code is generated, it is transferred to the hardware environment 2 and implemented or executed there.

[0033] Fig. Figure 2 schematically shows the process of storage planning using an SMT solver.

[0034] In step S1, successive calculation steps for calculating a neural network are initially specified, each of which is assigned at least one input data block and at least one output data block as memory areas in the working memory with defined sizes.

[0035] This example shows Fig. Section 4 describes the sequence of computation steps for an example neural network in which nine convolutional layers, Op1-Op9, are computed. The individual computation steps for calculating the computational layers of a neural network are typically executed sequentially on a hardware environment according to a generated code. This means that the generated code defines a sequence for the hardware environment in which the input data is processed and the output data of each computation step is generated. Each computation step retrieves the input data from one or more input data blocks in main memory and stores the resulting output data in one or more output data blocks in main memory. Input data blocks and output data blocks represent memory regions of main memory that correspond to a contiguous address space.Additionally, network parameters for the calculation of the individual calculation steps (layers) can be temporarily stored in the main memory to allow quick access to the network parameters.

[0036] In step S2, computation steps for calculating weight prefetching layers are identified. The calculation of weight prefetching layers is fundamentally suitable for weight prefetching where network parameters are preloaded into one or more network parameter blocks, each representing a memory area in main memory.

[0037] In step S3, weight prefetching is assigned to the weight prefetching layers for a multitude of combinations, or it is not. These combinations specify which weight prefetching layers should be considered in the storage planning and which should not.

[0038] In step S4, memory planning is performed for each combination. This planning is tailored to the successive computation steps of the neural network, taking into account the specific combination of weight prefetching layers for which weight prefetching is planned. For each weight prefetching layer, one or more network parameter blocks are reserved in a memory area of ​​the main memory. Memory planning can be performed using an SMT solver, which aims to minimize the maximum amount of memory required. Simultaneously, the required computation time can also be determined using memory planning, allowing a Pareto curve to be calculated based on the minimum and maximum memory requirements and the resulting computation time.

[0039] In step S5, a suitable combination can then be selected from the combinations for which a memory plan has been executed, and this combination can be taken into account in step S6 during code generation.

[0040] Fig. Figure 3 shows an example of such a Pareto curve, where the points represent possible combinations of weight-prefetching layers and are determined by their computation time / runtime and maximum required memory space.

[0041] The above methods can be very effectively combined with a tiling procedure for computing a convolutional neural network. According to a tiling factor, one or more computational steps for calculating a convolutional layer can be subdivided into multiple tiling steps, for which weight prefetching can be applied or omitted depending on the provided combinations. This increases the possibilities for selectively applying weight prefetching to the computational steps of a neural network. Thus, in addition to deciding which weight prefetching layers should be applied to, the above method also allows for determining which tiling factor is preferentially chosen for a particular convolutional layer.

Claims

[1] Computer-implemented method for performing memory planning for code generation to determine a code for the computation of a neural network in a hardware environment (2), comprising the following steps: - Providing (S1) successive computation steps of the neural network, wherein several of the computation steps provide for the computation of a weight prefetching layer for which weight prefetching is applicable; - Assigning (S3) a possible weight prefetching to at least some of the multiple weight prefetching layers to obtain several different combinations of weight prefetching layers, wherein the weight prefetching involves preloading network parameters into a working memory of the hardware environment for the respective weight prefetching layer; - Performing (S4) memory planning for the code to be created for the several different combinations, wherein the respective memory planning for the successive computation steps of the neural network is carried out taking into account the respective combination of the weight prefetching layers for which weight prefetching is provided, wherein for the respective memory planning for each computation step the memory area of ​​the respective input data block, output data block and at least one network parameter block in main memory is determined and a maximum required memory space of main memory is determined; - Select (S5) one of the memory schedules depending on the maximum required memory space. [2] Method according to claim 1, wherein performing the memory planning includes applying an optimization method in which the objective function takes into account a minimization of the maximum required memory space in main memory. [3] Method according to claim 1 or 2, wherein the selection of the memory layout is further carried out depending on the number of weight prefetching layers for which weight prefetching is provided, wherein in particular the memory layout for which the number of weight prefetching layers for which weight prefetching is provided is maximized is selected. [4] Method according to claim 1 or 2, wherein the memory planning of the combination of weight-prefetching layers is selected in which the memory planning results in an increase in the maximum required memory space compared to the maximum required memory space in an implementation of a neural network without weight-prefetching of less than a predetermined relative proportion and at the same time the number of weight-prefetching layers in which weight-prefetching is provided is maximized. [5] Method according to claim 1 or 2, wherein the memory planning of the combination of weight prefetching layers is selected from those combinations of weight prefetching layers which lie on a Pareto curve of combinations of weight prefetching layers for which weight prefetching is provided, wherein the Pareto curve indicates the smallest maximum required memory space with respect to a correspondingly shortest computation time for the calculation of all computation steps. [6] A method according to any one of claims 1 to 5, wherein the combinations of weight prefetching layers further consider tiling computation steps when providing tiling of weight prefetching layers suitable for tiling, in particular for convolution layers, depthwise convolution layers, pooling layers, and all element-wise layers, with a plurality of possible tiling factors, so that for each combination of weight prefetching layers and sequences of tiling computation steps determined by the tiling factor for which weight prefetching is provided, memory planning is carried out and a maximum required memory space is determined. [7] Method according to any one of claims 1 to 6, wherein code generation for the hardware environment (2) is performed based on the selected memory plan (S6) and the generated code is implemented there. [8] Apparatus for carrying out one of the methods according to any one of claims 1 to 7. [9] Computer program product comprising instructions which, when the program is executed by at least one data processing device, cause it to perform the steps of the method according to any one of claims 1 to 7. [10] Machine-readable storage medium comprising instructions which, when executed by at least one data processing device, cause it to perform the steps of the method according to any one of claims 1 to 7.

Images