Method and apparatus for memory planning for code generation of program code for artificial neural network computing in hardware environment

Abstract

A computer-implemented method of performing a memory plan for code generation for determining code for neural network computations in a hardware environment has the steps of: providing successive computing steps of a neural network, wherein a plurality of steps set up computations of a weight prefetch layer that can apply weight prefetch; assigning possible weight prefetches to at least a portion of the plurality of weight prefetch layers to obtain a plurality of different combinations thereof, the weight prefetches setting a preload of the network parameters into the hardware environment working memory for the respective weight prefetch layers; the memory planning is performed for the code to be created for the combination, and the respective memory planning is performed for successive calculation steps taking into account the respective combination of the weight prefetch layers for which the weight expectations are set. For the calculation step, memory areas of corresponding input data blocks and output data blocks and network parameter blocks are specified in a working memory, and the maximum required storage space of the working memory is solved; one of the memory plans is selected according to the maximum required memory space.

Description

Technical Field

[0001] This invention relates to program code implementation in a hardware environment, such as a control device controlled by a microcontroller. The invention further relates to a method for memory planning, which processes input data, output data, and network parameters. Background Technology

[0002] Specific hardware environments, such as microcontrollers in control devices, require the creation of matching executable code to account for the characteristics and limitations of that specific hardware environment. Consequently, the available size of the working memory that the microcontroller can directly access may be limited, or memory shifting or copying operations from flash memory or external storage to working memory may be particularly expensive due to hardware limitations.

[0003] The computation steps of the corresponding layer used for neural network computation may require huge memory requirements because for each computation layer, at least one input data block and at least one output data block must be available for recall and stored in working memory that can be utilized by the microcontroller.

[0004] Existing code generators specify during memory planning which area of ​​working memory each data block required for a computation layer will be stored in. During memory planning, in addition to assigning input and output data blocks to memory areas, corresponding memory areas are also allocated to data generated within the computation layer during computation.

[0005] Weight prefetching is a technique used to accelerate the computation of deterministic layers in neural networks. These layers are characterized by the fact that each element of the trained parameters is applied not only to elements of the input data, but to multiple layers. In many cases, a fully connected layer corresponds to simple matrix multiplication. An N-dimensional input vector is multiplied by an M×N weight matrix to produce an M-dimensional output vector. In these cases, weight prefetching is meaningless because each element of the weight matrix is ​​used only once. In other cases, the input data corresponds to a P×N matrix, which, by multiplying by an N×M matrix, produces a P×M matrix. In this case, each element of the N×M matrix is ​​used P times, and weight prefetching is meaningful. Layers that satisfy this premise are particularly—if applied to a range of data—convolutional and deep convolutional layers, but also LSTM layers and fully connected layers (weight prefetching layers).

[0006] Weight prefetching is particularly beneficial for systems without caches. This takes full advantage of the fact that, for example, each element in the kernel is used multiple times during computation for the convolution of the convolutional layer.

[0007] ETAS GmbH’s code generator, “Embedded AI Coder,” uses weight prefetching to accelerate neural networks containing the layers just listed.

[0008] Typically, the trained network parameters of a layer are stored in the hardware environment's data memory, such as flash memory, and then loaded into registers when these parameters are needed for computation. When using weight prefetching, all the network parameters for the quantized neural network—kernel parameters, bias parameters, and quantization parameters—are copied from data memory to working memory before computation of the corresponding layer begins. Then, all subsequent calls retrieve data from working memory instead of from data memory, which is significantly faster.

[0009] After the computation of the corresponding layer is completed, the memory used for the network parameters of the relevant layer can be used again for other data.

[0010] However, the additional consumption of storage space in working memory due to preloading network parameters in weight prefetching can significantly increase the demand for the maximum required storage space in working memory.

[0011] However, very often, the increase in maximum required memory is caused by only one or a few layers in the neural network, while the working memory remains unused in the computation of other layers that utilize weight prefetching. Weight prefetching on these layers does not negatively impact the amount of maximum required working memory.

[0012] The objective of this invention is to provide an improved memory management for faster artificial neural network computation, wherein the maximum required working memory remains limited. Summary of the Invention

[0013] This task is accomplished by the method for memory planning for performing code generation for code used in neural network computation as described in claim 1, and by the apparatus as described in the parallel claims.

[0014] Alternative design options are given in the dependent claims.

[0015] According to the first aspect, a computer implementation method for setting up memory planning for executing code generation for determining neural network computation in a hardware environment is provided, the method comprising the following steps: - Provides a series of computation steps for the neural network, wherein multiple computation steps in the computation steps set up the computation of a weight prefetch layer, for which weight prefetching can be applied; - Assign possible weight prefetchings to at least a portion of the plurality of weight prefetching layers to obtain a plurality of different combinations of weight prefetching layers, wherein the weight prefetching sets network parameters for the corresponding weight prefetching layer to be preloaded into the working memory of the hardware environment; - Memory planning is performed for the code to be created for the multiple different combinations, wherein, taking into account the corresponding combination of weight prefetch layers for which weight expectations are set, corresponding memory planning is performed for the successive computation steps of the neural network, wherein, for the corresponding memory planning, a memory region for the corresponding input data block, the output data block and at least one network parameter block are specified in the working memory for each computation step, and the maximum required storage space of the working memory is determined. - Select one of the memory plans based on the maximum required storage space of the working memory.

[0016] The computational steps of a computational layer in a neural network are typically computed serially on a hardware environment. This means that the generated code for the hardware environment specifies the following order: input data is processed and output data for the corresponding computational step is generated in this order. Each computational step fetches input data from one or more input data blocks in working memory and stores the output data generated in one or more output data blocks into working memory. Input data blocks and output data blocks represent memory regions in working memory corresponding to contiguous address spaces. Additionally, network parameters used for computation in each computational step (layer) can be cached in working memory to obtain fast access to network parameters.

[0017] Weight prefetching is an optimization technique used in neural network implementations to improve the efficiency and speed of computational steps. Here, the required network parameters are loaded from the data memory (which cannot be accessed during computational steps) into the working memory in time before they are actually needed.

[0018] Prefetching reduces latency caused by loading weights and makes more efficient use of computational power. Weight prefetching is particularly useful for layers / computation steps in neural networks that repeatedly access network parameters. Layers that meet this requirement are especially—if applied to a range of data—convolutional and deep convolutional layers, but also LSTM layers and fully connected layers (weight prefetching layers).

[0019] Memory planning specifies, before the code for performing computational steps is generated, which memory regions in the working memory will contain the corresponding input and output data blocks. If weight prefetching is configured for the computational steps, the memory planning also considers placing one or more corresponding network parameter blocks to store the network parameters.

[0020] By utilizing weight prefetching layers for which weights are prefetched in different combinations to perform multiple memory planning for neural network computations, the maximum required storage space in the working memory can be determined individually. This enables the reliable selection of the following combinations of weight prefetching layers for which weights are prefetched: in these combinations, the combination of a negative increase in the maximum required storage space for the application and a positive, faster implementation time for the application is optimal.

[0021] Memory planning can be configured by first determining the maximum required storage space for a neural network implementation (computation step order) without weight prefetching. Since each weight prefetching leads to a computational speedup in a computation step, weight prefetching can now be configured for the following combinations of weight prefetching layers: in which the memory planning does not yield an increase in the maximum required storage space compared to the maximum required storage space in a neural network implementation (computation step order) without weight prefetching. This implementation is faster than the original implementation without weight prefetching, while maintaining the same maximum required storage space, and is therefore objectively better.

[0022] Alternatively, weight prefetching can be configured for the following combinations of weight prefetching layers: in which memory planning results in a smaller increase in the maximum required storage space compared to the maximum required storage space in a neural network implementation (computation step order) without weight prefetching, and simultaneously, the number of weight prefetching layers configured with weight prefetching is maximized.

[0023] Furthermore, Pareto curves can be created for combinations of weight prefetch layers for which weight prefetching is configured, illustrating the minimum to maximum required storage space associated with the shortest computation time for all computation steps. This allows for better selection of combinations of weight prefetch layers for which weight prefetching is configured.

[0024] The Pareto curve offers the advantage that it allows for the selection of an implementation with the best possible trade-off between the maximum required storage space and the computation time duration for all computational steps.

[0025] The above method can be very effectively combined with a tiling approach for reducing the working memory required for computations in convolutional neural networks. This involves dividing computational steps for a series of layers to which tiling can be applied into multiple tiling steps based on a tiling factor. Weight prefetching can be set or not set for each tiling step, corresponding to a given combination. This increases the number of combinations / the possibility of selectively applying weight prefetching to computational steps of the neural network.

[0026] Therefore, it is also possible to configure the weight prefetch layer combination to consider the block computation steps using multiple possible block factors when setting up block prefetch layers suitable for block division, especially convolutional layers, depthwise convolutional layers, pooling layers, and all layers that work element-wise. This allows for memory planning to be performed for each combination for which weight prefetching is set up, and the maximum required storage space of the working memory to be determined, for the sequence of block computation steps determined by the block factors.

[0027] In other words, it involves creating a weight prefetching layer that is not assigned to a block and is either assigned or not assigned to a weight prefetching layer, and a block computation step for a weight prefetching layer that is assigned to a block using a determined block factor, wherein the block computation step is assigned or not assigned to a weight prefetching layer. For a weight prefetching layer that is assigned to a block using a determined block factor, the combination further includes consideration of multiple block factors.

[0028] In particular, the execution of memory planning can include the application of optimization methods, where the objective function considers minimizing the maximum required storage space.

[0029] Memory planning can be implemented in a manner known per se using a so-called SMT Solver. This memory planning begins by defining a list of relevant memory regions that have not yet been allocated to address areas in working memory, representing the successive computational steps of the neural network and the input data blocks, network parameter blocks, and output data blocks. Attached Figure Description

[0030] The embodiments are described in more detail below with reference to the accompanying drawings. Wherein: Figure 1 A schematic diagram of the platform used for code generation and implementation in the hardware environment is shown; Figure 2 This schematically illustrates a flowchart for illustrating a method used in memory planning to determine a weight prefetch layer for which weight prefetching is set; Figure 3 An exemplary Pareto curve is shown for a combination of weighted prefetch layers that have been weighted and prefetched. Figure 4The structure of a convolutional neural network with 9 convolutional layers is shown. Detailed Implementation

[0031] Figure 1 A block diagram is shown of a platform 1 used to perform code generation and implement the generated program code in a hardware environment 2. The hardware environment corresponds, for example, to a control device with a microcontroller, microprocessor, etc. Code generation is performed on a conventional computer 3 or workstation with a pre-defined neural network configuration. The computer 3 is configured to perform memory planning and code generation, wherein memory planning first involves placing memory regions for accommodating at least one input data block and at least one output data block for each computational step of the neural network. The network parameter block or multiple network parameter blocks include all network parameters required for the computation of the corresponding computational step, such as the kernel values, weights, and bias values ​​of convolutional layers.

[0032] If the code is generated, it is transferred to hardware environment 2 and implemented or executed there.

[0033] Figure 2 This schematically illustrates the process of memory planning using an SMT solver.

[0034] In step S1, successive computation steps are first given in advance for neural network computation, and each computation step is assigned at least one input data block and at least one output data block as a memory region of defined size in the working memory.

[0035] For example, Figure 4 The diagram illustrates the computational steps for an exemplary neural network, in which nine convolutional layers Op1-Op9 are computed. The individual computational steps of the computational layers used for neural network computation are typically computed serially in a hardware environment, corresponding to the generated code. This means that the generated code, for this hardware environment, specifies the following order: processing input data and producing output data for the corresponding computational step in this order. Each computational step extracts input data from one or more input data blocks in working memory and stores the resulting output data in one or more output data blocks into working memory. Input data blocks and output data blocks represent memory regions in working memory corresponding to contiguous address spaces. Additionally, network parameters for computation in each computational step (layer) may be cached in working memory to provide fast access to these parameters.

[0036] In step S2, the computation steps used for the weight prefetch layer computation are identified. The weight prefetch layer computation is, in principle, applicable to weight prefetching, in which network parameters are preloaded into one or more network parameter blocks that respectively represent memory regions in the working memory.

[0037] In step S3, weight prefetching is assigned to or not assigned to the weight prefetching layer for multiple combinations. The combination refers to which weight prefetching layers in the memory planning should utilize weight prefetching and which should not.

[0038] In step S4, memory planning is performed for each of the combinations, wherein, considering the corresponding combinations of weight prefetch layers for which weight expectations have been set, memory planning is performed for the successive computation steps of the neural network. For this purpose, one or more network parameter blocks are reserved in the memory region of the working memory for each of the weight prefetch layers for which weight expectations have been set. Memory planning can be performed using a so-called SMT solver, which in principle follows the minimization of the maximum required memory space in the working memory as the optimization objective. Simultaneously, the required computation time can also be determined using memory planning, enabling the determination of a Pareto curve formed by the minimized maximum required memory space and the resulting computation time.

[0039] Then in step S5, a corresponding combination can be selected from the combinations for which memory planning has been implemented, and the combination can be considered in code generation in step S6.

[0040] Figure 3 An exemplary Pareto curve is shown where points indicate possible combinations of weighted prefetch layers to which weights are applied, and are determined by their computational duration / runtime and maximum required storage space, respectively.

[0041] The above method can be very effectively combined with block-based methods for convolutional neural network computation, where one or more computational steps can be divided into multiple block computational steps based on a block factor for convolutional layer computation. Weight prefetching can be set or not set for each block computational step corresponding to a provided combination. This increases the possibility of selectively applying weight prefetching to computational steps of the neural network. Thus, the above method can, in addition to determining which weight prefetching layers should be applied to, also specify which block factor should preferably be selected for a given convolutional layer.

Claims

1. A computer-implemented method for performing memory planning for code generation, the code generation being used to determine code for neural network computation in a hardware environment (2), the method comprising the following steps: - Provides (S1) the successive computational steps of the neural network, wherein, The calculation steps in the calculation steps set up the calculation of the weight prefetch layer, and weight prefetching can be applied to the weight prefetch layer; - Assign possible weight prefetchings (S3) to at least a portion of the plurality of weight prefetching layers to obtain a plurality of different combinations of weight prefetching layers, wherein the weight prefetching sets network parameters for the corresponding weight prefetching layer to be preloaded into the working memory of the hardware environment; - Memory planning for code execution (S4) to be created for the multiple different combinations, wherein, taking into account the corresponding combination of weight prefetch layers for which weight expectations are set, corresponding memory planning is performed for the successive computation steps of the neural network, wherein, for the corresponding memory planning, a memory region for a corresponding input data block, an output data block and at least one network parameter block is specified in the working memory for each computation step, and the maximum required storage space of the working memory is determined; - Select one of the memory plans based on the maximum required storage space of the working memory (S5).

2. The method according to claim 1, wherein, The execution of the memory planning includes applying an optimization method, wherein the objective function considers minimizing the maximum required storage space in the working memory.

3. The method according to claim 1 or 2, wherein, Furthermore, the selection of the memory plan is performed based on the number of weight prefetch layers for which weight expectations have been set, wherein, in particular, the memory plans selected are those for which the number of weight prefetch layers for which weight expectations have been set becomes the maximum.

4. The method according to claim 1 or 2, wherein, The memory planning for selecting the following combinations of weight prefetch layers results in an increase of less than a predetermined relative share of the maximum required storage space compared to the maximum required storage space in a neural network implementation without weight prefetching, while maximizing the number of weight prefetch layers with weight prefetching.

5. The method according to claim 1 or 2, wherein, Memory planning for selecting the following weight prefetch layer combinations from these combinations: the weight prefetch layer combinations lie on the Pareto curve of the combinations of weight prefetch layers for which weight prefetching has been set, wherein the Pareto curve illustrates the minimum maximum required storage space associated with the shortest computation time corresponding to the computation of all computation steps.

6. The method according to any one of claims 1 to 5, wherein, When setting up block-based weight prefetching layers suitable for block-based processing, particularly convolutional layers, depthwise convolutional layers, pooling layers, and all layers that operate element-wise, the weight prefetching layer combination further considers the block computation steps using multiple possible block factors, such that memory planning is performed for each combination for which weight prefetching is set up, for the sequence of weight prefetching layers and block computation steps determined by the block factors, and the maximum required storage space of the working memory is determined.

7. The method according to any one of claims 1 to 6, wherein, Based on the selected memory plan, execute (S6) code generation for the hardware environment (2) and implement the generated code there.

8. An apparatus for performing the method according to any one of claims 1 to 7.

9. A computer program product comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 7.

10. A machine-readable storage medium comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 7.

Images