Method and apparatus for memory planning for code generation for creating program code for artificial neural network computing in hardware environment

Abstract

The invention relates to a computer-implemented method for executing memory planning for code generation for generating code for neural network computing in a hardware environment.

Description

Technical Field

[0001] This invention relates to the implementation of program code in a hardware environment, such as a control device controlled by a microcontroller. The invention further relates to a method for memory planning, which processes input data, output data, and model parameters. Background Technology

[0002] Specific hardware environments, such as microcontrollers in control devices, require the creation of matching executable code to account for the characteristics and limitations of that specific hardware environment. Therefore, the available size of working memory that a microcontroller or acceleration hardware can directly access may be limited, or memory movement or copying operations from data storage such as flash memory or external storage to working memory may be particularly expensive due to hardware limitations.

[0003] The computation steps of the corresponding network layers used for neural network computation may require huge memory because for each computation step, the input data block, model parameter block, and output data block must be callable and available to the microcontroller in the working memory.

[0004] Existing code generators specify during memory planning which area of ​​working memory the data blocks required for each computation step will be stored in. During memory planning, in addition to assigning input data blocks, output data blocks, and necessary model parameters to memory areas in working memory, corresponding memory areas are also allocated to data generated during computation steps.

[0005] Traditional code generators for neural networks typically do not start with limited working memory and usually allocate different memory regions for storing input data blocks, network parameter blocks, and output data blocks for each computational step that is computed sequentially. Therefore, it has been common practice to date to freely distribute model parameters across available memory in order to minimize the overall memory requirement. However, this approach can lead to the need to copy model parameters segment by segment into different memory regions of the working memory before performing computational steps.

[0006] In particular, copying memory regions from data memory to working memory and between different memory regions within working memory is typically a time-consuming memory operation, which necessitates that memory planning have the objective of reducing the total computation time caused by the implementation time of memory operations.

[0007] To minimize the maximum required working memory, a tiling algorithm can be applied, where the computation of a neural network layer is divided into separate, sequential individual computational steps. Tiling is suitable for element-wise operations, convolutional layer computations, pooling layer computations, etc. These separate computational portions are then concatenated together in a final concatenation step.

[0008] Block partitioning impacts memory planning because loading the input data and / or model parameters of the layer to be computed into working memory segment by segment for successive computation steps and then removing them after each individual computation step reduces the total maximum memory requirement for all neural network layers. This makes block partitioning particularly useful on systems with limited working memory. However, dividing computation steps for a specific neural network layer into individual computation steps in the case of block partitioning requires an increased number of memory operations to copy blocks of model parameters into working memory for each individual computation step. Therefore, the number of memory operations increases significantly when block partitioning is applied.

[0009] The objective of this invention is to provide a method for performing memory planning for code generation for creating code for artificial neural network computation in a hardware environment, wherein the number of memory operations can be reduced. Summary of the Invention

[0010] This task is solved by the method of memory planning for code generation for performing computations of layers of a neural network for a hardware environment, as described in claim 1, and by the device according to the parallel claims.

[0011] Other design options are described in the dependent claims.

[0012] According to the first aspect, a computer implementation method is provided for executing memory planning for code generation for determining neural network computation in a hardware environment, the method comprising the following steps: - Provides successive computational steps for network layers of a neural network, wherein for each computational step, the sizes of an input data block and an output data block are determined, and the sizes of one or more model parameter blocks are determined according to the type of computational step, wherein the one or more model parameter blocks have model parameters for the corresponding computational step. - Determine at least one rule for memory planning for a specific computational step that uses model parameters and for which block computation steps should be set up, wherein the block computation steps are set up to perform block computation for a specific computational step among multiple individual computational steps for processing segments of input data in a block of input data, wherein the rule specifies that the model parameters for all individual computational steps are stored in a prescribed memory area. - Memory planning is performed with regard to the established rules, in which memory regions for corresponding input data blocks, output data blocks, and model parameter blocks are specified in the working memory for each computation step.

[0013] - Execute code generation, wherein the model parameter block is copied to the working memory only before the first single computation step is performed, and the computation of the remaining single computation steps is performed with the model parameters in the model parameter block as a reference.

[0014] Furthermore, performing memory planning may include applying optimization methods while taking into account the established rules and, if necessary, additional rules, where the objective function considers minimizing the total memory requirement in the working memory.

[0015] The individual computational steps used in the computational layers of a neural network are typically performed serially on the hardware environment. This means that the generated code for the hardware environment specifies the following order: input data is processed in this order, and output data for the corresponding computational step is produced. Each computational step extracts input data from one or more input data blocks and stores the resulting output data in one or more output data blocks. Input data blocks and output data blocks represent memory regions arranged in a contiguous address space within the working memory.

[0016] As the computational steps that must be performed to compute neural networks, memory operations often occur between actual layer computations. These memory operations involve copying or moving memory regions into or within the working memory. These memory operations are often time-intensive and, moreover, have a significant time share independent of the size of the memory region to be copied or moved.

[0017] For specific computational steps in neural networks that have high memory requirements, block partitioning can be applied, where the computation of a neural network layer is divided into separate, sequential individual computational steps. Block partitioning is suitable for element-wise operations, convolutional layer computations, pooling layer computations, etc. Then, the partial results of the separate computations are linked together by the final concatenation layer computation.

[0018] If computational steps for a specific layer of a neural network are traditionally divided into blocks, at least one block of input data is provided in the working memory, and a block of model parameters used to perform operations on a portion of the input data block is copied into the working memory. A single computational step on that portion of the input data block is then performed using the model parameters. The resulting block of output data, containing partial output data, is cached in a memory area of ​​the working memory. This process is then repeated for subsequent partial regions. This corresponds to the partitioning of each partial region of the input data block being performed separately, resulting in multiple output data blocks distributed in the working memory, each containing partial output data for a computational step used to compute a specific layer. These partial output data are then concatenated with each other in computational steps constructed as concatenated layers.

[0019] In particular, the copying operation for replicating the model parameter block in the working memory is performed again before each individual computation step of the input data block, even though the model parameters contained therein are the same for each processing of the relevant partial region. Therefore, the memory planning is configured to utilize the corresponding model parameter block for all individual computation steps, and this corresponding model parameter block is copied to the working memory only once before the first block computation step. The model parameter block copied there then remains unchanged and protected for a considerable period until the computation performed through the individual computation steps is completed.

[0020] Memory operations that are typically performed before each block-specific computation step are applied only to the first part of the input data block before the block-specific computation step in the case of code generation, and are omitted for subsequent block-specific computation steps.

[0021] Furthermore, the entire input data block can be copied to the working memory before the start of each individual computation step in the block computation step. Alternatively, only a portion of the input data block can be read into the working memory before each individual computation step in the block. Here, the model parameter block stored in the working memory before the first block computation step remains unchanged.

[0022] It can be configured that, in the creation of rules for memory planning for a specific computation step, the rule further specifies the lifetime of the corresponding model parameter block used to store model parameters, wherein the lifetime specifies that the model parameter block remains stored in working memory and accessible for the block computation steps during the duration of the single computation step. Attached Figure Description

[0023] The embodiments are described in more detail below with reference to the accompanying drawings. Wherein: Figure 1A schematic diagram of the platform used for code generation and implementation in the hardware environment is shown; Figure 2 A flowchart illustrating the methods used for code generation for a hardware environment is shown; and Figure 3 shows a diagram of the memory usage of the working memory during computation in a single computation step of a convolutional layer. Detailed Implementation

[0024] Figure 1 A block diagram is shown of a platform 1 used to perform code generation and implement the generated program code in a hardware environment 2. This hardware environment corresponds, for example, to a control device with a microcontroller, microprocessor, etc. Code generation is performed on a conventional computer 3 or workstation with a pre-defined neural network configuration. The computer 3 is configured to perform memory planning and code generation, wherein memory planning first configures memory regions in the working memory for each computational step of the neural network to accommodate at least one input data block, one output data block, and at least one model parameter block. The model parameter block includes all model parameters required for the computation of the corresponding computational steps of the neural network layers.

[0025] If the code is generated, it is transferred to hardware environment 2 and implemented or executed there.

[0026] exist Figure 2 The diagram shows a flowchart illustrating methods for performing memory planning and for providing code generation for implementing neural networks.

[0027] In step S1, a neural network with computational steps is first pre-defined. These computational steps define the type of the layer to be computed, the input data block, and the resulting output data block for that neural network. Further model parameters can be pre-defined for each network layer; these parameters are used to compute the data elements of the output data block from the data elements of the input data block, depending on the type of the network layer.

[0028] Now, in step S2, memory planning is performed, which allocates memory regions in the working memory to each memory region required in the implementation of calculation steps for input data, output data, or model parameters. This includes the input and output data of each network layer, which are primarily the model parameters of the network layers.

[0029] Planning is performed using an SMT planner. The planner understands the size and lifetime of memory regions for each computational step in the network layer. Additionally, the planner manages a large number of rules that describe what effective memory planning should look like.

[0030] If B is a set of memory regions and b∈B is a single memory region with offset Ob and size Sb, then the following is an example for a rule managed by the planner: • RAM size All memory regions must be located within available memory.

[0031] • All memory regions must be distinct from each other. Solving these rules provides memory planning.

[0032] To implement this invention, model parameter blocks used in each individual computation step of the block computation step are assigned to separate memory regions. The lifetime of these model parameter blocks is chosen such that they exist in working memory throughout the computation of all block computation steps. This matching of lifetimes is sufficient to ensure that the SMT planner calculates a memory plan that leads to the code in the next step, in which the invention is implemented.

[0033] exist Figure 3a and 3b The results of memory planning in the traditional method and the above-mentioned method are shown by comparison. Figure 3a This demonstrates how the model parameters of the model parameter block are copied to separate memory regions for each individual calculation step O1-O4 of the block calculation step, and then the corresponding output data blocks AB1, AB2, AB3 are calculated from each segment of the input data block EB and stored in the working memory. Subsequently, in the calculation step of the splicing layer, the output data blocks AB1, AB2, AB3 are concatenated to form the output data K.

[0034] Figure 3b This demonstrates how the model parameters of the model parameter block MB are copied to a separate memory region for only the first single calculation step O1 of the block calculation step. Subsequently, when the model parameters in the model parameter block MB are accessed, the corresponding output data blocks AB1, AB2, and AB3 are calculated from each segment of the input data block EB and stored in the working memory. Then, the output data blocks AB1, AB2, and AB3 are concatenated in the calculation step of the splicing layer.

[0035] In step S3, code generation is performed for each block computation step. This code sets up a single memory operation for copying the model parameter block MB, assuming it is copied to the working memory only once. Therefore, the generated code must ensure that the parameters are copied to the working memory only once.

Claims

1. A computer-implemented method for performing memory planning for code generation for generating neural network computations in a hardware environment, the method comprising the following steps: - Provide (S1) successive computation steps for the network layers of the neural network, wherein for each computation step, the sizes of the input data block (EB), the output data block (AB), and the sizes of one or more model parameter blocks (MB) are determined according to the type of the computation step, wherein the one or more model parameter blocks (MB) have model parameters for the corresponding computation step. - Define rules for memory planning for specific computation steps that use model parameters and for which block computation steps should be set. The specific computation steps are configured to perform block computation for a specific computation step among multiple individual computation steps that process input data in segments of the input data block (EB), wherein the rules specify that model parameters for all individual computation steps (O1-O4) are stored in a designated memory area. - Memory planning (S2) is performed taking into account the established rules, in which, for each computational step, memory regions in the working memory are specified for the corresponding input data blocks (EB), output data blocks (AB1, AB2, AB3), and model parameter blocks (MB). - Execute (S3) code generation, wherein the model parameter block (MB) is copied to the working memory only before the first single calculation step (O1) is performed, and the calculations of the remaining single calculation steps (O1-O3) are performed using the model parameters in the model parameter block (MB).

2. The method of claim 1, wherein the execution of memory planning includes applying an optimization method, wherein the objective function considers minimizing the total storage requirement in the working memory.

3. The method according to any one of claims 1 to 2, wherein in the creation of the rules for memory planning for a particular computation step, the rules further specify the lifetime of the corresponding model parameter block (MB) for storing the model parameters, wherein the lifetime specifies that the model parameter block (MB) remains stored in working memory and accessible for the block computation step during the duration of the single computation step.

4. The method according to any one of claims 1 to 3, wherein the specific computational steps for applying block partitioning include element-level operations, convolutional layer or pooling layer computations.

5. An apparatus for performing the method according to any one of claims 1 to 4.

6. A computer program product comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 4.

7. A machine-readable storage medium comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 4.

Images