Method and apparatus for memory planning for code generation of program code for computation of artificial neural networks in a hardware environment

Abstract

This performs memory planning for code generation, determining the code for computation in the neural network. [Solution] The method provides sequential computation steps for a neural network, determines the size of the input data block and the size of the output data block for each computation step, and calculates memory planning conditions for each predetermined computation step in which the input data block in the memory area is at least partially contained within the output data block. The conditions indicate that the memory area allocated to the output data block of the computation step preceding the predetermined computation step is contained within the memory area of ​​the output data block of the predetermined computation step. The method also performs memory planning for each computation step, taking the calculated conditions into consideration, to set the memory areas of the respective input data block and output data block in main memory.

Description

Technical Field

[0001] The present invention relates to the implementation of program code in a hardware environment, such as a microcontroller-controlled control device and the like. The present invention further relates to a method for memory planning for handling input data, output data, and network parameters.

Background Art

[0002] Technical Background A predetermined hardware environment, for example, a microcontroller in a control device, requires the creation of an adjusted executable program code in consideration of the characteristics and limitations of the predetermined hardware environment. Therefore, in particular, the available memory size of the main memory directly accessible by the microcontroller may be limited, or the memory shift operation or memory copy operation from the flash memory or external memory may be particularly costly due to the hardware.

[0003] The calculation steps for the calculation of the corresponding layer of the neural network may require a considerable amount of required memory. This is because for each calculation layer, an input data block, a network parameter block, and an output data block are callable and must be stored in the main memory in a state available by the microcontroller.

[0004] Existing code generators set in which area of the main memory in the memory the data blocks required for each calculation layer are stored during memory planning. During memory planning, in addition to the allocation of input data blocks and output data blocks to the memory area, a corresponding memory area is also allocated to the data generated during the calculation in the calculation layer.

[0005] Conventional neural network code generators typically assume that the entire input and output data blocks must reside in completely separate memory areas.

[0006] However, conventional memory planning methods do not incorporate information about the type of computation layer. Therefore, they do not consider whether the output data block of one computation step corresponds to part of the input data block of a subsequent computation step. This is the case, for example, when two input data blocks are interconnected by concatenation during computation in a concatenation layer.

[0007] In particular, since both copying memory regions from flash memory or external memory to main memory, and copying memory regions between memory regions within main memory, are typically time-consuming memory operations, memory planning must aim to reduce the processing time of these memory operations. [Overview of the project] [Problems that the invention aims to solve]

[0008] The object of the present invention is to provide improved memory management for computation of artificial neural networks that can reduce the number of memory operations. [Means for solving the problem]

[0009] Disclosure of the invention The above problems are solved by the method for performing memory planning for code generation of a code for computation of a neural network as described in claim 1, and by the apparatus as described in the other independent claims.

[0010] Further embodiments are described in each dependent claim.

[0011] According to the first aspect, a computer-implemented method for performing memory planning for code generation to determine the code for computation of a neural network, comprising the following steps, namely: - A step of providing sequential computation steps for a neural network, wherein the size of the input data block and the size of the output data block are determined for each computation step. - A step of calculating memory planning conditions for each predetermined calculation step in which an input data block in a memory area is at least partially included in an output data block, wherein the conditions indicate that the memory area allocated to the output data block of a calculation step preceding the predetermined calculation step is included in the memory area of ​​the output data block of the predetermined calculation step. - A step of performing memory planning, which involves setting the memory areas of the respective input and output data blocks in main memory for each calculation step, taking into account the calculated conditions. A method including this is constructed.

[0012] In particular, performing memory planning may involve applying optimization methods whose objective function aims to minimize the number of memory operations.

[0013] The individual computation steps for calculations in the computational layer of a neural network are typically computed serially in the hardware environment. This means that the code for the generated hardware environment sets the order in which the input data is processed and the output data for each computation step is generated. Each computation step takes input data from one or more input data blocks and stores the resulting output data in one or more output data blocks. The input data blocks and output data blocks represent memory regions in main memory that correspond to contiguous address spaces.

[0014] The hardware environment includes a computing unit, main memory, and data memory. The code generated for the neural network computation is executed in the computing unit in the form of computation steps, where input data blocks and output data blocks are used to supply input data and store output data in main memory. Accessing data from main memory has a short access time, while accessing data memory requires a longer access time. Memory planning aims to reduce access time or overall computation time by supplying and arranging input data blocks and output data blocks in main memory, while simultaneously considering or limiting the maximum available memory capacity of main memory.

[0015] Depending on the computational steps that must be performed for the neural network calculation, memory operations, including copying, moving, or shifting memory regions, frequently occur between the actual layer calculations. Memory operations are often time-consuming and have a large time component that is independent of the size of the memory region to be copied or shifted.

[0016] In the above method, with respect to memory planning, within the code generation framework for calculating the code for neural network computation, for example, for predetermined computation steps such as concatenation, slicing, and padding, the output data blocks of preceding computation steps are positioned in the address area of ​​main memory so that subsequent computation steps can be executed immediately by allocating corresponding address pointers (pointers) without requiring memory operations, or so that only one memory operation with a lower cost is required.

[0017] In this way, the sequence of two calculation steps can be optimized so that time-consuming memory operations are not required between the memory area of ​​the output data block of the preceding calculation step and the memory area of ​​the input data block of the subsequent calculation step.

[0018] In particular, the memory planning is configured to place the output data block of a preceding calculation step, which subsequently performs a predetermined calculation step, within the memory area of ​​main memory so that this output data block can be used entirely or partially as the output data block of that predetermined calculation step. In other words, the input data block of a predetermined calculation step includes an immutable portion of the output data block according to the settings of the memory planning, thereby making memory operations omittable and requiring only the use of a single address pointer.

[0019] Memory planning can be performed using a so-called SMT solver, a method known in itself. Here, memory planning begins with a list of sequentially consecutive computation steps that define the neural network, belonging to the memory region of input and output data blocks that has not yet been allocated to an address region in main memory. This memory region is determined solely by its size. For each pair of computation steps that are not necessarily directly consecutive, a corresponding condition is added to the SMT solver indicating whether the output and input data blocks should be located within the same address region.

[0020] If the calculation step is not concatenation, slicing, or padding, and each input data block can be stored line by line, then the memory areas for the input and output data blocks of each calculation step must be stored separately.

[0021] If a predetermined calculation step is a concatenation step, the condition can be configured to indicate that one or more calculation steps preceding the concatenation step supply output data blocks in a memory area directly adjacent to the memory area to which the output data of the preceding calculation steps should be concatenated in the concatenation step.

[0022] When a predetermined calculation step is a padding step, the condition can be configured such that the output data block is supplied into a memory area directly adjacent to the memory area occupied by the padding pattern or the memory area into which the padding pattern will be written later by a calculation step preceding the padding step.

[0023] When a predetermined calculation step is a slicing step, the condition can be configured such that the output data block is supplied into a memory area that partially corresponds to the output data block of the predetermined calculation step by a calculation step preceding the slicing step.

[0024] When a predetermined calculation step is concatenation, slicing or padding, an attempt is made to arrange the output data block of the preceding calculation step such that the corresponding other memory area belonging to the input data block of the predetermined calculation step is arranged (concatenated) directly above or directly below the output data block of the preceding calculation step (within the address space).

[0025] In a predetermined calculation step corresponding to padding, the memory area occupied by the padding pattern can be arranged above or below the output data block.

[0026] In a predetermined calculation step corresponding to slicing, a cut-out from the corresponding output data block of the preceding calculation step is assumed as the input data block, which can be done by allocation of the corresponding address and size. Since the output data block corresponds to a part of the input data block, no copy operation is necessary.

[0027] In the following, embodiments will be described in more detail with reference to the accompanying drawings. The drawings show the following.

Brief Description of the Drawings

[0028] [Figure 1]This is a schematic diagram illustrating a platform for code generation and implementation in a hardware environment. [Figure 2] This diagram schematically illustrates a flowchart that describes a method for performing memory planning for computation steps that include concatenation, slicing, or padding. [Figure 3a] This is a schematic diagram showing a preferred memory planning for two sequential computation steps of a neural network without applying the method shown in Figure 2. [Figure 3b] This is a schematic diagram showing the preferred memory planning for two sequential computation steps of a neural network when the method shown in Figure 2 is applied. [Modes for carrying out the invention]

[0029] Description of the Embodiment Figure 1 shows a block diagram of platform 1 for executing code generation in hardware environment 2 and implementing the generated program code. The hardware environment corresponds to, for example, a control device having a microcontroller, microprocessor, or similar. Code generation is performed in a conventional computer 3 or workstation, under the configuration settings of the neural network. Computer 3 is configured to perform memory planning and code generation, where memory planning first performs the arrangement of memory areas to accommodate at least one input data block and at least one output data block for each computation step of the neural network. The model parameter block contains all the model parameters required for the computation of each computation step, for example, the weights of the fully connected layer, i.e., bias values.

[0030] Once the code is generated, it is transmitted to hardware environment 2, where it is implemented or executed.

[0031] In the memory planning framework described below, the memory areas for input and output data blocks for a given calculation step can be configured to be located within the address area of ​​the main memory such that, for each calculation step, access is performed in a manner that at least partially overlaps with the memory area of ​​the input data block for that calculation step. This significantly reduces the memory capacity of the main memory.

[0032] Figure 2 shows a schematic diagram of the memory planning flow using an SMT solver.

[0033] In step S1, a series of sequential computation steps for the neural network are first set up, and each of these computation steps is allocated as a memory area with a defined size for the input data block and the output data block.

[0034] In step S2, a predetermined calculation step is identified, which includes concatenation, slicing, or padding. This predetermined calculation step has the characteristic that at least a portion of the output data block of a preceding calculation step is included in the input data block or corresponds entirely to the input data block.

[0035] From this, we can derive the conditions for a predetermined calculation step for memory planning.

[0036] The conditions are, -For a concatenation step, the output data block should be supplied by the calculation step preceding the concatenation step to a memory area directly adjacent to the memory area to which the output data of the preceding calculation step is to be concatenated; -For a padding step, the output data block should be supplied by a calculation step preceding the padding step to a memory area directly adjacent to the memory area occupied by the padding pattern or the memory area to which the padding pattern can be written; - For a slicing step, the slicing step should set an input data block that belongs to the address area corresponding to the section of output data to be selected from the output data block of the preceding calculation step; - For residual types of calculation steps, the memory areas for the input data block and the output data block should be located in separate memory areas; That is the case.

[0037] Each memory region is assigned a lifetime that specifies how long it must be occupied, particularly for how many of the subsequent computation steps its data elements must not be overwritten, and ultimately indicates that the memory region will remain occupied until it is no longer needed.

[0038] In step S3, memory planning is performed using an SMT solver. The optimization aims to reduce the number of time-consuming memory operations.

[0039] Here, for each calculation step preceding one of the calculation steps described above, the output data block is positioned so that it is possible to supplement the data memory area associated with the given calculation step. In the case of concatenation, this corresponds to another memory area to be concatenated to the memory area of ​​the output data block of the preceding calculation step, or in the case of padding, it corresponds to adding padding patterns above and below the memory area of ​​the output data block of the preceding calculation step.

[0040] In the case of slicing, the input data block for a given calculation step is defined as a subset of the memory area of ​​the output data block of the preceding calculation step.

[0041] In particular, when main memory is limited, it is not always possible to optimize all of a given calculation steps with respect to the number of memory operations, as described above. Therefore, a regression optimization method can be performed using an SMT solver, which allocates different memory regions of the output data blocks of preceding calculation steps for a given calculation step.

[0042] For all predetermined calculation steps, maximum optimization is performed so that, in order to obtain the output data of the output data block of a predetermined calculation step, it is not necessary to perform a copy operation of the output data block of the preceding calculation step. This is achieved when the output data block of a preceding calculation step is written to the main memory area. To achieve this, the data of the output data block of a predetermined calculation step can be defined using only the corresponding address allocation (pointer).

[0043] Next, memory planning is used to generate the code.

[0044] Figures 3a and 3b show the positioning of memory regions for input data blocks EB1, EB2 and output data blocks AB1, AB2 in an example of a convolutional layer, followed by a concatenation layer that combines the input data block EB1. From Figure 3a, it can be seen that although the output data block of the convolutional layer supplements other input data for the concatenation layer, another copy operation is performed to supply the output data block of the concatenation layer.

[0045] Figure 3b shows that the output data block AB1 of the convolutional layer is located entirely within the address area of ​​the memory region of the input data block EB2 of the connected layer, and since the input data block EB2 is a portion of the output data block AB2, a copy operation is unnecessary to supply the output data block AB2.

Claims

1. A computer-implemented method for performing memory planning for code generation to determine the code for computation of a neural network, comprising the following steps: - Step (S1) provides sequential computation steps of the neural network, wherein the size of the input data blocks (EB1, EB2) and the size of the output data blocks (AB1, AB2) are determined for each computation step, - A step (S2) of calculating memory planning conditions for each predetermined calculation step in which an input data block (EB1) in a memory area is at least partially included in an output data block (AB2), wherein the conditions indicate that the memory area allocated to the output data block (AB1) of a calculation step preceding the predetermined calculation step is included in the memory area of ​​the output data block (AB2) of the predetermined calculation step, - A step (S3) of performing memory planning, which involves setting the memory areas of the respective input data blocks (EB1, EB2) and output data blocks (AB1, AB2) in the main memory for each calculation step, taking into account the calculated conditions. A method that includes this.

2. The method according to claim 1, wherein the execution of the memory planning includes applying an optimization method in which the objective function minimizes the number of memory operations.

3. The method according to claim 1 or 2, wherein the input data blocks (EB1, EB2) and the output data blocks (AB1, AB2) each represent or are allocated to a memory region having sequentially consecutive ascending addresses.

4. If the predetermined calculation step is a concatenation step, the condition indicates that the calculation step preceding the concatenation step supplies output data blocks (AB1, AB2) into a memory area directly adjacent to the memory area to be concatenated in the concatenation step. The method according to any one of claims 1 to 3.

5. If the predetermined calculation step is a padding step, the condition indicates that the calculation step preceding the padding step supplies the output data block into a memory area directly adjacent to the memory area occupied by the padding pattern or to the memory area to which the padding pattern is subsequently written. The method according to any one of claims 1 to 4.

6. If the predetermined calculation step is a slicing step, the condition indicates that the calculation step preceding the slicing step supplies the output data block to a memory area that partially corresponds to the output data block of the predetermined calculation step. The method according to any one of claims 1 to 5.

7. The method according to any one of claims 1 to 6, wherein code generation for the hardware environment (2) is performed based on the results of the memory planning and implemented at that location.

8. An apparatus for carrying out the method according to any one of claims 1 to 7.

9. A computer program product that includes instructions for causing a data processing device to perform a step of the method according to any one of claims 1 to 7, when the program is executed by such data processing device.

10. A machine-readable storage medium that includes, when executed by at least one data processing device, an instruction causing the data processing device to perform a step of the method according to any one of claims 1 to 7.

Images