Method and apparatus for code generation for program code for computing artificial neural network in hardware environment
By determining the maximum overlap area between input and output data blocks for each computational step of a computational neural network, storage planning is optimized, solving the problem of inefficient use of storage resources in existing technologies, and achieving a significant reduction in storage requirements and efficient use of resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-10-17
- Publication Date
- 2026-04-21
AI Technical Summary
Existing code generators fail to make efficient use of limited memory resources when generating program code for computing neural networks, resulting in excessive storage requirements, especially when allocating input and output data blocks for computational layers, where storage areas are not efficiently overlapped.
By determining the maximum overlap area between input and output data blocks for each computation step, storage planning is optimized so that input and output data blocks partially or completely overlap in the storage area, only rewriting data elements that are no longer used, thus reducing storage requirements.
It significantly reduces the storage locations required for computing neural networks, improves the efficiency of storage resource utilization, and reduces the overall demand for working memory.
Smart Images
Figure CN121900737A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to implementing program code in a hardware environment, such as a control device controlled by a microcontroller. Furthermore, this invention relates to a method for performing memory planning for processing input data, output data, and network parameters. Background Technology
[0002] Specific hardware environments, such as microcontrollers in control devices, require the creation of adapted executable code to account for the characteristics and limitations of that specific hardware environment. Consequently, the available size of the working memory that the microcontroller can directly access may be limited, or memory shifting or copying operations from flash or external storage to working memory may be particularly costly, depending on the hardware.
[0003] The computational steps used to compute layers of a neural network can require significant storage because at least one block of input data and at least one block of output data must be stored in working memory in a way that is callable and usable by the microcontroller for each computational layer.
[0004] Existing code generators specify during storage planning which area of working memory each data block required for a computation layer will be stored in. During storage planning, in addition to allocating input and output data blocks to storage areas, corresponding storage areas are also assigned to data formed within the computation layer during computation.
[0005] Traditional code generators for neural networks are typically not based on restricted data memory and often use completely separate memory blocks for each computational layer in successive computational layers to provide input and output data blocks.
[0006] The objective of this invention is to provide improved memory management for computational artificial neural networks, wherein the total number of storage locations required in the working memory can be reduced. Summary of the Invention
[0007] This task is accomplished by the method for performing storage planning for code generation for computing neural networks as described in claim 1, and by the apparatus according to the parallel claims.
[0008] Other design options are described in the dependent claims.
[0009] According to the first aspect, a method for configuring a computer implementation for determining code generation execution storage planning for computing neural networks is provided, comprising the following steps: - Provide successive computational steps for the neural network, wherein for each computational step, the size of one or more input data blocks and one or more output data blocks is determined; - For each computation step, determine the maximum overlap area between the respective input data blocks and the respective output data blocks; - Execute storage planning based on the maximum overlapping area for each computation step, wherein the storage areas for the corresponding input data blocks and output data blocks are specified in the working memory.
[0010] The computational steps of a computational layer used to compute a neural network are typically computed serially on a hardware environment. This means that the code generated for the hardware environment specifies the order in which input data is processed and the corresponding output data of the computational layer is produced. Input data for a computational step typically exists as sequentially stored data elements (bytes, words). Furthermore, due to serial processing, the input data is processed in the order it is stored in the storage area. This can lead to situations where, depending on the type of computational step, data elements of the input data used at the beginning of the computation are not further used or invoked in further processing of the corresponding computational layer of the neural network. This occurs, for example, in computational steps such as those computed in convolutional layers, depthwise convolutions, pooling operations, and computational steps with element-wise operations.
[0011] The above method, in accordance with the storage planning requirements for code generation, sets up input data blocks and output data blocks in overlapping storage areas, such that when the input data is processed element by element, the data blocks of the separately generated data elements of the output data cover a portion of the storage area of the input data blocks.
[0012] However, the rewriting is performed only in such a way that data elements that are still used for computational operations in the corresponding computational steps are not rewritten; instead, only data elements that have been used for operations and computational steps and are no longer used are rewritten. This significantly reduces the storage requirements for computational steps in computational layers of neural networks, which are configured to process input data element-wise.
[0013] The extent to which input data blocks and output data blocks can overlap depends on the memory access patterns of the computation steps. For example, in the convolution computation step of a convolutional layer, the size of the effective kernel and the intermediate memory present if necessary determine the degree of overlap between the input and output data blocks.
[0014] The effective kernel size is derived from the actual kernel size and possible expansion. For example, in a computational step involving element-wise addition of two or more input data blocks, other computational steps can immediately and completely rewrite that input data block (one of the input data blocks).
[0015] The advantage of the above method is that it can greatly reduce the storage space required to compute neural networks.
[0016] The calculations for storage planning begin with a list of storage blocks that should be intermediately stored in corresponding storage areas of the working memory according to the queued calculation steps. A list of assignment attributes is provided for each storage block, which, for example, describes the storage block's lifetime and the operation to which it is assigned.
[0017] For code generation, storage planning is determined for each computation step by iterating over the computation steps: in which storage region should the data blocks used for input and output data be stored. Traditionally, it is important to note here that all storage regions used for data blocks must be distinct, such that there is no overlap between storage regions during their lifetime.
[0018] The potential overlap between the storage region used for the input data block and the storage region used for the output data block is called the overlap region, and it depends on the type of computation step or the layer to be computed in the neural network. Different types of computation steps can be used to determine the overlap of the storage regions.
[0019] The overlapping region corresponds to the area where the end (maximum address) of the storage region of the output data block and the beginning (minimum address) of the storage region of the input data block can overlap. It is assumed here that the storage region is read or written at substantially increasing addresses from the beginning address of the storage region to the end address, which is the end address of the storage region.
[0020] For element-wise operations, such as addition, multiplication, and subtraction of elements with the same index from two input data blocks, the storage area for the output data block can be set to completely overlap with the storage area of one of the input data blocks, because the associated result can overwrite the input data elements used for the computed operation accordingly.
[0021] Similar to element-wise operations, the computational step for applying an activation function (such as ReLU (not for SoftMax)) computes the data elements of the output data block for each data element of the input data block. Therefore, as in the case of element-wise computational steps, the storage area set for the output data block can completely overlap with the storage area used for the input data block.
[0022] The computational steps used for convolutional or pooling layers have more complex memory access patterns. On one hand, data elements for the output data block are computed from multiple data elements of the input data block; on the other hand, padding and stride complicate the relationship between the storage regions of the input and output data blocks. Furthermore, in these cases, the degree of maximum overlap depends on the specific implementation of the network layer and is therefore only derivable if the network layer is known. Both types of computational steps, convolution and pooling, compute the corresponding data elements for the output data block from a sliding window that moves over the input data block. The maximum distance between the data elements of the output data block and the first required data element of the input data block is decisive for the magnitude of the maximum overlap of the storage regions.
[0023] The related calculation principles are generally based on the following: memory is organized by rows and columns, and storage regions have consecutive address ranges. In principle, the storage region of the input data block is processed from the lowest address to the highest address within that region. Let x and y be the address coordinates of the data elements in the input data block, where x corresponds to the column address and y corresponds to the row address. Furthermore, assume a stride... x and stride y To manipulate the step size in the x and y directions, pad x and pad y Corresponding to the padding operation in the x and y directions, ReLU is the corrected linear unit function, and the input... x input y and input ch Let the width, height, and number of channels be defined for the input data. Then, for all operations of convolution and pooling with the above access patterns, the index of the required first data element in memory can be calculated according to the following formula: If using output x , output y and output ch Given the width, height, and number of channels representing the output data, the index of the output element in memory is derived as follows: .
[0024] In order to calculate the largest possible overlap from these formulas, additional information about the specific implementation of the calculation steps is still needed.
[0025] In many implementations, the input data needed to compute the output elements is first collected in contiguous storage areas. Since this input data has then been read, it can be rewritten if necessary for later output elements. As an additional optimization step, data is typically pre-collected not only for a single output element, but for a specific implementation-dependent number of output elements. This increases the maximum possible overlap.
[0026] Without pre-collecting data, the maximum possible overlap is calculated as follows: in In the case of pre-collection, the maximum possible overlap correspondingly increases the number of intermediate storage elements.
[0027] The maximum overlap (max_overlap) now describes the displacement by which the end region of the storage area of the output data block and the beginning region (highest address) of the storage area of the input data block can overlap to the maximum extent. The maximum overlap is determined by the maximum value of all distances between the storage addresses of the data elements obtained in the output data block and the first data element (lowest address) accessed in the input data block for all data elements, relative to the relevant calculations.
[0028] Determining the maximum overlap (max_overlap) allows for greater flexibility in arranging storage regions for output data blocks during storage planning for code generation. Attached Figure Description
[0029] The preferred embodiments are explained in more detail below with reference to the accompanying drawings. Wherein: Figure 1 A schematic diagram is shown of a storage plan for computing a neural network in multiple computational steps with non-overlapping storage regions. Figure 2 A schematic diagram of a platform used for code generation and implementation in a hardware environment is shown; Figure 3 A flowchart illustrating a method for memory planning within the scope of code generation for computing neural networks in a hardware environment with limited storage space; and Figure 4 This diagram illustrates the arrangement of storage areas for input and output data blocks in the case of potential overlap between storage areas used for data blocks. Detailed Implementation
[0030] Within the scope of code generation, storage planning is required. This storage plan, for each computational step of the neural network, pre-defines storage areas for one or more input data blocks and for one or more output data blocks within their respective lifetimes. Here, storage planning is conventionally performed, only specifying that storage areas for input and output data blocks are released when their lifetimes expire. Here, the lifetime is always given until the end of the computational step or a subsequent computational step. Figure 1 The diagram schematically illustrates the storage segments arranged in a traditional storage plan, showing the input data blocks EB1, EB2, EB3, and EB4 used for different computation steps O1, O2, O3, and O4, and the resulting output data blocks AB1, AB2, AB3, and AB4.
[0031] Figure 2 A block diagram of platform 1 is shown for performing code generation and the implementation of the generated program code in hardware environment 2. Code generation is performed on a conventional computer 3 or workstation with a pre-defined neural network configuration. Computer 3 is configured to perform storage planning and code generation, wherein storage planning first allocates storage areas for each computational step of the neural network to accommodate at least one input data block, at least one output data block, and at least one model parameter block. The model parameter block includes all model parameters required to compute the corresponding computational step, such as the weights and biases of fully connected layers.
[0032] If the code is generated, it is transmitted to hardware environment 2 and implemented or executed there.
[0033] Within the scope of the method described below, it can now be specified that the storage area for the output data block used in the computation step is set to at least partially overlap with the storage area for the input data block used in the computation step. This can significantly save storage space because the storage areas for the input and output data blocks do not need to be set to not overlap at all.
[0034] Storage planning is typically carried out iteratively in the form of optimization methods, and can have objectives such as reducing storage locations overall, minimizing copying and shifting operations of storage regions, etc.
[0035] Through the following combination Figure 3 The flowchart describes a possible approach where, for a specific type of computational step, an overlapping area is set between the storage regions used for input and output data blocks, increasing flexibility in storage planning. To this end, a maximum overlap area is pre-defined for each computational step.
[0036] For example, in Figure 4 China for Figure 1The neural network's multiple computational steps illustrate how the storage regions used for input data blocks EB1, EB2, EB3, EB4 and output data blocks AB1, AB2, AB3, AB4 can be arranged when they at least partially overlap. It is observed that there is a significantly lower storage requirement for the working memory M, and the total required working memory locations can be reduced by the saved component S.
[0037] Figure 3 The flowchart illustrates the method for memory planning. In principle, for each computation step, it is defined which storage region of the working memory the resulting output data blocks AB1, AB2, AB3, and AB4 can / should be stored in. The input data blocks for this computation step are either initially loaded into the working memory M, or their storage region is derived from previous computation steps as output data blocks. For this purpose, it is necessary to define or determine, in addition to unused storage regions, a maximum overlap region. The storage regions for output data blocks AB1, AB2, AB3, and AB4 can maximize overlap with the storage regions for input data blocks EB1, EB2, EB3, and EB4 used for the corresponding computation steps. This increases the available storage region that can be used to store the corresponding output data blocks AB1, AB2, AB3, and AB4.
[0038] In step S1, the neural network to be implemented is first given in advance in the form of a sequence of computation steps. The computation steps can typically include common functions of neural networks, such as convolutional layers, fully connected layers, pooling layers, cascaded layers, etc.
[0039] In step S2, for each calculation step, it is determined how large the maximum overlap area can be between the input data blocks EB1, EB2, EB3, EB4 and the output data blocks AB1, AB2, AB3, AB4.
[0040] If the computation step is an element-wise operation, where two or more segments of the same size in the input data blocks EB1, EB2, EB3, EB4, or two or more input data blocks EB1, EB2, EB3, EB4 are element-wise correlated, then the result of the element-wise operation can directly overwrite the storage location of a data element in one of the segments of the input data blocks EB1, EB2, EB3, EB4, or one of the input data blocks EB1, EB2, EB3, EB4, because the corresponding data element is not used in subsequent operations. The maximum overlap area then corresponds to the size of a segment of the input data blocks EB1, EB2, EB3, EB4, or one of the input data blocks EB1, EB2, EB3, EB4. Therefore, the storage area of the output data block can completely overlap with one of the segments of the input data blocks EB1, EB2, EB3, EB4 that are computed together.
[0041] If the computation step is a convolution or pooling computation step, multiple elements of the input data blocks EB1, EB2, EB3, EB4 are processed into a single data element of the output data blocks AB1, AB2, AB3, AB4. In this case, the size of the overlapping region can be determined by calculating the maximum distance between the address of the data element in the output data blocks AB1, AB2, AB3, AB4 and the address of the data element accessed first for a single computation among the multiple data elements used in that computation step. The computational criteria associated with this are generally based on the following: memory is organized by rows and columns, and storage regions have consecutive address ranges of successive addresses. In principle, the storage regions of the input data blocks EB1, EB2, EB3, EB4 are processed from the lowest address to the highest address within that storage region.
[0042] Let x and y be the address coordinates of data elements in the input data blocks EB1, EB2, EB3, and EB4, where x corresponds to the column address and y corresponds to the row address. Furthermore, assume a stride... x and stride y To manipulate the step size in the x and y directions, pad x and pad y Corresponding to the operation of padding in the x and y directions, ReLU is the rectified linear-unit function, and the input... x input y and input ch Let the width, height, and number of channels be defined for the input data. Then, for all operations of convolution and pooling with the above access patterns, the address of the required first data element in memory can be calculated according to the following formula: If using output x , output y and output ch Given the width, height, and number of channels representing the output data, the address of the output element in memory is obtained as follows: (2) Without prior collection of input data, the maximum possible overlap, max_overlap, is calculated as follows: in In the case of pre-collection, the maximum possible overlap correspondingly increases the number of intermediate stored elements.
[0043] This calculation is performed for each data element of the input data block, and the maximum value of the difference between such values is determined. The obtained maximum value corresponds to the size of the maximum overlap region max_overlap, which represents the storage region at the beginning of the storage region of the input data block, and the ends of the output data blocks AB1, AB2, AB3, and AB4 can overlap with the beginning of the storage region.
[0044] If the maximum overlap region has been determined for each computational step, memory planning is performed in step S3, for example, using an optimization method that considers the total required size of the working memory as the objective function. Specifically, the objective function can be given in advance: minimizing the total required size of the working memory. Memory planning can be performed using an SMT solver in a manner known per se.
[0045] In step S4, based on the storage plan, code generation is performed for hardware environment 2 and the code generation is implemented there.
Claims
1. A computer implementation for generating and performing storage planning for determining code for computing a neural network for use in a hardware environment (2), comprising the following steps: - Provide (S1) successive computation steps of the neural network, wherein for each computation step the size of at least one input data block (EB1, EB2, EB3, EB4) and at least one output data block (AB1, AB2, AB3, AB4) is determined; - For each computation step, determine the maximum overlap area between the input data blocks (EB1, EB2, EB3, EB4) and the output data blocks (AB1, AB2, AB3, AB4); - For each computation step, a (S3) storage plan is performed based on the maximum overlap region for each computation step, wherein the storage areas for the corresponding input data blocks (EB1, EB2, EB3, EB4) and output data blocks (AB1, AB2, AB3, AB4) are specified in the working memory.
2. The method of claim 1, wherein performing memory planning includes applying an optimization method, wherein the objective function considers minimizing the total working memory (M) required.
3. The method according to claim 1 or 2, wherein the corresponding input data blocks (EB1, EB2, EB3, EB4) and the corresponding output data blocks (AB1, AB2, AB3, AB4) are respectively designated or allocated to the storage regions, the storage regions having successively increasing addresses, wherein the calculation steps access the data elements of the input data blocks (EB1, EB2, EB3, EB4) in particular at increasing addresses.
4. The method according to any one of claims 1 to 3, wherein the maximum overlapping region corresponds to the region where the end (maximum address) of the storage region of the corresponding output data block (AB1, AB2, AB3, AB4) overlaps with the beginning (minimum address) of the corresponding storage region of the input data block (EB1, EB2, EB3, EB4), wherein the storage region is read from or written to in particular at addresses that substantially increase from the beginning address to the end address of the storage region.
5. The method according to any one of claims 1 to 4, wherein the calculation steps of setting up single calculations element by element, in particular the addition, multiplication and subtraction of elements with the same index, are allocated for the complete overlap of the storage area of the output data block (AB1, AB2, AB3, AB4) with the segment of the input data block (EB1, EB2, EB3, EB4) or the storage area of one of the input data blocks.
6. The method according to any one of claims 1 to 5, wherein a maximum overlap (max_overlap) is assigned to the computation steps used for the convolutional layer or pooling layer, the maximum overlap being derived from the following calculation criteria: in Where x and y correspond to the address coordinates of the data elements in the input data block, where x corresponds to the column address and y corresponds to the row address, and stride x and stride y Corresponding to the step size calculated in both the x and y directions, pad x and pad y Corresponding to the operation of padding in the x and y directions, ReLU corresponds to the modified linear unit function, and the input... x input y and input ch The width, height, and number of channels corresponding to the input data, and the output... x , output y and output ch The width, height, and number of channels corresponding to the output data.
7. The method according to any one of claims 1 to 6, wherein code generation for the hardware environment (2) is performed and implemented thereon based on the results of the storage planning.
8. An apparatus for performing the method according to any one of claims 1 to 7.
9. A computer program product comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 7.
10. A machine-readable storage medium comprising instructions that, when executed by at least one data processing device, cause the data processing device to perform the steps of the method according to any one of claims 1 to 7.