Apparatus and computer-implemented method for determining memory plan for performing operation

Abstract

Apparatus and computer-implemented method for determining a memory plan (200) for performing an operation, in particular an operation of an artificial neural network, in which a list of memory regions required for performing the operation is created, in which, for the operation, which memory regions must be present in a first memory (202) for performing the corresponding operation is determined from the list, and which memory regions are allowed to be present in the second memory (204) during execution of the respective operation, wherein the memory plan (200) is determined depending on whether the memory regions must be present in the main memory for execution or are allowed to be present in the secondary memory.

Description

Technical Field

[0001] The present invention relates to an apparatus and a computer-implemented method for determining memory planning for performing operations, particularly operations of artificial neural networks. Background Technology

[0002] Memory planning is an integral part of a neural network code generator. It is responsible for memory management and specifies which memory addresses hold data during neural network computation.

[0003] DE 3232675 A1 discloses a method for controlling data access in a computer and a data control system for performing the method. Summary of the Invention

[0004] A computer implementation method for determining memory planning for performing operations on an artificial neural network specifies the creation of a list of memory regions required to perform the operations. Based on this list, it is determined which memory regions must reside in a first memory for the operation to be performed, and which memory regions are permitted to reside in a second memory during the operation. The memory planning is determined based on whether the memory regions must reside in the first memory for execution or are permitted to reside in the second memory. This list enables efficient computation of the memory plan, which allows for the simultaneous management of multiple memories and minimizes the number of copies required between memory regions.

[0005] For each operation, determine the allocation relationship of the memory region to the first memory or the second memory.

[0006] The allocation relationship is determined to minimize the number of times memory regions are moved between the first and second memories. This reduces the number of bytes that must be moved.

[0007] For example, the assignment relation can be determined using a solver based on satisfiability modulo theories.

[0008] For each operation, memory regions that do not need to be located in the first memory are marked in a list, and memory planning is determined based on the list of marked memory regions.

[0009] For example, memory regions for performing operations are provided according to memory planning, wherein memory regions are transferred from a first memory to a second memory, or from a second memory to a first memory, according to the memory planning. This reduces the required size of the first memory and thus makes it possible to use on hardware with limited memory size in the first memory. This method has the advantage of efficiently utilizing multiple memories.

[0010] An apparatus for determining memory planning for performing operations, particularly operations of an artificial neural network, includes at least one processor and at least one memory, wherein the at least one memory includes instructions executable by the at least one processor, and when these instructions are executed by the at least one processor, the method is run.

[0011] A computer program may be proposed, comprising computer-readable instructions that, when executed by a computer, run the method. Attached Figure Description

[0012] Further advantageous embodiments can be seen from the following description and accompanying drawings. In the drawings: Figure 1 This shows a portion of the network architecture of an artificial neural network. Figure 2 An exemplary memory layout is shown. Figure 3 A flowchart showing the steps of a method for determining memory planning is shown. Figure 4 An exemplary list of memory regions is shown. Figure 5 An example of memory region allocation is shown. Figure 6 A schematic diagram of a device for determining memory planning is shown. Detailed Implementation

[0013] exist Figure 1 The diagram shows a portion of the network architecture 100 of an artificial neural network.

[0014] The artificial neural network consists of a first layer 102 and a second layer 104.

[0015] The first layer 102 is configured to map the input x of the first layer 102 to the input of the second layer 104.

[0016] The second layer 104 is configured to map the input of the second layer 104 to the output of the second layer 104.

[0017] The function F(x) is applied to the input x through the first layer 102 and the second layer 104.

[0018] This artificial neural network includes, for example, skip connections. Skip connections can skip only one or more layers of the network. For example, skip connections allow the outputs of inner layers of the artificial neural network to be recomputed at a later time point. This, for example, combats vanishing gradients and simplifies the optimization process during the training of the artificial neural network.

[0019] An example of a skip connection in network architecture 100 is connection 106 to operand 108. Operand 108 applies the operation to the input x and the result of the function F(x). Connection 106 represents a skip connection that skips the first layer 102 and the second layer 104.

[0020] In this example, operand 108 is addition, i.e., F(x) + x. Operand 108 can also be other operations, such as multiplication or subtraction.

[0021] Skip connections, especially when performing computations on embedded hardware, create increased memory requirements. For example, during computation, temporary results (e.g., input x) are held in memory across multiple operations used for computation (e.g., for calculating function F(x)). Therefore, with limited available memory, the remaining memory used for computation operations reduces the memory required to store temporary results.

[0022] An extreme case is a network architecture in which the input of an artificial neural network is reused to compute the last layer of that artificial neural network.

[0023] In this network architecture, only a limited portion of the memory is actually available throughout the entire computation time. This is achieved through... Figure 2 The exemplary memory planning 200 shown is to avoid this.

[0024] The exemplary memory plan 200 specifies two memories for data: a first memory 202 and a second memory 204. Each of these memories has a memory block. A memory block defines a memory region for data.

[0025] The exemplary memory plan 200 specifies four operations: a first operation 206, a second operation 208, a third operation 210, and a fourth operation 212.

[0026] Input 214 is processed using a first operation 206. The first operation 206 generates a first output 216. Input 214 is allocated to a first memory 202. The first output 216 is also allocated to the first memory 202.

[0027] The first output 216 is processed using the second operation 208. The second operation 208 generates a second output 218. Input 214 is allocated to the second memory 204. The second output 218 is allocated to the first memory 202.

[0028] The second output 218 is processed using the third operation 210. The third operation 210 generates the third output 220. Input 214 is allocated to the second memory 204. The third output 220 is allocated to the first memory 202.

[0029] The fourth operation 212 processes the input 214 and the third output 220. The fourth operation 212 produces a fourth output 222. The input 214 is allocated to the first memory 202. The fourth output 222 is allocated to the first memory 202.

[0030] Memory planning 200 specifies that operations are executed sequentially in time, and the output of one operation constitutes the input of the operation that follows that operation.

[0031] Input 214 is used in the first operation 206 and the fourth operation 212.

[0032] Input 214 is temporarily stored in the second memory 204 for calculating the second operation 208 and the third operation 210. By temporarily storing input 214 in the second memory 204, space is created in the first memory 202 for storing the second output 218 and the third output 222. Input 214 is stored in the first memory 202 for calculating the first operation 206 and the fourth operation 212.

[0033] For example, memory planning 200 specifies that the input 214 for calculating the first operation 206 is stored in the first memory block 222, while the first output 216 is stored in the second memory block 224.

[0034] For example, memory planning 200 specifies that the first output 216 for calculating the second operation 208 is stored in the third memory block 226, while the second output 218 is stored in the fourth memory block 228.

[0035] For example, memory planning 200 specifies that input 214 is stored in a fifth memory block 230 during the computation of the second operation 208.

[0036] For example, memory planning 200 specifies that the second output 218 used to calculate the third operation 210 is stored in the sixth memory block 232, while the third output 222 is stored in the seventh memory block 234.

[0037] For example, memory planning 200 specifies that input 214 is stored in the eighth memory block 236 during the computation of the third operation 210.

[0038] For example, memory planning 200 specifies that the third output 222 used to calculate the fourth operation 212 is stored in the ninth memory block 238, the input 214 is stored in the tenth memory block 238, and the fourth output 222 is stored in the eleventh memory block 244.

[0039] It can be stipulated that, when calculating memory planning 200, the number of copies required between memory blocks should be minimized by using the same memory region for the same data, if possible. This reduces the overhead caused by memory migration and achieves significant memory savings.

[0040] For example, memory planning 200 specifies that input 214 is copied from the first memory 202 to a storage block of the second memory 204 after the first operation 206, remains in the same storage block of the second memory 204 during the computation of the second operation 208 and the third operation 210, and is copied to a storage block of the first memory 202 for the computation of the fourth operation 212.

[0041] For example, memory planning 200 specifies that the first output 216 is copied to a memory block of the first memory 202 after the first operation 206, and is read from the same memory block in order to compute the second operation 208.

[0042] For example, memory planning 200 specifies that the first output 216 is copied to a memory block of the first memory 202 after the first operation 206, and is read from the same memory block in order to compute the second operation 208.

[0043] Figure 3 A flowchart illustrating the steps of a method for determining memory planning is shown. The method is described using memory planning 200 as an example. In this method, main memory is used for computational operations, and secondary memory is used for temporary storage. The main memory is, for example, a first memory 202. The secondary memory is, for example, a second memory 204.

[0044] The method includes step 302.

[0045] In step 302, a list of memory regions is created.

[0046] These memory regions have a specified lifetime. This lifetime indicates at what points in time the memory region must exist in the total memory. The first memory 202 and the second memory 204 constitute the total memory.

[0047] Then proceed to step 304.

[0048] In step 304, it is determined which memory regions must exist in the main memory (i.e., the first memory 202) to perform the corresponding operation.

[0049] One or more memory regions that do not need to be in main memory for the corresponding operation can potentially be moved to secondary memory in order to perform the corresponding operation.

[0050] In step 304, it is determined which memory regions are allowed to exist in the secondary storage (i.e., the second memory 204) during the execution of the corresponding operation.

[0051] For example, these memory regions are marked accordingly in the list.

[0052] Then proceed to step 306.

[0053] In step 306, the memory planning is determined based on whether the memory region must exist in the main memory for execution or is allowed to exist in the secondary memory.

[0054] For example, for each operation, determine the allocation relationship of memory regions to main memory or to secondary memory.

[0055] For example, determining the allocation relationship that minimizes the number of transfers of memory regions between main memory and secondary memory.

[0056] This assignment relation is determined, for example, using a satisfiability modulus-theoretic solver (SMT-solver). An example of an SMT solver is the SMT solver Z3: Z3: An efficient SMT solver by LM de Moura and N. Bjørner (described in CR Ramakrishnan and J. Rehof, editors, TACAS, Lecture Notes in Computer Science, Vol. 4963, pp. 337–340, Springer, 2008).

[0057] This method can be set to step 308.

[0058] In step 308, a memory region for performing operations is provided according to memory planning 200.

[0059] This means that the memory region is transferred from the first memory 202 to the second memory 204 according to the memory plan 200.

[0060] This means that the memory region is transferred from the second memory 204 to the first memory 202 according to the memory plan 200.

[0061] exist Figure 4 Example 400 illustrates a list with a first operation 206, a second operation 208, a third operation 212, and a result 402. A checkmark indicates the required lifetime of data marked with a checkmark. A cross indicates the time during which data marked with a cross does not require memory.

[0062] For memory planning 200, the first operation 206, the second operation 208, the third operation 210, and the fourth operation 212 are marked as the lifetimes of input 214. For memory planning 200, the first operation 206 and the second operation 208 are marked as the lifetimes of the first output 216. For memory planning 200, the second operation 208 and the third operation 210 are marked as the lifetimes of the second output 218. For memory planning 200, the third operation 210 and the fourth operation 212 are marked as the lifetimes of the third output 220. For memory planning 200, the fourth operation 212 and the result 402 are marked as the lifetimes of the fourth output 222.

[0063] exist Figure 5 Example 500 illustrates the assignment relationship with first operation 206, second operation 208, third operation 212 and result 402.

[0064] In this allocation relationship, data marked with the number 1 is allocated to a memory region in main memory (e.g., first memory 202). Data marked with the number 2 is allocated to a memory region in secondary memory (e.g., second memory 204). Data marked with a cross does not require memory.

[0065] For memory planning 200, input 214 is allocated number 1 (i.e., first memory 202) during the lifetime of the first operation 206 and the fourth operation 212, and input 214 is allocated number 2 (i.e., second memory 204) during the lifetime of the second operation 208 and the third operation 210.

[0066] For memory planning 200, the first output 216 is allocated the number 1 (i.e., first memory 202) during the lifetime of the first operation 206 and the second operation 208. For memory planning 200, the second output 218 is allocated the number 1 (i.e., first memory 202) during the lifetime of the second operation 208 and the third operation 210. For memory planning 200, the third output 220 is allocated the number 1 (i.e., first memory 202) during the lifetime of the third operation 210 and the fourth operation 212. For memory planning 200, the fourth output 222 is allocated the number 1 (i.e., first memory 202) during the lifetime of the fourth operation 212 and the result 402. At other times, input 214 and the corresponding output are assigned crosses, i.e., no memory is allocated.

[0067] This means that memory planning 200 specifies a memory transfer 502 from the first memory 202 to the second memory 204 between the first operation 206 and the second operation 208. This means that memory planning 200 specifies a memory transfer 504 from the second memory 204 to the first memory 202 between the third operation 210 and the fourth operation 212.

[0068] exist Figure 6 The diagram schematically illustrates an apparatus 600 for determining a memory plan 200 for performing operations of an artificial neural network having a network architecture 100.

[0069] The device 600 includes at least one processor 602 and at least one memory 604.

[0070] The at least one memory 604 includes a first memory 202 and a second memory 204.

[0071] The at least one memory 604 includes instructions executable by the at least one processor 602, which, when executed by the at least one processor 602, run a method for determining memory planning 200.

[0072] The method and apparatus 600 are exemplarily described for determining a memory plan 200 for two memories. The apparatus 600 can accommodate more than two memories. The method is correspondingly applied to more than two memories.

Claims

1. A computer-implemented method for determining a memory plan (200) for performing operations of an artificial neural network, characterized in that, creating (302) a list (400) of memory regions required for performing the operations, wherein from the list it is determined (304) for the operations which memory regions have to be present in a first memory (202) for performing the respective operation and which memory regions are allowed to be present in a second memory (204) during performing the respective operation, wherein the memory plan (200) is determined (306) from whether a memory region has to be present in the first memory (202) for performing or is allowed to be present in the second memory (204), wherein for each operation, an assignment relationship (500) of the memory regions to the first memory (202) or the second memory (204) is determined (306), wherein for each operation, memory regions not necessarily present in the first memory (202) are marked (304) in the list (400), wherein the memory plan (200) is determined (306) from the list (400) of marked memory regions, wherein the assignment relationship (500) is determined (306) which minimizes the number of transfers of memory regions between the first memory (202) and the second memory (204).

2. The method of claim 1, wherein, determining (306) the assignment relationship (500) with a satisfiability modulo theories solver.

3. The method according to any of the preceding claims, characterized in that, providing (308) memory regions for performing operations according to the memory plan (200), wherein memory regions are transferred from the first memory (200) to the second memory (204) or from the second memory (204) to the first memory (202) according to the memory plan (200).

4. An apparatus (600) for determining a memory plan (200) for performing operations, in particular operations of an artificial neural network (606), characterized in that The apparatus (600) comprises at least one processor (602) and at least one memory (604), wherein the at least one memory (604) includes instructions executable by the at least one processor (602), which when executed by the at least one processor (602) perform the method according to any of the preceding claims.

5. A computer program, characterised in that, The computer program comprises computer readable instructions which when executed by the computer perform the method according to any of claims 1 to 3.

Images