Device and computer-implemented method for determining a memory plan for performing operations

Abstract

Device and computer-implemented method for determining a memory plan (200) for performing operations, in particular of an artificial neural network, wherein a list of memory areas required for performing the operations is created, wherein, depending on the list for the operations, it is determined which memory areas must be present in a first memory (202) for performing the respective operation and which memory areas may be present in a second memory (204) during the execution of the respective operation, wherein the memory plan (200) is determined depending on whether a memory area must be present in main memory for execution or may be present in secondary memory.

Description

State of the art

[0001] The invention relates to a device and a computer-implemented method for determining a memory plan for performing operations, in particular an artificial neural network.

[0002] Memory planning is an integral part of a code generator for neural networks. It is responsible for managing memory and determines at which addresses in memory data is stored during the computation of the neural network. Disclosure of the invention

[0003] A computer-implemented method for determining a memory layout for executing operations, particularly of an artificial neural network, involves creating a list of memory locations required for the operations. Depending on this list, the method determines which memory locations must be present in a primary memory for the execution of each operation and which memory locations may be present in a secondary memory during the execution of the respective operation. The memory layout is determined based on whether a memory location must be present in main memory for execution or may be present in secondary memory. The list enables efficient computation of the memory layout, allowing for the simultaneous management of multiple memory locations and minimizing the number of copies required between them.

[0004] It may be provided that for each operation, an assignment of the memory areas is determined either to the first memory or to the second memory.

[0005] It may be possible to determine the allocation that minimizes the number of memory transfers between the first and second storage locations. This reduces the amount of bytes that need to be transferred.

[0006] For example, the assignment is determined using a satisfiability modulo theories solver.

[0007] It may be provided that for each operation, the memory areas in the list that do not need to be in the first memory are marked, whereby the memory plan is determined depending on the list with the marked memory areas.

[0008] Memory areas are allocated, for example, for executing operations according to the memory plan, whereby memory areas are transferred from the first memory to the second memory or vice versa, according to the memory plan. This reduces the size of the required first memory and thus enables its use on hardware with limited primary memory capacity. This approach has the advantage of efficiently utilizing multiple memory locations.

[0009] A device for determining a memory plan for executing operations, in particular of an artificial neural network, comprises at least one processor and at least one memory, wherein the at least one memory comprises instructions executable by the at least one processor, the execution of which by the at least one processor carries out the procedure.

[0010] A computer program may be provided, wherein the computer program includes computer-readable instructions, the execution of which by a computer carries out the procedure.

[0011] Further advantageous embodiments can be found in the following description and the drawing. The drawing shows: Fig. 1. a part of the network architecture of an artificial neural network, Fig. 2 an example storage plan, Fig. 3. A flowchart showing the steps of a procedure for determining the memory plan. Fig. 4. An exemplary list of memory areas, Fig. 5. An example of an allocation of memory areas, Fig. 6 a schematic representation of a device for determining the memory plan.

[0012] In Fig. Figure 1 shows part of a network architecture 100 of an artificial neural network.

[0013] The artificial neural network comprises a first layer 102 and a second layer 104.

[0014] The first layer 102 is configured to map an input x of the first layer 102 to an input of the second layer 104.

[0015] The second layer 104 is configured to map the input of the second layer 104 to an output of the second layer 104.

[0016] The first layer 102 and the second layer 104 apply a function F(x) to the input x.

[0017] The artificial neural network includes, for example, skip connections. A skip connection can skip one or more layers of the network. For instance, a skip connection allows an output from an inner layer of the artificial neural network to be reintroduced into the computation at a later time. This counteracts, for example, vanishing gradients and simplifies the optimization process during the training of the artificial neural network.

[0018] An example of a skip connection in network architecture 100 is a connection 106 to an operand 108. The operand 108 applies an operation to the input x and the result of the function F(x). The connection 106 represents a skip connection that bypasses the first layer 102 and the second layer 104.

[0019] In the example, operand 108 is addition, i.e., F(x) + x. Operand 108 could be another operation, e.g., multiplication or subtraction.

[0020] Skip connections, especially when performing calculations on embedded hardware, result in increased memory requirements. For example, during a calculation, a temporary result, such as the input x, is held in memory across multiple operations, such as the calculation of the function F(x). With limited available memory, the memory available for the operations is therefore reduced by the amount of memory required to store the temporary result.

[0021] An extreme case is a network architecture in which the input of the artificial neural network is reused to compute the last layer of the artificial neural network.

[0022] In this network architecture, only a limited portion of the memory is available for practically the entire computation time. This is achieved through a [missing information - likely a specific feature or feature]. Fig. The exemplary storage plan 200 shown was avoided.

[0023] The example memory plan 200 provides two storage locations for data: a first storage location 202 and a second storage location 204. Each storage location has storage blocks. A storage block defines a storage area for the data.

[0024] The exemplary memory plan 200 provides for four operations: a first operation 206, a second operation 208, a third operation 210 and a fourth operation 212.

[0025] The first operation 206 processes an input 214. The first operation 206 generates a first output 216. The input 214 is assigned to the first memory 202. The first output 216 is assigned to the first memory 202.

[0026] The second operation 208 processes the first output 216. The second operation 208 generates a second output 218. The input 214 is assigned to the second memory 204. The second output 218 is assigned to the first memory 202.

[0027] The third operation, 210, processes the second output, 218. The third operation, 210, generates a third output, 220. Input 214 is assigned to the second memory, 204. The third output, 220, is assigned to the first memory, 202.

[0028] The fourth operation, 212, processes input 214 and the third output 220. The fourth operation, 212, generates a fourth output, 222. Input 214 is assigned to the first memory, 202. The fourth output, 222, is assigned to the first memory, 202.

[0029] The memory plan 200 provides that the operations take place sequentially, with the output of one operation representing the input of a subsequent operation.

[0030] Input 214 is used in the first operation 206 and the fourth operation 212.

[0031] Input 214 is temporarily stored in the second memory 204 for the calculation of the second operation 208 and the third operation 210. By temporarily storing input 214 in the second memory 204, space is freed up in the first memory 202 for storing the second output 218 and the third output 222. Input 214 is stored in the first memory 202 for the calculation of the first operation 206 and the fourth operation 212.

[0032] The memory plan 200 provides, for example, that the input 214 for the calculation of the first operation 206 is stored in a first memory block 222 and the first output 216 is stored in a second memory block 224.

[0033] The memory plan 200 provides, for example, that the first output 216 for calculating the second operation 208 is stored in a third memory block 226 and the second output 218 is stored in a fourth memory block 228.

[0034] The memory plan 200 provides, for example, that the input 214 is stored in a fifth memory block 230 during the calculation of the second operation 208.

[0035] The memory plan 200 provides, for example, that the second output 218 for calculating the third operation 210 is stored in a sixth memory block 232 and the third output 222 is stored in a seventh memory block 234.

[0036] The memory plan 200 provides, for example, that the input 214 is stored in an eighth memory block 236 during the calculation of the third operation 210.

[0037] The memory plan 200 provides, for example, that the third output 222 for calculating the fourth operation 212 is stored in a ninth memory block 238, the input 214 in a tenth memory block 238 and the fourth output 222 in an eleventh memory block 244.

[0038] It may be possible to minimize the number of copies required between memory blocks when calculating memory plan 200 by using the same memory area for the same data whenever possible. This reduces the additional overhead of memory transfers while enabling significant memory savings.

[0039] For example, memory plan 200 provides that after the first operation 206, input 214 is copied from the first memory 202 into a memory block in the second memory 204, remains in the same memory block in the second memory 204 during the calculation of the second operation 208 and the third operation 210, and is copied into a memory block in the first memory 202 for the calculation of the fourth operation 212.

[0040] For example, memory plan 200 provides that the first output 216 is copied into a memory block of the first memory 202 after the first operation 206 and is read from the same memory block to calculate the second operation 208.

[0041] For example, memory plan 200 provides that the first output 216 is copied into a memory block of the first memory 202 after the first operation 206 and is read from the same memory block to calculate the second operation 208.

[0042] In Fig. Figure 3 is a flowchart illustrating the steps of a procedure for determining a memory plan. The procedure is described using memory plan 200 as an example. The procedure uses main memory for calculating operations and intermediate memory for temporary storage. The main memory is, for example, the first memory 202. The intermediate memory is, for example, the second memory 204.

[0043] The procedure includes step 302.

[0044] Step 302 creates a list of memory areas.

[0045] The memory areas have a specified lifetime. This indicates at what times the memory area must be present in the overall memory.

[0046] Then, step 304 is executed.

[0047] In step 304, it is determined which memory areas in main memory, i.e. in the first memory 202, must be present for the execution of the respective operation.

[0048] A memory area or areas that do not need to be in main memory for a given operation can potentially be moved to the cache for the execution of that operation.

[0049] In step 304, it is determined which memory areas may be present in the intermediate memory, i.e., in the second memory 204, during the execution of the respective operation.

[0050] For example, the memory areas in the list are marked accordingly.

[0051] Then step 306 is executed.

[0052] In step 306, the memory plan is determined depending on whether a memory area must be present in main memory for execution, or may be present in secondary memory.

[0053] For example, for each operation, an allocation of memory areas is determined, either to main memory or to temporary storage.

[0054] For example, the allocation is determined that minimizes the number of transfers of memory areas between main memory and secondary memory.

[0055] The assignment is determined, for example, using a satisfiability modulo theories solver, or SMT solver. An example of an SMT solver is the SMT solver Z3: LM de Moura and N. Bjørner. Z3: An efficient smt solver. In CR Ramakrishnan and J. Rehof, editors, TACAS, volume 4963 of Lecture Notes in Computer Science, pages 337-340. Springer, 2008.

[0056] The procedure may include a step 308.

[0057] In step 308, the memory areas for executing the operations according to memory plan 200 are made available.

[0058] This means that the memory areas are transferred from the first memory 202 to the second memory 204 according to memory plan 200.

[0059] This means that the memory areas will be transferred from the second memory 204 to the first memory 202 according to memory plan 200.

[0060] In Fig. Example 400 is given for the list with the first operation 206, the second operation 208, the third operation 212, and a result 402. The checkmark indicates a required lifetime for the data marked with the checkmark. The cross indicates times when no memory is required for the data marked with the cross.

[0061] For memory plan 200, the first operation 206, the second operation 208, the third operation 210, and the fourth operation 212 are marked as the lifetime of input 214. For memory plan 200, the first operation 206 and the second operation 208 are marked as the lifetime of the first output 216. For memory plan 200, the second operation 208 and the third operation 210 are marked as the lifetime of the second output 218. For memory plan 200, the third operation 210 and the fourth operation 212 are marked as the lifetime of the third output 220. For memory plan 200, the fourth operation 212 and the result 402 are marked as the lifetime of the fourth output 222.

[0062] In Fig. 5 is an example 500 for the assignment with the first operation 206, the second operation 208, the third operation 212 and the result 402.

[0063] In the mapping, the number 1 indicates that the data marked with the number 1 is assigned a memory location in main memory, e.g., in the first memory location 202. In the mapping, the number 2 indicates that the data marked with the number 2 is assigned a memory location in temporary memory, e.g., in the second memory location 202. In the mapping, a cross indicates that no memory is required for the data marked with the cross.

[0064] For the memory plan 200, the input 214 is assigned the number 1 for the lifetime during the first operation 206 and the fourth operation 212, i.e. the first memory 202, and the number 2 for the lifetime during the second operation 208 and the third operation 210, i.e. the second memory 204.

[0065] For memory plan 200, the first output 216 is assigned the number 1, i.e., the first memory 202, for the lifetime of the first operation 206 and the second operation 208. For memory plan 200, the second output 218 is assigned the number 1, i.e., the first memory 202, for the lifetime of the second operation 208 and the third operation 210. For memory plan 200, the third output 220 is assigned the number 1, i.e., the first memory 202, for the lifetime of the third operation 210 and the fourth operation 212. For memory plan 200, the fourth output 222 is assigned the number 1, i.e., the first memory 202, for the lifetime of the fourth operation 212 and during the result 402. At all other times, a cross, i.e., no memory, is assigned to the input 214 and the respective outputs.

[0066] This means that memory plan 200 provides for a memory transfer of 502 from the first memory 202 to the second memory 204 between the first operation 206 and the second operation 208. This also means that memory plan 200 provides for a memory transfer of 504 from the second memory 204 to the first memory 202 between the third operation 210 and the fourth operation 212.

[0067] In Fig. 6 is a device 600 for determining the memory plan 200 for executing operations of the artificial neural network with the network architecture 100 schematically represented.

[0068] The device 600 comprises at least one processor 602 and at least one memory 604.

[0069] At least one memory 604 comprises the first memory 202 and the second memory 204.

[0070] The at least one memory 604 comprises instructions executable by the at least one processor 602, the execution of which by the at least one processor 602 carries out the procedure for determining the memory plan 200.

[0071] The method and the device 600 are described as an example for determining the memory layout 200 for two memories. The device 600 can provide for more than two memories. The method is applied accordingly for more than two memories. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] L. M. de Moura and N. Bjørner. Z3: An efficient smt solver. In C. R. Ramakrishnan and J. Rehof, editors, TACAS, volume 4963 of Lecture Notes in Computer Science, pages 337-340. Springer, 2008

[0055]

Claims

[1] Computer-implemented method for determining a memory plan (200) for performing operations, in particular of an artificial neural network, characterized by , that a list (400) of memory areas required for the execution of the operations is created (302), wherein depending on the list for the operations it is determined (304) which memory areas must be present in a first memory (202) for the execution of the respective operation and which memory areas may be present in a second memory (204) during the execution of the respective operation, wherein the memory plan (200) is determined depending on (306) whether a memory area must be present in main memory for execution or may be present in secondary memory. [2] Method according to claim 1, characterized by , that for each operation an assignment (500) of the memory areas is determined either to the first memory (202) or to the second memory (204) (306). [3] Method according to claim 2, characterized by , that the allocation (500) is determined (306) which minimizes a number of transfers of memory areas between the first memory (202) and the second memory (204). [4] Method according to one of claims 2 or 3, characterized by , that the assignment (500) is determined using a satisfiability modulo theories solver (306). [5] Method according to any one of the preceding claims, characterized by , that for each operation the memory areas in the list (400) are marked (304) which do not have to be in the first memory (202), whereby the memory plan (200) is determined depending on the list (400) with the marked memory areas (306). [6] Method according to any one of the preceding claims, characterized by, that the memory areas are provided for the execution of the operations according to the memory plan (200) (308), wherein memory areas are transferred from the first memory (202) to the second memory (204) or from the second memory (204) to the first memory (202) according to the memory plan (200). [7] Device (600) for determining a memory plan (200) for performing operations, in particular of an artificial neural network (606), characterized by , that the device (600) comprises at least one processor (602) and at least one memory (604), wherein the at least one memory (604) comprises instructions executable by the at least one processor (602), the execution of which by the at least one processor (602) is carried out according to one of the preceding claims. [8] Computer program, characterized by, that the computer program comprises computer-readable instructions, the execution of which by a computer results in the procedure according to one of claims 1 to 6.

Images