Memory allocation method and device for NPU heterogeneous platform
By combining time and space information on the NPU heterogeneous platform for static memory allocation, the problem of NPU's large memory usage when performing deep learning computing tasks in neural network model is solved, and memory multiplexing and efficiency improvement are achieved.
Patent Information
- Application Number
- CN202411344921.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-01-03
AI Technical Summary
NPUs occupy a lot of memory and have low processing efficiency when performing deep learning computing tasks of neural network models.
Memory allocation is performed by obtaining the input and output tensors of operators that support running inference acceleration on the NPU heterogeneous platform, and multiplexing the memory of its input tensors based on operator configuration information. Tensor memory allocation is performed from large to small, static memory allocation is performed based on time and space information, and the use of cache pools is optimized.
It effectively improves processing efficiency, realizes memory reuse, and reduces a large amount of memory consumption.
Smart Images
Figure CN119248669A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neural network technology, and in particular to a memory allocation method and device for a NPU heterogeneous platform. Background Art
[0002] NPU (Neural network processor unit) is a special processor that uses circuits to simulate human neurons and synaptic structures to handle specific tasks.
[0003] As the size of neural network models increases, the memory required by the NPU for deep learning calculations is increasing. The NPU may allocate memory to store various data for deep learning calculations of neural network models. For example, the memory allocated to each model includes: input memory for input data, weight memory for weight data, instruction memory for instruction data, temporary memory for output data of each network layer, and output memory for the overall output of each model.
[0004] In actual model inference, the overall memory allocated when the NPU performs deep learning computing tasks of the neural network model is large, but memory resources are limited. Summary of the invention
[0005] In view of this, the present invention provides a memory allocation method and device for an NPU heterogeneous platform, which is used to solve the problem that the current NPU occupies a large amount of memory and has low processing efficiency when executing deep learning computing tasks of a neural network model.
[0006] In order to solve the above technical problems, in a first aspect, the present invention provides a memory allocation method for an NPU heterogeneous platform, the method comprising:
[0007] Acquire a first target operator in the NPU heterogeneous platform, and perform memory allocation on an input tensor and an output tensor of the first target operator and a first target tensor, wherein the first target operator is an operator supporting inference acceleration, and the first target tensor is a tensor used by the operator supporting inference acceleration;
[0008] Acquire a second target operator in the NPU heterogeneous platform, reuse the memory of the output tensor of the second target operator with the memory of the input tensor of the second target operator, where the second target operator is an operator that supports in-place operation;
[0009] According to the operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform, memory is allocated for each tensor of the third target operator one by one, and the third target operator is an operator other than the first target operator and the second target operator. The operator configuration information includes the operator width, and the operator width is the minimum amount of memory required to run the third target operator.
[0010] Optionally, the step of allocating memory for each tensor of each of the third target operators one by one in descending order of operator widths in the operator configuration information of the third target operator in the NPU heterogeneous platform includes:
[0011] Obtain all second target tensors existing in the memory when the third target operator is running and recorded in the operator configuration information;
[0012] For each of the second target tensors, check whether there is a target cache matching the second target tensor in the existing cache pool. If the target cache exists, the target cache is allocated to the second target tensor for reuse. If the target cache does not exist, a new cache is reapplied for allocation to the second target tensor, and the reapplied new cache is stored in the cache pool.
[0013] Optionally, the step of searching, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool comprises:
[0014] Searching in an existing buffer pool whether there is a first buffer whose size is greater than or equal to the size of the second target tensor;
[0015] If so, the first cache whose size difference with the second target tensor is the smallest is determined as the target cache matching the second target tensor.
[0016] Optionally, the step of searching, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool comprises:
[0017] Searching in the existing cache pool whether there is a second cache that does not conflict with the usage period of the second target tensor;
[0018] If so, the second cache whose usage period is shortest with that of the second target tensor is determined as the target cache matching the second target tensor.
[0019] Optionally, before the step of allocating memory for each tensor of each of the third target operators one by one in descending order of operator widths in the operator configuration information of the third target operator in the NPU heterogeneous platform, the method further includes:
[0020] According to the topological structure of the network model in the NPU heterogeneous platform, determine the tensors existing in the memory when each operator in the topological structure is running;
[0021] Determine the operator width and usage period of each operator according to the tensors present in the memory when each operator is running;
[0022] The operator configuration information of each operator is generated according to the operator width and the usage period of each operator.
[0023] In a second aspect, the present invention further provides a memory allocation device for an NPU heterogeneous platform, the device comprising:
[0024] A first memory allocation module is used to obtain a first target operator in the NPU heterogeneous platform, and to perform memory allocation on an input tensor and an output tensor of the first target operator, and a first target tensor, wherein the first target operator is an operator supporting inference acceleration, and the first target tensor is a tensor used by the operator supporting inference acceleration;
[0025] A second memory allocation module is used to obtain a second target operator in the NPU heterogeneous platform, and reuse the memory of the output tensor of the second target operator with the memory of the input tensor of the second target operator, where the second target operator is an operator that supports in-place operation;
[0026] The third memory allocation module is used to allocate memory to each tensor of each third target operator in the NPU heterogeneous platform one by one in the order of the operator width in the operator configuration information of the third target operator from large to small, wherein the third target operator is an operator other than the first target operator and the second target operator, and the operator configuration information includes the operator width, which is the minimum amount of memory required to run the third target operator.
[0027] Optionally, the third memory allocation module includes:
[0028] A first acquisition unit, configured to acquire all second target tensors existing in the memory when the third target operator is running and recorded in the operator configuration information;
[0029] An allocation unit is used to search, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool; if the target cache exists, the target cache is allocated to the second target tensor for reuse; if the target cache does not exist, a new cache is re-applied for allocation to the second target tensor, and the re-applied new cache is stored in the cache pool.
[0030] Optionally, the allocation unit includes:
[0031] A first search subunit is used to search in an existing cache pool whether there is a first cache whose size is greater than or equal to the second target tensor size;
[0032] The first determining subunit is configured to determine, if any, a first cache having a smallest size difference with the second target tensor as a target cache matching the second target tensor.
[0033] Optionally, the allocation unit includes:
[0034] A second search subunit is used to search in the existing cache pool whether there is a second cache that does not conflict with the usage period of the second target tensor;
[0035] The second determining subunit is configured to determine, if any, a second cache having a shortest usage period difference with the second target tensor as a target cache matching the second target tensor.
[0036] Optionally, the device further comprises
[0037] A first determination module is used to determine, according to the topological structure of the network model in the NPU heterogeneous platform, the tensors existing in the memory when each operator in the topological structure is running;
[0038] A second determination module is used to determine the operator width and usage period of each operator according to the tensors existing in the memory when each operator is running;
[0039] The generating module is used to generate the operator configuration information of each operator according to the operator width and the use period of each operator.
[0040] The beneficial effects of the above technical solution of the present invention are as follows:
[0041] In the embodiment of the present invention, static memory allocation is performed based on the combination of time and space information of the NPU heterogeneous platform, which can effectively improve processing efficiency, realize memory reuse, and reduce a large amount of memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1This is a flow chart of a memory allocation method for an NPU heterogeneous platform in Embodiment 1 of the present invention;
[0043] Figure 2 A schematic diagram of a topological structure in Embodiment 1 of the present invention;
[0044] Figure 3 This is a schematic diagram of the structure of a memory allocation device for an NPU heterogeneous platform in Embodiment 2 of the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0046] NPU is a special processor that uses circuits to simulate human neurons and synaptic structures to handle specific tasks. Model reasoning is actually the process of reading the model structure and weight information, performing calculations based on the model input, and outputting the calculation results. As the scale of neural network models increases, the memory required by the NPU to perform deep learning calculations is also increasing.
[0047] In actual model reasoning, resources are limited, and it is necessary to improve the efficiency of reasoning and save memory space. In this case, static memory allocation is often used to plan and allocate the memory usage of each operator in the reasoning process in advance to avoid repeated requests for memory release during actual calculation and reasoning, thereby improving efficiency. However, how to plan memory to save memory resources is a big problem at this time.
[0048] The inventors of the present application have found that the algorithm currently used by the NPU is to traverse the operators one by one according to the order of the network topology, and to count the reference counts of the memory occupied by the input and output of each operator to determine whether the current operator has reusable memory / buffer. If the cache reference count is 0, it means that the cache is not occupied by other tensors and can be reused, which can reduce a lot of memory consumption.
[0049] The above strategy takes into account the time information of the neural network (according to the network topological order, in fact, the topological order is the execution order of the neural network, so this order represents the execution time information of the operators in the neural network). Although a lot of memory consumption has been optimized compared to the case of no memory reuse, there is still room for optimization due to the lack of consideration of the spatial information of the operator itself (that is, the size information of the related tensors).
[0050] Therefore, the present application no longer allocates static memory completely in accordance with the network topology order, but allocates static memory based on a full combination of time and space information.
[0051] Specifically, in the memory allocation of NPU heterogeneous platform, the code that generates the neural network model structure file first changes the initial input tensor of the model to an input type operator, whose input tensor number is 0 and output is the number of tensors that should be input. Since the input of an operator is the output of the previous operator, it is actually guaranteed that the output of each operator is allocated, that is, the static memory allocation of the entire model is completed. In the implementation of NPU heterogeneous platform operators, operators are divided into different types. The first type is the operator that NPU directly supports to run inference acceleration. The memory information occupied by the input and output tensors of this part of operators will be planned in the compiler. The other part is the operator that supports in-place operation, that is, the output can reuse the input memory without trampling, such as sinh, etc. The last type is the ordinary operator. NPU can neither run directly (the compiler will not allocate memory) nor reuse the input memory. It is necessary to consider finding a suitable buffer (cache / cache area) from the cache pool or applying for a new buffer. Memory allocation needs to comprehensively consider the characteristics of these operators and pursue the most resource-saving and efficient way.
[0052] Therefore, please refer to Figure 1 , Figure 1 A flowchart of a memory allocation method for an NPU heterogeneous platform provided in Embodiment 1 of the present invention is provided. The method comprises the following steps:
[0053] Step 11: Obtain a first target operator in the NPU heterogeneous platform, and perform memory allocation on an input tensor and an output tensor of the first target operator, as well as a first target tensor, wherein the first target operator is an operator that supports running inference acceleration, and the first target tensor is a tensor used by the operator that supports running inference acceleration.
[0054] In this embodiment, the traversal can be performed in the topological order of the neural network model, and the first target operator in the neural network model in the NPU heterogeneous platform is first obtained, wherein the first target operator is an operator that the NPU directly supports for computational inference acceleration. For this type of operator, memory allocation can be performed directly for its input tensor and output tensor according to the allocation strategy planned by the compiler, that is, memory is directly applied for. In addition, memory allocation is also performed for the first target tensor in the above manner, wherein the first target tensor is a tensor used by the first target operator, that is, the first target tensor is a tensor used by an operator that the NPU directly supports for computational inference acceleration, or the consumer of the first target tensor is the first target operator.
[0055] Step 12: Obtain a second target operator in the NPU heterogeneous platform, and reuse the memory of the output tensor of the second target operator with the memory of the input tensor of the second target operator, where the second target operator is an operator that supports in-place operation.
[0056] After completing the memory allocation of the first target operator, continue to traverse in the topological order of the neural network model to obtain the second target operator in the neural network model in the NPU heterogeneous platform, wherein the second target operator is an operator that supports in-place operation. For this type of operator, the memory of its input tensor and output tensor can be reused without trampling, that is, the memory of the input tensor of the second target operator can be reused as the memory of the output tensor of the second target operator, thereby achieving full utilization of the memory of the input tensor of the second target operator.
[0057] Step 13: According to the operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform, memory is allocated for each tensor of the third target operator one by one in descending order, the third target operator is other operators except the first target operator and the second target operator, and the operator configuration information includes the operator width, and the operator width is the minimum amount of memory required to run the third target operator.
[0058] In this embodiment, the third target operator in the neural network model in the NPU heterogeneous platform is completely traversed again, and the memory is allocated to each tensor corresponding to each third target operator one by one according to the operator width in the operator configuration information corresponding to the third target operator from large to small, wherein the third target operator is other operators in the neural network model in the NPU heterogeneous platform except the first target operator and the second target operator. Specifically, this memory allocation is allocated in combination with the priority of spatial information. When allocating, the outer layer first traverses the operator configuration information of all third target operators, arranges the operator configuration information of all third target operators in order from large to small according to the operator width, and traverses the third target operators one by one; the inner layer traverses the tensor array that must exist in the memory when the current third target operator is running, and then allocates memory to each tensor. Among them, the operator width refers to the minimum amount of memory required to run a third target operator, that is, when running a third target operator, the sum of the memory occupancy corresponding to the tensor array that must exist is the width of the third target operator.
[0059] The memory allocation method for the NPU heterogeneous platform provided in the embodiment of the present invention performs static memory allocation based on the combination of time and space information of the NPU heterogeneous platform, which can effectively improve processing efficiency, realize memory reuse, and reduce a large amount of memory consumption.
[0060] The following example illustrates the memory allocation method of the above NPU heterogeneous platform.
[0061] Please refer to Figure 2 , Figure 2 The schematic diagram of a topological structure provided in the embodiment of the present application is as follows, and the topological structure of a neural network model in the NPU heterogeneous platform is exemplarily introduced. Figure 2 As shown, there is a topological structure A→C→D→B, that is, the topological structure contains operators A, B, C, and D. The outputs of operator A are A_out1 and A_out2, and A_out2 is used as the input of operator C. The output of operator C is C_out, and C_out is used as the input of operator D. When operator C is running, operator A has finished running and has generated two outputs A_out1 and Aout_2. Then the minimum amount of memory required for operator C to run is the space required for the two tensors A_out1 and Aout_2 that must exist in the memory at this time plus the output C_out of C. This space is the width of the operator, and these three tensors are the tensors that must exist in the memory when the C operator is running. Therefore, the operator width is the amount of space required to measure the minimum space required for the current operator to run.
[0062] The way to calculate the width of the operator is as follows: first, count the usage cycle of each tensor, that is, the index of the operator that produces this tensor and the index of the operator that consumes this tensor. The index refers to the index of the operator in the network topology. For example, in the topology structure A→C→D→B, the indexes of the four operators are 0, 1, 2, and 3 respectively, then the usage cycle of operator A_out2 is (0, 1), that is, the producer is 0 and the consumer is 1, or the producer of operator A_out2 is A and the consumer is C. Then count all tensor records tensor_record, including the name, size, and usage cycle of the corresponding tensor.
[0063] The operator configuration information of all operators is counted according to the usage cycle, that is, the layer_profile record, including the operator name, operator width, and the tensor array that must exist in the memory when the current operator is running (arranged from large to small according to the tensor size). All operator configuration information is arranged in order from large to small according to the operator width. At this time, the operator width recorded by the largest operator configuration information can represent the minimum memory usage required by this model.
[0064] In one optional specific implementation manner, the step of allocating memory for each tensor of each of the third target operators one by one in descending order of operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform includes:
[0065] Obtain all second target tensors existing in the memory when the third target operator is running and recorded in the operator configuration information;
[0066] For each of the second target tensors, check whether there is a target cache matching the second target tensor in the existing cache pool. If the target cache exists, the target cache is allocated to the second target tensor for reuse. If the target cache does not exist, a new cache is reapplied for allocation to the second target tensor, and the reapplied new cache is stored in the cache pool.
[0067] In this embodiment, the operator configuration information of the third target operator records all the second target tensors that must exist in the memory when the third target operator is running, and then memory allocation is performed for all the second target tensors corresponding to the same third target operator. During the specific allocation, for each second target tensor, first search in the existing cache pool whether there is a target cache that matches the second target tensor. If the target cache exists, the target cache is allocated to the second target tensor for reuse. If the target cache does not exist, a new cache is re-applied for the second target tensor, that is, the new cache applied for is allocated to the second target tensor, and the re-applied new cache is stored in the cache pool. Therefore, when allocating memory, the spatial information of the operator is taken into account, and while saving memory resources, the operating efficiency is ensured.
[0068] In some embodiments of the present application, for each second target tensor, the step of searching in an existing cache pool whether there is a target cache matching the second target tensor includes:
[0069] Searching in an existing buffer pool whether there is a first buffer whose size is greater than or equal to the size of the second target tensor;
[0070] If so, the first cache whose size difference with the second target tensor is the smallest is determined as the target cache matching the second target tensor.
[0071] In this embodiment, specifically, when searching for a target cache that matches the second target tensor in an existing cache pool, first determine whether there is a first cache greater than or equal to the size of the second target tensor in the existing cache pool. If the caches in the existing cache pool are all smaller than the size of the second target tensor, it means that the existing cache cannot be reused for the second target tensor, and there is no target cache that matches the second target tensor. If there is a first cache greater than or equal to the size of the second target tensor in the existing cache pool, then among all first caches that meet the conditions, select the first cache with the smallest size difference with the second target tensor as the target cache, that is, the target cache that matches the second target tensor. In this way, it can be ensured that the size of the reused target cache is appropriate, and while completing memory reuse, it will not occupy caches with too large a difference, thereby reducing the amount of memory occupied.
[0072] In some other embodiments of the present application, the step of searching, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool comprises:
[0073] Searching in the existing cache pool whether there is a second cache that does not conflict with the usage period of the second target tensor;
[0074] If so, the second cache whose usage period is shortest with that of the second target tensor is determined as the target cache matching the second target tensor.
[0075] In this embodiment, in addition to matching according to the size of the cache in the cache pool and the size of the second target tensor, it is also possible to match according to the use cycle of the cache in the cache pool and the use cycle of the second target tensor. Specifically, the existing cache pool is searched to see if there is a second cache that does not conflict with the use cycle of the second target tensor. If the use cycle of the cache in the existing cache pool conflicts with the use cycle of the second target tensor, it means that the existing cache cannot be reused for the second target tensor, and there is no target cache that matches the second target tensor. If there is a second cache in the existing cache pool that does not conflict with the use cycle of the second target tensor, then among all second caches that meet the conditions, the second cache with the shortest difference in use cycle with the second target tensor is selected as the target cache, that is, the target cache that matches the second target tensor. In this way, it can be ensured that the reused target cache time is appropriate, and while completing memory reuse, the cache will not be occupied for too long, reducing the memory occupancy time.
[0076] Exemplarily, there are buffer_1, buffer_2, and buffer_3 in the existing cache pool, among which the usage cycles of tensors using buffer_1 are (12, 14) and (18, 20), the usage cycles of tensors using buffer_2 are (9, 10) and (19, 21), the usage cycle of tensors using buffer_3 is (16, 18), and the usage cycle of the second target tensor is (15, 16). The minimum difference between its usage cycle and the usage cycle of tensors in buffer_1 is 15-14=1, and the minimum difference between its usage cycle and the usage cycle of tensors in buffer_2 is 19-16=3. There is a conflict with the usage cycle of tensors already allocated in buffer_3, so buffer_1 should be selected for use.
[0077] In some embodiments of the present application, before the step of allocating memory to each tensor of each of the third target operators one by one in the descending order of the operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform, the method further includes:
[0078] According to the topological structure of the network model in the NPU heterogeneous platform, determine the tensors existing in the memory when each operator in the topological structure is running;
[0079] Determine the operator width and usage period of each operator according to the tensors present in the memory when each operator is running;
[0080] The operator configuration information of each operator is generated according to the operator width and the usage period of each operator.
[0081] In this embodiment, before performing memory allocation for the third target operator, the operator configuration information of the operator in the NPU heterogeneous platform needs to be generated. Specifically, first, according to the topological structure of the neural network model in the NPU heterogeneous platform, the tensors that must exist in the memory when each operator in the topological structure is running are determined, and then according to the tensors that must exist in the memory when each operator is operated, the operator width and usage cycle of each operator are determined, and the operator configuration information of the operator can be generated according to the determined operator width and usage cycle.
[0082] In summary, in the embodiments of the present invention, static memory allocation is performed based on the combination of time and space information of the NPU heterogeneous platform, which can effectively improve processing efficiency, realize memory reuse, and reduce a large amount of memory consumption.
[0083] See also Figure 3 , Figure 3 3 is a schematic diagram of the structure of a memory allocation device for an NPU heterogeneous platform provided in Embodiment 2 of the present invention. The device 30 includes:
[0084] A first memory allocation module 31 is used to obtain a first target operator in the NPU heterogeneous platform, and to perform memory allocation on an input tensor and an output tensor of the first target operator, and a first target tensor, wherein the first target operator is an operator supporting inference acceleration, and the first target tensor is a tensor used by the operator supporting inference acceleration;
[0085] A second memory allocation module 32 is used to obtain a second target operator in the NPU heterogeneous platform, and reuse the memory of the output tensor of the second target operator with the memory of the input tensor of the second target operator, where the second target operator is an operator that supports in-place operation;
[0086] The third memory allocation module 33 is used to allocate memory to each tensor of each third target operator in the NPU heterogeneous platform one by one in the order of the operator width in the operator configuration information of the third target operator from large to small, the third target operator is other operators except the first target operator and the second target operator, and the operator configuration information includes the operator width, and the operator width is the minimum amount of memory required to run the third target operator.
[0087] Optionally, the third memory allocation module includes:
[0088] A first acquisition unit, configured to acquire all second target tensors existing in the memory when the third target operator is running and recorded in the operator configuration information;
[0089] An allocation unit is used to search, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool; if the target cache exists, the target cache is allocated to the second target tensor for reuse; if the target cache does not exist, a new cache is re-applied for allocation to the second target tensor, and the re-applied new cache is stored in the cache pool.
[0090] Optionally, the allocation unit includes:
[0091] A first search subunit is used to search in an existing cache pool whether there is a first cache whose size is greater than or equal to the second target tensor size;
[0092] The first determining subunit is configured to determine, if any, a first cache having a smallest size difference with the second target tensor as a target cache matching the second target tensor.
[0093] Optionally, the allocation unit includes:
[0094] A second search subunit is used to search in the existing cache pool whether there is a second cache that does not conflict with the usage period of the second target tensor;
[0095] The second determining subunit is configured to determine, if any, a second cache having a shortest usage period difference with the second target tensor as a target cache matching the second target tensor.
[0096] Optionally, the device further comprises
[0097] A first determination module is used to determine, according to the topological structure of the network model in the NPU heterogeneous platform, the tensors existing in the memory when each operator in the topological structure is running;
[0098] A second determination module is used to determine the operator width and usage period of each operator according to the tensors existing in the memory when each operator is running;
[0099] The generating module is used to generate the operator configuration information of each operator according to the operator width and the use period of each operator.
[0100] In the embodiment of the present invention, static memory allocation is performed based on the combination of time and space information of the NPU heterogeneous platform, which can effectively improve processing efficiency, realize memory reuse, and reduce a large amount of memory consumption.
[0101] The embodiment of the present invention is a product embodiment corresponding to the above-mentioned method embodiment 1, so it will not be described in detail here. Please refer to the above-mentioned embodiment 1 for details.
[0102] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A memory allocation method for a NPU heterogeneous platform, characterized in that: include: Acquire a first target operator in the NPU heterogeneous platform, and perform memory allocation on an input tensor and an output tensor of the first target operator and a first target tensor, wherein the first target operator is an operator supporting inference acceleration, and the first target tensor is a tensor used by the operator supporting inference acceleration; Acquire a second target operator in the NPU heterogeneous platform, reuse the memory of the output tensor of the second target operator with the memory of the input tensor of the second target operator, where the second target operator is an operator that supports in-place operation; According to the operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform, memory is allocated for each tensor of the third target operator one by one, and the third target operator is an operator other than the first target operator and the second target operator. The operator configuration information includes the operator width, and the operator width is the minimum amount of memory required to run the third target operator.
2. The method according to claim 1, characterized in that The step of allocating memory to each tensor of each third target operator one by one in descending order of operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform includes: Obtain all second target tensors existing in the memory when the third target operator is running and recorded in the operator configuration information; For each of the second target tensors, check whether there is a target cache matching the second target tensor in the existing cache pool. If the target cache exists, the target cache is allocated to the second target tensor for reuse. If the target cache does not exist, a new cache is reapplied for allocation to the second target tensor, and the reapplied new cache is stored in the cache pool.
3. The method according to claim 2, characterized in that The step of searching, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool comprises: Searching in an existing buffer pool whether there is a first buffer whose size is greater than or equal to the size of the second target tensor; If so, the first cache whose size difference with the second target tensor is the smallest is determined as the target cache matching the second target tensor.
4. The method according to claim 2, characterized in that: The step of searching, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool comprises: Searching in the existing cache pool whether there is a second cache that does not conflict with the usage period of the second target tensor; If so, the second cache whose usage period is shortest with that of the second target tensor is determined as the target cache matching the second target tensor.
5. The method according to claim 1, characterized in that Before the step of allocating memory to each tensor of each of the third target operators one by one in descending order of operator width in the operator configuration information of the third target operator in the NPU heterogeneous platform, the method further includes: According to the topological structure of the network model in the NPU heterogeneous platform, determine the tensors existing in the memory when each operator in the topological structure is running; Determine the operator width and usage period of each operator according to the tensors present in the memory when each operator is running; The operator configuration information of each operator is generated according to the operator width and the usage period of each operator.
6. A memory allocation device for an NPU heterogeneous platform, characterized in that: include: A first memory allocation module is used to obtain a first target operator in the NPU heterogeneous platform, and to perform memory allocation on an input tensor and an output tensor of the first target operator, and a first target tensor, wherein the first target operator is an operator supporting inference acceleration, and the first target tensor is a tensor used by the operator supporting inference acceleration; A second memory allocation module is used to obtain a second target operator in the NPU heterogeneous platform, and reuse the memory of the output tensor of the second target operator with the memory of the input tensor of the second target operator, where the second target operator is an operator that supports in-place operation; The third memory allocation module is used to allocate memory to each tensor of each third target operator in the NPU heterogeneous platform one by one in the order of the operator width in the operator configuration information of the third target operator from large to small, wherein the third target operator is an operator other than the first target operator and the second target operator, and the operator configuration information includes the operator width, which is the minimum amount of memory required to run the third target operator.
7. The device according to claim 6, characterized in that The third memory allocation module comprises: A first acquisition unit, configured to acquire all second target tensors existing in the memory when the third target operator is running and recorded in the operator configuration information; An allocation unit is used to search, for each of the second target tensors, whether there is a target cache matching the second target tensor in the existing cache pool; if the target cache exists, the target cache is allocated to the second target tensor for reuse; if the target cache does not exist, a new cache is re-applied for allocation to the second target tensor, and the re-applied new cache is stored in the cache pool.
8. The device according to claim 7, characterized in that The distribution unit comprises: A first search subunit is used to search in an existing cache pool whether there is a first cache whose size is greater than or equal to the second target tensor size; The first determining subunit is configured to determine, if any, a first cache having a smallest size difference with the second target tensor as a target cache matching the second target tensor.
9. The device according to claim 7, characterized in that The distribution unit comprises: A second search subunit is used to search in the existing cache pool whether there is a second cache that does not conflict with the usage period of the second target tensor; The second determining subunit is configured to determine, if any, a second cache having a shortest usage period difference with the second target tensor as a target cache matching the second target tensor.
10. The device according to claim 6, characterized in that The device also includes A first determination module is used to determine, according to the topological structure of the network model in the NPU heterogeneous platform, the tensors existing in the memory when each operator in the topological structure is running; A second determination module is used to determine the operator width and usage period of each operator according to the tensors existing in the memory when each operator is running; The generating module is used to generate the operator configuration information of each operator according to the operator width and the use period of each operator.
Citation Information
Cited By
Neural network on-chip resource scheduling method and device, electronic equipment and medium
CN119862153A