Memory management method and device and AI computing equipment

Abstract

The embodiment of the invention discloses a memory management method and device and AI computing equipment, the method is applied to an AI system, the AI system comprises a computational graph, the computational graph comprises a target sub-graph, and the target sub-graph comprises N operators and M tensors; the target sub-graph corresponds to K dynamic shape sub-graphs; the method comprises the following steps: acquiring life cycle information and first memory demand information of the M tensors in a first dynamic shape subgraph range; the first dynamic shape sub-graph is a sub-graph to be executed currently in the K dynamic shape sub-graphs; according to the life cycle information, determining a memory reuse condition of the M tensors in the range of the first dynamic shape subgraph; and according to the memory reuse condition and the first memory demand information, allocating and using memories for the M tensors in the range of the first dynamic shape subgraph from a memory pool. The memory reuse rate of the AI computing equipment can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a memory management method, device and AI computing equipment. Background Art

[0002] Due to factors such as cost, area, and power consumption, the device memory of artificial intelligence (AI) accelerators is usually very limited. With the gradual development of AI technology, the memory requirements of AI models have increased rapidly, and the memory resources of AI accelerators have become a key factor limiting the deployment of AI models.

[0003] In order to reduce the AI ​​accelerator memory occupied by the AI ​​framework, various AI frameworks and AI compilers have adopted memory reuse technology. During the operation of the AI ​​model, the memory of the tensor is allocated, released, and reused according to the life cycle of the tensor. However, for some AI frameworks, memory reuse still cannot achieve better results, that is, the memory reuse rate is low. For example, in the dynamic shape (Shape) scenario, memory reuse is more challenging than static shape (Shape). How to improve the memory reuse rate in the dynamic shape scenario is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a memory management method, apparatus, and AI computing device, which can be used to improve the memory reuse rate of the AI ​​computing device.

[0005] In a first aspect, an embodiment of the present application provides a memory management method, which can be applied to an AI system, wherein the AI ​​system includes a computational graph, the computational graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, and the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than or equal to 1; the method may include: obtaining life cycle information and first memory requirement information of the M tensors within the scope of a first dynamic shape subgraph; the first dynamic shape subgraph is the subgraph currently to be executed among the K dynamic shape subgraphs; based on the life cycle information, determining the memory reuse of the M tensors within the scope of the first dynamic shape subgraph; based on the memory reuse and the first memory requirement information, allocating memory for use for the M tensors within the scope of the first dynamic shape subgraph from a memory pool.

[0006] An embodiment of the present application provides a memory management method for an AI acceleration device. The memory management method analyzes the lifecycle information of tensors within a dynamic shape subgraph range of a target subgraph under different inputs, and based on the lifecycle information and the memory requirements of each tensor, analyzes the memory reuse of M tensors in the target subgraph within the first dynamic shape subgraph range, and finally obtains a reuse relationship between the memory required by the M tensors, so that sufficient memory reuse can be achieved between these tensors without overlapping lifecycles. Furthermore, since the embodiment of the present application analyzes the life cycle of tensors within the scope of a dynamic shape subgraph, and since the life cycle relationship of tensors between different dynamic shape subgraphs is determined (because the connection relationship between operators within the target subgraph is determined), only the data input between different dynamic shape subgraphs is different (such as the size and dimension of the data), only the memory size required by each tensor between different dynamic shape subgraphs changes, while the reuse of memories does not change. Therefore, once the reuse of memories of tensors is determined for a dynamic shape subgraph in the target subgraph, the reuse relationship between the memories of tensors within the scope of other dynamic shape subgraphs of the target subgraph is equivalent to being determined. Thereafter, based on the changes in the memory requirement size that may change for each dynamic shape subgraph, the corresponding memory size can be updated on the basis of the memory reuse situation of the first construction, thereby satisfying the memory reuse requirements of the target subgraph under all dynamic shapes. In the embodiments of the present application, by pre-analyzing the life cycle information and memory requirement information of each tensor in the target subgraph and uniformly planning the memory reuse, the memory reuse rate can be improved in the dynamic Shape scenario to support the deployment of larger dynamic Shape models on AI accelerators with the same memory capacity.

[0007] In a possible implementation, the obtaining of the life cycle information and the first memory requirement information of the M tensors within the first dynamic shape subgraph includes: counting the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, wherein each tensor corresponds to a life cycle T; counting the memory requirements of the M tensors within the first dynamic shape subgraph to obtain M memory sizes S, wherein each tensor corresponds to a memory size S. In an embodiment of the present application, the life cycle of each of the M tensors within the first dynamic shape subgraph is counted to obtain M life cycles T, wherein the life cycle of the tensor ends with the completion of the dependent calculation, and the memory will be released accordingly (for example, it will be garbage collected), that is, the life cycle of the tensor refers to the time from the input to the output of the tensor. The memory requirement corresponding to each tensor is the memory required for the input and output data of the tensor when executing the operation. That is, each tensor corresponds to a life cycle and a required memory size. It should be noted that the memory requirements of the same tensor in different dynamic shape subgraph ranges may be different, so the embodiment of the present application counts and allocates the memory requirements of each tensor in different dynamic shape subgraph ranges.

[0008] In a possible implementation, the memory reuse of the M tensors within the first dynamic shape subgraph is determined based on the lifecycle information, including: dividing the M tensors into X tensor sets according to the lifecycle information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the lifecycles of any two tensors in each tensor set, and X is an integer greater than or equal to 1; setting the tensors in each of the tensor sets to reuse the same memory block. In an embodiment of the present application, the M tensors in the target subgraph are divided, and the division principle is to divide the tensors with no overlap between lifecycles into the same tensor set, so as to indicate that memory reuse can be performed between all tensors in the tensor set, and by reusing the memory block with the largest memory size in the tensor set, it can be achieved that all tensors in the tensor set meet the memory usage size. That is to say, the memory requirement size of the tensor with the largest memory requirement in the tensor set is used as the memory block size corresponding to the tensor set. Since the life cycles of these tensors do not overlap, the memory of any of these tensors can be released after the life cycle of any of these tensors ends. Therefore, when any tensor in the tensor set needs to use the memory block, there is no memory competition between these tensors, that is, each tensor in the tensor set can exclusively use the memory block corresponding to the set at the same time. In other words, the memory block corresponding to the tensor set can be reused among all tensors in the set, and can meet the memory required by each tensor in the set. In this way, it can be ensured that each tensor has only one corresponding memory block at the same time, and the memory size can meet the maximum memory requirement of the tensor that has currently appeared, and each memory block in the memory pool has a corresponding tensor, which can avoid the peak memory exceeding the actual needs of the AI ​​model.

[0009] In a possible implementation, dividing the M tensors into X tensor sets according to the lifecycle information includes: performing X rounds of tensor division on the M tensors to obtain X tensor sets, and each round of tensor division obtains a tensor set; wherein each round of the tensor division includes: determining, from the currently remaining tensors, a second tensor having no lifecycle overlap with the first tensor in an execution order; and then determining, in an execution order, a third tensor having no lifecycle overlap with the second tensor, until the currently remaining tensors are traversed and the determined tensors are divided into the same tensor set. In an embodiment of the present application, M tensors are divided in turn through X rounds to obtain X tensor sets. The process mainly involves first determining a first tensor as a calibration tensor from the M tensors, and then determining a second tensor with no life cycle overlap with the first tensor in the order of the life cycle start time of the tensors (for example, from front to back or from back to front). Next, a third tensor with no life cycle overlap with the second tensor is determined, and so on, until all M tensors are traversed and the tensor set of the first round is obtained. By analogy, the remaining tensors that have not been divided into the first tensor set are further divided in the above manner; the tensor set of the second round, the tensor set of the third round, and ... the tensor set of the Xth round are successively obtained, that is, until all tensors are divided into tensor sets, and finally the X tensor sets are obtained. In summary, there must be no memory conflict problem between any two tensors in the same tensor set, and the effect of reusing the same memory block between all tensors in a tensor set is achieved. It can be understood that the size of the memory block needs to cover the memory size required by all tensors in the tensor set.

[0010] In one possible implementation, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor are sequentially from the earliest to the latest; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor are sequentially from the latest to the earliest. In an embodiment of the present application, in the process of dividing the tensor set as described above, it is necessary to compare the life cycles of the tensors in the same direction to see if there is any overlap, such as from the beginning to the end, or from the end to the beginning, so as to ensure that there is no memory reuse conflict between the tensors within the determined tensor set.

[0011] In a possible implementation, the memory is allocated from the memory pool for the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information, including: sending a tensor memory application request to the memory pool for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information; allocating memory for each tensor from the memory pool according to the memory application request of each tensor. In the embodiment of the present application, memory application can be performed for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation, that is, the corresponding memory is applied for in advance before a certain tensor is executed, for example, a memory application request is sent to the memory pool once for each tensor, because there is memory reuse in the embodiment of the present application, and the memory required by a certain tensor needs to be released after being used by a tensor that does not overlap with its life cycle for reuse, so it is necessary to apply for memory for each tensor to be executed.

[0012] In a possible implementation, according to the memory reuse situation and the first memory requirement information, within the scope of the first dynamic shape subgraph, a tensor memory application request is sent to the memory pool for each of the M tensors, including: assigning a corresponding memory identifier to each of the tensor sets; within the scope of the first dynamic shape subgraph, using the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor. In an embodiment of the present application, a unique memory identifier can be assigned to each tensor set, that is, when all tensors in the tensor set apply for memory, the unique memory identifier corresponding to the tensor set is used. The reason is that since all tensors in the tensor set reuse the same memory block, it is only necessary to assign a unified identifier to the reused memory block, so that when applying for memory for a certain tensor, the memory identifier can be used to locate the same memory block, thereby achieving the purpose of reusing the same memory block between tensors with no overlapping life cycles. Furthermore, since the same tensor may have different memory requirements in different Shape situations, in an embodiment of the present application, it is necessary to re-determine the corresponding reusable memory size S for each tensor set under different dynamic shape subgraph ranges. MAX .

[0013] In a possible implementation, the method further includes: when the N operators are all executed within the scope of the first dynamic shape subgraph, obtaining the second memory requirement information of the M tensors within the scope of the second dynamic shape subgraph; and allocating memory for the M tensors within the scope of the second dynamic shape subgraph from the memory pool according to the memory reuse situation and the second memory requirement information. When the first dynamic shape subgraph of the target subgraph is executed, the execution of the second dynamic shape subgraph of the next round is continued. It should be noted that in the embodiment of the present application, since the memory reuse mode of the second dynamic shape subgraph is consistent with the first dynamic shape subgraph executed in the previous round (the reason is that the life cycle of the M tensors of the target subgraph will not change under different dynamic shape subgraphs, but the corresponding input, that is, the memory required for the tensor, may change), therefore, when executing different dynamic shape subgraphs, the division of the tensor set does not need to be repeated, that is, the memory reuse situation (such as the aforementioned X tensor set) obtained in the previous round of dynamic shape subgraphs under the same target subgraph can be used. Optionally, the new memory requirement information for each tensor needs to be updated under the new dynamic shape subgraph scope.

[0014] In a possible implementation, allocating memory for use from the memory pool to the M tensors within the second dynamic shape subgraph according to the memory reuse situation and the second memory requirement information includes: within the second dynamic shape subgraph, taking the largest S of the memory sizes S required by each tensor in each tensor set as the reusable memory size S of all tensors in the corresponding tensor set. MAX; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; according to the memory application request of each tensor, memory is allocated for use from the memory pool for each tensor. The embodiment of the present application can solve the problem that the memory required for the same tensor may be of different sizes due to different inputs between different dynamic shape subgraphs of the target subgraph, because, for the same target subgraph, the relationship between the life cycles of its tensors under different dynamic shape subgraphs is essentially the same, that is, the composition of the tensor sets divided by the M tensors under different dynamic shape subgraphs is actually the same, that is, the same target subgraph can use the same tensor set division scheme, and therefore the memory reuse of the same target subgraph under different dynamic shape subgraphs is consistent. Therefore, when executing N operators within different dynamic shape subgraphs, the memory reuse of the dynamic shape subgraph executed in the previous round can be continued. However, since the input of different dynamic shape subgraphs has changed, the memory size required by the tensor may also have changed. Therefore, the memory information required by each tensor in the new round of dynamic shape subgraphs can be counted, and based on the previously determined tensor set for memory reuse, the reusable memory size S corresponding to each tensor set can be updated. MAX , thereby satisfying the memory requirements of each tensor within the current dynamic shape subgraph.

[0015] In a possible implementation, allocating memory for each tensor from the memory pool according to the memory application request of each tensor includes: determining whether there is a memory block corresponding to the memory identifier in the memory application request in the current memory pool; if a memory block corresponding to the memory identifier already exists in the memory pool, further determining whether the size of the memory block is greater than or equal to the memory size S in the memory application request; if greater than or equal to, allocating the existing memory block to the tensor corresponding to the memory application request for reuse; if less than, reapplying for a memory block greater than or equal to S from the memory pool. MAXThe memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released. In the embodiment of the present application, when applying for memory for each tensor, it can be determined whether a memory block corresponding to the memory identifier already exists in the current memory pool based on the memory block identifier carried in the received memory application request. If so, it means that other tensors in the same tensor set have been applied for it before, that is, when a later tensor in the tensor set is applied for again, it is determined that it already exists, and the memory block can be reused immediately. Furthermore, when it is determined that a memory block corresponding to the memory identifier exists in the memory pool, it is necessary to further determine whether the size of the existing memory block is greater than or equal to the S carried in the memory application request. MAX If it is greater than or equal to, it means that the current memory block can be used directly. If it is less than, it means that the memory block is not enough and you need to re-apply for the memory block corresponding to the memory identifier, and the size needs to be increased to the S carried in the memory application request. MAX Or it is greater than S MAX That is, by judging whether the size of the memory block corresponding to the identifier corresponding to the tensor set is sufficient, the function of real-time expansion or release of memory is realized according to the execution status of operators under different dynamic shape subgraphs.

[0016] In a possible implementation, allocating memory for each tensor from the memory pool according to the memory application request of each tensor includes: if there is no memory block corresponding to the memory identifier in the memory pool, applying for a memory block with a size greater than or equal to S from the memory pool MAX In this embodiment of the application, when there is no memory block corresponding to the memory identifier carried in the memory application request in the memory pool, it is necessary to re-apply from the memory pool, and the size of the application is the memory size S carried in the memory application. MAX Or it is larger than the memory size S MAX , so that the memory requested by the memory application request can be satisfied, and the memory requirements of other tensors in the tensor set can also be satisfied.

[0017] In a second aspect, an embodiment of the present application provides a memory management device, which is applied to an AI system, wherein the AI ​​system includes a computation graph, the computation graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, and the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than or equal to 1; the device includes;

[0018] An information acquisition module, used to acquire life cycle information and first memory requirement information of the M tensors within a first dynamic shape subgraph; the first dynamic shape subgraph is a subgraph currently to be executed among the K dynamic shape subgraphs;

[0019] An information analysis module, used to determine the memory reuse of the M tensors within the first dynamic shape subgraph according to the life cycle information;

[0020] A memory management module is used to allocate memory for the M tensors within the first dynamic shape subgraph from a memory pool according to the memory reuse situation and the first memory requirement information.

[0021] In a possible implementation, the information acquisition module is specifically used to: count the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, where each tensor corresponds to a life cycle T; count the memory requirements of the M tensors within the first dynamic shape subgraph to obtain M memory sizes S, where each tensor corresponds to a memory size S.

[0022] In a possible implementation, the information analysis module is specifically used to: divide the M tensors into X tensor sets according to the life cycle information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the life cycles of any two tensors in each tensor set, and X is an integer greater than or equal to 1; and set the tensors in each of the tensor sets to reuse the same memory block.

[0023] In a possible implementation, the information analysis module is specifically used to: perform X rounds of tensor division on the M tensors to obtain X tensor sets, and each round of tensor division obtains a tensor set; wherein each round of tensor division includes: determining, from the currently remaining tensors, a second tensor that has no life cycle overlap with the first tensor in accordance with the execution order; and then determining, in accordance with the execution order, a third tensor that has no life cycle overlap with the second tensor, until the currently remaining tensors are traversed and the determined tensors are divided into the same tensor set.

[0024] In a possible implementation, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from early to late; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from late to early.

[0025] In one possible implementation, the memory management module is specifically used to: send a tensor memory application request to the memory pool for each of the M tensors within the scope of the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information; and allocate memory for each tensor from the memory pool according to the memory application request of each tensor.

[0026] In a possible implementation, the memory management module is specifically used to: assign a corresponding memory identifier to each of the tensor sets; within the first dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor.

[0027] In a possible implementation, the information acquisition module is also used to obtain the second memory requirement information of the M tensors within the scope of the second dynamic shape subgraph when the N operators are all executed within the scope of the first dynamic shape subgraph; the memory management module is also used to allocate memory for the M tensors within the scope of the second dynamic shape subgraph from the memory pool according to the memory reuse situation and the second memory requirement information.

[0028] In a possible implementation, the memory management module is specifically used to: within the second dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each tensor set as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; according to the memory application request of each tensor, memory is allocated for use for each tensor from the memory pool.

[0029] In a possible implementation, the memory management module is specifically used to: determine whether there is a memory block corresponding to the memory identifier in the memory application request in the current memory pool; if a memory block corresponding to the memory identifier already exists in the memory pool, further determine whether the size of the memory block is greater than or equal to the memory size S in the memory application request; if greater than or equal to, allocate the existing memory block to the tensor corresponding to the memory application request for reuse; if less than, re-apply for a memory block greater than or equal to S from the memory pool MAXThe memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released.

[0030] In a possible implementation, the memory management module is specifically configured to: if there is no memory block corresponding to the memory identifier in the memory pool, apply for a memory block with a size greater than or equal to S from the memory pool; MAX The memory block is used as the memory block corresponding to the memory identifier.

[0031] In a third aspect, an embodiment of the present application provides an AI computing device, which is applied to an AI system, wherein the AI ​​system includes a computational graph, the computational graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, and the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than 1; the device may include a processor and an AI accelerator, the AI ​​accelerator includes an accelerator memory; the processor is used to: obtain the life cycle of the M tensors within the first dynamic shape subgraph; cycle information and first memory requirement information; the first dynamic shape subgraph is the subgraph currently to be executed among the K dynamic shape subgraphs; according to the life cycle information, determining the memory reuse of the M tensors within the scope of the first dynamic shape subgraph; according to the memory reuse and the first memory requirement information, allocating memory for the M tensors within the scope of the first dynamic shape subgraph from the accelerator memory; the AI ​​accelerator is used to: execute the M tensors within the corresponding dynamic shape subgraph based on the memory allocated by the processor to each of the tensors.

[0032] In a possible implementation, the processor is specifically configured to: count the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, wherein each tensor corresponds to a life cycle T;

[0033] The memory requirements of the M tensors within the first dynamic shape subgraph are counted to obtain M memory sizes S, where each tensor corresponds to one memory size S.

[0034] In a possible implementation, the processor is specifically used to: divide the M tensors into X tensor sets according to the life cycle information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the life cycles of any two tensors in each tensor set, and X is an integer greater than or equal to 1; and set the tensors in each of the tensor sets to reuse the same memory block.

[0035] In a possible implementation, the processor is specifically used to: perform X rounds of tensor division on the M tensors to obtain X tensor sets, and each round of tensor division obtains a tensor set; wherein each round of the tensor division includes: each round of the tensor division includes: from the currently remaining tensors, determining a second tensor that has no life cycle overlap with the first tensor in accordance with the execution order; and then determining a third tensor that has no life cycle overlap with the second tensor in accordance with the execution order, until the currently remaining tensors are traversed and the determined tensors are divided into the same tensor set.

[0036] In a possible implementation, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from early to late; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from late to early.

[0037] In a possible implementation, the processor is specifically used to: send a tensor memory application request to the accelerator memory for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information; and allocate memory for each tensor from the accelerator memory according to the memory application request of each tensor.

[0038] In a possible implementation, the processor is specifically used to: assign a corresponding memory identifier to each of the tensor sets; within the first dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor.

[0039] In a possible implementation, the processor is further used to: when the N operators are all executed within the first dynamic shape subgraph, obtain the second memory requirement information of the M tensors within the second dynamic shape subgraph; and allocate memory for use for the M tensors within the second dynamic shape subgraph from the accelerator memory according to the memory reuse situation and the second memory requirement information.

[0040] In a possible implementation, the processor is specifically configured to: within the second dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each tensor set as the reusable memory size S of all tensors in the corresponding tensor set. MAX; For each of the M tensors, a memory application request is initiated, wherein the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; according to the memory application request of each tensor, memory is allocated for use for each tensor from the accelerator memory.

[0041] In a possible implementation, the processor is specifically used to: determine whether there is a memory block corresponding to the memory identifier in the memory application request in the current accelerator memory; if a memory block corresponding to the memory identifier already exists in the accelerator memory, further determine whether the size of the memory block is greater than or equal to the memory size S in the memory application request; if greater than or equal to, allocate the existing memory block to the tensor corresponding to the memory application request for reuse; if less than, reapply for a memory block greater than or equal to S from the accelerator memory MAX The memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released.

[0042] In a possible implementation, the processor is specifically configured to: if there is no memory block corresponding to the memory identifier in the accelerator memory, apply for a memory block with a size greater than or equal to S from the accelerator memory; MAX The memory block is used as the memory block corresponding to the memory identifier.

[0043] In a fourth aspect, the present application provides a system on chip (SOC), which may include a memory management device provided by any one of the implementations in the second aspect above.

[0044] In a fifth aspect, the present application provides a system on chip (SoC), which includes an AI computing device in any one of the implementations of the third aspect and a memory coupled to the AI ​​computing device. The SoC may be composed of a chip or may include a chip and other discrete devices.

[0045] In a sixth aspect, the present application provides a chip system, which may include: an AI computing device provided by any one of the implementation methods in the third aspect above, a controller, memory or other functional devices coupled to the AI ​​computing device, etc.

[0046] In a seventh aspect, the present application provides a memory management device, which has the function of implementing any one of the memory management methods in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0047] In an eighth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the memory management method process described in any one of the above-mentioned first aspects is implemented.

[0048] In a ninth aspect, an embodiment of the present application provides a computer program, which includes instructions. When the instructions are executed by a processor, the processor can execute the memory management method process described in any one of the above-mentioned first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0050] Figure 1A It is a schematic diagram of the memory management system framework of TensorFlow in the prior art.

[0051] Figure 1B It is a schematic diagram of the memory management system framework of Pytorch in the prior art.

[0052] Figure 2A It is a schematic diagram of the software and hardware architecture of a computation graph compilation and execution system provided in an embodiment of the present application.

[0053] Figure 2B It is a structural diagram of an AI computing device provided in an embodiment of the present application.

[0054] Figure 3 A schematic diagram of the structure of a target subgraph provided in an embodiment of the present application.

[0055] Figure 4A It is a flowchart of a memory management method provided in an embodiment of the present application.

[0056] Figure 4B It is a flowchart of another memory management method provided in an embodiment of the present application.

[0057] Figure 5A A schematic diagram of partitioning a tensor set provided in an embodiment of the present application.

[0058] Figure 5B A schematic diagram of partitioning another tensor set provided in an embodiment of the present application.

[0059] Figure 6 A schematic diagram of a memory allocation process provided in an embodiment of the present application.

[0060] Figure 7A flowchart of memory allocation based on life cycle and memory requirements is provided for an embodiment of the present application.

[0061] Figure 8 A schematic diagram of the structure of a memory management device provided in an embodiment of the present application.

[0062] Fig. 9 A schematic diagram of the structure of another AI computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0064] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0067] First, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0068] (1) Operator: An operator is a general term for a mathematical operation, such as the roll operator, matrix multiplication operator, etc. Specifically, an operator can be considered as a mapping from a function space to a function space O: X → X. In a broad sense, operators can be extended to any space, such as the inner product space.

[0069] (2) Tensor. In the field of artificial intelligence, a tensor is a multidimensional array. Specifically, a tensor can be defined as a set of numbers that satisfy a certain transformation law under a given basis. These numbers written as a vertical row are first-order tensors; written as a two-dimensional array is a second-order tensor; written as a three-dimensional array is a third-order tensor, and so on. Tensors with dimensions higher than the third order are called high-order tensors.

[0070] (3) The shape of the tensor. The literal meaning of shape is shape. In AI, HPC and other applications, computationally intensive operations (called operators) such as matrix multiplication, convolution, and vector operations are required. These operators are mathematically a function calculation of multidimensional arrays (i.e. tensors). Each dimension of a multidimensional array has a specific value, such as the height H and width W of a two-dimensional array; the height H, width W, and depth D of a three-dimensional array. These values ​​of different dimensions are collectively referred to as the shape of the multidimensional array (i.e. tensor), or shape.

[0071] The shape of a tensor refers to the shape given to the tensor when it is defined. For example, the shape of a 5*5 matrix is ​​5 in length and 5 in width; for another example, the tensor of the convolution kernel (filter) used for convolution calculation is defined as 3*3, and its shape is 3 in length and 3 in width. As an example, suppose the shape of the pixels of the convolved image is 6*6 and the depth is 3 (representing the three primary colors of R / G / B); the convolution kernel is a 3*3*3 cube. Therefore, the convolution kernel is a tensor with a shape of (3,3,3), which can be recorded as Tensor1; the convolved image can be represented as a tensor with a shape of (6,6,3), which can be recorded as Tensor2. The operation of convolving the two tensors Tensor1 and Tensor2 is called an operator, recorded as Convolution.

[0072] (4) Dynamic shape of tensors. Since each tensor expresses a physical meaning, such as the pixels of an image, a convolution kernel, etc. If the shape of the tensor to be output at each layer does not change during the training or inference calculation of the model, then the operator used in this type of calculation can be called a fixed-shape operator, and the tensor to be output is a tensor of fixed shape; conversely, if the shape of the tensor to be output at each layer can change during the training or inference calculation of the model, then the operator used in this type of calculation can be called a dynamic-shape operator, and the tensor to be output is a tensor of dynamic shape.

[0073] (5) Dynamic shape compiler: a static compiler that supports compiling operators of arbitrary shapes.

[0074] First, in order to facilitate the understanding of the embodiments of the present application, the technical problems to be solved by the present application are further analyzed and proposed. In the prior art, there are multiple technical solutions for the memory management method of TensorFlow, and two technical solutions are listed below. The details are as follows:

[0075] Solution 1: TensorFlow storage management solution

[0076] See also Figure 1A , Figure 1A The following is a schematic diagram of the memory management system framework of TensorFlow in the prior art. When the accelerator is initialized, TensorFlow uniformly applies for all the memory of the accelerator, and then organizes and manages it by Bin and Chunk. Chunks of equal size are placed in a bin, and the chunk size is 256*2^n.

[0077] Application Process:

[0078] 1. Find a minimum allocatable chunk in bins; if chunk size / req>=2, split the chunk, return one of them, and insert the other into bins;

[0079] 2. The allocated chunk will be marked as occupied.

[0080] Release process:

[0081] 1. Find out whether the previous / following chunk of the chunk in the bins is free, and try to merge it into a larger chunk;

[0082] Disadvantage: The process occupies all the memory of the accelerator and cannot be shared among multiple process instances.

[0083] Solution 2: Pytorch's memory management solution

[0084] See also Figure 1B , Figure 1B The following is a schematic diagram of the memory management system framework of Pytorch in the prior art. The memory management solution of Pytorch is as follows:

[0085] 1. Apply for memory in units of integer multiples of 512 bytes;

[0086] 2. Maintain the device memory pool. Tensor memory is requested from the memory pool and returned to the memory pool when released. When the memory pool is insufficient, Pytorch will request memory expansion from the device.

[0087] 3. Track the life cycle of the tensor through reference counting, and release the memory back to the memory pool immediately when the count reaches zero.

[0088] Disadvantages: As mentioned above Figure 1B As shown in the figure, when training / inference is performed through different Shapes, the memory requirements of the same tensor in each iteration are different, and different memory needs to be requested from the memory pool. Finally, there will be some memory blocks in the memory pool that are not actually used, resulting in an increase in the peak memory occupied by the model.

[0089] In summary, both of the above two memory management solutions have their own defects, which cannot fully utilize the memory blocks in the memory pool, and there are problems such as large peak memory occupied by the model. Therefore, the technical problems to be solved by this application may include the following aspects: providing a memory management method to improve the memory utilization rate in the AI ​​system and reduce the peak memory occupied by the model.

[0090] Any of the memory management methods provided in this application can be applied to the training or reasoning scenarios of AI models. Figure 2A , Figure 2A This is a schematic diagram of the hardware and software architecture of a computational graph compilation and execution system provided in an embodiment of the present application. The system architecture can be used to compile a neural network model. The system includes but is not limited to a compilation layer, a running layer, and an execution layer. Exemplarily, the hardware corresponding to each layer of the system can be an AI compiler, an AI running manager, and an AI accelerator. The following introduces each layer in the above architecture, the devices involved, and the corresponding functions. Among them,

[0091] 1. Graph compilation layer → AI compiler

[0092] The graph compilation layer receives the computation graph from the programming front-end framework and compiles the computation graph to obtain the compiled data of the computation graph. The compiled data of the computation graph is used to execute the computation graph and may include all the data required to execute the computation graph. Here, compilation refers to converting a high-level language into a binary language that can be recognized by a computer.

[0093] like Figure 2A As shown in, based on the training / inference scripts developed by AI developers, the artificial intelligence compiler (AI compiler) 101 compiles the user's training / inference scripts into tasks that can be executed on AI accelerators (such as GPU / TPU / NPU, etc.). For each task, it may include the task itself and the memory requirements required for task execution. For example, the AI ​​compiler compiles the specified task, including graph conversion and optimization, operator fusion optimization, or compiling operators to generate computing tasks that can be executed on AI processors. For another example, in the graph compilation layer, the AI ​​compiler 101 first sub-graphs the computational graph, thereby dividing the computational graph into multiple sub-graph tasks (such as the target sub-graph in this application).

[0094] 2. Graphics Operation Layer → AI Operation Manager (AIRuntime)

[0095] The graph operation layer can perform dependency management and specific scheduling for the subgraph tasks or node subtasks compiled by the graph compilation layer. During the runtime, the AI ​​operation manager 102 calls the subgraph tasks compiled by the AI ​​compiler 101. These tasks are scheduled for execution on the artificial intelligence runtime (AIRuntime) layer. When scheduling execution, the AI ​​operation manager 102 will apply for / release memory on the AI ​​accelerator (AI Accelerator) 20 and reuse the memory resources of the AI ​​accelerator 20 (i.e., accelerator memory) as much as possible to support running larger AI models on the AI ​​accelerator 20.

[0096] It should be noted that runtime usually refers to the execution environment of the program code during runtime, that is, the state and operation of the computer when the program is running. In the field of computer science, runtime is usually distinguished from compile time. Compile time refers to the process of translating source code into machine code that the computer can understand, while runtime refers to the process of running the program on the computer.

[0097] Graph Execution Layer → AI Accelerator

[0098] The graph execution layer may include different types of execution units or acceleration units, that is, the execution layer may include heterogeneous execution units. Different types of execution units may be responsible for executing subgraph tasks or node subtasks of corresponding task types. After executing the received task, each execution unit may return the task execution result to the AI ​​operation manager 102. The graph execution layer involves the memory reuse corresponding to the present application, that is, the present application allocates and reuses the accelerator memory inside the AI ​​accelerator 20 in the graph execution layer. The AI ​​accelerator 20 in the present application executes the operators of the subgraph within the corresponding dynamic shape subgraph based on the used memory allocated for each tensor in the above-mentioned AI operation manager 102.

[0099] Based on the above system architecture, the present application embodiment provides an AI computing device for executing any memory management method in the present application embodiment. Figure 2B , Figure 2B It is a structural diagram of an AI computing device provided by an embodiment of the present application. The AI ​​computing device 30 includes a processor 10 and an AI accelerator 20, wherein the processor 10 further includes an AI compiler 101 and an AI operation manager 102; the AI ​​compiler 101 may specifically include a graph compilation and operator compilation unit 1011 and a tensor lifecycle analysis unit 1012, the AI ​​operation manager 102 may specifically include a task scheduling and execution unit 1021 and a memory pool management unit 1022, and the AI ​​accelerator 20 may include an AI acceleration module 201 and an accelerator memory 202. Among them,

[0100] The graph compilation and operator compilation unit 1011 is responsible for compiling the AI ​​computation graph into an executable task sequence and performing necessary analysis, conversion & optimization during compilation. Operator compilation is responsible for compiling the nodes on the AI ​​computation graph into executable files that can be executed by the accelerator. Exemplarily, the computation graph is input into the graph compilation unit. Specifically, the graph compilation unit converts the computation graph into an intermediate representation and loads it into the operator compilation unit, which then generates a binary file executed in the AI ​​accelerator 20.

[0101] The tensor lifecycle analysis unit 1012 is used to analyze the memory lifecycle of the tensor. In order to support the reuse of tensor memory in dynamic Shape scenes in the embodiment of the present application, it is necessary to change the current reuse mode of tensor memory in dynamic Shape scenes without considering the lifecycle. It is necessary to add the analysis of the tensor lifecycle in the AI ​​compiler 101, and determine whether the memory can be reused between tensors according to the execution order of each operator in the calculation graph. Optionally, the memory reuse algorithm can also use the same or similar algorithm as the static Shape.

[0102] The task scheduling and execution unit 1021 allocates memory and other resources required for task execution on the AI ​​accelerator 20 according to the output of the graph compilation and operator compilation unit 1011, and schedules the execution of the task.

[0103] Memory pool management unit 1022: The memory used by each tensor in the dynamic Shape calculation graph needs to be obtained by managing the memory pool. When scheduling and executing tasks, the memory pool management unit 1022 is requested / released; the memory pool management unit 1022 is responsible for requesting / releasing memory from the accelerator memory 202 in the AI ​​accelerator 20, and allocating the memory in the accelerator memory 202 to the task scheduling and execution unit 1021.

[0104] AI acceleration module 201, AI acceleration module 201 may include corresponding operation circuits, vector processing units, control units, etc. The operation circuit is an electronic circuit that can perform mathematical operations such as multiplication and addition involved in the operator; the vector processing unit can further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc.; the control unit can be used to control the data access, instruction access, etc. involved in the above operation process. It can be understood that the above structure only provides an exemplary structure of an AI acceleration module, and the embodiment of the present application does not specifically limit the structure inside the AI ​​acceleration module. Optionally, the AI ​​accelerator 20 can be used for AI training or for AI reasoning. In summary, the AI ​​accelerator 20 can be mounted on the host processor 10 (Host CPU) as a coprocessor, and the Host CPU assigns tasks.

[0105] The accelerator memory 202 is used to store the relevant data of operators and tensors involved in the subgraph tasks in the embodiments of the present application. For example, input tensor data, output tensor data, etc. Optionally, the accelerator memory 202 is private to the AI ​​accelerator 20.

[0106] The structure illustrated in the above embodiment does not constitute a specific limitation of the AI ​​computing device 30. In other embodiments of the present application, the AI ​​computing device 30 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components illustrated in the above figure may be implemented in hardware, software, or a combination of software and hardware.

[0107] The present application relates to managing memory reuse of subgraphs in a computation graph during execution. To facilitate understanding of the embodiments of the present application, the computation graphs and subgraphs involved in the embodiments of the present application are further described below by way of example.

[0108] When using common deep learning frameworks such as TensorFlow, Pytorch and other AI frameworks for deep learning task training, you first need to build a calculation graph through programming, then use the data as input, calculate through the calculation operations specified in the calculation graph, and finally get the calculation results. This can get rid of the limitations of the programming language, help decouple the front and back ends, and present a more intuitive graphical effect. It can be understood that the calculation graph is a way to describe the calculation process of the neural network using a graph structure. In practical applications, the calculation graph model consists of nodes and directed edges. The nodes represent operators (Operation, Op), or operators, and the directed edges represent the dependencies between calculations. In the calculation of deep learning tasks based on deep frameworks (including training and prediction), the eigenvalues ​​flow one by one along the operations in the calculation graph nodes. Here, the eigenvalues ​​are usually multidimensional matrices and can be called tensors. In an embodiment of the present application, the target subgraph can be abstracted into a directed graph composed of N operators, which are connected to each other through M directed edges, and each directed edge can carry tensor data with dynamically adjusted shape. When the input data is different, the tensor data on the same directed edge will be different. For the sake of ease of description, in the present application, the directed edges carrying tensor data between operators are collectively referred to as tensors. That is, in an embodiment of the present application, the tensor connecting operators at least includes two dimensional characteristics of the tensor data itself and the specific flow direction of the tensor data. It can be understood that, in the graph structure, the tensor in the present application is the data flow (Dataflow) between operators.

[0109] In applications such as AI or high-performance computing (HPC), computationally intensive operations (called operators) such as matrix multiplication, convolution, and vector operations are required. These operators are mathematically a function calculation of multidimensional arrays (or tensors). Each dimension of a multidimensional array has a specific value, such as the height H and width W of a two-dimensional array; the height H, width W, and depth D of a three-dimensional array. These values ​​of different dimensions are collectively referred to as the shape of the multidimensional array (i.e., tensor), or shape.

[0110] For example, see Figure 3 , Figure 3 A schematic diagram of the structure of a target subgraph provided in an embodiment of the present application, Figure 3In the target subgraph, there are 9 operators and 9 tensors. Specifically, the 9 operators are operator A, operator B, operator C, operator D, operator E, operator F and operator G; the 9 tensors are tensor T1, tensor T2, tensor T3, tensor T4, tensor T5, tensor T6, tensor T7, tensor T8 and tensor T9. It can be understood that the 9 tensor data are respectively carried on 9 directed edges. For example, the output tensor data of operator A is carried on the directed edge between operator A and operator B to form tensor T1, the output tensor data of operator A is carried on the directed edge between operator A and operator C to form tensor T2, and the output tensor data of operator B is carried on the directed edge between operator B and operator G to form tensor T5. The composition of other tensors is analogous and will not be repeated here. In addition, taking T2 between operator A and operator C as an example, the pointing direction of T2 is from operator A to operator C, then T2 indicates that the output tensor data generated by operator A is the input tensor data of operator B.

[0111] The present application also involves memory reuse within a dynamic shape subgraph range of the same subgraph under different inputs. To facilitate understanding of the embodiments of the present application, the dynamic shape subgraph involved in the embodiments of the present application is further described below by way of example.

[0112] In the embodiment of the present application, the target subgraph presents different shape changes when the shape of the input tensor changes, that is, corresponding to different dynamic shape subgraphs. For example, the data size and data batches to be trained for models in the fields of natural language processing and computer vision are constantly changing. Therefore, all shape information cannot be known at compile time, and compilation is required for each shape to obtain relevant information of the target dynamic shape subgraph to be executed in the current round.

[0113] In dynamic shape scenarios, the shape of the specific tensor to be output cannot be known in advance (for example, before neural network training), but the specific value of the tensor shape will be confirmed at the AI ​​runtime layer (AI Runtime), that is, when running to the moment when a certain operator is to be calculated, the system knows what the value of the shape of the tensor to be output at the current moment is. For example, at runtime, the system knows that the shape of the tensor to be output is a matrix of 378*125. Therefore, based on this determined 378*125 shape information, the memory size required for the tensor in the next life cycle can be calculated. Among them, tensor data has a shape, shape refers to the length (number of elements) of each axis of the tensor, and rank refers to the number of axes of the tensor. The axis dimension can be used to refer to a specific dimension of the tensor, and the size or size can refer to the total number of items of the tensor, that is, the product shape vector. It indicates how many elements there are in each layer (corresponding to the axis dimension). For example, [2,3,4] means there are 2 elements in the first layer, 3 elements in the second layer, and 4 elements in the third layer. From this, we can know that the tensor has a total of 2×3×4=24 elements.

[0114] See also Figure 4A , Figure 4A It is a flowchart of a memory management method provided by an embodiment of the present application. The memory management method is applied to an artificial intelligence AI system. The AI ​​system includes a computation graph, the computation graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, and the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than or equal to 1. The memory management method can be implemented by the above Figure 2B The method may include the following steps S401-S402. Optionally, see Figure 4B , Figure 4B is a flow chart of another memory management method provided by an embodiment of the present application, and the method may further include step S404 and step S405.

[0115] Step S401: acquiring life cycle information and first memory requirement information of the M tensors within a first dynamic shape subgraph; the first dynamic shape subgraph is a subgraph currently to be executed among the K dynamic shape subgraphs.

[0116] Step S402: Determine the memory reuse status of the M tensors within the first dynamic shape subgraph according to the life cycle information.

[0117] Step S403: Allocate memory from a memory pool for the M tensors within the first dynamic shape subgraph range according to the memory reuse situation and the first memory requirement information.

[0118] An embodiment of the present application provides a memory management method for an AI acceleration device. The memory management method analyzes the lifecycle information of tensors within a dynamic shape subgraph range of a target subgraph under different inputs, and based on the lifecycle information and the memory requirements of each tensor, analyzes the memory reuse of M tensors in the target subgraph within the first dynamic shape subgraph range, and finally obtains a reuse relationship between the memory required by the M tensors, so that sufficient memory reuse can be achieved between these tensors without overlapping lifecycles. Furthermore, since the embodiment of the present application analyzes the life cycle of tensors within the scope of a dynamic shape subgraph, and since the life cycle relationship of tensors between different dynamic shape subgraphs is determined (because the connection relationship between operators within the target subgraph is determined), only the data input between different dynamic shape subgraphs is different (such as the size and dimension of the data), only the memory size required by each tensor between different dynamic shape subgraphs changes, while the reuse of memories does not change. Therefore, once the reuse of memories of tensors is determined for a dynamic shape subgraph in the target subgraph, the reuse relationship between the memories of tensors within the scope of other dynamic shape subgraphs of the target subgraph is equivalent to being determined. Thereafter, based on the changes in the memory requirement size that may change for each dynamic shape subgraph, the corresponding memory size can be updated on the basis of the memory reuse situation of the first construction, thereby satisfying the memory reuse requirements of the target subgraph under all dynamic shapes. In the embodiments of the present application, by pre-analyzing the life cycle information and memory requirement information of each tensor in the target subgraph and uniformly planning the memory reuse, the memory reuse rate can be improved in the dynamic Shape scenario to support the deployment of larger dynamic Shape models on AI accelerators with the same memory capacity.

[0119] Optionally, the memory pool involved in the method embodiment of the present application can be an accelerator memory in an AI accelerator, or it can be other memory resources used to execute subgraph tasks in the above-mentioned calculation graph (for example, including the execution of N operators under the above-mentioned different shapes), which is not specifically limited here.

[0120] Step S404: when the N operators are all executed within the first dynamic shape subgraph range, second memory requirement information of the M tensors within the second dynamic shape subgraph range is obtained.

[0121] Step S405: Allocate memory from the memory pool for the M tensors within the second dynamic shape subgraph range according to the memory reuse situation and the second memory requirement information.

[0122] When the first dynamic shape subgraph of the target subgraph is executed, the execution of the second dynamic shape subgraph of the next round is continued. It should be noted that in the embodiment of the present application, since the memory reuse method of the second dynamic shape subgraph is consistent with the first dynamic shape subgraph executed in the previous round (the reason is that the life cycle of the M tensors of the target subgraph will not change under different dynamic shape subgraphs, but the corresponding input, that is, the memory required by the tensor, may change), therefore, when executing different dynamic shape subgraphs, the division of the tensor set does not need to be repeated, that is, the memory reuse obtained in the previous round of dynamic shape subgraphs under the same target subgraph (such as the X tensor sets obtained above) can be used. Optionally, it is necessary to update the new memory requirement information of each tensor under the new dynamic shape subgraph range.

[0123] In a possible implementation, the obtaining of the life cycle information and the first memory requirement information of the M tensors within the first dynamic shape subgraph includes: counting the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, wherein each tensor corresponds to a life cycle T; counting the memory requirements of the M tensors within the first dynamic shape subgraph to obtain M memory sizes S, wherein each tensor corresponds to a memory size S. In an embodiment of the present application, the life cycle of each of the M tensors within the first dynamic shape subgraph is counted to obtain M life cycles T, wherein the life cycle of the tensor ends with the completion of the dependent calculation, and the memory will be released accordingly (for example, it will be garbage collected), that is, the life cycle of the tensor refers to the time from the input to the output of the tensor. The memory requirement corresponding to each tensor is the memory required for the input and output data of the tensor when executing the operation. That is, each tensor corresponds to a life cycle and a required memory size. It should be noted that the memory requirements of the same tensor in different dynamic shape subgraph ranges may be different, so the embodiment of the present application counts and allocates the memory requirements of each tensor in different dynamic shape subgraph ranges.

[0124] In a possible implementation, the memory reuse of the M tensors within the first dynamic shape subgraph is determined based on the lifecycle information, including: dividing the M tensors into X tensor sets according to the lifecycle information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the lifecycles of any two tensors in each tensor set, and X is an integer greater than or equal to 1; setting the tensors in each of the tensor sets to reuse the same memory block. In an embodiment of the present application, the M tensors in the target subgraph are divided, and the division principle is to divide the tensors with no overlap between lifecycles into the same tensor set, so as to indicate that memory reuse can be performed between all tensors in the tensor set, and by reusing the memory block with the largest memory size in the tensor set, it can be achieved that all tensors in the tensor set meet the memory usage size. That is to say, the memory requirement size of the tensor with the largest memory requirement in the tensor set is used as the memory block size corresponding to the tensor set. Since the life cycles of these tensors do not overlap, the memory of any of these tensors can be released after the life cycle of any of these tensors ends. Therefore, when any tensor in the tensor set needs to use the memory block, there is no memory competition between these tensors, that is, each tensor in the tensor set can exclusively use the memory block corresponding to the set at the same time. In other words, the memory block corresponding to the tensor set can be reused among all tensors in the set, and can meet the memory required by each tensor in the set. In this way, it can be ensured that each tensor has only one corresponding memory block at the same time, and the memory size can meet the maximum memory requirement of the tensor that has currently appeared, and each memory block in the memory pool has a corresponding tensor, which can avoid the peak memory exceeding the actual needs of the AI ​​model.

[0125] In a possible implementation, the M tensors are divided into X sets of tensors according to the lifecycle information, including: dividing the M tensors through X rounds of tensor division to obtain X sets of tensors, and each round of tensor division obtains a set of tensors; wherein each round of the tensor division includes: each round of the tensor division includes: from the currently remaining tensors, determine the second tensor that has no lifecycle overlap with the first tensor according to the execution order; then determine the third tensor that has no lifecycle overlap with the second tensor according to the execution order, until the currently remaining tensors are traversed, and the determined tensors are divided into the same tensor set. Optionally, the first tensor, the second tensor and the three tensors are the tensors executed in sequence among the M tensors; or, the first tensor, the second tensor and the three tensors are the tensors executed in order from the back to the front among the M tensors. It should be noted that each tensor set may contain more or fewer tensors than the first tensor, the second tensor, and the third tensor mentioned above. The above description is only an illustrative explanation of the division method and does not limit the specific number of tensors contained in the tensor set.

[0126] For example, see Figure 5A , Figure 5A A schematic diagram of dividing a tensor set provided in an embodiment of the present application. In the embodiment of the present application, M tensors are divided in the order of execution from the start time of their life cycle to the end. Assume Figure 5A The 7 blocks from left to right represent 7 tensors. The upper and lower lengths of the blocks represent the life cycle of the tensor. According to the overlap of the memory life cycle of the tensor, the first block ( Figure 5A The first one from left to right in the round, and the other blocks are also defined in this order, which will not be repeated later) is the first tensor, then the second tensor of this round is the fifth block, the third tensor of this round is the sixth block, and the third tensor of this round is the seventh block; at this time, the round has traversed all tensors and obtained the first tensor set, and the first tensor set contains Figure 5A The first round - first tensor, the first round - second tensor, the first round - third tensor, the first round - third tensor. The next step is to come to the next round of division. From the tensors left after the previous round of division, the second square and the third square are used as the first tensor and the second tensor of this round to obtain the second tensor set, and the second tensor set contains Figure 5A The second round in the first tensor, the second round in the second tensor, and so on, to obtain a third set of tensors, which may include the fourth block, that is, Figure 5A The third round in - the first tensor.

[0127] For example, see Figure 5B , Figure 5BAnother schematic diagram of dividing a tensor set provided in an embodiment of the present application is shown in FIG. In the embodiment of the present application, M tensors are divided in the order of execution from the start time of their life cycle to the first. Figure 5A , and finally four tensor sets are obtained after four rounds of division, namely, the first tensor set, including the first round-first tensor, the first round-second tensor, and the first round-third tensor; the second tensor set, including the second round-first tensor and the second round-second tensor; the third tensor set, including the third round-first tensor; and the fourth tensor set, including the fourth round-first tensor. That is, in the process of dividing the tensor set above, it is necessary to compare whether the life cycle of the tensors overlaps in the same direction, such as from the beginning time of the life cycle of the tensor from the front to the back, or from the back to the front, so as to ensure that there is no memory reuse conflict between the tensors in the determined tensor set.

[0128] In an embodiment of the present application, M tensors are divided in turn through X rounds to obtain X tensor sets. The process mainly involves first determining a first tensor as a calibration tensor from the M tensors, and then determining a second tensor with no life cycle overlap with the first tensor in the order of the life cycle start time of the tensors (for example, from front to back or from back to front). Next, a third tensor with no life cycle overlap with the second tensor is determined, and so on, until all M tensors are traversed and the tensor set of the first round is obtained. By analogy, the remaining tensors that have not been divided into the first tensor set are further divided in the above manner; the tensor set of the second round, the tensor set of the third round, and ... the tensor set of the Xth round are successively obtained, that is, until all tensors are divided into tensor sets, and finally the X tensor sets are obtained. In summary, there must be no memory conflict problem between any two tensors in the same tensor set, and the effect of reusing the same memory block between all tensors in a tensor set is achieved. It can be understood that the size of the memory block needs to cover the memory size required by all tensors in the tensor set.

[0129] In a possible implementation, the memory is allocated from the memory pool for the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information, including: sending a tensor memory application request to the memory pool for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information; allocating memory for each tensor from the memory pool according to the memory application request of each tensor. In the embodiment of the present application, memory application can be performed for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation, that is, the corresponding memory is applied for in advance before a certain tensor is executed, for example, a memory application request is sent to the memory pool once for each tensor, because there is memory reuse in the embodiment of the present application, and the memory required by a certain tensor needs to be released after being used by a tensor that does not overlap with its life cycle for reuse, so it is necessary to apply for memory for each tensor to be executed.

[0130] In a possible implementation, according to the memory reuse situation and the first memory requirement information, within the scope of the first dynamic shape subgraph, a tensor memory application request is sent to the memory pool for each of the M tensors, including: assigning a corresponding memory identifier to each of the tensor sets; within the scope of the first dynamic shape subgraph, using the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor. In an embodiment of the present application, a unique memory identifier can be assigned to each tensor set, that is, when all tensors in the tensor set apply for memory, the unique memory identifier corresponding to the tensor set is used. The reason is that since all tensors in the tensor set reuse the same memory block, it is only necessary to assign a unified identifier to the reused memory block, so that when applying for memory for a certain tensor, the memory identifier can be used to locate the same memory block, thereby achieving the purpose of reusing the same memory block between tensors with no overlapping life cycles. Furthermore, since the same tensor may have different memory requirements in different Shape situations, in an embodiment of the present application, it is necessary to re-determine the corresponding reusable memory size S for each tensor set under different dynamic shape subgraph ranges. MAX .

[0131] In a possible implementation, allocating memory for use from the memory pool to the M tensors within the second dynamic shape subgraph according to the memory reuse situation and the second memory requirement information includes: within the second dynamic shape subgraph, taking the largest S of the memory sizes S required by each tensor in each tensor set as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; according to the memory application request of each tensor, memory is allocated for use from the memory pool for each tensor. The embodiment of the present application can solve the problem that the memory required for the same tensor may be of different sizes due to different inputs between different dynamic shape subgraphs of the target subgraph, because, for the same target subgraph, the relationship between the life cycles of its tensors under different dynamic shape subgraphs is essentially the same, that is, the composition of the tensor sets divided by the M tensors under different dynamic shape subgraphs is actually the same, that is, the same target subgraph can use the same tensor set division scheme, and therefore the memory reuse of the same target subgraph under different dynamic shape subgraphs is consistent. Therefore, when executing N operators within different dynamic shape subgraphs, the memory reuse of the dynamic shape subgraph executed in the previous round can be continued. However, since the input of different dynamic shape subgraphs has changed, the memory size required by the tensor may also have changed. Therefore, the memory information required by each tensor in the new round of dynamic shape subgraphs can be counted, and based on the previously determined tensor set for memory reuse, the reusable memory size S corresponding to each tensor set can be updated. MAX , thereby satisfying the memory requirements of each tensor within the current dynamic shape subgraph.

[0132] In a possible implementation, allocating memory for each tensor from the memory pool according to the memory application request of each tensor includes: determining whether there is a memory block corresponding to the memory identifier in the memory application request in the current memory pool; if a memory block corresponding to the memory identifier already exists in the memory pool, further determining whether the size of the memory block is greater than or equal to the memory size S in the memory application request; if greater than or equal to, allocating the existing memory block to the tensor corresponding to the memory application request for reuse; if less than, reapplying for a memory block greater than or equal to S from the memory pool. MAX The memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released. Figure 6 , Figure 6A memory allocation process diagram is provided for an embodiment of the present application. When a memory application request carries a memory identifier corresponding to a tensor set to which a current tensor belongs, such as Chunk-ID, and a memory size actually required by the tensor, such as S, the memory pool management unit can determine whether a memory block corresponding to Chunk-ID already exists in the memory pool based on the Chunk-ID carried in the application. If so, the size of the memory block is further compared with the size of S. If the memory block is greater than or equal to the size of S, the memory block can be directly reused. If the memory block is smaller than the size of Chunk-size, a new memory block needs to be applied for, and the size is greater than or equal to the size of Chunk-size corresponding to Chunk-ID, and the old memory block needs to be released.

[0133] In the embodiment of the present application, when applying for memory for each tensor, it is possible to determine whether a memory block corresponding to the memory identifier already exists in the current memory pool based on the memory block identifier carried in the received memory application request. If so, it means that other tensors in the same tensor set have already applied for it. That is, when a later tensor in the tensor set is applied for again, if it is determined that it already exists, the memory block can be reused immediately. Furthermore, when it is determined that a memory block corresponding to the memory identifier exists in the memory pool, it is necessary to further determine whether the size of the existing memory block is greater than or equal to the S carried in the memory application request. MAX If it is greater than or equal to, it means that the current memory block can be used directly. If it is less than, it means that the memory block is not enough and you need to re-apply for the memory block corresponding to the memory identifier, and the size needs to be increased to the S carried in the memory application request. MAX Or it is greater than S MAX That is, by judging whether the size of the memory block corresponding to the identifier corresponding to the tensor set is sufficient, the function of real-time expansion or release of memory is realized according to the execution status of operators under different dynamic shape subgraphs.

[0134] In a possible implementation, allocating memory for each tensor from the memory pool according to the memory application request of each tensor includes: if there is no memory block corresponding to the memory identifier in the memory pool, applying for a memory block with a size greater than or equal to S from the memory pool MAX The memory block is used as the memory block corresponding to the memory identifier.

[0135] In the embodiment of the present application, when there is no memory block corresponding to the memory identifier carried in the memory application request in the memory pool, it is necessary to re-apply from the memory pool, and the size of the application is the memory size S carried in the memory application. MAX Or it is larger than the memory size SMAX , so that the memory requested by the memory application request can be satisfied, and the memory requirements of other tensors in the tensor set can also be satisfied.

[0136] The following is an exemplary description of the implementation of memory management in the embodiment of the present application. Specifically, N is 3, M is 3, and K is 3. Figure 7 , Figure 7 A flowchart of memory allocation based on life cycle and memory requirements is provided in an embodiment of the present application. Figure 7 In the above example, the target subgraph (Sub-GraphA) includes at least Figure 7 The three dynamic shape subgraphs in are shape1, shape2 and shape3, and the target subgraph includes operator A (OpA), operator B (OpB), operator C (OpC), and the tensors between operators include Tensor0 (the tensor from operator A to operator B), Tensor1 (the tensor from operator B to operator C), and Tensor2 (the output tensor of operator C).

[0137] exist Figure 7 In shape1, the actual memory requirement of Tensor0 is 3M, the actual memory requirement of Tensor1 is 7M, and the actual memory requirement of Tensor2 is 2M. It can be seen that in this shape1, there is an overlap in the life cycle between Tensor0 and Tensor1, because Figure 7 From the above, we can see that Tensor0 is used as the input of Tensor1, which means that Tensor1 needs to use the data of Tensor0 when participating in the execution of operator B; similarly, there is an overlap in the life cycle between Tensor1 and Tensor2, because Tensor1 is used as the input of Tensor2, which means that Tensor2 needs to use the data of Tensor1 when participating in the execution of operator C. It can also be seen that there is no overlap in the life cycle between Tensor0 and Tensor2, because there is no relationship between input and output. Therefore, Tensor0 and Tensor2 can reuse the same memory block, and since the maximum memory requirement size corresponding to Tensor0 and Tensor2 is 3M, for Tensor0 and Tensor2 within the range of shape1, 3M is used as the reusable S of Tensor0 and Tensor2. max , which is 3M; and since Tensor1 cannot reuse memory, a memory block greater than or equal to 7M is allocated to it separately.

[0138] exist Figure 7In shape2, the actual memory requirement of Tensor0 is 3.5M, the actual memory requirement of Tensor1 is 8.2M, and the actual memory requirement of Tensor2 is 2.3M. Since shape2 and shape1 belong to the same target subgraph (Sub-Graph A), that is, the operators and tensors inside them are the same, only the shapes of the tensors are different. Therefore, the memory reuse relationship corresponding to shape2 is consistent with that of shape1, that is, Tensor0 and Tensor2 still reuse the same memory block, and Tensor1 uses the memory block alone. However, since the inputs of Tensor0 and Tensor2 have changed, the corresponding memory requirements have also changed. At this time, the maximum memory requirement size is 3.5M. Therefore, for Tensor0 and Tensor2 within the range of shape2, 8.2M is used as the reusable S of Tensor0 and Tensor2. max , which is 3.5M; and since Tensor1 cannot reuse memory, a memory block greater than or equal to 8.2M is allocated to it separately.

[0139] exist Figure 7 In shape3, the actual memory requirement of Tensor0 is 5M, the actual memory requirement of Tensor1 is 11.7M, and the actual memory requirement of Tensor2 is 3.4M. Similarly, since shape3 and shape1 belong to the same target subgraph (Sub-Graph A), that is, the operators and tensors inside them are the same, only the shapes of the tensors are different. Therefore, the memory reuse relationship corresponding to shape3 is consistent with that of shape1, that is, Tensor0 and Tensor2 still reuse the same memory block, but because the inputs of Tensor0 and Tensor2 have changed, the corresponding memory requirements have also changed. At this time, the maximum memory requirement size is 5M. Therefore, for Tensor0 and Tensor2 within the range of shape3, 5M is used as the reusable S of Tensor0 and Tensor2. max , which is 5M. Since Tensor1 cannot reuse memory, a memory block greater than or equal to 11.7M is allocated to it separately.

[0140] It should be noted that in the above shape1, shape2 and shape3, since the corresponding memory block has been applied to the memory pool and the corresponding memory identifier has been assigned when shape1 is executed, when shape2 or shape3 is executed, since the memory block identifier corresponding to the tensor has not changed, the memory block applied in shape1 can continue to be used. However, due to the required S max has changed, so what happens is Figure 7 The process of reapplying memory blocks and releasing old memory blocks shown in , namely 3M→3.5M→5M, 7M→8.2M→11.7M. It can be understood that in Figure 7 The light gray blocks in DeviceMemory represent the memory blocks corresponding to Tensor0 and Tensor2, and the dark gray blocks represent the memory blocks corresponding to Tensor1. The same blocks are used to represent the memory blocks of the same tensor in different shapes1, shape2, and shape3, indicating that the memory identifiers of the memory blocks corresponding to the same tensor in different shapes have not changed.

[0141] Based on the above Figure 7 The specific memory reuse situation described below is further combined with Figure 2B The relevant modules in describe the relevant processes when the tensor to be executed is requested or allocated memory.

[0142] 1. Analysis of tensor life cycle:

[0143] Combination Figure 2B The tensor lifecycle analysis unit 1012 in the AI ​​computing device 30 analyzes the tensor lifecycle within the subgraph for each dynamic Shape subgraph, or further combines the memory size information to determine whether memory reuse can be performed between tensors. For details, please refer to the above Figure 7 The specific reuse principle of memory in Figure 7 blocks of the same color in the

[0144] For a tensor with reusable memory, the unique identifier chunk_id of the allocated memory block is recorded for the tensor.

[0145] For tensors that can reuse memory, the maximum memory of these tensors is used as the memory size chunk_size of the memory block.

[0146] 2. Memory pool management:

[0147] Combination Figure 2B The memory pool management unit 1022 in the AI ​​computing device 30 and Figure 6According to the judgment process described in , during operation, the memory pool management unit 1022 receives the memory allocation request of the task scheduling and execution unit 1021, and performs memory allocation according to the tensor identifier and memory size of the request.

[0148] The memory application request carries the unique identifier chunk_id of the memory chunk corresponding to the tensor and the actual memory chunk size S required by the tensor.

[0149] Determine whether the chunk exists in the memory pool. If not, request chunk_size memory from the accelerator.

[0150] If the chunk already exists, determine whether the size of the chunk is smaller than the actual memory requirement S.

[0151] If the size of the existing chunk is greater than or equal to the actual memory requirement S, the chunk is reused directly.

[0152] If the size of the existing chunk is smaller than the actual memory requirement S, then reapply for memory greater than or equal to chunk_size and release the memory of the old chunk.

[0153] It can be understood that for different subgraphs (such as Figure 7 In the subgraph B (SubGraphB), the tensor lifecycle analysis unit 1012 can further analyze the execution order between the subgraphs. If the subgraphs are executed serially, the tensor lifecycles between the subgraphs will not overlap, so the memory blocks can also be reused according to the above method.

[0154] See also Figure 8 , Figure 8 A schematic diagram of the structure of a memory management device provided in an embodiment of the present application, the device 50 can be applied to an AI system, the AI ​​system includes a computation graph, the computation graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than or equal to 1; the memory management device 50 may include;

[0155] The information acquisition module 501 is used to acquire the life cycle information and the first memory requirement information of the M tensors within the scope of the first dynamic shape subgraph; the first dynamic shape subgraph is the subgraph currently to be executed among the K dynamic shape subgraphs;

[0156] An information analysis module 502, configured to determine the memory reuse of the M tensors within the first dynamic shape subgraph according to the life cycle information;

[0157] The memory management module 503 is used to allocate memory for the M tensors within the first dynamic shape subgraph from a memory pool according to the memory reuse situation and the first memory requirement information.

[0158] In a possible implementation, the information acquisition module is specifically used to: count the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, where each tensor corresponds to a life cycle T; count the memory requirements of the M tensors within the first dynamic shape subgraph to obtain M memory sizes S, where each tensor corresponds to a memory size S.

[0159] In a possible implementation, the information analysis module is specifically used to: divide the M tensors into X tensor sets according to the life cycle information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the life cycles of any two tensors in each tensor set, and X is an integer greater than or equal to 1; and set the tensors in each of the tensor sets to reuse the same memory block.

[0160] In a possible implementation, the information analysis module is specifically used to: perform X rounds of tensor division on the M tensors to obtain X tensor sets, and each round of tensor division obtains a tensor set; wherein each round of tensor division includes: determining, from the currently remaining tensors, a second tensor that has no life cycle overlap with the first tensor in accordance with the execution order; and then determining, in accordance with the execution order, a third tensor that has no life cycle overlap with the second tensor, until the currently remaining tensors are traversed and the determined tensors are divided into the same tensor set.

[0161] In a possible implementation, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from early to late; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from late to early.

[0162] In a possible implementation, the information analysis module is specifically used to: divide the M tensors into X tensor sets according to the life cycle information and in combination with the first memory requirement information, wherein each of the X tensor sets includes one or more tensors, there is no overlap between the life cycles of any two tensors in each tensor set, and the memory size required for a tensor with an earlier life cycle start time among the one or more tensors is greater than or equal to the memory size required for a later tensor, and X is an integer greater than or equal to 1.

[0163] In one possible implementation, the memory management module is specifically used to: send a tensor memory application request to the memory pool for each of the M tensors within the scope of the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information; and allocate memory for each tensor from the memory pool according to the memory application request of each tensor.

[0164] In a possible implementation, the memory management module is specifically used to: assign a corresponding memory identifier to each of the tensor sets; within the first dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor.

[0165] In a possible implementation, the information acquisition module is also used to obtain the second memory requirement information of the M tensors within the scope of the second dynamic shape subgraph when the N operators are all executed within the scope of the first dynamic shape subgraph; the memory management module is also used to allocate memory for the M tensors within the scope of the second dynamic shape subgraph from the memory pool according to the memory reuse situation and the second memory requirement information.

[0166] In a possible implementation, the memory management module is specifically used to: within the second dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each tensor set as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; according to the memory application request of each tensor, memory is allocated for use for each tensor from the memory pool.

[0167] In a possible implementation, the memory management module is specifically used to: determine whether there is a memory block corresponding to the memory identifier in the memory application request in the current memory pool; if a memory block corresponding to the memory identifier already exists in the memory pool, further determine whether the size of the memory block is greater than or equal to the memory size S in the memory application request; if greater than or equal to, allocate the existing memory block to the tensor corresponding to the memory application request for reuse; if less than, re-apply for a memory block greater than or equal to S from the memory pool MAXThe memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released.

[0168] In a possible implementation, the memory management module is specifically configured to: if there is no memory block corresponding to the memory identifier in the memory pool, apply for a memory block with a size greater than or equal to S from the memory pool; MAX The memory block is used as the memory block corresponding to the memory identifier.

[0169] It can be understood that the specific implementation of the memory management method described in the embodiments of the present application can refer to the above Figure 2A-Figure 7 The relevant descriptions in the relevant embodiments described in will not be repeated here.

[0170] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of another AI computing device provided in an embodiment of the present application, wherein the AI ​​computing device is applied to an AI system, wherein the AI ​​system includes a computation graph, wherein the computation graph includes a target subgraph, wherein the target subgraph includes N operators and M tensors; wherein the target subgraph corresponds to K dynamic shape subgraphs, wherein the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M, and K are all integers greater than or equal to 1; wherein the AI ​​computing device 60 includes a processor 601 and an AI accelerator 602, wherein the AI ​​accelerator 602 includes an accelerator memory 6020;

[0171] Processor 601 is used to: obtain life cycle information and first memory requirement information of the M tensors within a first dynamic shape subgraph; the first dynamic shape subgraph is a subgraph currently to be executed among the K dynamic shape subgraphs; determine the memory reuse of the M tensors within the first dynamic shape subgraph according to the life cycle information; and allocate memory for the M tensors within the first dynamic shape subgraph from the accelerator memory according to the memory reuse and the first memory requirement information;

[0172] The AI ​​accelerator 602 is configured to execute the M tensors within a corresponding dynamic shape subgraph range based on the memory used by the processor for each tensor.

[0173] In a possible implementation, the processor 601 is specifically configured to: count the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, where each tensor corresponds to a life cycle T;

[0174] The memory requirements of the M tensors within the first dynamic shape subgraph are counted to obtain M memory sizes S, where each tensor corresponds to one memory size S.

[0175] In a possible implementation, the processor 601 is specifically configured to: divide the M tensors into X tensor sets according to the life cycle information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the life cycles of any two tensors in each tensor set, and X is an integer greater than or equal to 1; and set the tensors in each of the tensor sets to reuse the same memory block.

[0176] In a possible implementation, the processor 601 is specifically configured to: perform X rounds of tensor division on the M tensors to obtain X tensor sets, and each round of tensor division obtains a tensor set; wherein each round of tensor division includes: determining, from the currently remaining tensors, a second tensor having no life cycle overlap with the first tensor in accordance with the execution order; and then determining, in accordance with the execution order, a third tensor having no life cycle overlap with the second tensor, until the currently remaining tensors are traversed and the determined tensors are divided into the same tensor set.

[0177] In a possible implementation, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from early to late; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from late to early.

[0178] In a possible implementation, the processor 601 is specifically configured to: divide the M tensors into X tensor sets according to the lifecycle information and in combination with the first memory requirement information, wherein each of the X tensor sets includes one or more tensors, and there is no overlap between the lifecycles of any two tensors in each tensor set, and a memory size required for a tensor with an earlier lifecycle start time among the one or more tensors is greater than or equal to a memory size required for a later tensor, and X is an integer greater than or equal to 1.

[0179] In one possible implementation, the processor 601 is specifically used to: send a tensor memory application request to the accelerator memory for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information; and allocate memory for each tensor from the accelerator memory according to the memory application request of each tensor.

[0180] In a possible implementation, the processor 601 is specifically configured to: assign a corresponding memory identifier to each of the tensor sets; within the first dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set;MAX ; For each of the M tensors, a memory application request is initiated, and the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor.

[0181] In a possible implementation, the processor 601 is further used to: when the N operators are all executed within the first dynamic shape subgraph, obtain the second memory requirement information of the M tensors within the second dynamic shape subgraph; and allocate memory for use for the M tensors within the second dynamic shape subgraph from the accelerator memory according to the memory reuse situation and the second memory requirement information.

[0182] In a possible implementation, the processor 601 is specifically configured to: within the second dynamic shape subgraph, use the largest S of the memory sizes S required by each tensor in each of the tensor sets as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, wherein the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; according to the memory application request of each tensor, memory is allocated for use for each tensor from the accelerator memory.

[0183] In one possible implementation, the processor 601 is specifically used to: determine whether there is a memory block corresponding to the memory identifier in the memory application request in the current accelerator memory; if a memory block corresponding to the memory identifier already exists in the accelerator memory, further determine whether the size of the memory block is greater than or equal to the memory size S in the memory application request; if greater than or equal to, allocate the existing memory block to the tensor corresponding to the memory application request for reuse; if less than, reapply for a memory block greater than or equal to S from the accelerator memory MAX The memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released.

[0184] In a possible implementation, the processor 601 is specifically configured to: if there is no memory block corresponding to the memory identifier in the accelerator memory, apply for a memory block with a size greater than or equal to S from the accelerator memory; MAX The memory block is used as the memory block corresponding to the memory identifier.

[0185] It can be understood that the specific implementation of the memory management method described in the embodiments of the present application can refer to the above Figure 2A-Figure 7 The relevant descriptions in the relevant embodiments described in will not be repeated here.

[0186] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program includes part or all of the steps of any one of the steps recorded in the above method embodiments.

[0187] An embodiment of the present application also provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can execute part or all of the steps of any one of the above method embodiments.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, some steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. Among them, the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviated: ROM) or random access memory (Random Access Memory, abbreviated: RAM) and other media that can store program codes.

[0194] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A memory management method, characterized in that: Applied to an AI system, the AI ​​system includes a computation graph, the computation graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than or equal to 1; the method includes: Acquire life cycle information and first memory requirement information of the M tensors within a first dynamic shape subgraph; the first dynamic shape subgraph is a subgraph currently to be executed among the K dynamic shape subgraphs; Determining, according to the life cycle information, memory reuse of the M tensors within the first dynamic shape subgraph; According to the memory reuse situation and the first memory requirement information, memory is allocated from a memory pool for the M tensors within the range of the first dynamic shape subgraph.

2. The method according to claim 1, characterized in that The obtaining of the life cycle information and the first memory requirement information of the M tensors within the first dynamic shape subgraph includes: Counting the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, where each tensor corresponds to a life cycle T; The memory requirements of the M tensors within the first dynamic shape subgraph are counted to obtain M memory sizes S, where each tensor corresponds to one memory size S.

3. The method according to claim 1 or 2, characterized in that: The determining, according to the life cycle information, memory reuse of the M tensors within the first dynamic shape subgraph includes: According to the life cycle information, the M tensors are divided into X tensor sets, wherein each of the X tensor sets includes one or more tensors, and the life cycles of any two tensors in each tensor set do not overlap, and X is an integer greater than or equal to 1; Set the tensors in each tensor set to reuse the same memory block.

4. The method according to claim 3, characterized in that The dividing the M tensors into X tensor sets according to the life cycle information includes: The M tensors are subjected to X rounds of tensor division to obtain X sets of tensors, and each round of tensor division obtains a set of tensors; wherein, Each round of tensor partitioning includes: determining a second tensor whose life cycle has no overlap with the first tensor from the currently remaining tensors according to the execution order; then determining a third tensor whose life cycle has no overlap with the second tensor according to the execution order, until all the currently remaining tensors are traversed and the determined tensors are partitioned into the same tensor set.

5. The method according to claim 4, characterized in that The start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from early to late; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from late to early.

6. The method according to any one of claims 3 to 5, characterized in that: The allocating memory for the M tensors within the first dynamic shape subgraph from a memory pool according to the memory reuse situation and the first memory requirement information includes: According to the memory reuse situation and the first memory requirement information, within the range of the first dynamic shape subgraph, sending a tensor memory application request to the memory pool for each of the M tensors; According to the memory application request of each tensor, memory is allocated to each tensor from the memory pool.

7. The method according to claim 6, characterized in that The sending a tensor memory application request to the memory pool for each of the M tensors within the first dynamic shape subgraph according to the memory reuse situation and the first memory requirement information includes: Allocate a corresponding memory identifier to each of the tensor sets; In the first dynamic shape subgraph, the largest S among the memory sizes S required by each tensor in each tensor set is used as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, where the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor.

8. The method according to any one of claims 6 to 7, further comprising: When the N operators are all executed within the scope of the first dynamic shape subgraph, the second memory requirement information of the M tensors within the scope of the second dynamic shape subgraph is obtained; and according to the memory reuse situation and the second memory requirement information, memory is allocated from the memory pool for the M tensors within the scope of the second dynamic shape subgraph.

9. The method according to claim 8, characterized in that The allocating memory for the M tensors within the second dynamic shape subgraph range from the memory pool according to the memory reuse situation and the second memory requirement information includes: In the second dynamic shape subgraph, the largest S among the memory sizes S required by each tensor in each tensor set is used as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, where the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; According to the memory application request of each tensor, memory is allocated to each tensor from the memory pool.

10. The method according to any one of claims 6 to 9, characterized in that: Allocating memory for each tensor from the memory pool according to the memory application request of each tensor includes: Determine whether there is a memory block corresponding to the memory identifier in the memory application request in the current memory pool; If a memory block corresponding to the memory identifier already exists in the memory pool, further determining whether the size of the memory block is greater than or equal to the memory size S in the memory application request; If it is greater than or equal to, the existing memory block is allocated to the tensor corresponding to the memory application request for reuse; If it is less than, re-apply for a size greater than or equal to S from the memory pool MAX The memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released.

11. The method according to claim 10, characterized in that The allocating memory for each tensor from the memory pool according to the memory application request of each tensor includes: If the memory block corresponding to the memory identifier does not exist in the memory pool, a memory block with a size greater than or equal to SMAX is applied for from the memory pool as the memory block corresponding to the memory identifier.

12. An AI computing device, characterized in that: Applied to an AI system, the AI ​​system includes a computation graph, the computation graph includes a target subgraph, the target subgraph includes N operators and M tensors; the target subgraph corresponds to K dynamic shape subgraphs, and the K dynamic shape subgraphs are subgraphs of the target subgraph under K different inputs; wherein N, M and K are all integers greater than or equal to 1; the computing device includes a processor and an AI accelerator, and the AI ​​accelerator includes an accelerator memory; The processor is configured to: Acquire life cycle information and first memory requirement information of the M tensors within a first dynamic shape subgraph; the first dynamic shape subgraph is a subgraph currently to be executed among the K dynamic shape subgraphs; Determining, according to the life cycle information, memory reuse of the M tensors within the first dynamic shape subgraph; Allocate memory for the M tensors within the first dynamic shape subgraph from the accelerator memory according to the memory reuse situation and the first memory requirement information; The AI ​​accelerator is used to: Based on the used memory allocated by the processor to each of the tensors, the N operators are executed within the corresponding dynamic shape subgraph range.

13. The computing device according to claim 12, characterized in that The processor is specifically used for: Counting the life cycles of the M tensors within the first dynamic shape subgraph to obtain M life cycles T, where each tensor corresponds to a life cycle T; The memory requirements of the M tensors within the first dynamic shape subgraph are counted to obtain M memory sizes S, where each tensor corresponds to one memory size S.

14. The computing device according to claim 12 or 13, characterized in that The processor is specifically used for: According to the life cycle information, the M tensors are divided into X tensor sets, wherein each of the X tensor sets includes one or more tensors, and the life cycles of any two tensors in each tensor set do not overlap, and X is an integer greater than or equal to 1; Set the tensors in each tensor set to reuse the same memory block.

15. The computing device according to claim 14, characterized in that The processor is specifically used for: The M tensors are subjected to X rounds of tensor division to obtain X sets of tensors, and each round of tensor division obtains a set of tensors; wherein, Each round of tensor partitioning includes: determining a second tensor whose life cycle has no overlap with the first tensor from the currently remaining tensors according to the execution order; then determining a third tensor whose life cycle has no overlap with the second tensor according to the execution order, until all the currently remaining tensors are traversed and the determined tensors are partitioned into the same tensor set.

16. The computing device according to claim 15, characterized in that The start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from early to late; or, the start time of the life cycle of the first tensor, the second tensor, and the Zhang San tensor is from late to early.

17. The computing device according to any one of claims 14 to 16, characterized in that: The processor is specifically used for: According to the memory reuse situation and the first memory requirement information, within the first dynamic shape subgraph range, for each of the M tensors, a tensor memory application request is sent to the accelerator memory; According to the memory application request of each tensor, memory is allocated for each tensor from the accelerator memory.

18. The computing device according to claim 17, characterized in that The processor is specifically used for: Allocate a corresponding memory identifier to each of the tensor sets; In the first dynamic shape subgraph, the largest S among the memory sizes S required by each tensor in each tensor set is used as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, where the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor.

19. The computing device according to any one of claims 17-18, wherein the processor is further configured to: When the N operators are all executed within the first dynamic shape subgraph range, obtaining second memory requirement information of the M tensors within the second dynamic shape subgraph range; According to the memory reuse situation and the second memory requirement information, memory is allocated from the accelerator memory for the M tensors within the second dynamic shape subgraph range.

20. The computing device of claim 19, wherein: The processor is specifically used for: In the second dynamic shape subgraph, the largest S among the memory sizes S required by each tensor in each tensor set is used as the reusable memory size S of all tensors in the corresponding tensor set. MAX ; For each of the M tensors, a memory application request is initiated, where the memory application request includes the memory identifier corresponding to the tensor set to which the tensor belongs and the memory size S required by the tensor; According to the memory application request of each tensor, memory is allocated for each tensor from the accelerator memory.

21. The computing device according to any one of claims 17 to 20, characterized in that: The processor is specifically used for: Determine whether there is a memory block corresponding to the memory identifier in the memory application request in the current accelerator memory; If a memory block corresponding to the memory identifier already exists in the accelerator memory, further determining whether the size of the memory block is greater than or equal to the memory size S in the memory application request; If it is greater than or equal to, the existing memory block is allocated to the tensor corresponding to the memory application request for reuse; If it is less than, re-apply for a size greater than or equal to S from the accelerator memory MAX The memory block is used as the memory block corresponding to the memory identifier, and the existing memory block is released.

22. The computing device of claim 21, wherein: The processor is specifically used for: If there is no memory block corresponding to the memory identifier in the accelerator memory, a size greater than or equal to S is requested from the accelerator memory. MAX The memory block is used as the memory block corresponding to the memory identifier.

23. A computer-readable storage medium, characterized in that: The computer-readable medium is used to store program codes, and when the program codes are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

24. A computer program, characterized in that The computer program comprises instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 11.

Images