Operator memory allocation method for complex structure deep learning model reasoning
By allocating memory according to the priority of the memory allocation of operators in the deep learning model, the problems of long inference time and low memory utilization of complex structure deep learning models are solved, and efficient memory utilization and inference speed are achieved.
Patent Information
- Application Number
- CN202411462343.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-14
AI Technical Summary
When the prior art deals with complex structure deep learning models, it is difficult to effectively shorten the inference time, and the memory utilization rate is not high, resulting in inferential efficiency.
By statistically computing the types of operators in the graph, the memory allocation priority of each operator is determined, and the memory allocation is performed in combination with deep learning to accelerate the storage structure of kernel NNA and the life cycle of operators, and the memory allocation is performed, breaking the traditional limitation of dependent computing order.
It realizes efficient memory allocation in complex structure deep learning models, reduces data transfer, improves memory utilization, shortens inference time, and improves model inference speed.
Smart Images
Figure CN119440814A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to memory allocation, and in particular to an operator memory allocation method for reasoning of a complex structure deep learning model. Background Art
[0002] With the popularity of ChatGPT, deep learning models have begun to enter people's daily lives. For applications, the most critical thing is model reasoning. Model reasoning can be seen as a traversal of the computational graph. The core of it is to traverse the computational graph (such as Figure 3 The memory allocation problem of each node (also called operator) in the network is shown in Figure 2.
[0003] For memory allocation, we first introduce the current mainstream memory hierarchy, such as Figure 2 As shown. Under current storage technology, the memory hierarchy is usually a pyramid type. The top memory has advanced technology and extremely fast storage speed, but it is expensive and has a small capacity; the bottom memory has a slow storage speed, but it is cheap and has a large capacity. In model reasoning, it is necessary to reasonably allocate each operator to the memory to ensure data storage and maximize memory utilization, so that the reasoning results can be obtained accurately and efficiently.
[0004] At present, the most commonly used memory allocation method is to allocate according to the calculation order of the calculation graph. This method can achieve the output of reasoning results well, but it does not work in terms of data movement and memory utilization, and when the model structure becomes more complex, the reasoning time will increase sharply. At the same time, some optimization algorithms and optimization methods based on this method, or for special network structures and special hardware models, cannot achieve both universality and efficiency due to the large amount of calculation.
[0005] The following introduces a traditional operator memory allocation method. The memory hierarchy is as follows Figure 2 As shown, L1_buffer is reserved for calculation and is not allocated. The allocation objects are L2_buffer, OCM and DDR. The method of allocating operator memory according to the calculation order of the calculation graph is as follows:
[0006] 1) Determine the calculation order of the calculation graph;
[0007] 2) Determine the memory requirements of each operator;
[0008] 3) Determine the life cycle of each operator, for example Figure 3 In the example, after operator 3 is calculated, it needs to wait until operator 4 and operator 5 are calculated before releasing the occupied memory space.
[0009] 4) Based on the calculation order, the memory required by the operator is applied in the order of L2_buffer, OCM, and DDR. It is only applied once, and the occupied memory cannot be applied. If the life cycle ends, the occupied memory space will be released.
[0010] The above method is simple to implement, but it does not fully utilize memory and is prone to unreasonable allocation, wasting reasoning time. At the same time, some existing optimization algorithms are based on this method, and do not jump out of the premise of the calculation order of the calculation graph. When encountering a deep learning model with a complex structure, there is a problem that both versatility and efficiency cannot be taken into account. Summary of the invention
[0011] 1. Technical issues to be resolved
[0012] In view of the above-mentioned shortcomings of the prior art, the present invention provides an operator memory allocation method for reasoning of deep learning models with complex structures, which can effectively overcome the defect of the prior art that it is difficult to effectively shorten the reasoning time when applied to deep learning models with complex structures.
[0013] (II) Technical solution
[0014] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0015] A method for allocating operator memory for inference of a complex structure deep learning model comprises the following steps:
[0016] S1. Types of operators in the statistical computation graph;
[0017] S2. Determine the memory allocation priority of each operator;
[0018] S3. Determine the life cycle of each operator;
[0019] S4. Select operators that have not been memory allocated and have the highest memory allocation priority, and allocate memory based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator;
[0020] S5. Repeat S4 until the memory allocation of all operators is completed.
[0021] Preferably, S2 determines the memory allocation priority of each operator, including:
[0022] The memory allocation priority of each operator is determined based on the data exchange, implementation method and computational workload of each operator.
[0023] Preferably, determining the life cycle of each operator in S3 includes:
[0024] The life cycle of each operator is determined based on the calculation execution order and calculation dependencies.
[0025] Preferably, in S4, operators of the type that have not been memory allocated and have the highest memory allocation priority are selected, and memory allocation is performed based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator, including:
[0026] When allocating memory, the memory of two nodes is allocated at a time based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator;
[0027] Among them, the two nodes include the operator itself and its parent node. If there are multiple parent nodes, memory is allocated to all parent nodes in sequence.
[0028] Preferably, when performing memory allocation, the memory of two nodes is allocated at a time based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator, including:
[0029] Each time memory is allocated, the operator and its parent node are requested in the order of L2_buffer, OCM, and DDR. If the operator itself has completed memory allocation, the operator is skipped; if the parent node has completed memory allocation, the parent node is skipped.
[0030] At the same time, each time memory is allocated, operators with computational dependencies must not occupy their own allocated memory.
[0031] Preferably, S4 is repeated in S5 until memory allocation of all operators is completed, including:
[0032] S6. Assign the output position of each operator to the input position of the corresponding child node;
[0033] S7. Process the input nodes and output nodes of the deep learning model.
[0034] Preferably, the storage structure of the deep learning acceleration core NNA includes L1_buffer, L2_buffer, and L3_buffer from the upper layer to the lower layer;
[0035] L1_buffer, which belongs to the memory inside the single-core NNA, is the storage buffer closest to the computing component. It is used to cache the input data and output results of the computing component and perform fast data access and data exchange with the computing component.
[0036] L2_buffer, a memory for interaction between a single-core NNA and multiple L1_buffers, is a storage buffer with a larger space at the next level of L1_buffer. It is used to respond to read and write requests from L1_buffer and interact with L3_buffer. To ensure efficient data supply to computing components, it has data transmission capabilities with corresponding data bandwidth and data throughput;
[0037] L3_buffer is a memory outside the single-core NNA and is an internal Share SRAM shared by the multi-core NNA cores and is used for data sharing among the multi-core NNAs.
[0038] Preferably, the L3_buffer includes OCM and DDR;
[0039] OCM, used to store input data, intermediate results, and output results of deep learning models, has a slow data exchange speed with L1_buffer;
[0040] DDR is used to store instruction data, weight data, and bias data of deep learning models, and has a slow data exchange speed with L1_buffer.
[0041] (III) Beneficial effects
[0042] Compared with the prior art, the operator memory allocation method for complex structure deep learning model reasoning provided by the present invention has the following beneficial effects:
[0043] 1) The present invention breaks the premise that the memory allocation order is based on the calculation order of operators in the calculation graph, and uses the memory allocation priority of the operator instead of the calculation order to perform memory allocation, so that the memory allocation is no longer affected by various complex model branches and structures, and the problem is converted into determining the operator priority in the model. There is no need to optimize the model structure, and no additional processing is required for each layer. It can adapt to various large-scale complex structure deep learning models;
[0044] 2) The present invention allocates memory according to the memory allocation priority of the operator. In the face of different hardware architectures, operator implementation methods and operator types, efficient memory allocation can be achieved by adjusting the memory allocation priority of the operator. It can cope with various large-scale deep learning models with complex structures, and takes into account both versatility and efficiency. Based on the deep learning acceleration core NNA, an inference time of 15.36ms is achieved on yolov5s, while the inference time of the traditional memory allocation in calculation order is 18.83ms, which is improved by 3.47ms, an increase of 22.6%. Therefore, the present invention can effectively reduce data movement, improve memory utilization, shorten the inference time, and ultimately achieve a significant improvement in the model inference speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 It is a schematic diagram of the process of the present invention;
[0047] Figure 2 is a schematic diagram of an existing memory hierarchy;
[0048] Figure 3 A schematic diagram of a computational graph;
[0049] Figure 4 This is a comparison chart of the inference time of the present invention and the traditional operator memory allocation method on yolov5s. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] A memory allocation method for operators in deep learning model reasoning with complex structures, such as Figure 1 As shown, S1, the types of operators in the statistical calculation graph.
[0052] S2. Determine the memory allocation priority of each operator, including:
[0053] The memory allocation priority of each operator is determined according to the data exchange, implementation method and calculation amount of each operator (for example, the larger the calculation amount, the higher the memory allocation priority of the operator).
[0054] S3. Determine the life cycle of each operator, including:
[0055] Determine the life cycle of each operator based on the calculation execution order and calculation dependencies (such as Figure 3 As shown in FIG. 1 , after the calculation of operator 3 is completed, it is necessary to wait until the calculations of operators 4 and 5 are completed before releasing the occupied memory space).
[0056] S4. Select operators that have not been allocated memory and have the highest memory allocation priority, and allocate memory based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator, including:
[0057] When allocating memory, the memory of two nodes is allocated at a time based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator;
[0058] Among them, the two nodes include the operator itself and its parent node. If there are multiple parent nodes, memory is allocated to all parent nodes in sequence.
[0059] Specifically, when allocating memory, the memory of two nodes is allocated at a time based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator, including:
[0060] Each time memory is allocated, the operator and its parent node are requested in the order of L2_buffer, OCM, and DDR. If the operator itself has completed memory allocation, the operator is skipped; if the parent node has completed memory allocation, the parent node is skipped.
[0061] At the same time, each time memory is allocated, operators with computational dependencies must not occupy their own allocated memory (such as Figure 3 As shown in the figure, when operator 3 is alive, operator 4 and operator 5 are also alive and have allocated memory, then operator 3 shall not occupy the memory already allocated by operator 4 and operator 5).
[0062] S5. Repeat S4 until the memory allocation of all operators is completed.
[0063] S4 is repeated in S5 until the memory allocation of all operators is completed, including:
[0064] S6. Assign the output position of each operator to the input position of the corresponding child node;
[0065] S7. Process the input nodes and output nodes of the deep learning model.
[0066] like Figure 2 As shown, the storage structure of the deep learning acceleration core NNA includes L1_buffer, L2_buffer, and L3_buffer from the upper layer to the lower layer;
[0067] L1_buffer, which belongs to the memory inside the single-core NNA, is the storage buffer closest to the computing component. It is used to cache the input data and output results of the computing component and perform fast data access and data exchange with the computing component.
[0068] L2_buffer, a memory for interaction between a single-core NNA and multiple L1_buffers, is a storage buffer with a larger space at the next level of L1_buffer. It is used to respond to read and write requests from L1_buffer and interact with L3_buffer. To ensure efficient data supply to computing components, it has data transmission capabilities with corresponding data bandwidth and data throughput;
[0069] L3_buffer is a memory outside the single-core NNA and is an internal Share SRAM shared by the multi-core NNA cores and is used for data sharing among the multi-core NNAs.
[0070] Specifically, L3_buffer includes OCM and DDR;
[0071] OCM, used to store input data, intermediate results, and output results of deep learning models, has a slow data exchange speed with L1_buffer;
[0072] DDR is used to store instruction data, weight data, and bias data of deep learning models, and has a slow data exchange speed with L1_buffer.
[0073] The core point of the technical solution of the present application is to allocate memory according to the memory allocation priority of the operator. When faced with various factors such as various large-scale complex structure deep learning models, the hardware structure of different deep learning acceleration cores NNA, operator implementation methods and the characteristics of the operator itself, it is only necessary to adjust the memory allocation priority of the operator according to the above factors to achieve efficient memory allocation.
[0074] The following is a brief explanation using yolov5s as an example. The memory allocation priority of operators in yolov5s is shown in the following table:
[0075] Table 1 Memory allocation priority of operators in yolov5s
[0076]
[0077]
[0078] Considering the characteristics of the operator, the concat operator in Table 1 is set to the highest priority. Based on the hardware structure of the deep learning acceleration core NNA, the concat operator does not need to be calculated or interact with the computing components, so it only needs to be moved and combined. The highest priority ensures that the input and output of the concat operator are both on the L2_buffer, so the local optimum of the concat operator can be achieved.
[0079] Considering the computational complexity of the operator, the conv2d_swish operator and the conv2d operator in Table 1 are at a lower priority. Based on the hardware structure of the deep learning acceleration core NNA, the convolution operator has a large computational complexity, and the weights and inputs need to be used in batches multiple times. Therefore, prioritizing them to the L2_buffer helps the convolution operator approach the local optimum.
[0080] Considering the hardware structure, the add operator, pool operator, and resize operator in Table 1 are at the lower priority. Based on the hardware structure of the deep learning acceleration core NNA and the influence of the L2_buffer data placement mode, the implementation method of some operators requires allocation in OCM, and the computational complexity is also relatively small.
[0081] The memory allocation priority of operators in yolov5s is shown in Table 1, where the conv2d_swish operator uses the characteristics of the activation function to perform operator fusion. When compared below, the traditional operator memory allocation method also uses this technology.
[0082] The technical solution of this application is used to allocate memory for yolov5s. The entire model is intercepted to the permute operator. Subsequent NMS and other operations are performed using the CPU. The operator memory allocation of yolov5s is shown in the following table:
[0083] Table 2 yolov5s operator memory allocation table
[0084]
[0085]
[0086]
[0087]
[0088] Based on the deep learning acceleration core NNA and different bus utilization, the inference time of yolov5s is counted. The data type is fp16. The specific results are as follows Figure 4 As shown (where * represents the technical solution of the present application, and ● represents the traditional method of allocating memory according to the calculation order of operators in the calculation graph). Figure 4 It can be seen that the technical solution of the present application comprehensively surpasses the traditional operator memory allocation method under different bus utilization rates. Experiments have proved that the technical solution of the present application can improve memory utilization and shorten the inference time for complex structure deep learning models.
[0089] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for allocating operator memory for inference of complex structure deep learning models, characterized by: The following steps are involved: S1. Types of operators in the statistical computation graph; S2. Determine the memory allocation priority of each operator; S3. Determine the life cycle of each operator; S4. Select operators that have not been memory allocated and have the highest memory allocation priority, and allocate memory based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator; S5. Repeat S4 until the memory allocation of all operators is completed.
2. The operator memory allocation method for complex structure deep learning model reasoning according to claim 1, characterized in that: S2 determines the memory allocation priority of each operator, including: The memory allocation priority of each operator is determined based on the data exchange, implementation method and computational workload of each operator.
3. The operator memory allocation method for complex structure deep learning model reasoning according to claim 2, characterized in that: S3 determines the life cycle of each operator, including: The life cycle of each operator is determined based on the calculation execution order and calculation dependencies.
4. The operator memory allocation method for complex structure deep learning model reasoning according to claim 3, characterized in that: S4 selects operators that have not been allocated memory and have the highest memory allocation priority, and allocates memory based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator, including: When allocating memory, the memory of two nodes is allocated at a time based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator; Among them, the two nodes include the operator itself and its parent node. If there are multiple parent nodes, memory is allocated to all parent nodes in sequence.
5. The operator memory allocation method for complex structure deep learning model reasoning according to claim 4, characterized in that: When performing memory allocation, the memory of two nodes is allocated at a time based on the storage structure of the deep learning acceleration core NNA and the life cycle of each operator, including: Each time memory is allocated, the operator and its parent node are requested in the order of L2_buffer, OCM, and DDR. If the operator itself has completed memory allocation, the operator is skipped; if the parent node has completed memory allocation, the parent node is skipped. At the same time, each time memory is allocated, operators with computational dependencies must not occupy their own allocated memory.
6. The operator memory allocation method for complex structure deep learning model reasoning according to claim 5, characterized in that: S4 is repeated in S5 until the memory allocation of all operators is completed, including: S6. Assign the output position of each operator to the input position of the corresponding child node; S7. Process the input nodes and output nodes of the deep learning model.
7. The operator memory allocation method for complex structure deep learning model reasoning according to claim 5, characterized in that: The storage structure of the deep learning acceleration core NNA includes L1_buffer, L2_buffer, and L3_buffer from the upper layer to the lower layer; L1_buffer, which belongs to the memory inside the single-core NNA, is the storage buffer closest to the computing component. It is used to cache the input data and output results of the computing component and perform fast data access and data exchange with the computing component. L2_buffer, a memory for interaction between a single-core NNA and multiple L1_buffers, is a storage buffer with a larger space at the next level of L1_buffer. It is used to respond to read and write requests from L1_buffer and interact with L3_buffer. To ensure efficient data supply to computing components, it has data transmission capabilities with corresponding data bandwidth and data throughput; L3_buffer is a memory outside the single-core NNA and is an internal Share SRAM shared by the multi-core NNA cores and is used for data sharing among the multi-core NNAs.
8. The operator memory allocation method for complex structure deep learning model reasoning according to claim 7, characterized in that: The L3_buffer includes OCM and DDR; OCM, used to store input data, intermediate results, and output results of deep learning models, has a slow data exchange speed with L1_buffer; DDR is used to store instruction data, weight data, and bias data of deep learning models, and has a slow data exchange speed with L1_buffer.