Fine-grained scheduling-based deep learning model memory optimization method for small equipment

Through fine-grained scheduling segmentation and scheduling optimization of deep learning model operators on small devices, the problem of excessive memory usage is solved, and the possibility of deploying complex deep learning models on small devices is realized, which significantly reduces memory requirements and broadens the Application scope.

CN118916167APending Publication Date: 2024-11-08PEKING UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411069656.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When deploying deep learning models on small devices, it is difficult for the prior art to effectively optimize the memory usage of intermediate tensors, especially in networks that process high-resolution images or long-sequence data, resulting in excessive memory requirements and limiting the deployment of models and application.

Method used

Using a fine-grained scheduling method, the DNN operator with the largest peak memory occupancy is divided into fine-grained operators, and the life periods of these operators do not overlap through reasonable scheduling, thereby realizing memory sharing and breaking through the coarse-grained scheduling. Restrict, reduce memory usage. The specific steps include the coordinated work of the measuring device, simulator, analyzer, and optimizer. By building an axis connection diagram and inverting the subsequent traversal algorithm, searching for the segmentation scheme to optimize memory usage.

Benefits of technology

With no more than 5% performance overhead, the memory requirements of deep neural networks are significantly optimized, and the memory usage can be reduced by 80%, broadening the scope of application on small devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916167A_ABST
    Figure CN118916167A_ABST
Patent Text Reader

Abstract

The invention discloses a deep learning model memory optimization method based on fine-grained scheduling for embedded equipment, and belongs to the field of deep learning models. According to the method, a deep learning model DNN is input to an analyzer, and the analyzer analyzes a connection relationship between an operator and a circulation axis of an operator dependent on the operator according to a TE of each operator in the DNN, and outputs an ACG to an optimizer; the optimizer converts the DNN into a fine-grained DNN according to a segmentation scheme of the ACG search for the DNN, and schedules the fine-grained DNN; the fine-grained DNN and the scheduling of the fine-grained DNN are handed over to a simulator to evaluate the memory occupation of the fine-grained DNN, the execution performance of the fine-grained DNN is obtained through a measurer, memory and performance data are fed back to an optimizer to carry out iterative search so as to obtain the optimal fine-grained DNN and the scheduling of the fine-grained DNN, and execution codes of the DNN are generated according to the execution codes and are compiled and deployed on target equipment. According to the method, on the premise that the performance overhead does not exceed 5%, the memory requirements of numerous popular deep neural networks DNN during operation can be remarkably optimized, and the application range of the DNN on small equipment is widened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and relates to the deployment of deep learning models (also called deep neural networks DNN) on embedded devices (also called small devices), and specifically to a deep learning model memory optimization method based on fine-grained scheduling for small devices. Background Art

[0002] Deep Learning Model, also known as Deep Neural Network (DNN), is an artificial neural network composed of multiple layers of neurons. These networks can learn and simulate complex nonlinear relationships and are the core elements of Artificial Intelligence (AI), which has been widely used in recent years. Tensor, also known as multidimensional array (ND-Array), is the basic data representation in DNN. The dimension of a tensor is also called tensor axis (TensorAxis). An N-dimensional tensor has N tensor axes. All tensor axes of tensor X are denoted as Xa, and the i-th tensor axis is denoted as Xa. i . The shape of a tensor is a combination of the lengths of its axes. For example, a tensor X with a shape of 2×3 means that its first axis has a length of 2 and its second axis has a length of 3. Each element in a tensor is a scalar that can be accessed using a subscript index. For example, A[0,2] represents the scalar element at the 0th row and 2nd column of a two-dimensional tensor A. Operators are the basic operations performed on tensors in DNNs. An operator usually has several input tensors and one output tensor. Operators in DNNs can be simple mathematical operations such as addition and multiplication, or more complex operations such as matrix multiplication, convolution, and pooling. Most DNN operators can be represented by tensor expressions (TEs). Tensor expressions usually contain loop axes, tensors, summation symbols, and other scalar operation symbols. For example, matrix multiplication can be represented as C[m,n]=∑ k A[m,k]×B[k,n], where m,n,k are cyclic axes, A,B,C are tensors (A and B are input tensors, C is the output tensor). The cyclic axis that indexes the output tensor in TE is called the spatial axis, which corresponds one-to-one to the tensor axis of the output tensor. The cyclic axis involved in the summation symbol is called the temporal axis. For example, in the previous example, m,n are spatial axes, and k is the temporal axis. All spatial (temporal) axes of operator X are denoted as Xs(Xt), and the i-th spatial (temporal) axis is denoted as Xs i (Xt i). DNN can be represented as a directed acyclic graph (DAG). The nodes in DAG are operators of DNN, and the edges are data dependencies between operators. If the output tensor of operator u is an input tensor of operator v, there is an edge between u and v. The scheduling of DNN refers to the execution order of operators in DAG, which needs to satisfy the data dependencies between operators. If there is an edge from u to v, the execution order of u must be before v in the scheduling.

[0003] The practice of deploying DNNs on small devices such as microcontroller units (MCUs) is becoming increasingly common, laying the foundation for the widespread application of artificial intelligence in the field of the Internet of Things (IoT). These small devices generally have limited storage capacity, typically no more than 2MB of static random access memory (SRAM) and a few MB of expandable read-only flash memory. For example, the STM32F767ZI MCU is equipped with 512KB of SRAM and an initial capacity of 2MB of expandable Flash. When deploying DNNs on small devices, the model weights are usually stored in the relatively large Flash memory, while the intermediate tensors generated during inference (execution) must be allocated to the limited SRAM. However, even for networks designed specifically for resource-constrained devices, the memory requirements of intermediate tensors are quite large compared to the limited SRAM of small devices. For example, networks such as MobileNetV2 and BERT-Tiny need to process high-resolution images or long sequences of tokens to ensure accuracy, which increases the size of tensors. In addition, some networks adopt complex structures to improve the expressiveness of the model, which leads to the generation of more intermediate tensors during the reasoning process, further increasing the demand for memory.

[0004] Several libraries and frameworks have been developed for deploying deep neural networks (DNNs) on small devices, but they pay insufficient attention to optimizing the use of intermediate tensor memory. CMSIS-NN provides DNN operator implementations for small devices, but does not provide DAG scheduling and memory management. TensorFlow-Lite Micro provides a basic scheduling framework and simple memory management for the deployment of DNNs on small devices, but does not have effective memory optimization methods. TinyEngine and other works reduce memory usage by manually writing patch-based operators, but this method introduces significant latency overhead and is limited to simple convolutional neural network (CNN) structures, making it difficult to apply to more complex DNNs, such as complex CNNs like NASNet or Transformer models like BERT-Tiny. Serenity and HMCOS and other studies have found that scheduling the execution order of operators in networks with complex structures can effectively reduce memory usage. However, these memory optimizations are limited because they only consider coarse-grained scheduling, that is, network scheduling based on DNN operators. Therefore, it is of great significance to study a memory optimization method for deep learning models based on fine-grained scheduling for small devices. Summary of the invention

[0005] In response to the above problems existing in the prior art, the present invention proposes a deep learning model memory optimization method based on fine-grained scheduling for small devices, which divides the DNN operators with the largest peak memory occupancy into fine-grained operators (the division of a part of the operators in the DNN is the division of the DNN), and reasonably schedules the execution of each DNN operator so that the life cycles of the split operators do not overlap, thereby allowing those operators to share the same memory, breaking through the coarse-grained scheduling limitations of previous work and further reducing memory occupancy.

[0006] The technical solution of the present invention is as follows:

[0007] A method for deep learning model memory optimization based on fine-grained scheduling for small devices is provided. Given a target device, a performance overhead limit (a predetermined performance overhead limit: the excess ratio of the execution time of the optimized DNN on the device relative to the execution time of the original DNN), a search time limit and an original DNN represented in ONNX format as input, the specific steps include:

[0008] 1) Compile and deploy the measurer to the target device. The measurer measures the performance of each operator of the original DNN, that is, the execution time. The measured value is fed back to the simulator. The simulator calculates the total execution time of the original DNN, and calculates the total execution time limit of the optimized fine-grained DNN in combination with the performance overhead limit of a predetermined ratio, and feeds the time limit value back to the optimizer;

[0009] 2) The simulator calculates the peak memory based on the scheduling order of each operator in the original DNN and the dependencies between operators, and feeds the calculated value back to the optimizer. Specifically, the active period of each operator output tensor is calculated based on the scheduling sequence and dependencies. Let the execution time of operator v be t1, and the time when all operators that depend on v are executed is t2. Then the time interval [t1, t2) is called the active period of v. The sum of the output tensors of all operators active at the same time point is the memory usage at that time point. The maximum value of the memory usage at all time points is the peak memory usage of the DNN.

[0010] 3) The analyzer analyzes the connection relationship between the cyclic axes of each operator of the original DNN, constructs and outputs the axis connection graph ACG to the optimizer. Specifically, the analyzer analyzes the connection relationship between the cyclic axes of the operator and the operators it depends on according to the tensor expression TE of each operator in the DNN. For a matrix multiplication operator C in the DNN and its dependent operators A and B, its TE is C[m,n]=Σ k A[m,k]×B[k,n], where m, n, k are cyclic axes, m is the first spatial axis of C and also indexes the first spatial axis of A, n is similar, and k is the first time axis of C and also indexes the second spatial axis of A and the first spatial axis of B. Then there is a connection relationship between the cyclic axes: A.s1→C.s1, A.s2→C.t1, B.s1→C.t1, B.s2→C.s2, where A.s1 represents the first spatial axis of operator A, B.s1 represents the first spatial axis of operator B, C.s1 represents the first spatial axis of operator C, A.s2 represents the second spatial axis of operator A, B.s2 represents the second spatial axis of operator B, C.s2 represents the second spatial axis of operator C, and C.t1 represents the first time axis of operator C; then use the cyclic axes of different operators as nodes and the connection relationships as edges to construct ACG;

[0011] 4) The optimizer searches for a partitioning scheme for the DNN based on the ACG, and partitions the original DNN into fine-grained DNNs. Each partitioning scheme consists of several connected components CC of ACG and a partitioning factor PF. The original DNN is partitioned into fine-grained DNNs along the ACG CC. Each operator that the ACGCC passes through is partitioned into several fine-grained operators of the same shape along the corresponding cyclic axis. The length of each fine-grained operator on the partitioned cyclic axis is equal to PF.

[0012] 5) The optimizer applies the reverse subsequent traversal RPO algorithm to the segmented fine-grained DNN to obtain the operator scheduling sequence;

[0013] 6) The optimizer outputs the fine-grained DNN of step 4) and the scheduling sequence of step 5) to the simulator; the simulator calculates the peak memory during the execution of the fine-grained DNN according to the scheduling order of each operator in the fine-grained DNN and the dependencies between operators, and calls the measurer to obtain the performance of a single operator, stores its performance data, calculates the sum of the performance of each operator, and then feeds the peak memory and the sum of the performance of each operator back to the optimizer;

[0014] 7) The optimizer eliminates the segmentation schemes with performance overhead exceeding a predetermined ratio based on the fed-back memory and performance data; further iterates the search for segmentation schemes, i.e. repeats steps 4) to this step, until the search time exceeds the given search time limit. The optimizer selects the currently optimal segmentation scheme, i.e. obtains the fine-grained DNN and its scheduling sequence with performance loss not exceeding the given limit and the lowest memory usage, and finally generates the DNN execution code based on this, compiles and deploys it on the target device.

[0015] Furthermore, the step 1) compiles and deploys the measurement device to the target device by burning it via USB.

[0016] Furthermore, the performance overhead limit of step 1) is set at a predetermined ratio, and the predetermined ratio is adjustable and is 5% by default.

[0017] Furthermore, in step 4), the optimizer searches for a partitioning scheme for the DNN based on the ACG, and considers some nodes in the ACG, wherein the connection length of the connected component CC of the considered some nodes is at least 2.

[0018] Furthermore, in step 6), the simulator calls the measurer to obtain the performance of a single operator and stores the performance data. When encountering operators of the same type and with the same shape, the simulator will directly use the stored performance data without performing measurement.

[0019] Furthermore, the step 7) generates the execution code of the DNN, which is C++ code.

[0020] The present invention discloses a method for optimizing the memory of a deep learning model based on fine-grained scheduling for small devices. The method divides the DNN operators with the largest peak memory usage into fine-grained operators, and reasonably schedules the execution of each operator of the fine-grained DNN. Under the premise of maintaining a performance overhead of no more than 5%, the memory requirements of many popular deep neural networks DNN can be significantly optimized during operation. Compared with the current most advanced technology, the memory usage can be reduced by up to 80%. This significant memory optimization enables many DNNs that were originally unable to be deployed on small devices due to memory limitations to be successfully deployed on these devices. This technological breakthrough not only optimizes the efficiency of resource utilization, but also broadens the application scope of DNN on small devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the main components of the present invention;

[0022] Figure 2 A flowchart of a deep learning model memory optimization method based on fine-grained scheduling for small devices according to the present invention;

[0023] Figure 3 It is a schematic diagram of the DAG of (a Block of) the DNN input in a specific embodiment of the present invention, the TE of each operator, and the ACG of the Block;

[0024] Figure 4 It is a schematic diagram of a fine-grained DNN obtained by the DNN segmentation scheme 1 and a peak memory obtained after scheduling in a specific embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of a fine-grained DNN obtained by segmentation scheme 2 and a peak memory obtained after scheduling in a specific embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of the fine-grained DNN obtained by the DNN segmentation scheme 3 in a specific embodiment of the present invention and the peak memory obtained after scheduling. DETAILED DESCRIPTION

[0027] The present invention will be further clearly and completely described below through specific embodiments in conjunction with the accompanying drawings.

[0028] The present invention provides a deep learning model memory optimization method based on fine-grained scheduling for small devices. Figure 1 The schematic diagram of the main components of the present invention is as follows:

[0029] A. Analyzer: Analyzes the input DNN to clarify the dependencies of the loop axes of its operators and constructs an ACG based on them. Specifically, it includes:

[0030] a) Input DNN represented in OpenNeural Network Exchange (ONNX) format;

[0031] b) According to the TE of each operator in the DNN, the connection relationship between the operator and the cyclic axis of the operator it depends on is analyzed. For example, for the matrix multiplication operator C and its dependent operators A and B, its TE is C[m,n]=∑ kA[m,k]×B[k,n], where m is the first spatial axis of C and indexes the first spatial axis of A. n is similar, and k is the first temporal axis of C and indexes the second spatial axis of A and the first spatial axis of B. Then there are the following cyclic axis connection relationships: A.s1→C.s1, A.s2→C.t1, B.s1→

[0032] C.t1, B.s2→C.s2;

[0033] c) Using the cyclic axes of different operators as nodes and the connection relationships obtained through analysis as edges, construct ACG.

[0034] B. Optimizer: Based on the ACG obtained by the analyzer, the original DNN can be converted into a finer-grained DNN by searching for the relevant partition scheme (PS), and scheduled to evaluate the memory optimization effect, and run on the target device to evaluate the performance overhead, including:

[0035] a) Each partitioning scheme consists of several ACG connected components (CC) and partition factors (PF), which divide the original DNN into fine-grained DNNs along the ACG CC.

[0036] Each operator in ACG CC is divided into several fine-grained operators of the same shape along the corresponding axis, and the length of each fine-grained operator on the divided axis is equal to PF;

[0037] b) Apply the Reversed Post-Order (RPO) algorithm to the segmented fine-grained DNN

[0038] (Reverse the sequence obtained by post-order traversal of DAG) to obtain the operator scheduling sequence, and give the fine-grained DNN and the scheduling sequence to the simulator to evaluate its runtime peak memory usage and running performance;

[0039] c) Finally, the fine-grained DNN and its schedule with the lowest memory usage under the given performance loss limit are selected as output, and the corresponding C++ code is generated based on it, compiled and deployed to the target device;

[0040] The optimizer will consider the segmentation scheme for different blocks in the original DNN. Block is the basic composition pattern of DNN. A DNN is usually composed of several stacked blocks with similar structures.

[0041] C. Simulator: Input a given DNN and schedule sequence and evaluate its peak memory and execution performance, including:

[0042] a) Calculate the peak memory of the DNN based on the scheduling order of each operator in the input DNN and the dependencies between operators: Calculate the active period of each operator output tensor based on the scheduling sequence and dependencies (assuming that the execution time of operator v is t1, and the time when all operators that depend on v are executed is t2, then the time interval [t1, t2) is called the active period of v). The sum of the sizes of the output tensors of all operators active at the same time point is the memory usage at that time point, and the maximum value of the memory usage at all time points is the peak memory usage of the DNN;

[0043] b) Estimate the overall performance of the DNN based on the performance of each operator: Call the measurer to obtain the performance of the fine-grained DNN / single operator and store its performance data. When encountering the same type of operators with the same shape, the stored performance data will be used directly without measurement.

[0044] D.Measurer: Given an operator type and shape, measures operator performance.

[0045] a) Deployed on a small device on the target and communicates with the host via USB;

[0046] b) receiving a measurement request from the simulator of the host, the request including the type and shape of the operator of the DNN;

[0047] c) According to the type and shape of the operator, the corresponding code is executed, the execution performance is measured, and the performance data is returned to the simulator of the host.

[0048] The present invention discloses a deep learning model memory optimization method based on fine-grained scheduling for small devices. Given a target device STM32H7A3ZI-Q microprocessor (Micro Controller Unit, MCU), a performance overhead limit of no more than 5%, a search time limit of 2 minutes, and an input DAG (such as DAG) of the original DNN represented in ONNX format, the memory optimization method is implemented. Figure 3 As shown on the left, a simple DNN is selected as a demonstration in this embodiment, and the memory optimization of the deep learning model based on fine-grained scheduling is performed on it. The specific process is as follows Figure 2 As shown, the steps are as follows:

[0049] 1) Compile and deploy the measurer to the target device (burned via USB). The measurer measures the performance (execution time) of each operator of the original DNN, and the measured value is fed back to the simulator. The simulator calculates the total execution time as 184.549us (the measurer only feeds back the execution time of a single operator, and the total execution time is calculated in the simulator). Combined with the performance overhead limit of 5%, it can be seen that the total execution time after optimization cannot exceed 184.549*(1+5%)=193.776us. The total execution time limit value is fed back to the optimizer.

[0050] 2) The simulator calculates the peak memory based on the scheduling order of each operator in the original DNN and the dependencies between operators, and feeds the calculated value back to the optimizer;

[0051] Figure 3 The Linear on the left represents the linear mapping operator, which is actually a matrix multiplication with transposition. However, the weight matrix (W1, W3 in the figure) is generally stored directly in Flash in small devices, and will not be read from SRAM memory for calculation, so it will not affect the peak memory of DNN deployed on small devices. Therefore, these weight matrices are not considered in the analysis. The 16→64 marked on Linear means that the second dimension of the input is mapped from 16 to 64, that is, the size of the weight matrix is ​​64×16. ReLU represents the linear rectification operator, which is a common nonlinear activation function in DNN.

[0052] Figure 3 The numbers on the left side of each operator node indicate the execution order after RPO scheduling. According to the scheduling order, the active period of each operator output matrix can be calculated (assuming that the execution time of each operator is 1): X0: [0, 4), X1: [0, 2), X2: [1, 3), X3: [2, 4), X4: [3, ∞); thus, the active tensor at each time point can be calculated: 0: {X0, X1}, 1: {X0, X1, X2}, 2: {X0, X2, X3}, 3: {X0, X3, X4}, 4: {X4}; thus, the peak memory of 768 (the sum of the sizes of X0, X1, and X2) can be calculated at time point 1 (or time interval [1, 2)). The scheduling of DNN and the calculation of peak memory by some simulators in the following embodiments are similar, so such calculations will be omitted later.

[0053] 3) The analyzer analyzes the connection relationship between the cyclic axes of each operator of the DNN, constructs and outputs the axis connection graph ACG to the optimizer;

[0054] Figure 3 The right side shows the Figure 3 The axis connection graph ACG constructed by the DNN on the left according to the TE of each operator is composed of TE: X1[m,n]=Σ k (X0[m,k]*W1[n,k]), we can get that there is a connection between the cyclic axes: X0.s1→X1.s1, X0.s2→X1.t1; from TE: X2[m,n]=max(X1[m,n],0), we can get that there is a connection: X1.s1→X2.s1, X1.s2→X2.s2; from TE: X3[m,n]=Σ k(X2[m,k]*W3[n,k]), we can get that there is a connection: X2.s1→X3.s1, X2.s2→X3.t1; from TE: X4[m,n]=X3[m,n]+X0[m,n], we can get that there is a connection: X3.s1→X4.s1, X3.s2→X4.s2, X0.s1→X4.s1, X0.s2→X4.s2.

[0055] 4) The optimizer searches for a DNN segmentation scheme based on ACG and divides the original DNN into fine-grained DNNs;

[0056] Figure 3 The nodes in the dark part of the axis connection graph ACG on the right represent the CCs that will be considered in the optimizer's splitting plan, which are called CC1 = {X1.s1, X2.s1, X3.s1, X4.s1} (X0 is the input, and its tensor axis is not considered as the splitting axis) and CC2 = {X1.s2, X2.s2, X3.t1}. Note that the connection lengths between the remaining loop axes are shorter, with the longest being only 1, while the longest connection length in CC1 is 3 (X1.s1→X2.s1→X3.s1→X4.s1), and the longest connection length in CC2 is 2 (X1.s2→X2.s2→X3.t1). CCs with a length less than 2 generally have no actual benefits after splitting, so they are not considered.

[0057] During the optimizer search process, a partitioning scheme is found as {(CC1, PF = 4)}. The fine-grained DNN after partitioning is as follows: Figure 4 As shown, the Slice operator represents slicing the tensor (the first axis) and outputting a part of the tensor, and the Concat operator represents merging multiple tensors. Because the input X0 of the subgraph covered by CC1 has an axis connected to the axis of CC1 (X0.s1→X1.s1), the Slice operator is needed to split the input tensor X0 as the input of each subgraph slice. Because the output of the subgraph covered by CC1 is X4, and the loop axis corresponding to X4 in CC1 is the spatial axis, the results of each subgraph slice need to be connected using Concat as the final output.

[0058] 5) The optimizer applies the reverse subsequent traversal RPO algorithm to the segmented fine-grained DNN to obtain the operator scheduling sequence; the operator scheduling sequence is as follows Figure 4 The numbers on the operator nodes are shown.

[0059] 6) The optimizer outputs the fine-grained DNN of step 4) and the scheduling sequence of step 5) to the simulator; the simulator calculates the peak memory during the execution of the fine-grained DNN according to the scheduling order of each operator in the fine-grained DNN and the dependencies between the operators, and calls the measurer to obtain the performance of a single operator, stores its performance data, calculates the sum of the performance of each operator, and then feeds the peak memory and the sum of the performance of each operator back to the optimizer; the peak memory obtained by the first search of the segmentation scheme {(CC1, PF = 4)} is 640, and the sum of the performance of each operator, that is, the execution time, is 188.24us.

[0060] 7) The optimizer excludes the segmentation schemes with performance overhead exceeding the predetermined ratio according to the fed-back memory and performance data; further iterates the search for segmentation schemes, i.e. repeats steps 4) to this step, until the search time exceeds the given search time limit. The optimizer selects the currently optimal segmentation scheme, i.e. obtains the fine-grained DNN and its scheduling sequence with the performance loss not exceeding the given limit and the lowest memory usage, and finally generates the DNN execution code based on it, compiles and deploys it on the target device;

[0061] The execution time of the first search segmentation scheme {(CC1, PF=4)} is 188.24us, which does not exceed the performance overhead limit of 193.776us, and further iterates to search for segmentation schemes. Figure 5 , Figure 6 The fine-grained DNN, scheduling sequence and peak memory obtained by searching two partitioning schemes {(CC2, PF = 32)} and {(CC1, PF = 4), (CC2, PF = 32)} during the search process are shown respectively. The optimal partitioning scheme finally selected is {(CC1, PF = 4), (CC2, PF = 32)}, with a peak memory of 384 and an execution time of 191.931us.

[0062] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A deep learning model memory optimization method based on fine-grained scheduling for small devices, characterized in that: Given the target device, performance overhead constraints, search time constraints, and the original DNN represented in ONNX format as input, the specific steps include: 1) Compile and deploy the measurer to the target device. The measurer measures the performance of each operator of the original DNN, that is, the execution time. The measured value is fed back to the simulator. The simulator calculates the total execution time of the original DNN, and calculates the total execution time limit of the optimized fine-grained DNN in combination with the performance overhead limit of a predetermined ratio, and feeds the time limit value back to the optimizer; 2) The simulator calculates the peak memory based on the scheduling order of each operator in the original DNN and the dependencies between operators, and feeds the calculated value back to the optimizer. Specifically, the active period of each operator output tensor is calculated based on the scheduling sequence and dependencies. Let the execution time of operator v be t1, and the time when all operators that depend on v are executed is t2. Then the time interval [t1, t2) is called the active period of v. The sum of the output tensors of all operators active at the same time point is the memory usage at that time point. The maximum value of the memory usage at all time points is the peak memory usage of the DNN. 3) The analyzer analyzes the connection relationship between the cyclic axes of each operator of the original DNN, constructs and outputs the axis connection graph ACG to the optimizer. Specifically, the analyzer analyzes the connection relationship between the cyclic axes of the operator and the operators it depends on according to the tensor expression TE of each operator in the DNN. For a matrix multiplication operator C in the DNN and its dependent operators A and B, its TE is C[m,n]=∑ k A[m,k]×B[k,n], where m, n, k are cyclic axes, m is the first spatial axis of C and also indexes the first spatial axis of A, n is similar, and k is the first time axis of C and also indexes the second spatial axis of A and the first spatial axis of B. Then there is a connection relationship between the cyclic axes: A.s1→C.s1, A.s2→C.t1, B.s1→C.t1, B.s2→C.s2, where A.s1 represents the first spatial axis of operator A, B.s1 represents the first spatial axis of operator B, C.s1 represents the first spatial axis of operator C, A.s2 represents the second spatial axis of operator A, B.s2 represents the second spatial axis of operator B, C.s2 represents the second spatial axis of operator C, and C.t1 represents the first time axis of operator C; then use the cyclic axes of different operators as nodes and the connection relationships as edges to construct ACG; 4) The optimizer searches for a partitioning scheme for the DNN based on the ACG, and partitions the original DNN into fine-grained DNNs. Each partitioning scheme consists of several connected components CC of ACG and a partitioning factor PF. The original DNN is partitioned into fine-grained DNNs along the ACG CC. Each operator that the ACGCC passes through is partitioned into several fine-grained operators of the same shape along the corresponding cyclic axis. The length of each fine-grained operator on the partitioned cyclic axis is equal to PF. 5) The optimizer applies the reverse subsequent traversal RPO algorithm to the segmented fine-grained DNN to obtain the operator scheduling sequence; 6) The optimizer outputs the fine-grained DNN of step 4) and the scheduling sequence of step 5) to the simulator. The simulator calculates the peak memory during the execution of the fine-grained DNN according to the scheduling order of each operator in the fine-grained DNN and the dependencies between operators, and calls the measurer to obtain the performance of a single operator, stores its performance data, calculates the sum of the performance of each operator, and then feeds the peak memory and the sum of the performance of each operator back to the optimizer; 7) The optimizer eliminates the segmentation schemes with performance overhead exceeding a predetermined ratio based on the fed-back memory and performance data; further iterates the search for segmentation schemes, i.e. repeats steps 4) to this step, until the search time exceeds the given search time limit. The optimizer selects the currently optimal segmentation scheme, i.e. obtains the fine-grained DNN and its scheduling sequence with performance loss not exceeding the given limit and the lowest memory usage, and finally generates the DNN execution code based on this, compiles and deploys it on the target device.

2. The method for optimizing memory of a deep learning model based on fine-grained scheduling for small devices according to claim 1, characterized in that: The step 1) compiles and deploys the measurement device to the target device by burning it via USB.

3. The method for optimizing memory of a deep learning model based on fine-grained scheduling for small devices according to claim 1, characterized in that: The step 1) limits the performance overhead by a predetermined ratio, and the predetermined ratio is adjustable and is 5% by default.

4. The method for optimizing memory of a deep learning model based on fine-grained scheduling for small devices according to claim 1, characterized in that: In step 4), the optimizer searches for a partitioning scheme for the DNN based on the ACG, and considers some nodes in the ACG, wherein the connection length of the connected component CC of the considered some nodes is at least 2.

5. The method for optimizing memory of a deep learning model based on fine-grained scheduling for small devices according to claim 1, characterized in that: In step 6), the simulator calls the measurer to obtain the performance of a single operator and stores the performance data. When encountering operators of the same type and with the same shape, the simulator will directly use the stored performance data without performing measurement.

6. The method for optimizing memory of a deep learning model based on fine-grained scheduling for small devices according to claim 1, characterized in that: The step 7) generates the execution code of the DNN, which is C++ code.

Citation Information

Cited By

  • DNN universal deployment method and system based on heterogeneous system

    CN119759595A