Deep learning memory optimization method for microcontroller

Abstract

The present invention belongs to the technical field of deep learning. Disclosed is a deep learning memory optimization method for a microcontroller. The present invention proposes a multi-operator fusion computing optimization method, by means of which a residual structure, a convolution-activation structure and a connection structure, which are commonly used in a deep learning model, are respectively optimized, so as to reduce memory usage during an inference process of the deep learning model, such that the deep learning model can operate on a microcontroller having a smaller memory. For the characteristic of multi-segment memory in a microcontroller, a memory layout optimization method for deep learning operators in multi-segment memory is proposed, so as to make deep learning operators use multi-segment memory to perform inference, thereby overcoming the disadvantage of only single-segment memory being used in the art and effectively improving memory usage efficiency.

Description

A deep learning memory optimization method for microcontrollers Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a deep learning memory optimization method for a microcontroller. Background Art

[0002] To achieve better performance, deep learning models require an increasing number of parameters and computational complexity, placing higher demands on device storage and computing resources. Transmitting data acquired on microcontrollers to servers for inference is difficult due to concerns about data privacy and network resources. Therefore, deploying and running deep learning models on microcontrollers has become a new development trend.

[0003] Microcontrollers have limited memory resources, typically ranging from a few KB to 1MB. Deploying and running lightweight deep learning on microcontrollers primarily faces this memory constraint. While existing lightweight deep learning inference frameworks for microcontrollers, such as TFLM, TinyEngine, and NNoM, have explored memory optimization during deep learning inference, they still have shortcomings.

[0004] For example, patent application CN115357381A discloses a memory optimization method and system for deep learning reasoning of embedded devices. It optimizes the memory usage of deep learning reasoning by monitoring whether the last node is input for the activation value and destroying the activation value in time, and saves the activation value data output by the qualified quantized node as int8 data type; however, this patent application only optimizes the memory used for activation values ​​in deep learning reasoning, and does not optimize the memory of other operators in deep learning reasoning, so the memory optimization effect is limited; quantizing the activation value data from int32 type to int8 type will cause precision loss, affecting the accuracy of the model.

[0005] Patent application CN112990421A discloses a method, device and storage medium for optimizing the operation process of a deep learning network. The method determines multiple operation orders corresponding to multiple computing layers included in the deep learning network, and then determines the memory overhead required for each operation order. Then, based on the memory overhead required for each operation order, the optimal operation order of the multiple computing layers is determined. This patent application reduces the memory overhead of deep learning by optimizing the calculation order of multiple layers, but does not optimize the calculation method between multiple layers, and the memory optimization is not thorough enough. The multi-segment memory situation is not taken into account during memory optimization, and the multi-segment memory of embedded devices such as microcontrollers cannot be effectively utilized.

[0006] Patent application CN114742211A discloses a convolutional neural network deployment and optimization method for microcontrollers, which mainly optimizes the memory calculation of a single convolution operator in deep learning. It only optimizes within the deep learning layer, does not optimize the layout of multiple operators in the memory, nor does it optimize the characteristics of the microcontroller's multi-segment memory. It cannot effectively utilize the multi-segment memory of embedded devices such as microcontrollers.

[0007] In summary, the existing lightweight deep learning inference framework for microcontrollers lacks more effective memory optimization for operator calculations and cannot simultaneously use multiple memory segments on the microcontroller, resulting in low memory utilization and many restrictions on deep learning models deployed on the microcontroller.

[0008] Summary of the Invention

[0009] To solve the above technical problems, the present invention provides a deep learning memory optimization method for microcontrollers, and proposes a multi-operator fusion computing optimization method for specific structures in deep learning models; based on the multi-segment memory characteristics of microcontrollers, a microcontroller memory layout optimization method is proposed to solve the problem of large memory usage during deep learning inference and meet the memory constraints of deploying deep learning models on microcontrollers.

[0010] The present invention provides a method for optimizing memory for deep learning in a microcontroller, comprising the following steps:

[0011] Step 1: On the microcontroller, a multi-operator fusion optimization method is used to fuse multiple operators of the deep learning model and optimize its memory usage;

[0012] Step 2: Use a deep learning operator memory layout optimization method for multi-segment memory to determine the optimal layout of multiple operators in the deep learning model in the microcontroller's multi-segment memory, and optimize the locations where multiple operators are stored in the memory.

[0013] Furthermore, the specific steps for optimizing the calculation of multiple operators of the deep learning model are as follows:

[0014] Step 1-1: Use the residual structure fusion calculation optimization method to fuse the calculations of Conv, Logistic, and Mul operators in the residual structure, and write the corresponding element structure of the final residual structure into the memory;

[0015] Step 1-2: Use the activation function fusion calculation optimization method to fuse the calculation of the current operator with the calculation of the activation function, and write the result of the fusion calculation into the output memory of the current operator;

[0016] Steps 1-3: Allocate the memory space required for the connection operator in advance, use the connection operator fusion calculation method, and store the feature maps calculated by the two input operators in the memory address corresponding to the connection operator output.

[0017] Furthermore, the fusion calculation optimization of the residual structure in the deep learning model is carried out. Specifically, each element in the output x of the Conv operator is first calculated. Each element obtained by calculation is not directly written into the output memory of Conv. Instead, Logistic and Mul calculations are performed on the element to obtain the final result of the element corresponding to the residual structure, and then it is written into the memory. At this time, the output of the residual structure is stored in the memory, but the entire process only uses one piece of memory, which effectively reduces the memory usage of the residual structure.

[0018] Furthermore, the convolution-activation structure in the deep learning model is optimized for fusion calculation. Specifically, when operators such as convolution are followed by an activation function, the calculation of the current operator and the activation function can be fused together. After the calculation of each output element of the current operator is completed, the activated value is calculated first, and then the activated value is written to the output memory of the current operator to complete a fusion calculation. In this way, the activation function no longer requires a separate memory space, reducing memory usage.

[0019] Furthermore, the connection structure in the deep learning model is optimized for fusion calculation. Specifically, the memory space required for the connection operator (Concat) is first allocated in advance, and the memory space is divided into two parts, which are used as the output memory space of the two input operators of Concat. Therefore, the feature maps calculated by the two input operators are directly stored in the memory address corresponding to the Concat output. There is no need to allocate memory space for the feature maps separately, thereby reducing memory usage.

[0020] Furthermore, we use deep learning operator memory for microcontroller multi-segment memory to perform layout optimization. The specific steps are as follows:

[0021] Step 2-1: Calculate the memory size and lifecycle of each operator in the deep learning model. The memory size is directly calculated based on the length, width, and height of the operator output. The lifecycle calculation requires traversing the operators in the model to find the first calculation time and the last time the operator was used.

[0022] Step 2-2: Sort all operators in descending order by memory usage. Operators with larger memory usage will be placed at the front and their memory locations will be calculated first. Operators with smaller memory usage will be placed at the back and their memory locations will be calculated later.

[0023] Step 2-3: Calculate the position of the sorted operators in memory. First, you need to maintain N linked lists, where N is the number of memory segments, which respectively represent the usage of each memory segment. These linked lists are sorted in descending order of memory access speed, that is, the first linked list represents the memory segment with the fastest memory access speed, and the Nth linked list represents the memory segment with the slowest memory access speed. Then, traverse the sorted operators and search for a suitable position in the N linked lists for each operator. If the current linked list memory space cannot meet the operator's requirements, jump to the next linked list to search. If a suitable position is found, insert the operator item into the corresponding position of the linked list.

[0024] The beneficial effects described in the present invention are as follows: the multi-operator fusion computing optimization method proposed in the present invention optimizes the residual structure, convolution-activation structure and connection structure in the deep learning model on the microcontroller respectively, reduces the memory usage in the deep learning model reasoning process, and enables the deep learning model to run on a microcontroller with smaller memory; based on the characteristics of the multi-segment memory of the microcontroller, a deep learning operator memory layout optimization method for multi-segment memory is proposed, the calculation order of each operator is determined according to the properties of the directed acyclic graph of the deep learning model, and the operator memory layout is optimized, which enables the deep learning operator to use multi-segment memory for reasoning, and determines the optimal layout of multiple operators in the deep learning model in the multi-segment memory of the microcontroller, thereby solving the shortcomings of the existing technology that only uses single-segment memory and effectively improving memory utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of the residual structure;

[0026] Figure 2 is a schematic diagram of residual structure memory usage;

[0027] Figure 3 is a schematic diagram of memory flow before and after residual structure fusion;

[0028] Figure 4 is a schematic diagram of memory usage before and after activation function fusion;

[0029] Figure 5 is a schematic diagram of memory flow before and after activation function fusion;

[0030] Figure 6 is a schematic diagram of the connection operator before and after fusion;

[0031] Figure 7 is a schematic diagram of memory usage before and after the connection operator fusion;

[0032] FIG8 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0033] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0034] As shown in FIG8 , the present invention provides a method for optimizing memory for deep learning in a microcontroller, comprising the following steps:

[0035] Step 1: On the microcontroller, a multi-operator fusion optimization method is used to fuse multiple operators of the deep learning model and optimize its memory usage;

[0036] Step 2: Use a deep learning operator memory layout optimization method for multi-segment memory to determine the optimal layout of multiple operators in the deep learning model in the microcontroller's multi-segment memory, and optimize the locations where multiple operators are stored in the memory.

[0037] The specific steps to optimize the calculation of multiple operators of the deep learning model are:

[0038] Step 1-1: Use the residual structure fusion calculation optimization method to fuse the calculations of Conv, Logistic, and Mul operators in the residual structure, and write the corresponding element structure of the final residual structure into the memory; the details are as follows:

[0039] Figure 1 is a schematic diagram of the residual structure. The output x of the Conv operator is calculated by the Logistic operator to obtain F(x). Then, through the Mul operator and shortcut connection, F(x) and x are Hadamard products, that is, the corresponding elements are multiplied. In this way, the residual structure is equivalent to H(x) = x⊙F(x), which increases the network depth while improving the network accuracy.

[0040] In the residual structure, the result x computed by the Conv operator needs to be used by the Logistic and Mul operators. Therefore, the Conv memory lifecycle must last until the Mul operator is calculated, a total of three time units. The Logistic memory lifecycle also needs to last until the Mul operator is calculated, a total of two time units. Because the Mul operator depends on the Conv and Logistic operators, and the Logistic operator in turn depends on the Conv operator, their memory cannot overlap spatially. Therefore, the memory usage of this residual structure is shown on the left side of Figure 2.

[0041] The present invention proposes a residual structure fusion calculation optimization method that combines the calculations of Conv, Logistic, and Mul operators. First, each element in the Conv operator output x is calculated. Each element obtained is not directly written into the Conv output memory. Instead, Logistic and Mul calculations are performed on the element to obtain the final result of the element corresponding to the residual structure, which is then written into the memory. At this time, the output of the residual structure is stored in the memory, but the entire process only uses one piece of memory, effectively reducing the memory usage of the residual structure. The memory usage of the fused residual structure is shown on the right side of Figure 2, and the memory usage is one-third of the original.

[0042] Figure 3 shows the memory data flow before and after residual structure fusion calculations. Before fusion, each operator reads input data from memory before calculation. The data is read from memory through the cache and then written to the CPU internal register. After the operator calculation is completed, the calculation result is written from the CPU internal register to the cache and then to memory. After the fusion, each element of the residual structure is calculated by each operator in turn. The calculation result is cached in the CPU internal register and serves as the input data for the next operator calculation. The corresponding element is not written to memory until all operators of the residual structure have been calculated.

[0043] Step 1-2: Use the activation function fusion calculation optimization method to fuse the calculation of the current operator with the calculation of the activation function, and write the fusion calculation result into the output memory of the current operator; the details are as follows:

[0044] The activation function typically calculates each element only based on the input value of that element, and has no dependencies on other adjacent elements. Therefore, if the current operator is followed by an activation function, the calculation of the current operator and the activation function can be fused together. After the calculation of each output element of the current operator is completed, the activated value is first calculated, and then the activated value is written to the output memory of the current operator, completing a fused calculation. This eliminates the need for separate memory space for the activation function, reducing memory usage, as shown in Figure 4.

[0045] In Figure 4, taking the Conv operator and ReLU activation function as an example, the left side is a schematic diagram of memory usage during the calculation process of the unfused convolution operator and activation function. The calculation result of the Conv operator is stored in the Conv memory space, and the calculation result of the activation function is stored in the ReLU memory space. The size is the same as Conv, and a total of 2 times the Conv memory space is required; the right side is a schematic diagram of memory usage after the activation function proposed in the present invention is fused. The convolution operator first calculates the activation value after calculating each element, and then writes it to the memory space Conv_ReLU, requiring only a memory space the size of Conv.

[0046] The memory data flow before and after the activation function fusion calculation is shown in Figure 5. Similar to the residual structure fusion, the fused activation function directly calculates the ReLU after calculating each element of the Conv operator, and the calculation result is finally written to the memory.

[0047] Steps 1-3: Allocate the memory space required for the join operator in advance, use the join operator fusion calculation method, and store the feature maps calculated by the two input operators in the memory address corresponding to the join operator output; the details are as follows:

[0048] The left side of Figure 6 is a schematic diagram before the connection operator (Concat) is fused, and the right side of Figure 6 is a schematic diagram after the connection operator (Concat) is fused. As can be seen from Figure 6, Concat merges the feature maps output by Conv_2 and Conv_3 into one feature map output. Before the connection operator is fused, the output feature map of Conv_1 is first calculated, and then Conv_2 and Conv_3 use the output of Conv_1 as the input feature map to calculate the output feature maps of Conv_2 and Conv_3 in sequence, and finally Concat merges the feature maps of Conv_2 and Conv_3 together. The present invention proposes a connection operator fusion calculation method, which previously allocates the memory space required for Concat in advance and divides the memory space into two parts, which are used as Conv_2 and Conv_3 output memory spaces respectively. Therefore, the feature maps calculated by Conv_2 and Conv_3 are directly stored in the memory address corresponding to the Concat output, and there is no need to separately allocate the memory space of the Conv_2 and Conv_3 feature maps, thereby reducing memory usage.

[0049] The left side of Figure 7 shows the memory usage before the connection operator fusion, and the right side of Figure 7 shows the memory usage after the connection operator fusion. It can be seen that before the connection operator fusion, Conv_2 and Conv_3 each need to allocate a piece of memory space to store their respective output feature maps, while after the connection operator fusion, Conv_2 and Conv_3 directly store the output feature maps in the corresponding position of the Concat output memory space, effectively reducing the memory usage of the fusion operator.

[0050] The specific steps for optimizing the memory locations of multiple operators using the deep learning operator memory layout optimization method for multi-segment memory are as follows:

[0051] Step 2-1: Calculate the memory size and lifecycle required by each operator in the deep learning model. The memory size can be directly calculated based on the length, width, and height parameters output by the operator. The calculation of the lifecycle requires traversing the operators in the model to find the first calculation time and the last time the operator was used.

[0052] Step 2-2: Sort all operators in descending order according to memory usage. Operators with large memory usage are placed in front and calculated first, while operators with small memory usage are placed in the back and calculated last. This is because calculating small blocks of memory first is more likely to cause memory fragmentation, while calculating large blocks of memory first can reduce memory fragmentation. Even if memory fragmentation occurs, small blocks of memory calculated later are more likely to fill these fragments, thereby improving memory usage efficiency.

[0053] Step 2-3: Calculate the memory location of the sorted operators. First, maintain N linked lists, where N is the number of memory segments, representing the usage of each segment. These linked lists are sorted in descending order of memory access speed. That is, the first linked list represents the fastest memory segment, and the Nth linked list represents the slowest. Next, traverse the sorted operators, searching each of the N linked lists for a suitable location. If the current linked list does not have enough memory for the operator, jump to the next linked list. If a suitable location is found, the operator entry is inserted into the corresponding linked list. This method prioritizes placing operators in memory segments with fast access speeds, improving memory usage efficiency while also speeding up the operator's computation.

[0054] The method of the present invention is described by the following examples:

[0055] The STM32H730VBT6 microcontroller contains two segments of memory, including 128KB of DTCM memory and 436KB of SRAM memory, totaling 564KB. Existing methods only use single-segment memory such as DTCM or SRAM, which cannot fully utilize the 564KB memory size, resulting in low memory utilization. The present invention proposes a multi-operator fusion optimization method and a deep learning operator memory layout optimization method for multi-segment memory, which effectively utilizes the characteristics of the microcontroller's multi-segment memory while reducing the memory usage of the deep learning model and improving memory utilization. The steps are as follows:

[0056] 1. Use TensorFlow to train the SqueezeNet deep learning model with an input size of 128×128×3, Adam as the optimizer, a learning rate of 0.00005, a cross-entropy loss function, and a ReLU6 activation function;

[0057] 2. Use multi-operator fusion optimization method on STM32H730VBT6 microcontroller to fuse convolution-activation structure and connection structure to reduce the memory usage of SqueezeNet deep learning model;

[0058] 3. Use the deep learning operator memory layout optimization method for multi-segment memory on the STM32H730VBT6 microcontroller to determine the optimal layout of multiple operators in the deep learning model in the microcontroller's multi-segment memory and optimize the location where multiple operators are stored in the memory;

[0059] 4. Use the STM32H730VBT6 microcontroller to collect image data from the OV7725 camera, preprocess the image data, and scale the resolution to the same size as the training dataset; then perform model inference calculations and post-process the output of the inference results to obtain the final prediction results.

[0060] Table 1 Comparison of optimization effects of the present invention on microcontrollers

[0061] Table 1 is a comparative test of the present invention with TFLM (TensorFlow Lite Micro) and X-CUBE-AI solutions. The present invention first uses a multi-operator fusion optimization method to reduce the total memory usage of the deep learning model, and then uses a deep learning operator memory layout optimization method for multi-segment memory to simultaneously utilize the DTCM and SRAM memory segments on the microcontroller (since DTCM memory is faster, DTCM memory is used first). Compared with the comparison solution, although its total memory usage is less than the total memory of the STM32H730VBT6 microcontroller, since it does not support multi-segment memory layout, only one segment of memory can be used and cannot be deployed. The present invention has a smaller total memory usage and supports multi-segment memory layout, so more models can be deployed on the microcontroller.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made using the contents of the present invention description and drawings are within the scope of protection of the present invention.

Claims

1. A deep learning memory optimization method for microcontrollers, characterized in that: The following steps are involved: Step 1: On the microcontroller, a multi-operator fusion optimization method is used to fuse multiple operators of the deep learning model and optimize its memory usage; Step 2: Use a deep learning operator memory layout optimization method for multi-segment memory to determine the optimal layout of multiple operators in the deep learning model in the multi-segment memory of the microcontroller, and optimize the locations where multiple operators are stored in the memory.

2. A deep learning memory optimization method for microcontrollers according to claim 1, characterized in that: The specific steps to optimize the calculation of multiple operators of the deep learning model are: Step 1-1, adopt the residual structure fusion calculation optimization method to fuse the calculations of Conv, Logistic, and Mul operators in the residual structure, and write the corresponding element structure of the final residual structure into the memory; Step 1-2: Use the activation function fusion calculation optimization method to fuse the calculation of the current operator with the calculation of the activation function, and write the result of the fusion calculation into the output memory of the current operator; Step 1-3: Allocate the memory space required for the connection operator in advance, use the connection operator fusion calculation method, and store the feature maps calculated by the two input operators in the memory address corresponding to the connection operator output.

3. A deep learning memory optimization method for microcontrollers according to claim 2, characterized in that: The residual structure in the deep learning model is optimized by fusion calculation. Specifically, each element in the Conv operator output x is calculated first, and Logistic and Mul calculations are performed on the element to obtain the final result of the element corresponding to the residual structure, and then the result is written into the memory. At this time, the output of the residual structure is stored in the memory.

4. The deep learning memory optimization method for microcontroller according to claim 2, characterized in that: The convolution-activation structure in the deep learning model is optimized by fusion calculation. Specifically, when the convolution operator is followed by an activation function, the calculation of the current operator and the calculation of the activation function are fused together. After the calculation of each output element of the current operator is completed, the activated value is calculated first, and then the activated value is written to the output memory of the current operator to complete a fusion calculation.

5. The deep learning memory optimization method for microcontroller according to claim 2, characterized in that: The connection structure in the deep learning model is optimized for fusion calculation. Specifically, the memory space required for the connection operator Concat is allocated in advance, and the memory space is divided into two parts, which are used as the output memory space of the two input operators of Concat respectively. The feature maps calculated by the two input operators are directly stored in the memory address corresponding to the Concat output.

6. A deep learning memory optimization method for microcontrollers according to claim 2, characterized in that: Use deep learning operator memory for multi-segment memory for layout optimization. The specific steps are as follows: Step 2-1: Calculate the memory size and life cycle required by each operator in the deep learning model. The memory size is directly calculated based on the length, width, and height of the operator output. The calculation of the life cycle requires traversing the operators in the model to find the first calculation time and the last use time of the operator. Step 2-2: Sort all operators in descending order according to the memory usage. Operators with large memory usage will be placed in front and their positions in memory will be calculated first; operators with small memory usage will be placed in the back and their positions in memory will be calculated later. Step 2-3: Calculate the position of the sorted operators in the memory. First, you need to maintain N linked lists, where N is the number of memory segments, which respectively represent the usage of each memory segment. These linked lists are arranged in descending order according to the memory access speed, that is, the first linked list represents the memory segment with the fastest memory access speed, and the Nth linked list represents the memory segment with the slowest memory access speed. Then, traverse the sorted operators and search for a suitable position in the N linked lists for each operator. If the current linked list memory space cannot meet the operator's requirements, jump to the next linked list to search. If a suitable position is found, insert the operator item into the corresponding position of the linked list.

Images