A microcontroller-oriented in-circular memory management method
By employing a circular memory management method, the problem of insufficient microcontroller memory was solved, achieving significant memory savings without altering model accuracy and computational correctness, and expanding the deployment capability of deep learning models on microcontrollers.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PEKING UNIV
- Filing Date
- 2024-01-08
- Publication Date
- 2026-07-21
Smart Images
Figure CN117931688B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to deploying deep learning algorithms on microcontrollers, and more specifically to a method for deploying deep learning algorithms on microcontrollers by utilizing a circular memory management method to save memory overhead. Background Technology
[0002] Deep learning algorithms have achieved levels surpassing human capabilities in image and speech recognition. With the widespread adoption of intelligent algorithms, more and more terminal devices are equipped with low-power computing devices, such as microcontroller units (MCUs). Microcontrollers consume extremely low power, allowing for long-term operation; applications such as mobile phone wake-up, watch gaze detection, and smart home monitoring can all be deployed using microcontrollers. However, microcontrollers have very limited memory, often only a few hundred KB, making it extremely difficult to deploy neural network models from deep learning algorithms on them. This is because neural network computations require large-scale tensor calculations, and even small models designed for terminal devices often have memory requirements exceeding the microcontroller's memory. For example, the MCUNet-320KB-ImageNet model proposed by MIT requires 320KB of memory for deployment, but many microcontroller devices have less memory; for instance, the STM32-F411RE only has 128KB of memory, making it impossible to deploy this model. To solve this problem, the only common approach is often to retrain the model to further compress the number of model parameters. However, this leads to a decrease in the model's capabilities and accuracy, and retraining is not feasible in all scenarios. In most deployment scenarios, there are no conditions for retraining (because training requires a large amount of computing resources and time).
[0003] When deploying a neural network model, the main memory overhead consists of two parts: the model parameters and the space occupied by intermediate results during runtime. Intermediate results often account for a large proportion of the overall overhead. Therefore, reducing memory overhead can start by adjusting the size of intermediate results without changing the number of parameters. This eliminates the need for retraining or parameter modifications, further reducing overall memory consumption.
[0004] In 2018, CMSIS-NN (CMSIS-NN: EfficientNeural Network Kernels for Arm Cortex-M CPUs), a neural network computing library developed and open-sourced by ARM, supported a variety of operators in a low-bit manner and reduced memory overhead. It supports important operators such as matrix multiplication and convolution, uses 8-bit data precision, and can reduce memory usage by 75% compared to single-precision floating-point numbers.
[0005] In 2020, Han Song's team at MIT proposed TinyEngine (MCUNet: Tiny Deep Learning on IoTDevices), which saves memory by having the input and output of depthwise convolutions share the same memory address. This method perfectly saves the memory overhead of network operators, but it only applies to depthwise convolutions. The memory overhead of other operators in the network, such as matrix multiplication and convolution, is not saved, becoming the memory bottleneck of the overall network. Summary of the Invention
[0006] For convenience, MCU is used to refer to a microcontroller, and DNN is used to refer to a neural network model in deep learning.
[0007] The purpose of this invention is to provide a recurrent memory management method for microcontrollers, which enables address space reuse of input and output tensor data for matrix multiplication and convolution operators in DNNs, saving DNN memory overhead without changing computational correctness and DNN accuracy. The target deployment hardware is an MCU.
[0008] The technical solution provided by this invention is as follows:
[0009] A circular memory management method for microcontrollers includes the following steps:
[0010] A. DNN computation preparation stage:
[0011] A.1DNN operator classification;
[0012] A.2 Constructing a virtual circular memory, which is to abstract the actual physical memory into a virtual circular memory in units of data elements;
[0013] A.3 Initialize the pointer positions for input and output data in the loop memory;
[0014] B. DNN computation stage:
[0015] B.1 First, determine whether the calculation is complete. If no calculation is required, end the process directly; otherwise, proceed to B.2.
[0016] B.2 Based on the current input pointer, read the input data, load it from memory into the register, and use vectorized read instructions to speed it up;
[0017] B.3 Perform calculations on the read data, using the MCU's vectorized calculation instructions for acceleration;
[0018] B.4 Determine whether the current intermediate calculation result should be stored out of the register. For matrix multiplication and convolution, a complete accumulation operation needs to be completed along the reduction dimension to obtain the complete calculation result. In this case, it will be stored out and proceed to B.5. Otherwise, proceed to B.7.
[0019] B.5 Store the intermediate results from the register based on the current output data pointer;
[0020] B.6 Updates the output data pointer by moving the pointer forward, with the movement amount equal to the number of data elements stored in B.5;
[0021] B.7 Update the current input data pointer by moving the pointer forward, with the movement amount equal to the number of data elements read in B.2, and then return to step B.1.
[0022] The beneficial effects of this invention are:
[0023] This invention abstracts physical memory into virtual circular memory based on data elements. By setting the starting address, computation order, and pointer movement details of the DNN operator input and output tensors in the virtual memory, it ensures that input and output share the same physical memory while maintaining computational correctness. Using the solution provided by this invention, memory requirements can be further reduced without affecting the accuracy of DNN computation, allowing more DNN models to be deployed on MCUs with limited memory space. Attached Figure Description
[0024] Figure 1 This is a flowchart of the circular memory management method of the present invention;
[0025] Figure 2 This is a flowchart of the DNN calculation part in this invention;
[0026] Figure 3 This is a schematic diagram illustrating the application of the present invention to matrix multiplication. Detailed Implementation
[0027] The specific implementation of this invention is as follows:
[0028] A. DNN computation preparation stage, corresponding process Figure 1 The part before the DNN computation:
[0029] A.1 DNN Operator Classification. Subsequent calculations are performed based on the DNN operator category. In this invention, matrix multiplication and convolution are the two main types of DNN operators considered. The DNN computation implementations and memory initializations for matrix multiplication and convolution operators differ.
[0030] A.2 Constructing Virtual Circular Memory. This invention requires abstracting actual physical memory into virtual circular memory, with data elements as the unit. This is achieved using a circular array, where the address is calculated as addr = vaddr % length. Here, addr is the actual physical memory address, vaddr is the abstracted virtual address, length is the length of the circular array, and % is the modulo operation.
[0031] A.3 Initialize the pointer positions for input and output data in the circular memory. The output pointer is denoted as `vaddr_out`, and the input pointer as `vaddr_in`. `vaddr_out` must be located before `vaddr_in`, meaning `vaddr_out <= vaddr_in`. The distance between these two positions needs to be precisely calculated to prevent data overlap errors during DNN computation. Furthermore, the distance between them should be less than or equal to the size of the output data to achieve memory conservation. During computation, the output results will gradually replace the positions of previously occupied input data. The replaced input data must be ensured to be no longer used. After calculating the distance `vaddr_in - vaddr_out`, setting `vaddr_out` to address 0 allows us to deduce the position of `vaddr_in`.
[0032] B. DNN computation section, corresponding process Figure 2 The process is the same for matrix multiplication and convolution operators. A convolution operator can be converted into a matrix multiplication operator.
[0033] B.1 First, determine if the calculation is complete. If no calculation is needed, end the process directly; otherwise, proceed to B.2.
[0034] B.2 Based on the current input pointer, read in the input data, load the data from memory into the register, and use vectorized read instructions to speed up the process.
[0035] B.3 Performs calculations on the read-in data (vector), using the MCU's vectorized calculation instructions for acceleration.
[0036] B.4 Determine whether the current intermediate calculation result should be stored out of the register. For matrix multiplication and convolution, a complete accumulation operation needs to be completed along the reduction dimension to obtain the complete calculation result. In this case, it will be stored out and proceed to B.5. Otherwise, proceed to B.7.
[0037] B.5 Store the intermediate results from the register based on the current output data pointer.
[0038] B.6 updates the output data pointer by moving the pointer forward, with the movement amount equal to the number of data elements stored in B.5.
[0039] B.7 Update the current input data pointer by moving it forward, with the movement amount equal to the number of data elements read in B.2. Then return to step B.1.
[0040] The initial distance between the input and output addresses, vaddr_in - vaddr_out, is calculated.
[0041] 1) For the matrix multiplication operator, the input shape is M, N, K, where M is the number of rows in the left matrix, K is the number of columns in the left and right matrices, N is the number of rows in the right matrix, and vaddr_in-vaddr_out=min(N,K)-1.
[0042] 2) For the convolution operator, the window size is RxS, the input shape is B, C, H, W, K, where B is the batch size, C is the number of input image channels, H and W are the number of rows and columns of the input image, and K is the number of output image channels. In this case, vaddr_in – vaddr_out = min(C, K) - 1.
[0043] For the implementation of the matrix multiplication operator, the formula for matrix multiplication is C[m,n] + = A[m,k] * B[n,k]. The pseudocode for implementing matrix multiplication in an MCU using this invention is as follows:
[0044] for(int m = 0; m <M;++m){
[0045] int8_t outVector[N] = {0};
[0046] int8_t inVector[K] = {0};
[0047] int8_t weightVector[K]={0};
[0048] VectorLoad(inVector,PtrInput);
[0049] for(int n = 0; n <N;++n){
[0050] VectorLoad(weightVector,PtrWeight);
[0051] outVector[n]=VectorDot(inVector,weightVector);
[0052] }
[0053] VectorStore(outVector,PtrOutput);
[0054] PtrOutput += N;
[0055] PtrInput += K;
[0056] PtrWeight + = K;
[0057]
[0058] VectorLoad loads vector data using the MCU instruction set, loading it from memory to a register. VectorStore stores vector data using MCU instructions, storing it from a register to memory. PtrInput is a pointer to the input data, PtrOutput is a pointer to the output data, and PtrWeight is a pointer to the weights. PtrWeight is initialized in the MCU's flash memory, so it does not affect memory usage. VectorDot performs vector dot product calculations.
[0059] During initialization, PtrInput is set at address min(N,K)-1 in the loop memory, and PtrOutput is set at address 0 in the loop memory. According to the pseudocode above, the movement range of the two pointers is max(MK,MN)+min(N,K)-1, which is always less than or equal to MK+MN. Therefore, it can save memory overhead. When M is much larger than K and N, the saving ratio approaches 50%, saving nearly half of the memory overhead.
[0060] like Figure 3 The matrix multiplication example in the example illustrates this: M=2, N=2, K=3. The weight parameters are stored in Flash memory and do not affect memory usage; only the input and output data reside in memory. Without the circular memory management technique of this invention, 10 units of storage space would need to be allocated. Using the method of this invention, only max(MK,MN) + min(N,K) - 1 = 7 units of memory need to be allocated. The address of the output data is at 0, and the address of the input data is at min(N,K) - 1 = 1. During calculation, as the calculation steps proceed, the output data gradually covers the position of the input data. However, due to the reasonable calculation order and pointer update order shown in the pseudocode, the correctness of the calculation remains unchanged; only memory usage is reduced.
[0061] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Claims
1. A circular memory management method for microcontrollers, comprising the following steps: A. DNN computation preparation stage: A.1DNN operator classification; A.2 Constructing a virtual circular memory, which is to abstract the actual physical memory into a virtual circular memory in units of data elements; A.3 Initialize the pointer positions for input and output data in the loop memory; B. DNN computation stage: B.1 First, determine whether the calculation is complete. If no calculation is required, end the process directly; otherwise, proceed to B.
2. B.2 Based on the current input data pointer, read the input data from memory into the register, using vectorized read instructions for acceleration; B.3 Perform calculations on the read data, using the MCU's vectorized calculation instructions for acceleration; B.4 Determine whether the current intermediate calculation result should be stored out of the register. If it should be stored out, proceed to B.5; otherwise, proceed to B.
7. B.5 Store the intermediate results from the register based on the current output data pointer; B.6 Updates the output data pointer by moving the pointer forward, with the movement amount equal to the number of data elements stored in B.5; B.7 Update the current input data pointer by moving the pointer forward, with the amount of movement equal to the number of data elements read in B.2, and then return to step B.
1.
2. The circular memory management method for microcontrollers as described in claim 1, characterized in that, Step A.1 is divided into two types of DNN operators: matrix multiplication and convolution.
3. The circular memory management method for microcontrollers as described in claim 1, characterized in that, Step A.2 is implemented using a circular array. The address calculation method in the circular array is addr = vaddr % length, where addr is the actual physical memory address, vaddr is the abstracted virtual address, length is the length of the circular array, and % is the modulo operation.
4. The circular memory management method for microcontrollers as described in claim 1, characterized in that, In step A.3, the output data pointer is denoted as vaddr_out, the input data pointer is denoted as vaddr_in, vaddr_out <= vaddr_in, and after calculating the distance vaddr_in - vaddr_out, vaddr_out is always set to address 0.
5. The circular memory management method for microcontrollers as described in claim 4, characterized in that, For the matrix multiplication operator, the method to calculate the distance vaddr_in-vaddr_out is as follows: the input shape is M, N, K, where M is the number of rows in the left matrix, K is the number of columns in the left and right matrices, N is the number of rows in the right matrix, and vaddr_in-vaddr_out = min(N,K)-1.
6. The circular memory management method for microcontrollers as described in claim 4, characterized in that, For the convolution operator, the method to calculate the distance vaddr_in-vaddr_out is as follows: the convolution window size is RxS, the input shape is B,C,H,W,K, where B is the batch size, C is the number of input image channels, H and W are the number of rows and columns of the input image, and K is the number of output image channels. vaddr_in–vaddr_out=min(C,K)-1.