Memory optimization method and system for embedded device deep learning inference

Abstract

The application discloses a memory optimization method and system for embedded device deep learning inference, belongs to the technical field of memory optimization, deep learning, inference and embedding, and aims to solve the technical problems of high memory occupation of activation values in the model inference stage, inflexible and inefficient memory processing and memory conflicts, and adopts the technical scheme that the method specifically comprises the following steps: analyzing a trained deep learning model: obtaining a network structure and model weights, and organizing the network structure and model weights into a directed acyclic graph data format; traversing the directed acyclic graph in a breadth-first manner; an activation value destruction strategy: adopting a mode of monitoring whether the activation value is input into the last node, and timely destroying the activation value; and saving activation value data of a quantization node meeting a condition as int8 data type.

Description

Technical Field

[0001] This invention relates to the fields of memory optimization, deep learning, inference, and embedded technology, specifically a memory optimization method and system for deep learning inference in embedded devices. Background Technology

[0002] In recent years, deep learning neural network models have been widely applied in many fields and have achieved excellent results. Model inference, as the stage where the model is used, is closely related to the hardware and environment in which the model runs, leading to the emergence of model inference frameworks adapted to different hardware. Furthermore, embedded devices have low power consumption and limited computing power and memory resources, placing high demands on the deployment of deep learning models. The most prominent problem in the model inference stage is the large memory consumption, especially for activation values, which conflicts with the limited memory resources of embedded devices. Currently, some inference frameworks support embedded devices, but because they are not specifically designed for embedded devices, their memory handling is not flexible and efficient enough, and the memory conflict problem remains serious. Summary of the Invention

[0003] The technical objective of this invention is to provide a memory optimization method and system for deep learning inference in embedded devices, in order to solve the problems of high memory consumption of activation values, insufficient flexibility and efficiency in memory processing, and memory conflicts during the model inference stage.

[0004] The technical objective of this invention is achieved as follows: a memory optimization method for deep learning inference in embedded devices, the method being as follows:

[0005] Analyze the trained deep learning model: obtain the network structure and model weights, and organize them into a directed acyclic graph data format;

[0006] Traverse the directed acyclic graph using a breadth-first approach;

[0007] Activation value destruction strategy: The strategy involves monitoring whether the activation value was the last node input and destroying the activation value in a timely manner.

[0008] Save the activation value data of the quantization node that meets the conditions as an int8 data type.

[0009] As a preferred approach, the pre-trained deep learning model is analyzed as follows:

[0010] Construct a mapping between nodes and node inputs (node2input), a mapping between nodes and node outputs (node2output), and an activation label set consisting of the input and output labels of all nodes.

[0011] Deduplication of activation value tag set: Remove duplicate activation value tags from the activation value tag set.

[0012] As a preferred approach, the directed acyclic graph is traversed using a breadth-first method, as follows:

[0013] Determine whether the node sequence of the computation graph of the trained deep learning model is sorted in width-first order:

[0014] If not, then the node sequence of the trained deep learning model computation graph will be converted into a width-first sorted order.

[0015] During inference, an activation map (activation_map) is constructed to store activation values. The keys of this map are all labels in the activation value label set, and the values ​​are the activation values ​​corresponding to those labels. Initially, the activation value corresponding to the first node label is the model input, and the activation values ​​corresponding to other labels are empty.

[0016] Traverse the directed acyclic graph in a breadth-first manner: obtain the input label corresponding to the node through input2node, and use the label to find the corresponding activation value in activation_map as the node input;

[0017] Execute the node to obtain its output. Get the output label of the node through node2output and use the label to store the node output in the corresponding position of activation_map.

[0018] More preferably, the activation value destruction strategy is as follows:

[0019] When reasoning through each node of the directed acyclic graph, determine whether that node is the last node of the input in the directed acyclic graph:

[0020] If it is the last node, after the execution of the node, the activation value corresponding to the node label in the activation_map is destroyed and the memory space occupied by it is released; where the last node of the input corresponding to the directed acyclic graph refers to the last node traversed in the input corresponding node.

[0021] Determine whether the input activation value corresponding to this node will generate an intermediate result during the calculation process:

[0022] If so, the input activation value corresponding to that node should be destroyed immediately after the intermediate result is generated.

[0023] Even better, when the input activation values ​​corresponding to a node are used for two-dimensional convolution calculation, the General Matrix Multiplication (GEMM) method is used to accelerate the calculation. The input activation values ​​corresponding to the node are expanded and transformed into a two-dimensional matrix. The input activation values ​​corresponding to the node are destroyed immediately after the transformation.

[0024] More preferably, when the trained deep learning model is a convolutional neural network model, when executing a node that requires quantization (i.e., before the next quantization), the corresponding output activation value of that node is calculated using the int8 data type, then the corresponding output activation value of that node is saved as the int8 data type, as follows:

[0025] In convolutional neural network models, a combination of 2D convolution and linear rectified function (RELU) is used (batch normalization is fused into 2D convolution). The output of the 2D convolution is of data type int8. The next RELU node is executed, and then the subsequent 2D convolution is executed. When the 2D convolution node is computed, int8 weights and int8 input are used to compute the 2D convolution, and then inverse quantization is performed. The quantization of the next 2D convolution node is also executed, resulting in an output of data type int8. That is, the quantization process of the next quantization node is performed at this node. Compared with the output of data type int32, the output of data type int8 greatly reduces the memory usage.

[0026] A memory optimization system for deep learning inference in embedded devices, the system comprising,

[0027] The parsing unit is used to parse the trained deep learning model, obtain the network structure and model weights, and organize them into a directed acyclic graph data format.

[0028] The execution unit is used to traverse the directed acyclic graph in a breadth-first manner.

[0029] The destruction unit is used to destroy the activation value in a timely manner by monitoring whether the activation value is the last node input.

[0030] The storage unit is used to save the activation value data of the quantization node that meets the conditions as an int8 data type.

[0031] Preferably, the parsing unit includes,

[0032] The building module is used to construct the mapping between nodes and node inputs (node2input), the mapping between nodes and node outputs (node2output), and the activation label set (activation_label_set) consisting of the input and output labels of all nodes.

[0033] The deduplication module is used to remove duplicate activation value tags from the set of activation value tags;

[0034] The traversal unit includes,

[0035] Module 1 is used to determine whether a node is the last node of the input in the directed acyclic graph when reasoning and traversing each node.

[0036] The destruction and memory release module is used to destroy the input activation value corresponding to the last node of the input in the directed acyclic graph and release the memory space it occupies; where the last node of the input in the directed acyclic graph refers to the last node traversed among the input nodes.

[0037] Module 2 is used to determine whether the input activation value corresponding to the node will generate an intermediate result during the calculation process.

[0038] An electronic device includes: a memory and at least one processor;

[0039] The memory contains computer programs;

[0040] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the memory optimization method for deep learning inference in embedded devices as described above.

[0041] The memory optimization method and system for deep learning inference in embedded devices of the present invention have the following advantages:

[0042] (i) This invention greatly reduces model memory usage by computing graph width traversal, flexible activation value destruction strategy and saving the activation value data of qualified quantized nodes as int8 data type, combined with the quantization method commonly used in deep learning inference of embedded devices, and solves the problems of high activation value memory usage, insufficient flexibility and efficiency of memory processing and memory contradiction.

[0043] (ii) Compared with the output of int32 data type, the present invention saves the activation value data of the quantization node that meets the conditions as int8 data type, and the output of int8 data type greatly reduces memory usage. Attached Figure Description

[0044] The invention will be further described below with reference to the accompanying drawings.

[0045] Appendix Figure 1 This is a flowchart of a memory optimization method for deep learning inference in embedded devices. Detailed Implementation

[0046] The memory optimization method and system for deep learning inference in embedded devices of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1:

[0048] As attached Figure 1 As shown in the figure, this embodiment provides a memory optimization method for deep learning inference in embedded devices, which is as follows:

[0049] S1. Analyze the trained deep learning model: obtain the network structure and model weights, and organize them into a directed acyclic graph data format;

[0050] S2. Traverse the directed acyclic graph using a breadth-first approach;

[0051] S3. Activation value destruction strategy: The strategy is to promptly destroy the activation value by monitoring whether it is the last node input for that activation value.

[0052] S4. Save the activation value data of the quantization node that meets the conditions as an int8 data type.

[0053] The specific steps for analyzing the trained deep learning model in step S1 of this embodiment are as follows:

[0054] S101. Construct the mapping between nodes and node inputs (node2input), the mapping between nodes and node outputs (node2output), and the activation label set (activation_label_set) consisting of the input and output labels of all nodes.

[0055] S102, Deduplication of Activation Value Tag Set: Deduplicate duplicate activation value tags in the activation value tag set.

[0056] In this embodiment, step S2, which uses a breadth-first approach to traverse the directed acyclic graph, is as follows:

[0057] S201. Determine whether the node sequence of the trained deep learning model computation graph is sorted in a width-first manner:

[0058] ① If not, proceed to step S202;

[0059] ② If so, proceed to step S203;

[0060] S202. Convert the node sequence of the trained deep learning model computation graph into a width-first sorted order, and then proceed to step S203.

[0061] S203. During inference, an activation map (activation_map) is constructed to store activation values. The keys of this map are all labels in the activation value label set, and the values ​​are the activation values ​​corresponding to those labels. Initially, the activation value corresponding to the first node label is the model input, and the activation values ​​corresponding to other labels are empty.

[0062] S204. Traverse the directed acyclic graph in a breadth-first manner: obtain the input label corresponding to the node through input2node, and use the label to find the corresponding activation value in activation_map as the node input;

[0063] S205. Execute the node to obtain its output. Obtain the node's output label through node2output and use the label to store the node's output in the corresponding location of the activation_map.

[0064] The activation value destruction strategy in step S3 of this embodiment is as follows:

[0065] S301. When reasoning through each node of a directed acyclic graph, determine whether that node is the last node of the input in the directed acyclic graph:

[0066] If it is the last node, then proceed to step S302;

[0067] S302. After the execution of this node, destroy the activation value corresponding to the node label in the activation_map and release the memory space it occupies; where the last node of the input in the directed acyclic graph refers to the last node traversed in the input corresponding nodes;

[0068] S303. Determine whether the input activation value corresponding to this node will generate an intermediate result during the calculation process:

[0069] If so, proceed to step 304;

[0070] S304. After generating intermediate results, immediately destroy the input activation value corresponding to the node.

[0071] In this embodiment, when the input activation value corresponding to a node is used for two-dimensional convolution calculation, the general matrix multiplication (GEMM) method is used to accelerate the calculation. The input activation value corresponding to the node is expanded and converted into a two-dimensional matrix. The input activation value corresponding to the node is destroyed immediately after the conversion.

[0072] In this embodiment, when the trained deep learning model is a convolutional neural network model, when executing a node that requires quantization (i.e., before the next quantization), the corresponding output activation value of that node is calculated using the int8 data type, the corresponding output activation value of that node is saved as the int8 data type, as follows:

[0073] In convolutional neural network models, a combination of 2D convolution and Rectified Linear Logic (RELU) is used (batch normalization is fused into the 2D convolution). The output of the 2D convolution is of data type int8. The next RELU node is executed, and then the subsequent 2D convolution is executed. When the 2D convolution node is computed, int8 weights and int8 input are used to compute the 2D convolution, and then inverse quantization is performed. The quantization of the next 2D convolution node is also executed, resulting in an output of data type int8. That is, the quantization process of the next quantization node is performed at this node. Compared with the output of data type int32, the output of data type int8 greatly reduces the memory usage.

[0074] Example 2:

[0075] This embodiment provides a memory optimization system for deep learning inference in embedded devices, the system comprising:

[0076] The parsing unit is used to parse the trained deep learning model, obtain the network structure and model weights, and organize them into a data format of acyclic graph;

[0077] The execution unit is used to traverse the directed acyclic graph in a breadth-first manner.

[0078] The destruction unit is used to destroy the activation value in a timely manner by monitoring whether the activation value is the last node input.

[0079] The storage unit is used to save the activation value data of the quantization node that meets the conditions as an int8 data type.

[0080] The parsing unit in this embodiment includes,

[0081] The building module is used to construct the mapping between nodes and node inputs (node2input), the mapping between nodes and node outputs (node2output), and the activation label set (activation_label_set) consisting of the input and output labels of all nodes.

[0082] The deduplication module is used to remove duplicate activation value tags from the set of activation value tags;

[0083] The traversal unit in this embodiment includes,

[0084] Module 1 is used to determine whether a node is the last node of the input in the directed acyclic graph when reasoning and traversing each node.

[0085] The destruction and memory release module is used to destroy the input activation value corresponding to the last node of the input in the directed acyclic graph and release the memory space it occupies; where the last node of the input in the directed acyclic graph refers to the last node traversed among the input nodes.

[0086] Module 2 is used to determine whether the input activation value corresponding to the node will generate an intermediate result during the calculation process.

[0087] Example 3:

[0088] This embodiment also provides an electronic device, including: a memory and a processor;

[0089] The memory stores the instructions executed by the computer.

[0090] The processor executes computer execution instructions stored in the memory, causing the processor to perform a memory optimization method for deep learning inference in an embedded device according to any embodiment of the present invention.

[0091] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0092] Memory is used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in memory, and by accessing data stored in memory. Memory may primarily consist of a program storage area, where model file data is directly written into the program.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A memory optimization method for embedded device deep learning inference, characterized in that, The method specifically comprises the following steps: Analyzing the trained deep learning model: obtaining the network structure and model weight, and organizing them into a data format of a directed acyclic graph; Using a breadth-first method to traverse the directed acyclic graph; specifically: judging whether the node sequence of the trained deep learning model calculation graph is sorted in a breadth-first manner: if not, converting the node sequence of the trained deep learning model calculation graph into a sequence arranged in a breadth-first sorting manner; during reasoning, an activation value mapping activation_map is constructed to store activation values, the keys of the mapping are all labels in the activation value label set, and the values of the mapping are the activation values corresponding to the labels; initially, the activation value corresponding to the first node label is the model input, and the activation values corresponding to other labels are empty; the directed acyclic graph is traversed in a breadth-first manner: the input label corresponding to a node is obtained through input2node, the corresponding activation value is found in activation_map through the label to obtain the node input; the node is executed to obtain the output of the node, the node output label is obtained through node2output, and the node output is stored in the corresponding position of activation_map using the label; An activation value destruction strategy; specifically: during the reasoning traversal of each node of the directed acyclic graph, judging whether the node is the last node of the input corresponding to the directed acyclic graph: if it is the last node, destroying the activation value corresponding to the node label in activation_map after the execution of the node, and releasing the memory space occupied by the activation value; wherein the last node of the input corresponding to the directed acyclic graph refers to the last traversed node in the input corresponding nodes; judging whether the input activation value corresponding to the node will generate an intermediate result during calculation: if so, the input activation value corresponding to the node should be destroyed immediately after the generation of the intermediate result; Saving the output activation value data of the quantized node meeting the condition as int8 data type; When the trained deep learning model is a convolutional neural network model, the output activation value corresponding to the node is saved as int8 data type when the node is executed, specifically as follows: In the convolutional neural network model, a combination of two-dimensional convolution + linear rectifier function is included; the output of the two-dimensional convolution is int8 data type, the next RELU node is executed, and then the subsequent two-dimensional convolution is executed; during the calculation of the two-dimensional convolution node, int8 weight and int8 input are used to calculate the two-dimensional convolution, and then inverse quantization is performed, and the quantization of the next two-dimensional convolution node is performed, to obtain the output of int8 data type, that is, the quantization process of the next quantized node is performed in the node.

2. The memory optimization method for embedded device deep learning inference of claim 1, wherein, Analyzing the trained deep learning model specifically comprises the following steps: Constructing a mapping node2input of nodes and node inputs, a mapping node2output of nodes and node outputs, and an activation value label set activation_label_set composed of input and output labels of all nodes; De-duplicating the activation value label set: de-duplicating the repeated activation value labels in the activation value label set. 3.The memory optimization method and system for embedded device deep learning inference according to claim 1 or 2, characterized in that, When the node corresponding input activation value is calculated by two-dimensional convolution, the general matrix multiplication is used to accelerate the calculation, the node corresponding input activation value is unfolded and converted into a two-dimensional matrix, and the node corresponding input activation value is destroyed immediately after conversion.

4. A memory optimization system for embedded device deep learning inference, characterized in that, The system comprises, The analysis unit is configured to analyze the trained deep learning model, obtain a network structure and model weights, and organize the network structure and model weights into a data format of a directed acyclic graph. The execution unit is configured to traverse the directed acyclic graph in a breadth-first manner, and specifically configured to: determine whether a node sequence of a computation graph of the trained deep learning model is sorted in a breadth-first manner; if not, rearrange the node sequence of the computation graph of the trained deep learning model in a breadth-first manner; during inference, construct an activation value mapping activation_map to store activation values, the keys of the mapping being all labels in a label set, and the values of the mapping being activation values corresponding to the labels; initially, an activation value corresponding to a first label is a model input, and activation values corresponding to other labels are empty; traverse the directed acyclic graph in a breadth-first manner: obtain input labels corresponding to a node through input2node, and find corresponding activation values in the activation_map through the labels to obtain node inputs; execute the node to obtain a node output, obtain a node output label through node2output, and store the node output in the activation_map through the label; and destroy the activation value corresponding to the label of the node in the activation_map after execution of the node, and release a memory space occupied by the activation value. The destruction unit is configured to, during traversal of each node of the directed acyclic graph during inference, determine whether the node is a last node of an input corresponding to the directed acyclic graph; if the node is the last node, destroy an activation value corresponding to a label of the node in the activation_map after execution of the node, and release a memory space occupied by the activation value; wherein the last node of the input corresponding to the directed acyclic graph refers to a last-traversed node in input corresponding nodes; and determine whether a node corresponding input activation value will generate an intermediate result during calculation; if so, destroy the node corresponding input activation value immediately after generation of the intermediate result. The saving unit is configured to save an output activation value of a quantization node that meets a condition as an int8 data type. When the trained deep learning model is a convolutional neural network model, an output activation value of a node that needs to be quantized is saved as an int8 data type, and specifically as follows: In the convolutional neural network model, a combination of two-dimensional convolution and a linear rectifier function is included; an output of the two-dimensional convolution is an int8 data type, a next RELU node is executed, and a subsequent two-dimensional convolution is executed; during calculation of the two-dimensional convolution node, int8 weights and int8 inputs are used to calculate the two-dimensional convolution, and inverse quantization is performed, and quantization of a next two-dimensional convolution node is performed to obtain an output of an int8 data type, that is, a quantization process of the next quantization node is performed in the node.

5. The memory optimization system for embedded device deep learning inference of claim 4, wherein, The analysis unit comprises, A constructing module is configured to construct a mapping node2input of nodes and node inputs, a mapping node2output of nodes and node outputs, and an activation label set activation_label_set composed of input and output labels of all nodes; A deduplication module is configured to perform deduplication processing on repeated activation labels in the activation label set.

6. An electronic device, comprising: The method comprises the following steps: a memory and at least one processor; wherein the memory stores a computer program; the at least one processor executes the computer program stored in the memory, so that the at least one processor executes the memory optimization method for embedded device deep learning inference as claimed in any one of claims 1 to 3.

Images